Framework for UE assisted performance monitoring

Abstract

There is in an aspect provided an apparatus for determining an association between at least one inference instance and a performance monitoring instance (PMI), wherein the association is defined based on a time window; determine a total number of valid PMIs, wherein a PMI is considered valid when a monitoring reference signal resource set corresponding to the PMI is carrying at least one predicted beam of a number of predicted beams having a quality metric higher than a quality metric of other predicted beams, and when the predicted beams having the quality metric higher than the quality metric of the other predicted beams correspond to the at least one inference instance; and use the total number when calculating a performance monitoring metric that represents a performance of an inference operation for beam prediction for the at least one inference instance

Description

Framework For UE Assisted Performance MonitoringCROSS-REFERENCE TO RELATED APPLICATIONS

[0001] This application claims priority from, and the benefit of, US provisional Application No. 63 / 697815, filed September 23, 2024, which is hereby incorporated by reference in its entirety.TECHNICAL FIELD

[0002] The examples and non-limiting example embodiments relate generally to communications and, more particularly, to a framework for UE assisted performance monitoring.BACKGROUND

[0003] A communication device may gain access to a communication network through an access network node.SUMMARY

[0004] In a first aspect of the present disclosure, there is provided a first apparatus. The first apparatus comprises at least one processor; and at least one memory storing instructions that, when executed by the at least one processor, cause the first apparatus at least to: determine an association between at least one inference instance and a performance monitoring instance, wherein the association between the at least one inference instance and the performance monitoring instance is defined based on a time window; determine a total number of valid performance monitoring instances, wherein a performance monitoring instance is considered as a valid performance monitoring instance when a monitoring reference signal resource set corresponding to the performance monitoring instance is carrying at least one predicted beam of a number of predicted beams having a quality metric that is higher than a quality metric of other predicted beams among a plurality of predicted beams, and when the predicted beams having the quality metric that is higher than the quality metric of the other predicted beams among the plurality of predicted beams correspond to the at least one inference instance; and use the total number of valid performance monitoring instances when calculating a performance monitoring metric that represents a performance of an inference operation for beam prediction for the at least one inference instance.

[0005] In a second aspect of the present disclosure, there is provided a second apparatus. The second apparatus comprises at least one processor; and at least one memory storing instructions that, when executed by the at least one processor, cause the second apparatus at least to determine a total number of valid performance monitoring instances associated with at least one inference instance of a time window; receive, from a user equipment, a total number of errors in performance monitoring that reflects a total number of faulty instance of beam prediction; and calculate a performance monitoring metric based on at least one of: the total number of valid performance monitoring instances or the total number of errors in performance monitoring.BRIEF DESCRIPTION OF THE DRAWINGS

[0006] The foregoing aspects and other features are explained in the following description, taken in connection with the accompanying drawings.

[0007] FIG. 1 is a block diagram of one possible and non-limiting system in which the example embodiments may be practiced.

[0008] FIG. 2 shows an example of associating a monitoring instance and an inference instance.

[0009] FIG. 3 shows an example of considering valid and invalid monitoring instances.

[0010] FIG. 4 shows an example signaling diagram based on the examples described herein.

[0011] FIG. 5 is an example apparatus configured to implement the examples described herein.

[0012] FIG. 6 shows a representation of an example of non-volatile memory media used to store instructions that implement the examples described herein.

[0013] FIG. 7 is an example method, based on the examples described herein.

[0014] FIG. 8 is an example method, based on the examples described herein.DETAILED DESCRIPTION OF EXAMPLE EMBODIMENTS

[0015] Turning to FIG. 1 , this figure shows a block diagram of one possible and non-limiting example in which the examples may be practiced. A user equipment (UE) 110, radio access network (RAN) node 170, and network element(s) 190 are illustrated. In the example of FIG. 1 , the user equipment (UE) 110 is in wireless communication with a wireless network 100. A UE is a wireless device that can access the wireless network 100. The UE 110 includes one or more processors 120, one or more memories 125, and one or more transceivers 130 interconnected through one or more buses 127. Each of the one or more transceivers 130 includes a receiver, Rx, 132 and a transmitter, Tx, 133. The one or more buses 127 may be address, data, or control buses, and may include any interconnection mechanism, such as a series of lines on a motherboard or integrated circuit, fiber optics or other optical communication equipment, and the like. The one or more transceivers 130 are connected to one or more antennas 128. The one or more memories 125 include computer program code 123. The UE 110 includes a module 140, comprising one of or both parts 140-1 and / or 140-2, which may be implemented in a number of ways. The module 140 may be implemented in hardware as module 140-1 , such as being implemented as part of the one or more processors 120. The module 140-1 may be implemented also as an integrated circuit or through other hardware such as a programmable gate array. In another example, the module 140 may be implemented as module 140-2, which is implemented as computer program code 123 and is executed by the one or more processors 120. For instance, the one or more memories 125 and the computer program code 123 may be configured to, with the one or more processors 120, cause the user equipment 110 to perform one or more of the operations as described herein. The UE 110 communicates with RAN node 170 via a wireless link 111.

[0016] The RAN node 170 in this example is a base station that provides access for wireless devices such as the UE 110 to the wireless network 100. The RAN node 170 may be, for example, a base station for 5G, also called New Radio (NR). In 5G, the RAN node 170 may be a NG-RAN node, which is defined as either a gNB or an ng-eNB. A gNB is a node providing NR user plane and control plane protocol terminations towards the UE, and connected via the NG interface (such as connection 131) to a 5GC (such as, for example, the network element(s) 190). The ng-eNB is a node providing E-UTRA user plane and control plane protocol terminations towards the UE, and connected via the NG interface (such as connection 131) to the 5GC. The NG-RAN node may include multiple gNBs, which may also include a central unit (CU) (gNB-CU) 196 and distributed unit(s) (DUs) (gNB-DUs), of which DU 195 is shown. Note that the DU 195 may include or be coupled to and control a radio unit (RU). The gNB-CU 196 is a logical node hosting radio resource control (RRC), SDAP and PDCP protocols of the gNB or RRC and PDCP protocols of the en-gNB that control the operation of one or more gNB-DUs. The gNB-CU 196 terminates the F1 interface connected with the gNB-DU 195. The F1 interface is illustrated as reference 198, although reference 198 also illustrates a link between remote elements of the RAN node 170 and centralized elements of the RAN node 170, such as between the gNB-CU 196 and the gNB-DU 195. The gNB-DU 195 is a logical node hosting RLC, MAC and PHY layers of the gNB or en-gNB, and its operation is partly controlled by gNB-CU 196. One gNB-CU 196 supports one or multiple cells. One cell may be supported with one gNB-DU 195, or one cell may be supported / shared with multiple DUs under RAN sharing. The gNB-DU 195 terminates the F1 interface 198 connected with the gNB- CU 196. Note that the DU 195 is considered to include the transceiver 160, e.g., as part of a RU, but some examples of this may have the transceiver 160 as part of a separate RU, e.g., under control of and connected to the DU 195. The RAN node 170 may also be an eNB (evolved NodeB) base station, for LTE (long term evolution), or any other suitable base station or node.

[0017] The RAN node 170 includes one or more processors 152, one or more memories 155, one or more network interfaces (NW l / F(s)) 161 , and one or more transceivers 160 interconnected through one or more buses 157. Each of the one or more transceivers 160 includes a receiver, Rx, 162 and a transmitter, Tx, 163. The one or more transceivers 160 are connected to one or more antennas 158. The one or more memories 155 include computer program code 153. The CU 196 may include the processor(s) 152, one or more memories 155, and network interfaces 161. Note that the DU 195 may also contain its own memory / memories and processor(s), and / or other hardware, but these are not shown.

[0018] The RAN node 170 includes a module 150, comprising one of or both parts 150-1 and / or 150-2, which may be implemented in a number of ways. The module 150 may be implemented in hardware as module 150-1 , such as being implemented as part of the one or more processors 152. The module 150-1 may be implemented also as an integrated circuit or through other hardware such as a programmable gate array. In another example, the module 150 may be implemented as module 150-2, which is implemented as computer program code 153 and is executed by the one or more processors 152. For instance, the one or morememories 155 and the computer program code 153 are configured to, with the one or more processors 152, cause the RAN node 170 to perform one or more of the operations as described herein. Note that the functionality of the module 150 may be distributed, such as being distributed between the DU 195 and the CU 196, or be implemented solely in the DU 195.

[0019] The one or more network interfaces 161 communicate over a network such as via the links 176 and 131 . Two or more gNBs 170 may communicate using, e.g., link 176. The link 176 may be wired or wireless or both and may implement, for example, an Xn interface for 5G, an X2 interface for LTE, or other suitable interface for other standards.

[0020] The one or more buses 157 may be address, data, or control buses, and may include any interconnection mechanism, such as a series of lines on a motherboard or integrated circuit, fiber optics or other optical communication equipment, wireless channels, and the like. For example, the one or more transceivers 160 may be implemented as a remote radio head (RRH) 195 for LTE or a distributed unit (DU) 195 for gNB implementation for 5G, with the other elements of the RAN node 170 possibly being physically in a different location from the RRH / DU 195, and the one or more buses 157 could be implemented in part as, for example, fiber optic cable or other suitable network connection to connect the other elements (e.g., a central unit (CU), gNB-CU 196) of the RAN node 170 to the RRH / DU 195. Reference 198 also indicates those suitable network link(s).

[0021] A RAN node / gNB can comprise one or more TRPs to which the methods described herein may be applied. FIG. 1 shows that the RAN node 170 comprises TRP 51 and TRP 52, in addition to the TRP represented by transceiver 160. Similar to transceiver 160, TRP 51 and TRP 52 may each include a transmitter and a receiver. The RAN node 170 may host or comprise other TRPs not shown in FIG. 1 .

[0022] A relay node in NR is called an integrated access and backhaul node. A mobile termination part of the IAB node facilitates the backhaul (parent link) connection. In other words, the mobile termination part comprises the functionality which carries UE functionalities. The distributed unit part of the IAB node facilitates the so called access link (child link) connections (i.e. for access link UEs, and backhaul for other IAB nodes, in the case of multi-hop IAB). In other words, the distributed unit part is responsible for certain base station functionalities. The IAB scenario may follow the so called split architecture, where the central unit hosts the higher layer protocols to the UE and terminates the control plane and user plane interfaces to the 5G core network.

[0023] It is noted that the description herein indicates that “cells” perform functions, but it should be clear that equipment which forms the cell may perform the functions. The cell makes up part of a base station. That is, there can be multiple cells per base station. For example, there could be three cells for a single carrier frequency and associated bandwidth, each cell covering one-third of a 360 degree area so that the single base station’s coverage area covers an approximate oval or circle. Furthermore, each cell can correspond to a single carrier and a base station may use multiple carriers. So if there are three 120 degree cells per carrierand two carriers, then the base station has a total of 6 cells.

[0024] The wireless network 100 may include a network element or elements 190 that may include core network functionality, and which provides connectivity via a link or links 181 with a further network, such as a telephone network and / or a data communications network (e.g., the Internet). Such core network functionality for 5G may include location management functions (LMF(s)) and / or access and mobility management function(s) (AMF(S)) and / or user plane functions (UPF(s)) and / or session management function(s) (SMF(s)). Such core network functionality for LTE may include MME (mobility management entity) / SGW (serving gateway) functionality. Such core network functionality may include SON (self-organizing / optimizing network) functionality. These are merely example functions that may be supported by the network element(s) 190, and note that both 5G and LTE functions might be supported. The RAN node 170 is coupled via a link 131 to the network element 190. The link 131 may be implemented as, e.g., an NG interface for 5G, or an S1 interface for LTE, or other suitable interface for other standards. The network element 190 includes one or more processors 175, one or more memories 171 , and one or more network interfaces (NW l / F(s)) 180, interconnected through one or more buses 185. The one or more memories 171 include computer program code 173. Computer program code 173 may include SON and / or mobility robustness optimization (MRO) functionality 172.

[0025] The wireless network 100 may implement network virtualization, which is the process of combining hardware and software network resources and network functionality into a single, software-based administrative entity, or a virtual network. Network virtualization involves platform virtualization, often combined with resource virtualization. Network virtualization is categorized as either external, combining many networks, or parts of networks, into a virtual unit, or internal, providing network-like functionality to software containers on a single system. Note that the virtualized entities that result from the network virtualization are still implemented, at some level, using hardware such as processors 152 or 175 and memories 155 and 171 , and also such virtualized entities create technical effects.

[0026] The computer readable memories 125, 155, and 171 may be of any type suitable to the local technical environment and may be implemented using any suitable data storage technology, such as semiconductor based memory devices, flash memory, magnetic memory devices and systems, optical memory devices and systems, non-transitory memory, transitory memory, fixed memory and removable memory. The computer readable memories 125, 155, and 171 may be means for performing storage functions. The processors 120, 152, and 175 may be of any type suitable to the local technical environment, and may include one or more of general purpose computers, special purpose computers, microprocessors, digital signal processors (DSPs) and processors based on a multi-core processor architecture, as non-limiting examples. The processors 120, 152, and 175 may be means for performing functions, such as controlling the UE 110, RAN node 170, network element(s) 190, and other functions as described herein.

[0027] In general, the various example embodiments of the user equipment 110 can include, but are notlimited to, cellular telephones such as smart phones, tablets, personal digital assistants (PDAs) having wireless communication capabilities, portable computers having wireless communication capabilities, image capture devices such as digital cameras having wireless communication capabilities, gaming devices having wireless communication capabilities, music storage and playback devices having wireless communication capabilities, internet appliances including those permitting wireless internet access and browsing, tablets with wireless communication capabilities, head mounted displays such as those that implement virtual / augmented / mixed reality, as well as portable units or terminals that incorporate combinations of such functions. The UE 110 can also be a vehicle such as a car, or a UE mounted in a vehicle, a UAV such as e.g. a drone, or a UE mounted in a UAV. The user equipment 110 may be a terminal device, such as mobile phone, mobile device, sensor device etc., the terminal device being a device used by the user or not used by the user.

[0028] UE 110, RAN node 170, and / or network element(s) 190, (and associated memories, computer program code and modules) may be configured to implement (e.g. in part) the methods described herein. Thus, computer program code 123, module 140-1 , module 140-2, and other elements / features shown in FIG. 1 of UE 110 may implement user equipment related aspects of the examples described herein. Similarly, computer program code 153, module 150-1 , module 150-2, and other elements / features shown in FIG. 1 of RAN node 170 may implement gNB / TRP related aspects of the examples described herein. Computer program code 173 and other elements / features shown in FIG. 1 of network element(s) 190 may be configured to implement network element related aspects of the examples described herein.

[0029] Having thus introduced a suitable but non-limiting technical context for the practice of the example embodiments, the example embodiments are now described with greater specificity.

[0030] Rel-18 / Rel-19 AI-ML for beam prediction relates to AI / ML-based beam management, two sub-use cases, and measurements and prediction based on two beam sets, namely Set A and Set B. AI / ML-based beam management includes leveraging AI / ML models to predict the best beam(s) based on a limited set of measurements. The two Sub-use cases include spatial-domain prediction and time-domain prediction. Spatial-domain prediction relates to beam prediction based on a limited set of measurements that does not contain any historical information. Time-domain prediction relates to beam prediction into the future based on a limited set of measurements that contains historical information. Set A is the complete set of beams over which the prediction operates, and Set B is the set of beams whose measurements are inputted to the AI / ML model (e.g., L1-RSRP, etc.). Set B can be: different from Set A (space-domain and time-domain prediction), a subset of Set A (space-domain and time-domain prediction), or the same as Set A (time-domain prediction).

[0031] BM-Case1 relates to spatial-domain DL Tx prediction for Set A of beams based on measurements results of Set B of beams. BM-Case2 relates to temporal DL Tx beam prediction for Set A of beams based on the historic measurements results of Set B of beams.

[0032] The examples described here relate to, for BM-Case1 and BM-Case 2 with a UE-side AI / ML model, supporting performance monitoring including Option 1 (NW-side performance monitoring) and Option 2 (UE-assisted performance monitoring). For NW-side performance monitoring (Option 1), the UE sends a report to NW (for the calculation of performance metric at NW), where measurement results from resource set for monitoring, e.g., L1-RSRP and / or RS index is supported as the content of the report (other contents may also be included in the report), and where the report is at least configured / triggered by the NW. UE-assisted performance monitoring (Option 2) is where the UE calculates one or more performance metrics (including determining how to report and what to report), and whether to trigger a report based on one or more events for NW-side performance monitoring (Option 1) or UE-assisted performance monitoring (Option 2).

[0033] The examples described herein further relate to, for a UE sided model in beam management, supporting an associated ID that can at least be configured within a CSI framework (the associated ID may be configured or indicated via other one or more signals and / or in other one or more procedures or frameworks.

[0034] For BM-Case1 and BM-Case2 with a UE-sided AI / ML model, for Option 2 (UE-assisted performance monitoring), alternatives include Alt 1 , Alt 2, and Alt 3. The examples described herein provide solutions for Alt 1 , Alt 2, and Alt. 3. Alt 1 relates to: Top 1 or Top K beam prediction accuracy (with or without margin) by comparing the prediction results and the Top 1 or Top K beam based on the measurements from a resource set / resources for monitoring. Alt 2 relates to: The L1-RSRP difference information based on actual measurement of the L1 -RSRP of one or more of Top K predicted beam, and L1 -RSRP measurements from a resource set / resources for monitoring. Alt 3 relates to: The RSRP difference information between the predicted RSRP and measured L1-RSRP of corresponding beam(s) of a resource set / resources for monitoring (resources for Set B for monitoring are not precluded), applicable when the model can predict RSRP.

[0035] Described herein are also examples related to, for Alt 1 , Alt 2, and Alt 3, other details including how to configure the resource set / resources for monitoring, including E.g. whether / how to use full set of Set A for measurement. If not, whether / how to obtain the measurement of the predicted Top 1 or Top K beam for calculating the prediction accuracy or the RSRP difference.

[0036] Information for beam reporting framework

[0037] Other information considers several technical aspects of the CSI framework, configuration, L1-RSRP reporting, and UCI bit sequence generation. All these aspects are described in technical specifications like TS 38.306, TR 38.831 , TS 38.214, and TS 38.212 and are summarized as follows (1-4):

[0038] 1) CSI reporting framework capability (TS 38.306 clause 4.2.7): This clause describes the capability of the UE to support CSI reporting. It includes parameters defining the maximum number of periodic / aperiodic CSI reports that can be configured per CC (Component Carrier), per BWP (Bandwidth Part) and per beam. Moreover, it specifies the concurrent CSI reports per CC that the UE can measure and process, including periodic, semi-persistent and aperiodic CSI, including beam reports.

[0039] 2) CSI report configuration (TR 38.831): Describes the configuration parameters used to set up periodic, aperiodic or semi-persistent CSI reports sent on the PUCCH or PUSCH for a particular cell or triggered by downlink control information (DCI). It includes fields such as report quantity, frequency domainconfiguration, time domain behaviour and channel measurement resource allocation that affect how the UE perform reports based on different configurations.

[0040] 3) L1 RSRP reporting (TS 38.214, clause 5.2.1): This clause defines how the UE calculates and reports L1-RSRP. It covers configurations involving CSI-RS resources, SS / PBCH block resources or both, detailing limitations on the number of CSI-RS resource sets and resources within those sets. It also explains how L1-RSRP is quantized and reported based on different scenarios, considering group-based reporting, differential reporting, and channel measurement timing with respect to SS / PBCH or NZP CSI-RS.

[0041] 4) UCI bit sequence generation (TS 38.212, clauses 6.3.1.1 and 6.3.2.1): This clause deals with the generation of UCI bit sequences for uplink transmission. It defines the specific order or mapping of CSI fields within a report for different reporting scenarios such as CRI / RSRP, SSBRI / RSRP or Capability Index reporting. It provides details on the structure of the CSI reports, including CRI, RSRP and Capability Index, for transmission within the UCI.

[0042] These descriptions are integral for defining how CSI is handled, reported, and utilized in the communication system by UE. Each section covers specific technical aspects, configurations and procedures related to CSI reporting, L1-RSRP calculation and UCI bit sequence generation, which are critical for establishing and maintaining the communication link between NW and UE, while enabling efficient use of CSI for data transmission and reception.

[0043] For UE-sided AI / ML based beam prediction use cases, e.g., BM-Case1 and BM-Case2, RAN1 agreed to support UE-assisted performance monitoring, where UE calculates performance monitoring metric(s) and reacts / reports to the gNB. To calculate the performance monitoring metrics, RAN1 discussed few alternatives and listed details for further study. From those alternatives, the following are mainly related to the case of beam prediction (with or without RSRP prediction).

[0044] Alt 1 : Top 1 or Top K beam prediction accuracy (with or without margin) by comparing the prediction results and the Top 1 or Top K beam based on the measurements from a resource set / resources for monitoring

[0045] Alt 2: The L1 -RSRP difference information based on actual measurement of the L1 -RSRP of one or more of Top K predicted beam, and L1-RSRP measurements from a resource set / resources for monitoring

[0046] Even though some discussions on calculating the metrics are available in Rel-18 study report, TR 38.843, those are not fully considering the practical limitations where Set A can is not transmitted in full or partial manner in order to allow UE calculating metrics. Also, the metrics in 3GPP define in the spec shall be practical than the metrics used in the TR. The examples and embodiments described herein relate to managing these issues and provide basic framework for enabling UE-assisted performance monitoring.

[0047] The following embodiments 1-11 are described herein (among other embodiments and examples described herein):

[0048] Embodiment 1 : A UE (that perform inference operation for beam prediction and / or calculate a performance monitoring metric for beam prediction) comprising: determining an association between at leastone inference instance and a performance monitoring instance, wherein the association between at least inference instance and the performance monitoring instance is defined based on a time window, determining a total number of valid performance monitoring instances (N1 ), wherein a performance monitoring instances is considered as a valid performance monitoring instance when the monitoring RS resource set corresponding to the performance monitoring instance is carrying at least one of the Top-K predicted beams corresponding to the at least one associated inference instance, using the total number of valid performance monitoring instances (N1) when calculating a performance monitoring metric that represents the performance of inference operation for beam prediction.

[0049] Embodiment 2: The UE as in any of embodiments 1 or 3-11 , where the total number of valid performance monitoring instances (N1) may be considered within a time period (Tm) or a max number of monitoring instances (Nmax) that is configured or defined to the UE when measuring RS resources corresponding to the monitoring RS resource set.

[0050] Embodiment 3: The UE as in any of embodiments 1-2 or 4-11, where the total number of valid performance monitoring instances Ni is further considered when determining a total number of errors in performance monitoring (N2) which reflects the total number of faulty instances of beam prediction.

[0051] Embodiment 4: The UE as in any of embodiments 1-3 or 5-11 , where the UE may neglect the reception of monitoring RS resource set when the Top-K predicted beams of the associated inference instance are known prior to the reception instance of the monitoring RS resource set.

[0052] Embodiment 5: The UE as in any of embodiments 1-4 or 6-11 , where the total number of valid performance monitoring instances Ni and the total number of errors in performance monitoring N2 is used together when calculating the performance monitoring metric.

[0053] Embodiment 6: The UE and gNB as in any of embodiments 1 -5 or 7-11 , where UE reporting of the total number of valid performance monitoring instances Ni and the total number of errors in performance monitoring N2 to the gNB, and the gNB calculating the performance monitoring metric based on N1 and N2.

[0054] Embodiment 7: The UE and gNB as in any of embodiments 1-6 or 8-11 , where UE reporting of the total number of errors in performance monitoring N2 to the gNB, and the gNB determining the total number of valid performance monitoring instances Ni and calculating the performance monitoring metric based on N1 and N2.

[0055] Embodiment 8: The UE as in any of embodiments 1-7 or 9-11 , where a faulty instance of the total number of errors in performance monitoring N2 is determined when none of Top-K predicted beams of the associated inference report appear in the Top-L measured beams of the associated monitoring RS resource set.

[0056] Embodiment 9: The UE as in any of embodiments 1 -9 or 10-11 , where a faulty instance of the total number of errors in performance monitoring N2 is determined when the best available beam of Top-K predicted beams of the associated inference report appear in the Top-L measured beams of the associated monitoringRS resource set.

[0057] Embodiment 10: The UE as claimed in any of claim 1-9 or 11 , where the time window can be configured or pre-defined and the time window is using a reference time instance when determining the association between inference instance and performance monitoring instance, wherein the reference time can be defined with respect to the reception of monitoring RS resource set, or defined with respect to the reception of measurements RS resource set for inference, or defined with respect to the reporting of inference report, or defined with respect to the reporting of monitoring report.

[0058] Embodiment 11 : The UE as claimed in any of claim 1 -X, when there are more than one inference instance for a performance monitoring instance within the time window, the UE is further defined to consider the latest inference instance or earliest inference instance with respect to the performance monitoring instance.

[0059] In the embodiments and examples described herein:

[0060] The UE may be configured with a time period (Tm) or a max number of monitoring instances (Nmax) to measure DL RSs corresponding to a monitoring RS resource set. When there is a transmission of DL RS resources (e.g., CSI-RS) corresponding to the monitoring resource set, the UE may receive DL RS resources and determine Top-L beam IDs and corresponding L1-RSRP of Top-L beam IDs of the monitoring resource set. L may be defined or configured to the UE. L can be equal to the size of monitoring resource set or smaller than that.

[0061] The UE determines a total number of valid performance monitoring instances Ni (Ni is selected from all monitoring instances of Tmor Nmax), wherein a performance monitoring instance (within Tmor Nmax) is considered as a valid performance monitoring instance (counted for Ni) when the monitoring RS resource set measured at a given time instance is carrying one or more of the Top-K predicted beams corresponding to an associated inference instance.

[0062] How to determine the association between inference instance and performance monitoring may be defined to the UE.

[0063] The association between inference instance and monitoring instance can be defined based on reference time instance and time window defined / configured to the UE. the reference time instance can be defined as (1-4): 1) The first or last symbol / slot that the UE is expected to receive RS resources of monitoring RS resource set (Q_x), or 2) The first or last symbol / slot that the UE is expected to receive RS resources of the measurement RS resource set (Set B), or 3) The reporting time instance, slot / symbol, that carrying the inference results (e.g., Top-K predicted beams), or 4) If the calculation of the performance monitoring metric is carried out by the gNB, the reporting time instance, slot / symbol, that carrying the monitoring results (e.g., Top- L beams). The time window can be configured or defined to the UE.

[0064] In one variant, when the time window size is defined / configured to the UE to determine the associated inference instance, and the time window from a first / last symbol / slot of monitoring RS resources can be used to determine the associated inference instance. In one example, the UE may consider the nearestinference reporting instance or Set B measurement instance to determine the associated inference instance. This example is shown in FIG. 2 showing time window 202 and time window 204. FIG. 2 shows an example of associating the monitoring instance and inference instance.

[0065] In another variant, the time window can also be defined with respect to the inference reporting instance or Set B reception instance, where monitoring instance is associated with respect to the inference instance.

[0066] For a given monitoring instance, when the monitoring RS resource set is not containing one or more of the Top-K predicted beams of latest / associated inference instance, the UE does not expect to count that as for the valid number of performance monitoring instance. This is shown in FIG. 3, which shows time window 302 and time window 304. FIG. 3 shows an example of considering valid and invalid monitoring instances.

[0067] The UE may neglect the reception of monitoring RS resource set when the Top-K predicted beams of the associated inference instance are known prior to the reception instance of the monitoring RS resource set.

[0068] Within the total number of valid performance monitoring instances Ni, the UE determine a total number of errors in performance monitoring N2 which reflects the total number of faulty instances of beam prediction, and this number is calculated when the following is satisfied, For Alt. 1 and Alt. 2:

[0069] For Alt 1 , for each valid performance instance, if none of Top-K predicted beams appear in the measured Top-L beams of monitoring RS resource set, the prediction at the UE may considered faulty and counted for N2. Otherwise, prediction is not considered as faulty and not counted for N2. In some variants, the UE may be defined to consider only the best among available Top-K predicted beams (e.g., Top-2 beam when Top-1 beam is not available in the monitoring RS resource set) when considering faulty counts.

[0070] For Alt.2, for each valid performance instance, if none of the measured L1 -RSRPs corresponding to Top-K predicted beams (in other words, highest available L1-RSRP of Top-K predicted beams) is within a margin of measured L1-RSRP of Top-1 / L beam of the monitoring RS resource set, the prediction at the UE may considered faulty and counted for N2. Otherwise, prediction is not considered as faulty and not counted for N2. Similar to Alt.1 , as L1-RSRP forTop-1 predicted beam may not be always available in the N1 instances. In such cases, the second-best predicted beam is selected for comparing L1 -RSRPs. That may follow up to Top-K. In one variant, when comparing L1-RSRP difference, Top-1 , 2,..K from prediction set may be respectively compared to the Top-1 ,2, ,.,K from monitoring RS resource set. In one variant, with or without counting N2, the UE may sum up the L1-RSRP difference (e.g., different between L1-RSRP of best available predicted beam and L1-RSRP of Top-1 beam of monitoring RS resource set) in dBs and average it by N1 to determine an metric of L1-RSRP difference.

[0071] After the end of Tmor Nmaxor when an threshold is reached for N2, the UE may report at least N2 to the gNB. In one variant, the UE determine that the beam prediction accuracy as the ratio (N1 -N2) / NI . In another variant, the UE determine that the beam prediction accuracy with L1-RSRP margin as (N1-N2) / N2 andmay further determine the average for L1-RSRP difference over N1 instances. The UE may configure to report either N2 or (N1 , N2) or a ratio determined (N1 , N2). In one variant, the ratio determined based on (N1 , N2) may be further checked with a look-up table to indicate an entry in the look-up table. In another variant, the UE may consider an threshold for N2 or threshold corresponding to ratio determined based on N1 and N2, and report an event based on that.

[0072] K = 1 can also be assumed with the same text above. Monitoring RS resource set can be transmitted as P-CSI-RS, SP-CSI-RS, or AP-CSI-RS.

[0073] An example signaling diagram is provided in Figure 3. FIG. 4 shows an exampling signaling diagram between the user equipment 110 and network (170, 190) including Steps 1-16.

[0074] Step 1 : The UE receives RRC configurations (e.g., that contain CSI-MeasConfig, CSI-ReportConfig, and other CSI measurement and reporting parameters), where at least one CSI-ReportConfig (e.g., CSI- ReportConfig_x) is enabling ML beam prediction at the UE side. Set A and Set B for beam prediction operation is provided to the UE by the same CSI-ReportConfig.

[0075] Step 2-3: The UE receives RRC configuration on performance monitoring, where the performance monitoring may be enabled by one or more of other CSI report configurations (CSI-ReportConfig_y) or by the same CSI report configuration (CSI-ReportConfig_x). The details on monitoring RS resource set Q_x, max number of performance monitoring instances (Nmax), and time window (T) to consider when determining the associated inference instance are also configured to the UE. In step 3, the UE may start monitoring of the inference operation.

[0076] Step 4-8 and Step 11 : Shows the steps of beam prediction (inference operation) associated with the CSI-ReportConfig_x.

[0077] Step 9-10: Monitoring RS resource set (Q_x) is transmitted by the gNB and the UE checks for an associated inference operation instance. Figure 1-2 provides some examples on how the UE determine the associated inference operation. In one variant, there may be additional signalling or an additional procedure involved to update the monitoring RS resource set (Q_x) and the updated monitoring RS resource set (Q_x) shall be applied to later Steps.

[0078] Step 12: The UE considers whether to consider monitoring instance in the N1 count or not. The details on what to consider here are provided herein.

[0079] Step 13-14: Assuming that the monitoring is counted towards N1 , the UE measure Q_x and determine whether Top-L of Q_x is related to the Top-K predicted beams. Details on how to check this is provided herein.

[0080] Step 15-16: When the performance monitoring is conducted for multiple monitoring instances, the UE can determine N2 and N1 , and report them to the gNB. The details on what to report are described previously.

[0081] The examples described herein may relate to 3GPP AI / ML work items, including those for Rel-19.

[0082] FIG. 5 is an example apparatus 500, which may be implemented in hardware, configured to implement the examples described herein. The apparatus 500 comprises at least one processor 502 (e.g. an FPGA and / or central processing unit), one or more memories 504 including computer program code 505, the computer program code 505 having instructions to carry out the methods described herein, wherein the at least one memory 504 and the computer program code 505 are configured to, with the at least one processor 502, cause the apparatus 500 to implement circuitry, a process, component, module, or function (implemented with control module 506) to implement the examples described herein. The one or more memories 504 may include a non-transitory memory, a transitory memory, a volatile memory (e.g. random access memory (RAM)), or a non-volatile memory (e.g. read-only memory (ROM)).

[0083] Reporting 530 implements the examples described herein related to a framework for UE-assisted performance monitoring.

[0084] The apparatus 500 includes a display and / or input / output interface 508, which includes user interface (Ul) circuitry and elements, that may be used to display aspects or a status of the methods described herein (e.g., as one of the methods is being performed or at a subsequent time), or to receive input from a user such as with using a keypad, camera, touchscreen, touch area, microphone, biometric recognition, one or more sensors, etc. The apparatus 500 includes one or more communication e.g. network (NW) interfaces (l / F(s)) 510. The communication l / F(s) 510 may be wired and / or wireless and communicate over the Internet / other network(s) via any communication technique including via one or more links 524. The link(s) 524 may be the link(s) 131 and / or 176 from FIG. 1. The link(s) 131 and / or 176 from FIG. 1 may also be implemented using transceiver(s) 516 and corresponding wireless link(s) 526. The communication l / F(s) 510 may comprise one or more transmitters or one or more receivers.

[0085] The transceiver 516 comprises one or more transmitters 518 and one or more receivers 520. The transceiver 516 and / or communication l / F(s) 510 may comprise standard well-known components such as an amplifier, filter, frequency-converter, (de)modulator, and encoder / decoder circuitries and one or more antennas, such as antennas 514 used for communication over wireless link 526.

[0086] The control module 506 of the apparatus 500 comprises one of or both parts 506-1 and / or 506-2, which may be implemented in a number of ways. The control module 506 may be implemented in hardware as control module 506-1 , such as being implemented as part of the one or more processors 502. The control module 506-1 may be implemented also as an integrated circuit or through other hardware such as a programmable gate array. In another example, the control module 506 may be implemented as control module 506-2, which is implemented as computer program code (having corresponding instructions) 505 and is executed by the one or more processors 502. For instance, the one or more memories 504 store instructions that, when executed by the one or more processors 502, cause the apparatus 500 to perform one or more of the operations as described herein. Furthermore, the one or more processors 502, the one or more memories 504, and example algorithms (e.g., as flowcharts and / or signaling diagrams), encoded as instructions,programs, or code, are means for causing performance of the operations described herein.

[0087] The apparatus 500 to implement the functionality of control 506 may be UE 110, RAN node 170 (e.g. gNB), or network element(s) 190 (e.g. LMF 190). Thus, processor 502 may correspond to processor(s) 120, processor(s) 152 and / or processor(s) 175, memory 504 may correspond to one or more memories 125, one or more memories 155 and / or one or more memories 171 , computer program code 505 may correspond to computer program code 123, computer program code 153, and / or computer program code 173, control module 506 may correspond to module 140-1 , module 140-2, module 150-1 , and / or module 150-2, and communication l / F(s) 510 and / or transceiver 516 may correspond to transceiver 130, antenna(s) 128, transceiver 160, antenna(s) 158, NW l / F(s) 161 , and / or NW l / F(s) 180. Alternatively, apparatus 500 and its elements may not correspond to either of UE 110, RAN node 170, or network element(s) 190 and their respective elements, as apparatus 500 may be part of a self-organizing / optimizing network (SON) node or other node, such as a node in a cloud.

[0088] The apparatus 500 may also be distributed throughout the network (e.g. 100) including within and between apparatus 500 and any network element (such as a network control element (NCE) 190 and / or the RAN node 170 and / or UE 110).

[0089] Interface 512 enables data communication and signaling between the various items of apparatus 500, as shown in FIG. 5. For example, the interface 512 may be one or more buses such as address, data, or control buses, and may include any interconnection mechanism, such as a series of lines on a motherboard or integrated circuit, fiber optics or other optical communication equipment, and the like. Computer program code (e.g. instructions) 505, including control 506 may comprise object-oriented software configured to pass data or messages between objects within computer program code 505, or computer program code (e.g. instructions) 505, including control 506 may include functional, scripting, or procedural code. The apparatus 500 need not comprise each of the features mentioned, or may comprise other features as well. The various components of apparatus 500 may at least partially reside in a common housing 528, or a subset of the various components of apparatus 500 may at least partially be located in different housings, which different housings may include housing 528.

[0090] FIG. 6 shows a schematic representation of non-volatile memory media 600a (e.g. computer / compact disc (CD) or digital versatile disc (DVD)) and 600b (e.g. universal serial bus (USB) memory stick) and 600c (e.g. cloud storage for downloading instructions and / or parameters 602 or receiving emailed instructions and / or parameters 602) storing instructions and / or parameters 602 which when executed by a processor allows the processor to perform one or more of the steps of the methods described herein. Instructions and / or parameters 602 may represent a computer readable medium.

[0091] FIG. 7 is an example method 700 based on the examples described herein. At 710, the method includes determining an association between at least one inference instance and a performance monitoring instance, wherein the association between the at least one inference instance and the performance monitoringinstance is defined based on a time window. At 720, the method includes determining a total number of valid performance monitoring instances, wherein a performance monitoring instance is considered as a valid performance monitoring instance when a monitoring reference signal resource set corresponding to the performance monitoring instance is carrying at least one predicted beam of a number of predicted beams having a quality metric that is higher than a quality metric of other predicted beams among a plurality of predicted beams, and when the predicted beams having the quality metric that is higher than the quality metric of the other predicted beams among the plurality of predicted beams correspond to the at least one inference instance. At 730, the method includes using the total number of valid performance monitoring instances when calculating a performance monitoring metric that represents a performance of an inference operation for beam prediction for the at least one inference instance. Method 700 may be performed with UE 110 or apparatus 500.

[0092] At 810, the method includes determining a total number of valid performance monitoring instances associated with at least one inference instance of a time window. At 820, the method includes receiving, from a user equipment, a total number of errors in performance monitoring that reflects a total number of faulty instance of beam prediction. At 830, the method includes calculating a performance monitoring metric based on at least one of: the total number of valid performance monitoring instances or the total number of errors in performance monitoring. Method 800 may be performed with RAN node 170, one or more network elements 190, or apparatus 500.

[0093] The following examples are provided and described herein.

[0094] Example 1. An apparatus including: at least one processor; and at least one memory storing instructions that, when executed by the at least one processor, cause the apparatus at least to: determine an association between at least one inference instance and a performance monitoring instance, wherein the association between the at least one inference instance and the performance monitoring instance is defined based on a time window; determine a total number of valid performance monitoring instances, wherein a performance monitoring instance is considered as a valid performance monitoring instance when a monitoring reference signal resource set corresponding to the performance monitoring instance is carrying at least one predicted beam of a number of predicted beams having a quality metric that is higher than a quality metric of other predicted beams among a plurality of predicted beams, and when the predicted beams having the quality metric that is higher than the quality metric of the other predicted beams among the plurality of predicted beams correspond to the at least one inference instance; and use the total number of valid performance monitoring instances when calculating a performance monitoring metric that represents a performance of an inference operation for beam prediction for the at least one inference instance.

[0095] Example 2. The apparatus of example 1 , wherein: the total number of valid performance monitoring instances are within a time period or a maximum number of monitoring instances; and the time period or the maximum number of monitoring instances are configured or defined for the apparatus when measuringreference signal resources corresponding to the monitoring reference signal resource set.

[0096] Example 3. The apparatus of any of examples 1 to 2, wherein the apparatus is further caused to: neglect reception of the monitoring reference signal resource set corresponding to a performance monitoring instance that is not considered as a valid performance monitoring instance when the number of prediction beams having the quality metric that is higher than the quality metric of the other predicted beams among the plurality of prediction beams are known prior to receiving the monitoring reference signal resource set.

[0097] Example 4. The apparatus of any of examples 1 to 3, wherein the apparatus is further caused to: determine a total number of errors in performance monitoring during the valid performance monitoring instances; wherein the total number of errors in performance monitoring reflects a total number of faulty instances of beam prediction.

[0098] Example 5. The apparatus of example 4, wherein the total number of valid performance monitoring instances and the total number of errors in performance monitoring are used together when calculating the performance monitoring metric.

[0099] Example 6. The apparatus of any of examples 4 to 5, wherein the apparatus is further caused to: report, to a network entity, the total number of valid performance monitoring instances; and report, to the network entity, the total number of errors in performance monitoring that reflects the total number of faulty instance of beam prediction.

[0100] Example 7. The apparatus of any of examples 4 to 6, wherein the apparatus is further caused to: determine a faulty instance of the total number of errors in performance monitoring when none of the predicted beams having the quality metric that is higher than the quality metric of the other predicted beams among the plurality of predicted beams that correspond to the at least one inference instance appear among a number of measured beams having a quality metric that is higher than a quality metric of other measured beams among a plurality of measured beams associated with the monitoring reference signal resource set.

[0101] Example 8. The apparatus of any of examples 4 to 7, wherein the apparatus is further caused to: determine a faulty instance of the total number of errors in performance monitoring when a best available beam of the predicted beams having the quality metric that is higher than a quality metric of other predicted beams among the plurality of predicted beams that correspond to the at least one inference instance does not appear among a number of measured beams having a quality metric that is higher than a quality metric of other measured beams among a plurality of measured beams associated with the monitoring reference signal resource set.

[0102] Example 9. The apparatus of any of examples 1 to 8, wherein the time window is configured or predefined, and the time window is based on a reference time instance when the apparatus determines the association between the at least one inference instance and the performance monitoring instance, wherein the reference time is defined with respect to: reception of the monitoring reference signal resource set, or reception of a measurements reference signal resource set for inference, or reporting of an inference report associatedwith the at least one inference instance, or reporting of a monitoring report.

[0103] Example 10. The apparatus of any of examples 1 to 9, wherein determining the association between the at least one inference instance and the performance monitoring instance comprises determining an association between a latest inference instance or an earliest inference instance and the performance monitoring instance, when there is more than one inference instance for the performance monitoring instance within the time window.

[0104] Example 11. An apparatus including: at least one processor; and at least one memory storing instructions that, when executed by the at least one processor, cause the apparatus at least to: determine a total number of valid performance monitoring instances associated with at least one inference instance of a time window; receive, from a user equipment, a total number of errors in performance monitoring that reflects a total number of faulty instance of beam prediction; and calculate a performance monitoring metric based on at least one of: the total number of valid performance monitoring instances or the total number of errors in performance monitoring.

[0105] Example 12. The apparatus of example 11 , wherein the apparatus is further caused to: receive, from the user equipment, a report that indicates the total number of valid performance monitoring instances.

[0106] Example 13. The apparatus of any of examples 11 to 12, wherein the total number of valid performance monitoring instances and the total number of errors in performance monitoring are used together when calculating the performance monitoring metric.

[0107] Example 14. The apparatus of any of examples 11 to 13, wherein a faulty instance of the total number of errors in performance monitoring is based on when none of predicted beams having a quality metric that is higher than a quality metric of other predicted beams among a plurality of predicted beams that correspond to the at least one inference instance appear among a number of measured beams having a quality metric that is higher than a quality metric of other measured beams among a plurality of measured beams associated with a monitoring reference signal resource set.

[0108] Example 15. The apparatus of any of examples 11 to 14, wherein a faulty instance of the total number of errors in performance monitoring is based on when a best available beam of predicted beams having ae quality metric that is higher than a quality metric of other predicted beams among a plurality of predicted beams that correspond to the at least one inference instance does not appear among a number of measured beams having a quality metric that is higher than a quality metric of other measured beams among a plurality of measured beams associated with a monitoring reference signal resource set.

[0109] Example 16. The apparatus of any of examples 11 to 15, wherein the apparatus is further caused to: transmit, to the user equipment, a configuration comprising the time window configured to be used by the user equipment to determine an association between the at least one inference instance and a performance monitoring instance comprising the valid performance monitoring instances.

[0110] Example 17. The apparatus of example 16, wherein the time window is based on a reference timeinstance that is defined with respect to: reception by the user equipment of the monitoring reference signal resource set, or reception by the user equipment of a measurements reference signal resource set for inference, or reporting of an inference report associated with the at least one inference instance, or reporting of a monitoring report.

[0111] Example 18. The apparatus of any of examples 11 to 17, wherein the time window is configured to be used by the user equipment to determine an association between the at least one inference instance and a performance monitoring instance comprising the valid performance monitoring instances, wherein the time window is based on a reference time instance that is defined with respect to: reception by the user equipment of the monitoring reference signal resource set, or reception by the user equipment of a measurements reference signal resource set for inference, or reporting of an inference report associated with the at least one inference instance, or reporting of a monitoring report.

[0112] Example 19. The apparatus of any of examples 11 to 18, wherein: the total number of errors in performance monitoring that reflects the total number of faulty instance of beam prediction is received from the user equipment based on an association between the at least one inference instance and a performance monitoring instance, and the association between the at least one inference instance and the performance monitoring instance comprises an association between an a latest inference instance or an earliest inference instance and the performance monitoring instance, when there is more than one inference instance for the performance monitoring instance within the time window.

[0113] Example 20. A method including: determining an association between at least one inference instance and a performance monitoring instance, wherein the association between the at least one inference instance and the performance monitoring instance is defined based on a time window; determining a total number of valid performance monitoring instances, wherein a performance monitoring instance is considered as a valid performance monitoring instance when a monitoring reference signal resource set corresponding to the performance monitoring instance is carrying at least one predicted beam of a number of predicted beams having a quality metric that is higher than a quality metric of other predicted beams among a plurality of predicted beams, and when the predicted beams having the quality metric that is higher than the quality metric of the other predicted beams among the plurality of predicted beams correspond to the at least one inference instance; and using the total number of valid performance monitoring instances when calculating a performance monitoring metric that represents a performance of an inference operation for beam prediction for the at least one inference instance.

[0114] Example 21. A method including: determining a total number of valid performance monitoring instances associated with at least one inference instance of a time window; receiving, from a user equipment, a total number of errors in performance monitoring that reflects a total number of faulty instance of beam prediction; and calculating a performance monitoring metric based on at least one of: the total number of valid performance monitoring instances or the total number of errors in performance monitoring.

[0115] Example 22. An apparatus including: means for determining an association between at least one inference instance and a performance monitoring instance, wherein the association between the at least one inference instance and the performance monitoring instance is defined based on a time window; means for determining a total number of valid performance monitoring instances, wherein a performance monitoring instance is considered as a valid performance monitoring instance when a monitoring reference signal resource set corresponding to the performance monitoring instance is carrying at least one predicted beam of a number of predicted beams having a quality metric that is higher than a quality metric of other predicted beams among a plurality of predicted beams, and when the predicted beams having the quality metric that is higher than the quality metric of the other predicted beams among the plurality of predicted beams correspond to the at least one inference instance; and means for using the total number of valid performance monitoring instances when calculating a performance monitoring metric that represents a performance of an inference operation for beam prediction for the at least one inference instance.

[0116] Example 23. An apparatus including: means for determining a total number of valid performance monitoring instances associated with at least one inference instance of a time window; means for receiving, from a user equipment, a total number of errors in performance monitoring that reflects a total number of faulty instance of beam prediction; and means for calculating a performance monitoring metric based on at least one of: the total number of valid performance monitoring instances or the total number of errors in performance monitoring.

[0117] Example 24. A computer readable medium including instructions stored thereon for performing at least the following: determining an association between at least one inference instance and a performance monitoring instance, wherein the association between the at least one inference instance and the performance monitoring instance is defined based on a time window; determining a total number of valid performance monitoring instances, wherein a performance monitoring instance is considered as a valid performance monitoring instance when a monitoring reference signal resource set corresponding to the performance monitoring instance is carrying at least one predicted beam of a number of predicted beams having a quality metric that is higher than a quality metric of other predicted beams among a plurality of predicted beams, and when the predicted beams having the quality metric that is higher than the quality metric of the other predicted beams among the plurality of predicted beams correspond to the at least one inference instance; and using the total number of valid performance monitoring instances when calculating a performance monitoring metric that represents a performance of an inference operation for beam prediction for the at least one inference instance.

[0118] Example 25. A computer readable medium including instructions stored thereon for performing at least the following: determining a total number of valid performance monitoring instances associated with at least one inference instance of a time window; receiving, from a user equipment, a total number of errors in performance monitoring that reflects a total number of faulty instance of beam prediction; and calculating aperformance monitoring metric based on at least one of: the total number of valid performance monitoring instances or the total number of errors in performance monitoring.

[0119] References to a ‘computer’, ‘processor’, etc. should be understood to encompass not only computers having different architectures such as single / multi-processor architectures and sequential or parallel architectures but also specialized circuits such as field-programmable gate arrays (FPGAs), application specific circuits (ASICs), signal processing devices and other processing circuitry. References to computer program, instructions, code etc. should be understood to encompass software for a programmable processor or firmware such as, for example, the programmable content of a hardware device whether instructions for a processor, or configuration settings for a fixed-function device, gate array or programmable logic device etc.

[0120] The memories as described herein may be implemented using any suitable data storage technology, such as semiconductor based memory devices, flash memory, magnetic memory devices and systems, optical memory devices and systems, non-transitory memory, transitory memory, fixed memory and removable memory. The memories may comprise a database for storing data.

[0121] The term “non-transitory,” as used herein, is a limitation of the medium itself (i.e., tangible, not a signal) as opposed to a limitation on data storage persistency (e.g., RAM vs. ROM).

[0122] As used herein, the term ‘circuitry’ may refer to the following: (a) hardware circuit implementations, such as implementations in analog and / or digital circuitry, and (b) combinations of circuits and software (and / or firmware), such as (as applicable): (i) a combination of processor(s) or (ii) portions of processor(s) / software including digital signal processor(s), software, and memories that work together to cause an apparatus to perform various functions, and (c) circuits, such as a microprocessor(s) or a portion of a microprocessor(s), that require software or firmware for operation, even if the software or firmware is not physically present. As a further example, as used herein, the term ‘circuitry’ would also cover an implementation of merely a processor (or multiple processors) or a portion of a processor and its (or their) accompanying software and / or firmware. The term ‘circuitry’ would also cover, for example and if applicable to the particular element, a baseband integrated circuit or applications processor integrated circuit for a mobile phone or a similar integrated circuit in a server, a cellular network device, or another network device.

[0123] It should be understood that the foregoing description is only illustrative. Various alternatives and modifications may be devised by those skilled in the art. For example, features recited in the various dependent claims could be combined with each other in any suitable combination(s). In addition, features from different example embodiments described above could be selectively combined into a new example embodiment. Accordingly, this description is intended to embrace all such alternatives, modifications and variances which fall within the scope of the appended claims.

[0124] The following acronyms and abbreviations that may be found in the specification and / or the drawing figures are given as follows (the abbreviations and acronyms may be appended / combined with each other or with other characters using e.g. a dash, hyphen, slash, letter, or number, and may be case insensitive):3GPP third generation partnership project4G fourth generation5G fifth generation5GC 5G core networkAl artificial intelligenceAlt alternativeAP-CSI-RS aperiodic CSI-RS BM beam managementConfig configurationCRI CSI-RS resource indicatorCSI channel state informationCSI-RS channel state information reference signalDCI downlink control informationDL downlink eNB evolved Node B (e.g., an LTE base station) en-gNB node providing NR user plane and control plane protocol terminations towards the UE, and acting as a secondary node in EN-DCE-UTRA evolved UMTS terrestrial radio access, i.e., the LTE radio access technology E-UTRAN E-UTRA networkF1 interface between the CU and the DU gNB generalized node B, base station for 5G / NR, i.e., a node providing NR user plane and control plane protocol terminations towards the UE, and connected via the NG interface to the 5GCIAB integrated access and backhaulID identifierK integer K (e.g. Top-K)L integer L (e.g. Top-L)L1-RSRP layer 1 reference signal received powerLTE long term evolution (4G)MAC medium access controlML machine learning ng or NG new generation ng-eNB new generation eNBNG-RAN new generation radio access networkNR new radioNZP non-zero powerPBCH physical broadcast channelP-CSI-RS persistent CSI-RSPDA personal digital assistantPDCP packet data convergence protocolPHY physical layerPUCCH physical uplink control channelPUSCH physical uplink shared channelRAN radio access networkRAN1 radio layer 1 , or RAN working group 1Rel releaseRLC radio link controlRRC radio resource controlRS reference signalRSRP reference signal received powerRx receive, or receiver, or receptionSDAP service data adaptation protocolSP semi-persistent (e.g. SP-CSI-RS)SP-CSI-RS semi-persistent CSI-RS SS synchronization signalSSB synchronization signal block, or synchronization signal and PBCH blockSSBRI SS / PBCH block resource indicatorTR technical reportTRP transmission reception pointTS technical specificationTx transmit, or transmitter, or transmissionUAV unmanned aerial vehicleUCI uplink control informationDE user equipment (e.g., a wireless, typically mobile device)Ul user interfaceUMTS Universal Mobile Telecommunications SystemUPF user plane functionUSB universal serial busX2 network interface between RAN nodes and between RAN and the core networkXn network interface between NG-RAN nodes

Claims

CLAIMSWhat is claimed is:

1. An apparatus comprising: at least one processor; and at least one memory storing instructions that, when executed by the at least one processor, cause the apparatus at least to: determine an association between at least one inference instance and a performance monitoring instance, wherein the association between the at least one inference instance and the performance monitoring instance is defined based on a time window; determine a total number of valid performance monitoring instances, wherein a performance monitoring instance is considered as a valid performance monitoring instance when a monitoring reference signal resource set corresponding to the performance monitoring instance is carrying at least one predicted beam of a number of predicted beams having a quality metric that is higher than a quality metric of other predicted beams among a plurality of predicted beams, and when the predicted beams having the quality metric that is higher than the quality metric of the other predicted beams among the plurality of predicted beams correspond to the at least one inference instance; and use the total number of valid performance monitoring instances when calculating a performance monitoring metric that represents a performance of an inference operation for beam prediction for the at least one inference instance.

2. The apparatus of claim 1 , wherein: the total number of valid performance monitoring instances are within a time period or a maximum number of monitoring instances; and the time period or the maximum number of monitoring instances are configured or defined for the apparatus when measuring reference signal resources corresponding to the monitoring reference signal resource set.

3. The apparatus of any of claims 1 to 2, wherein the apparatus is further caused to: neglect reception of the monitoring reference signal resource set corresponding to a performance monitoring instance that is not considered as a valid performance monitoring instance when the number of prediction beams having the quality metric that is higher than the quality metric of the other predicted beams among the plurality of prediction beams are known prior to receiving the monitoring reference signal resourceset.

4. The apparatus of any of claims 1 to 3, wherein the apparatus is further caused to: determine a total number of errors in performance monitoring during the valid performance monitoring instances; wherein the total number of errors in performance monitoring reflects a total number of faulty instances of beam prediction.

5. The apparatus of claim 4, wherein the total number of valid performance monitoring instances and the total number of errors in performance monitoring are used together when calculating the performance monitoring metric.

6. The apparatus of any of claims 4 to 5, wherein the apparatus is further caused to: report, to a network entity, the total number of valid performance monitoring instances; and report, to the network entity, the total number of errors in performance monitoring that reflects the total number of faulty instance of beam prediction.

7. The apparatus of any of claims 4 to 6, wherein the apparatus is further caused to: determine a faulty instance of the total number of errors in performance monitoring when none of the predicted beams having the quality metric that is higher than the quality metric of the other predicted beams among the plurality of predicted beams that correspond to the at least one inference instance appear among a number of measured beams having a quality metric that is higher than a quality metric of other measured beams among a plurality of measured beams associated with the monitoring reference signal resource set.

8. The apparatus of any of claims 4 to 7, wherein the apparatus is further caused to: determine a faulty instance of the total number of errors in performance monitoring when a best available beam of the predicted beams having the quality metric that is higher than a quality metric of other predicted beams among the plurality of predicted beams that correspond to the at least one inference instance does not appear among a number of measured beams having a quality metric that is higher than a quality metric of other measured beams among a plurality of measured beams associated with the monitoring reference signal resource set.

9. The apparatus of any of claims 1 to 8, wherein the time window is configured or predefined, and the time window is based on a reference time instance when the apparatus determines the association between the at least one inference instance and the performance monitoring instance, wherein the reference time is defined with respect to: reception of the monitoring reference signal resource set, or reception of a measurements reference signal resource set for inference, orreporting of an inference report associated with the at least one inference instance, or reporting of a monitoring report.

10. The apparatus of any of claims 1 to 9, wherein determining the association between the at least one inference instance and the performance monitoring instance comprises determining an association between a latest inference instance or an earliest inference instance and the performance monitoring instance, when there is more than one inference instance for the performance monitoring instance within the time window.11 . An apparatus comprising: at least one processor; and at least one memory storing instructions that, when executed by the at least one processor, cause the apparatus at least to: determine a total number of valid performance monitoring instances associated with at least one inference instance of a time window; receive, from a user equipment, a total number of errors in performance monitoring that reflects a total number of faulty instance of beam prediction; and calculate a performance monitoring metric based on at least one of: the total number of valid performance monitoring instances or the total number of errors in performance monitoring.

12. The apparatus of claim 11 , wherein the apparatus is further caused to: receive, from the user equipment, a report that indicates the total number of valid performance monitoring instances.

13. The apparatus of any of claims 11 to 12, wherein the total number of valid performance monitoring instances and the total number of errors in performance monitoring are used together when calculating the performance monitoring metric.

14. The apparatus of any of claims 11 to 13, wherein a faulty instance of the total number of errors in performance monitoring is based on when none of predicted beams having a quality metric that is higher than a quality metric of other predicted beams among a plurality of predicted beams that correspond to the at least one inference instance appear among a number of measured beams having a quality metric that is higher than a quality metric of other measured beams among a plurality of measured beams associated with a monitoring reference signal resource set.

15. The apparatus of any of claims 11 to 14, wherein a faulty instance of the total number of errors in performance monitoring is based on when a best available beam of predicted beams having ae quality metric that is higher than a quality metric of other predicted beams among a plurality of predicted beams that correspond to the at least one inference instance does not appear among a number of measured beamshaving a quality metric that is higher than a quality metric of other measured beams among a plurality of measured beams associated with a monitoring reference signal resource set.

16. The apparatus of any of claims 11 to 15, wherein the apparatus is further caused to: transmit, to the user equipment, a configuration comprising the time window configured to be used by the user equipment to determine an association between the at least one inference instance and a performance monitoring instance comprising the valid performance monitoring instances.

17. The apparatus of claim 16, wherein the time window is based on a reference time instance that is defined with respect to: reception by the user equipment of the monitoring reference signal resource set, or reception by the user equipment of a measurements reference signal resource set for inference, or reporting of an inference report associated with the at least one inference instance, or reporting of a monitoring report.

18. The apparatus of any of claims 11 to 17, wherein the time window is configured to be used by the user equipment to determine an association between the at least one inference instance and a performance monitoring instance comprising the valid performance monitoring instances, wherein the time window is based on a reference time instance that is defined with respect to: reception by the user equipment of the monitoring reference signal resource set, or reception by the user equipment of a measurements reference signal resource set for inference, or reporting of an inference report associated with the at least one inference instance, or reporting of a monitoring report.

19. The apparatus of any of claims 11 to 18, wherein: the total number of errors in performance monitoring that reflects the total number of faulty instance of beam prediction is received from the user equipment based on an association between the at least one inference instance and a performance monitoring instance, and the association between the at least one inference instance and the performance monitoring instance comprises an association between an a latest inference instance or an earliest inference instance and the performance monitoring instance, when there is more than one inference instance for the performance monitoring instance within the time window.

20. A method comprising: determining an association between at least one inference instance and a performance monitoring instance, wherein the association between the at least one inference instance and the performance monitoring instance is defined based on a time window;determining a total number of valid performance monitoring instances, wherein a performance monitoring instance is considered as a valid performance monitoring instance when a monitoring reference signal resource set corresponding to the performance monitoring instance is carrying at least one predicted beam of a number of predicted beams having a quality metric that is higher than a quality metric of other predicted beams among a plurality of predicted beams, and when the predicted beams having the quality metric that is higher than the quality metric of the other predicted beams among the plurality of predicted beams correspond to the at least one inference instance; and using the total number of valid performance monitoring instances when calculating a performance monitoring metric that represents a performance of an inference operation for beam prediction for the at least one inference instance.

21. A method comprising: determining a total number of valid performance monitoring instances associated with at least one inference instance of a time window; receiving, from a user equipment, a total number of errors in performance monitoring that reflects a total number of faulty instance of beam prediction; and calculating a performance monitoring metric based on at least one of: the total number of valid performance monitoring instances or the total number of errors in performance monitoring.

22. An apparatus comprising: means for determining an association between at least one inference instance and a performance monitoring instance, wherein the association between the at least one inference instance and the performance monitoring instance is defined based on a time window; means for determining a total number of valid performance monitoring instances, wherein a performance monitoring instance is considered as a valid performance monitoring instance when a monitoring reference signal resource set corresponding to the performance monitoring instance is carrying at least one predicted beam of a number of predicted beams having a quality metric that is higher than a quality metric of other predicted beams among a plurality of predicted beams, and when the predicted beams having the quality metric that is higher than the quality metric of the other predicted beams among the plurality of predicted beams correspond to the at least one inference instance; and means for using the total number of valid performance monitoring instances when calculating a performance monitoring metric that represents a performance of an inference operation for beam prediction for the at least one inference instance.

23. An apparatus comprising: means for determining a total number of valid performance monitoring instances associated with atleast one inference instance of a time window; means for receiving, from a user equipment, a total number of errors in performance monitoring that reflects a total number of faulty instance of beam prediction; and means for calculating a performance monitoring metric based on at least one of: the total number of valid performance monitoring instances or the total number of errors in performance monitoring.

24. A computer readable medium comprising instructions stored thereon for performing at least the following: determining an association between at least one inference instance and a performance monitoring instance, wherein the association between the at least one inference instance and the performance monitoring instance is defined based on a time window; determining a total number of valid performance monitoring instances, wherein a performance monitoring instance is considered as a valid performance monitoring instance when a monitoring reference signal resource set corresponding to the performance monitoring instance is carrying at least one predicted beam of a number of predicted beams having a quality metric that is higher than a quality metric of other predicted beams among a plurality of predicted beams, and when the predicted beams having the quality metric that is higher than the quality metric of the other predicted beams among the plurality of predicted beams correspond to the at least one inference instance; and using the total number of valid performance monitoring instances when calculating a performance monitoring metric that represents a performance of an inference operation for beam prediction for the at least one inference instance.

25. A computer readable medium comprising instructions stored thereon for performing at least the following: determining a total number of valid performance monitoring instances associated with at least one inference instance of a time window; receiving, from a user equipment, a total number of errors in performance monitoring that reflects a total number of faulty instance of beam prediction; and calculating a performance monitoring metric based on at least one of: the total number of valid performance monitoring instances or the total number of errors in performance monitoring.

Images