User equipment, network node, and methods for evaluating model performance

Abstract

The present disclosure relates to a method (400) performed by a user equipment, UE (220), in a radio network (100), the method comprising: calculating model performance metrics based on a received configuration, wherein the model performance metric reflects a performance of a beam prediction model, transmitting a report indicating the calculated model performance metric.

Description

[0001] USER EQUIPMENT, NETWORK NODE, AND METHODS FOR EVALUATING MODEL PERFORMANCE

[0002] TECHNICAL FIELD

[0003] The embodiments herein relate to a user equipment, (UE), a network node, and methods for evaluating model performance.

[0004] BACKGROUND

[0005] In radio networks today, large antenna arrays are used to improve performance at both receiver and transmitter sides. In particular, performance of radio channels between radio access nodes and UE.

[0006] The large antenna arrays use beamforming by controlling the use of antennas. To reduce hardware costs, large antenna arrays for high frequencies use time-domain analog beamforming. The core idea of analog beamforming is to share a single radio frequency chain between many (or, potentially, all) of the antenna elements. A limitation of analog beamforming is that it is only possible to transmit radio energy in using one beam (in one direction) at a given time.

[0007] The above limitation requires the network (NW) and UE to preform beam management procedures to establish and maintain suitable transmitter (Tx) / receiver (Rx) beam-pairs.

[0008] Conventionally this is typically performed by transmitting reference symbols in different antenna beams in downlink, measuring the reference signals at the UE, report reference signal characteristics to the serving network node and the network nodes takes a decision on beam pairs to be used. This has the drawback of introducing delay in selecting beam pairs.

[0009] There is therefore a need to improve beam management procedures.

[0010] SUMMARY

[0011] The above-described drawbacks are overcome by the subject matter described herein. Further advantageous implementation forms of embodiments herein are described herein, and set out in the appended set of claims.

[0012] An object of embodiments herein is to improve the performance of a radio network. According to an aspect of embodiments herein, the object is achieved by a method performed by a UE in a radio network. The UE calculates a model performance metric based on a received configuration. The model performance metric reflects a performance of a beam prediction model. The UE transmits a report indicating the calculated model performance metric.

[0013] According to an aspect of embodiments herein, the object is achieved by a method performed by a network node in a radio network. The network node generates a configuration for antenna beam prediction performance monitoring. The network node transmits a signal comprising the configuration to a UE.

[0014] According to an aspect the objective is achieved, according to embodiments herein, by providing a UE, and a network node configured to perform the methods herein, respectively.

[0015] . Reference will be made to the appended sheets of drawings that will first be described briefly.

[0016] BRIEF DESCRIPTION OF THE DRAWINGS

[0017] It should be appreciated that like reference numerals are used to identify like elements illustrated in one or more of the figures.

[0018] Fig. 1 A shows an example of a radio network. The radio network comprises a plurality of nodes configured to transmit radio signals in the radio network.

[0019] Fig. 1 B illustrates transmitter and receiver antenna beams according to one or more embodiments of the present disclosure.

[0020] Fig. 2 illustrates demand on accuracy for model performance metrics.

[0021] Fig. 3A shows signaling for antenna beam management according to one or more embodiments of the present disclosure.

[0022] Fig. 3B-C shows options for the AI / ML model.

[0023] Fig. 4 a flowchart of a method according to one or more embodiments of the present disclosure.

[0024] Fig. 5 shows a flowchart of a method according to one or more embodiments of the present disclosure.

[0025] Fig. 6 illustrates Inference-accuracy based real-time monitoring. Fig. 7 shows details of a network node according to one or more embodiments of the present disclosure.

[0026] Fig. 8 illustrates SSB beam selection as part of Initial access procedure according to the P1 scenario.

[0027] Fig. 9 illustrates CSI-RS Tx beam selection in Downlink according to a P2 scenario.

[0028] Fig. 10 illustrates UE Rx beam selection for corresponding CSI-RS Tx beam in DL according to the P3 scenario.

[0029] Fig. 11 illustrates an example where Set B is a subset of Set A.

[0030] Fig. 12: Illustrates an example where Set A is a set of narrow beams and Set B is a set of wide beams.

[0031] Fig. 13 illustrates a table with alternatives for Performance metric(s) of AI / ML model monitoring for BM-Case 1 and BM-Case 2.

[0032] A more complete understanding of embodiments herein will be afforded to those skilled in the art, as well as a realization of additional advantages thereof, by a consideration of the following detailed description of one or more embodiments. It should be appreciated that like reference numerals are used to identify like elements illustrated in one or more of the figures.

[0033] DETAILED DESCRIPTION

[0034] The present disclosure relates to beam management procedures for large array antenna systems. In particular, for systems using prediction of characteristics of transmitter or receiver beams or beam transmitter / receiver beam pairs utilizing Artificial Intelligence, Al, and / or Machine Learning, ML.

[0035] A problem with such prediction is that there is a large overhead in signaling, reference symbol, RS, transmission resources and user equipment, UE measurements needed for beam prediction performance metric calculations. To achieve sufficient statistics to determine, at the network, NW, side whether to activate / deactivate a model or not, there is a need to have a large amount of samples. The term sample denotes herein a comparison between values predicted by a model and ground truth data of the value, e.g., predicted signal strength and signal strength measured using a reference symbol. This comparison may include calculating model performance metrics.

[0036] Based on the NW received performance metrics, the NW can for example have a threshold on the required accuracy needed for activating the NW / UE-sided beam prediction model, for example the required accuracy needs to be above 50% for the model to be activated (or not deactivated). Then, a performance metric such as beam prediction accuracy above 60% might impact the selection of a set of radio parameters (e.g., how many beams to measure in Top- K) in a certain way, and above 70% in another way. For example, in case of very high accuracy, the NW might not need to perform extra measurements to find the top-1 beam in the BM use case in 3d generation Partnership Project, 3GPP. E.g., BM-Case1 : Spatial-domain DL beam prediction for Set A of beams based on measurement results of Set B of beams.

[0037] One problem occurs for the low-accuracy (erroneous) prediction models. The NW might spend an unnecessary number of resources for the purpose of performance metric calculation on models that are performing significantly bad, i.e., having a metric below e.g., 50%. For this case, the NW is not interested in activating and using such models.

[0038] Moreover, the NW is not interested in understanding whether an underperforming model achieves 10% or 20 %, it will anyway not activate such model. Note that 50% is an arbitrary selected number, for other models used for LIRLLC traffic, a model might need >90% accuracy to be activated.

[0039] In general, the NW should spend more resources to understand if a model accuracy is 90% or 95%, in comparison to whether the model have 5% or 10% accuracy.

[0040] It is thus a problem how to minimize the resource overhead, signaling overhead and UE battery consumption for performing beam prediction performance monitoring procedures.

[0041] The present disclosure is partially based on the realization that in case a beam prediction model performs badly and gives very large prediction errors, it might be enough with only one or two samples to determine that the model is underperforming and should be deactivated.

[0042] Hence, early detection and deactivation of an underperforming AI / ML model is utilized in embodiments herein.

[0043] Fig. 1A shows an example of a radio network 100. The radio network comprises a plurality of nodes N1-N21 configured to transmit radio signals in the radio network 100. Each node comprises one or more antennas and circuitry capable of performing radio transmission. In one non-limiting example the nodes are radio access nodes and / or user operated nodes such as UEs.

[0044] A radio signal transmission between nodes and / or UEs is then performed using certain transmission characteristics.

[0045] Fig. 1 B illustrates transmitter and receiver antenna beams according to one or more embodiments of the present disclosure. The network node 210 is in the example in Fig. 1 B shown with eight TX antenna beams TX1- TX8. The UE 220 is in the example in Fig. 1 B shown with eight RX antenna beams RX1-RX8. A suitable antenna pair, e.g., TX5 and RX1 may then be selected using antenna beam management procedures.

[0046] The large antenna array systems typically comprise plurality of receive antennas RX1-RX8 and a plurality of transmit antennas TX1-TX8. The respective antennas are electrically and communicatively coupled using a phase network.

[0047] In the present disclosure antenna beams where characteristics are measured are shown as dashed lines and remaining antenna beams are shown with solid lines.

[0048] In one example, beam characteristics of RS signals received from TX beams TX3, TX5 and TX7 are measured by the UE. The method of the present disclosure then predicts characteristics of antenna beams TX1-TX8 and then optionally ranks the antenna beams TX1- TX8 in terms of suitability for node-UE communication. E.g., the antenna beam providing the best signal strength and / or signal quality.

[0049] In one example, a model performance metrics can then be calculated by comparing the characteristics of the measured signals received from TX beams TX3, TX5 and TX7 and the predicted characteristics of TX beams TX3, TX5 and TX7.

[0050] In one aspect of the disclosure, a method is provided to enable the NW to trigger the UE to transmit a report comprising an early performance indication / model performance metrics (e.g., an indicator of the model performance, or performance metric calculation). The early transmitted report is beneficial for underperforming AI / ML beam prediction models, to reduce the signaling, reference symbol resources, UE measurement overhead and UE battery consumption in performance metric calculation. In other words, there is no need to send RS, measure RS, predict beams and report performance if the results of the beam prediction are too bad to be considered.

[0051] Fig. 2 illustrates demand on accuracy for model performance metrics.

[0052] Examples of embodiments herein utilizes the property that the NW is only interested in having accurate performance metrics of the model for the range of performance where it will activate the model, and that underperforming models can be rather quickly identified. This is highlighted in Fig. 2.

[0053] Examples of embodiments herein also provides means for the NW to quickly react to degradation of the UE sided beam prediction models by allowing an early indication of prediction degradation in the observed performance monitoring transparently performed by the UE.

[0054] To summarize, the present disclosure includes at least a method in the UE to calculate beam prediction performance metrics, and if the performance is lower than desirable (e.g., lower than a threshold), then the UE can send an early report to the NW.

[0055] The NW upon receiving such early report of an underperforming AI / ML model, disables the performance metric assessment in the UE, i.e., disable the transmission of reference symbols, RS, by a serving network node intended for this performance estimation. This also reduces the future RS overhead, signaling overhead and reduces UE battery consumption (since the UE disables the performance metric calculations as a reaction to this disabling from the NW or as a reaction of reporting an early report to the NW).

[0056] The present disclosure has at least the following advantages:

[0057] Reduced overhead from disabled RS transmissions

[0058] Reduced UE battery consumptions due to early abortion of performance monitoring.

[0059] Reduced NW energy consumption due to disabled RS transmissions.

[0060] Reduced interference to neighboring cells due to RS transmissions.

[0061] Reduce the impact of underperforming model on the NW by providing a mean to react faster to model performance degradation (e.g., NW requesting the UE to deactivate functionality / model upon reception of early performance indication)

[0062] Fig. 3A shows signaling for antenna beam management according to one or more embodiments of the present disclosure.

[0063] Action 310. The network node 210 generates a configuration for antenna beam prediction performance monitoring.

[0064] Action 315. The network node 210 then sends a first signal S1 comprising the configuration to the UE 220.

[0065] In one embodiment the NW, such as the network node 210, configures the UE 220 to monitor the functionality / model based on N time instances and report performance indication on a functionality level. In one example, the UE 220 is configured to perform N measurements, perform N predictions and calculate N model performance metrics or samples of performance. The N model performance metrics is then aggregated in a suitable manner to a total model performance metrics. The UE 220 is requested to report an indication of performance failure of the model, while the performance metric calculation is kept transparent from the NW. In another embodiment, the NW, such as the network node 210 configures the UE 220 to monitor the model based on N time instances and report performance metrics corresponding the model.

[0066] The NW, such as the network node 210 indicates / configures rules on when the UE 220 can trigger / report an early performance indication.

[0067] In a context of the example above, this means that an aggregated model performance metrics is generated using less than N samples.

[0068] The signaling for UE reporting / the configuration can comprise on or more of: indication if early performance indication is allowed or not definition of the events upon which the UE 220 may trigger an early performance indications.

[0069] Indication of the time / frequency resources that can be used for reporting

[0070] List of ranges, including indication for in which range the UE 220 should report an exact value based on all N time instances

[0071] A threshold value for the tolerated error (see example of real-time monitoring below)

[0072] A minimum number of samples / time instances the UE 220 shall use before making an assessment whether the metric is below or above the threshold value

[0073] Whether the UE 220 should abort ongoing performance monitoring if the tolerated error is below the threshold value (to save battery)

[0074] Action 320. The UE 220 then calculates model performance metrics, such as a model performance metric, based on the received configuration, wherein the model is optionally configured to rank a set of candidate beamforming beams. The model performance metric may e.g., reflect the performance the model.

[0075] The UE 220 has, before the calculation of model performance metrics, performed measurements on a set of beams, e.g., TX3, TX5 and TX7 shown in Fig. 1 B.

[0076] The predicted characteristics of beams TX1-TX8, e.g., measured and ranked Signal to Interference plus Noise Ratio, SI NR, and / or measured and ranked Reference Signal Strength Indicator, RSRP, is then compared to measured beams TX3, TX5 and TX7. It is understood that the measured characteristics of the beams may include any suitable characteristics, such as Reference Signal Received Quality, RSRQ, and / or Received Signal Strength Indicator, RSSI. In one example, the NW, such as the network node 210, generated configuration can be represented using a classifier. E.g., as a list indicating the corresponding ranges e.g. [0%, 50%, 75%, 85%, 90%, 99%, 100%]. The list can indicate in which range the UE 220 should report the exact value based on all measurement time instances (N). The number N can be included in the configuration sent to the UE 220.

[0077] In one further example, the NW, such as the network node 210, generated configuration can be represented as a uniform list of ranges with a single value indicating the resolution. For example, a value 10 divides the range of 10 uniform distributed bins [0%, 10%,20%, ... .90%, 100%].

[0078] In one further example, the NW, such as the network node 210, generated configuration can be a regression value, e.g., as a list indicating the mean / median / max / min error values [0-3 dB error, 3-5 dB error, 5dB->infinitive error]. The error may e.g., represent difference between measured and predicted characteristics. In embodiments, the list can indicate in which range the UE should report the exact value based on all measurement time instances (N).

[0079] NW selection of ranges

[0080] The NW, such as the network node 210, can for example select the ranges in the configuration based on its needed accuracy for the beam prediction model. For example, if it needs more than 50% accuracy to use said model. Moreover, it can select the upper ranges based on the configurations of the model, if the model is 99% accurate instead of 90% accurate, the NW, such as the network node 210, can rely more on the predictions instead of configuring additional measurements. For example, to use option 1) instead of option 2) in the scheme shown in Fig. 3B-C which reduces the latency from measurement of characteristics to data transmission.

[0081] Set A represents all predicted beams TX1-TX8 and set B represents measured beams TX3, TX5 and TX7 on which measurements of beam characteristics, such as RSRP and SINR, is performed.

[0082] Fig. 3B shows a first option where the AI / ML model predicts Top-1 beam.

[0083] Fig. 3C shows a second option where the AI / ML model predicts Top-K beams.

[0084] For the regression case, the NW, such as the network node 210, can for example select the error ranges based on whether it can use such metric in the link-adaptation, for example a Modulation-and-Coding Scheme, MCS, selection should need an RSRP metric estimate below x dB error. Optionally, if early transmission of model performance reports / reports are enabled, The UE 220 further performs the steps of:

[0085] Action 330. The UE 220 evaluates if the calculated model performance metrics meets criteria for early transmission of the report. This is also referred herein as triggering an event.

[0086] Action 340. The UE 220 then transmits the report if the calculated model performance metrics meets the criteria in a signal S3. In other words, report an early performance indicator.

[0087] Action 350. The network node 210 may optionally transmit a signal S4 comprising an updated configuration to the UE 220.

[0088] Action 360. If no early transmission of report is performed / no event is triggered, then the UE 220 sends the model performance report / report after a predetermined time and / or periodicity and / or a time instance in a signal S5. The predetermined time and / or periodicity and / or a time instance is comprised in and defined by the configuration received in signal S1.

[0089] Fig. 4 shows a flowchart of a method according to one or more embodiments of the present disclosure. The method 400 is performed by a UE 220 in the radio network 100. The method comprises the following steps, which steps may be taken in any suitable order. Steps that are optional are presented in dashed boxes in Fig. 4.

[0090] Step 410

[0091] The UE 220 calculates model performance metrics based on the received configuration, wherein the model is configured to rank a set of candidate beamforming beams TX1-TX8, RX1-RX8. In other words, the UE 220 calculates model performance metric based on the received configuration. The model performance metric reflects a performance of a beam prediction model. The received configuration may e.g., comprise the configuration comprised in signal S1 received from the network node 210, as described above. This step is related to Action 315 above.

[0092] In some embodiments, the received configuration is indicative of instructions to the UE 220 for calculating the model performance metric based on a first number ‘X’ of time instances. This may e.g., mean that the configuration configures the UE 220 to calculate the performance metric based on the first number ‘X’ of time instances.

[0093] In some embodiments, the UE 220 is configured, e.g., by the received configuration, with a set of ranges of one or more model performance metrics. The ranges in the configured set of ranges may e.g., be uniformly distributed between a minimum value and a maximum value or non-uniformly distributed between a minimum value and a maximum value.

[0094] Step 420 The UE 220 transmits a report comprising the calculated model performance metrics. The report may e.g., be sent to the network node 210. This may comprise aggregating model performance metrics calculated at each time instance. This is related to Action 360 above.

[0095] Additionally, or alternatively, the report is transmitted periodically. This can be performed using a predetermined time and / or periodicity and / or a time instance as further described in relation to Fig. 3A.

[0096] The report may, in some embodiments, indicate a specific range out of the configured set of ranges. The specific range may comprise the calculated performance metric.

[0097] Additionally or alternatively, the method further comprises:

[0098] Step 430:

[0099] In some embodiments, the UE 220 evaluates if the calculated model performance metrics meets criteria for early transmission of the report. In other words, if an event is triggered. This is related to Action 330 above.

[0100] In one embodiment, the criteria are indicative of one or more samples of performance being below a certain performance level.

[0101] The UE can in case the event / criteria is fulfilled, for example the model performance metrics is within a certain reporting range (e.g., in range 0<->50% accuracy). The event can be triggered, or the criteria can be met for example using RRC signaling. The NW can allocate pre-configured uplink resources which the UE can use for early transmission of the report comprising model performance metrices.

[0102] In another embodiment, the UE may request dedicated resources to report the performance indication. The information can be sent using UCI or PUSCH (RRC / MAC messages)

[0103] In one embodiment, the model performance report is only an indication of failure of the model performance procedure.

[0104] In one related embodiment, the indication of failure is signaled in a pre-configured single-bit PUCCH resource.

[0105] In another related embodiment, the indication of failure is signaled by a dedicated scheduling request (SR).

[0106] In one embodiment, the model performance failure report also contains information of a new preferred gNB beam. For example, if the model performance is poor, the predicted gNB beam reported in other gNB beam prediction reports associated with that model might not be valid. Hence, in this case, the model performance failure report also could include the top K gNB beams based on actual measurements, such that the network can attain / maintain a suitable beam pair link between the gNB and the UE.

[0107] Below are some examples of different performance level configuration that could trigger transmission of an early model performance metric report (if it happens for N samples of performance, where N can be 1 or larger than 1):

[0108] • RSRP of predicted best beam differs more than X dB compared to RSRP of measured best beam (where the predicted best beam and measured best beam can be two different beams)

[0109] • Predicted RSRP and measured RSRP of the predicted best beam differs more than X dB (i.e., the predicted RSRP and measured RSRP are compared for the same beam)

[0110] • Assume that the UE predicts RSRP for K top beams and that the UE measures RSRP for the same K top beams. In this case, an early performance metric report can be triggered / transmitted for example if:

[0111] -the RSRP difference between predicted RSRP and measured RSRP for any of the K top beams is above a certain threshold (for example assume that K=2, and that the difference in predicted RSRP and measured RSRP for the first beam is 3 dB, the predicted RSRP and measured RSRP for the second beam is 6 dB, and the threshold is 4 dB, then an early performance metric report event is triggered / transmitted since at least one of the K top beams have an RSRP difference above the threshold)

[0112] -the RSRP difference between predicted RSRP and measured RSRP for all of the K top beams is above a certain threshold (for example assume that K=2, and that the difference in predicted RSRP and measured RSRP for the first beam is 3 dB, the predicted RSRP and measured RSRP for the second beam is 6 dB, and the threshold is 4 dB, then no early performance metric report event is triggered / transmitted since at least one of the K top beams have an RSRP difference less than the threshold)

[0113] -if the average RSRP difference between predicted RSRP and measured RSRP for all of the K top beams is above a certain threshold (for example assume that K=2, and that the difference in predicted RSRP and measured RSRP for the first beam is 3 dB, the predicted RSRP and measured RSRP for the second beam is 6 dB, and the threshold is 4 dB, then an early performance metric report event is triggered / transmitted since the average RSRP error across all K top beams are above the threshold (3+6 / 2 = 4,5 >4).

[0114] In some embodiments, the evaluation is based on the model performance metric calculated based on a second number of time instances smaller than the first number ‘X’ of time instances. Step 440:

[0115] In some embodiments, the UE 220 transmits the report if the calculated model performance metrics meets the criteria. The early transmission of the report is performed by sending a signal S3, as shown in Fig. 3A. This is related to Action 340 above.

[0116] The report may, in some embodiments, indicate a specific range out of the configured set of ranges. The specific range may comprise the calculated performance metric.

[0117] Additionally, or alternatively, the method further comprises:

[0118] Step 450:

[0119] In some embodiments, the UE 220 receives an updated configuration. This is related to Action 350 above.

[0120] The optional step of providing the UE with a new performance metric can be performed e.g., be via RRC configuration or via DCI signaling.

[0121] An advantage of the latter method (DCI signaling) is that it facilitates faster changes. This can e.g., be useful if the NW for some reason at some moment decides to use a different beam than the one predicted by the UE; in such case a prediction performance measure in the UE based on BLER / data link quality (such as 3GPP Alt-2) could be misleading, and the NW may want to very temporarily change the performance metric calculation.

[0122] Additionally, or alternatively, the updated configuration is received via downlink control information, DCI, signaling and / or via radio resource control, RRC, signaling.

[0123] Additionally, or alternatively, the report is transmitted via radio resource control, RRC, signaling.

[0124] Additionally, or alternatively, the report is transmitted via Uplink Control Information, UCI, signaling and / or Physical Uplink Shared Channel, PUSCH signaling and / or Dedicated Scheduling Request signaling.

[0125] Additionally, or alternatively, the received configuration is indicative of instructions to the UE to calculate the model performance based on a number X of time instances and wherein the report is indicative of a functionality level.

[0126] Additionally, or alternatively, the number X is equal to one or greater than one.

[0127] Additionally, or alternatively, the received configuration is indicative of instructions to the UE to calculating the model performance based on a number X of time instances and wherein the report is indicative of a model specific performance measure.

[0128] Additionally, or alternatively, the received configuration is indicative of a selection of any of: -An indication if early transmission of the report is allowed or not.

[0129] -A definition of criteria upon which the UE may trigger early transmission of the report.

[0130] -an indication of the time / frequency resources that can be used for transmitting the report.

[0131] -A list of model performance ranges.

[0132] -A threshold value indicative of a tolerated prediction error of the model.

[0133] -A minimum number of measurement samples and / or time instances the UE shall use before evaluating the calculated model performance.

[0134] -An indication on whether the UE 220 should abort ongoing performance monitoring if the calculating model performance metrics is below the threshold value.

[0135] Additionally, or alternatively, the calculated model performance metrics is based on a distance in time between predictions not meeting evaluation criteria and / or or frequency of predictions not meeting the evaluation criteria.

[0136] In this embodiment, the distance between two consecutive error samples of performance (also called “error events"), is used in calculating the model performance metric. Here the distance refers to the difference in time between two error samples / events. The error event refers to an unsatisfactory AI / ML model output. For example, for beam prediction, an error event is an erroneous prediction by the AI / ML model, e.g., a predicted RSRP and a measured RSRP of the predicted best beam differs more than X dB.

[0137] In general, if an AI / ML model performs well, error events are rare, and the distance derrbetween two consecutive error events is large. Conversely, if an AI / ML model deteriorates, error events are more frequent, and the distance derrbetween two consecutive error events is smaller. Thus, distance derrcan be an early indicator of model performance degradation.

[0138] In one embodiment, the statistics of distance derris calculated and used to derive model performance metrics. The statistics can be the mean .d,err and standard deviation od.err of distance derrin a time window. Furthermore, the time window can be a sliding time window to keep track of the ongoing AI / ML model performance. The model monitoring metric can be a function of .d,err and od.err f(jxd,err, Od.err). If f(jxd,err, Od.err) is smaller (i.e. , worse) than a threshold Td.err then an early performance metric report is triggered to signal a warning sign of model deterioration; otherwise, the model is considered normal, and no warning sign is triggered.

[0139] The function f(|id,err, Od.err) can be: f(|id,err, Od.err) = | d,err+ A* Od.err, where A is a constant (e.g., A=1 or A=2). The threshold Td.err can be a fixed or pre-defined value, which is set based on the normal performance of the model and obtained using a reference dataset (e.g., test dataset or training dataset). Alternatively, the threshold Td.err can be a dynamically changing value, for example, Td.err—P ( 'd,err,max+ A Od.err.max ), Where (pd , err, max, Od, err, max ) corresponds to the best performing instance (i.e., largest distance) observed in a historical reference time period. Parameter p (p<1_ is a scaling parameter for setting the threshold, e.g., p=0.1 or 0.2.

[0140] Additionally, or alternatively, the calculated model performance metrics is further based on a severity classification of predictions not meeting the evaluation criteria.

[0141] In this embodiment, the model performance metric utilizes both (a) the distance between two consecutive error events and (b) the severity of the error event. This is based on the rationale that when an AI / ML model deteriorates, the error events tend to be more frequent, and the model output values tend to be further away from the ground truth. Thus, a model performance metric can be formulated to reflect both, which is used to determine whether an early performance metric report should be triggered / transmitted.

[0142] In one example, the model monitoring metric function F can be a function of the statistics of the distance derrbetween two consecutive errors, and the statistics of the distance between the predicted RSRP and the ground truth RSRP of beam prediction. One specific example is, F is a function of pd.err, Od.err PRSRP.err, ORSRP.err.

[0143] In one example,

[0144] Here (pd , err, max, Od, err, max ) corresponds to the mean and standard deviation of the best performing instance (largest distance) observed in a historical reference time period; and (pRSRP.em.max, ORSRP.err.max ) corresponds to the mean and standard deviation of the worst RSRP error (largest difference between the predicted RSRP and the ground truth RSRP) observed in a historical reference time period. Parameters wi and W2 are used to provide the relative weighting between distance of error events and severity of the error. If Fmd,err>sd,err>mRSRP,err,sRSRP,err) IsSmaller (i.e., WOPSe) than a threshold Td.RSRP.err then transmission of an early performance metric report is triggered to signal a warning sign of model deterioration; otherwise, the model is considered normal and no warning sign is triggered.

[0145] Additionally, or alternatively, the transmitted report comprises a result of a function converting the calculated model performance metrics to indicia, such as an index. The index may e.g., indicate a specific range of the configured set of ranges. The specific range may comprise the calculated model performance metric. This way only the index of the range has to be transmitted in the report, instead of the calculated performance metric. This may e.g., lead to a reduced signaling overhead.

[0146] In some embodiments, there is a function f that for any given number of samples and associated model performance metrics / quality measures converts it into an index (or a binary decision on whether to send an early performance metric report). The function f can be predefined in the specifications but possibly also with one or more parameters configured (or signaled) by the network.

[0147] The per-sample model performance metrics / quality measures may be binary (success / failure) or a multiple-step scale according to any of the previous embodiments. In case of binary measure, the measures can under measurement uncertainty be seen as a binomial distribution with parameters n (sample occasions so far) and p (per-sample probability of failure or success), which the function f can be based on.

[0148] Additionally, or alternatively, the function uses time stamps of measurement samples to weight samples when calculating the model performance metrics.

[0149] The function f can optionally in addition to quality measures take as input the (relative) timing (time stamps) of the samples. For example, the function f may give lower weight to earlier sample occasions and more weight to more recent sample occasions, accounting for the fact that the more recent samples may better reflect current performance under changing channel conditions.

[0150] In one non-limiting example, the NW configures the UE to estimate and / or rank the beam prediction accuracy. The NW can for example configure the UE with different resolutions for the performance metric calculation (such as the beam prediction accuracy). One example of different ranges of model performance is illustrated below for the accuracy metric:

[0151] Index 0 Index 1 Index 2 Index 3

[0152] 0->50 50->75 75-90 Report exact number In this example, the UE is configured to measure on 20 time instances. The UE will already after time instance 10 understand, if all 10 calculated model performance metrics indicate that the predictions have been erroneous, that the final result / model performance metrics will be part of the first range (0-50% / lndex 0). Hence, the UE can send an early feedback to the NW (index 0). The NW can then refrain from spending more RS resources on the monitoring procedure (saving 10 of the 20 time-instances in this simple example). The UE report an indication that its accuracy is in the first range (0->50%).

[0153] In another example, the network can configure the UE to trigger an early model performance metric report event if one or more samples are below a certain performance level. For example, the network can configure the UE to report an early model performance metric report event if the predicted RSRP of best beam and actually RSRP of the measured best beams TX3, TX5 and TX7 differs more than a certain threshold. The network can also configure the number of samples that this must happen before the early performance metric report event is triggered.

[0154] Additionally, or alternatively, the configuration comprises constraints for calculating the model performance metrics.

[0155] In one example, the NW can set constraints when a sample can be part of the performance metric calculations, for example the UE can only include a new sample in case it differs from previous sample with a certain threshold (thereby not including duplicated samples).

[0156] On some embodiments, the UE can be configured in addition to the current set of samples take into account also past measurements when calculating the performance metric.

[0157] In some embodiments, the UE 220 receives an indication to not calculate a model performance metric in remaining time instances of the first number ‘X’ of time instances. The indication may e.g., indicate to not calculate the model performance metric in remaining time instances of the first number ‘X’ of time instances in case an early report has been, or is to be, transmitted. The indication is received in any one out of the received configuration, the updated configuration, or separately from either of the received configuration and the updated configuration.

[0158] Additionally, or alternatively, the UE 220 receives a request from the network node 210 to deactivate the model responsive to the transmitted report being an early transmitted report.

[0159] Fig. 5 shows a flowchart of a method according to one or more embodiments of the present disclosure. The method 500 is performed by the network node 210, in the radio network 100. The method comprises the following steps, which steps may be taken in any suitable order. Steps that are optional are presented in dashed boxes in Fig. 5.

[0160] Step 510 The network node 210 generates a configuration for antenna beam prediction performance monitoring. This is related to Action 310 above.

[0161] Step 520

[0162] The network node 210 transmits a signal S1 comprising the configuration to the UE 220. This is related to Action 315 above.

[0163] Additionally, or alternatively, the method further comprises receiving a report indicative of beam prediction performance, such as indicative of a model performance metric of a beam prediction model. This is related to Action 340 and / or 360 above. This based on whether the report is an early report (Action 340) or not an early report (Action 360).

[0164] Additionally, or alternatively, the method further comprises transmitting an updated configuration. This is related to Action 350 above.

[0165] Additionally, or alternatively, the updated configuration is transmitted via downlink control information, DCI, signaling and / or via radio resource control, RRC, signaling.

[0166] Additionally, or alternatively, the report is received via radio resource control, RRC, signaling.

[0167] Additionally, or alternatively, the report is received via Uplink Control Information, UCI, signaling and / or Physical Uplink Shared Channel, PUSCH signaling and / or Dedicated Scheduling Request signaling.

[0168] Additionally, or alternatively, the transmitted configuration is indicative of instructions to the UE to calculate the model performance based on a number X of time instances and wherein the report is indicative of a functionality level.

[0169] Additionally, or alternatively, the number X is equal to one or greater than one.

[0170] Additionally, or alternatively, the transmitted configuration is indicative of instructions to the UE 220 to calculating the model performance based on a number X of time instances and wherein the report is indicative of a model specific performance measure.

[0171] Additionally, or alternatively, the transmitted configuration is indicative of a selection of any of: an indication if early transmission of the report is allowed or not, definition of criteria upon which the UE may trigger early transmission of the report, an indication of the time / frequency resources that can be used for transmitting the report, a list of model performance ranges, a threshold value indicative of a tolerated prediction error of the model, a minimum number of measurement samples and / or time instances the UE shall use before evaluating the calculated model performance, an indication on whether the UE (220) should abort ongoing performance monitoring if the calculating model performance metrics is below the threshold value.

[0172] Additionally, or alternatively, the transmitted configuration is indicative of instructions to the UE 220 to calculate a model performance metric based on a first number ‘X’ of time instances.

[0173] Additionally, or alternatively, the received report is an early transmitted report where the model performance metric of the beam prediction model is based on a second number of time instances smaller than the first number ‘X’ of time instances. This is related to Action 340 above.

[0174] Additionally, or alternatively, the network node sends an indication to not calculate a performance metric in remaining time instances of the first number ‘X’ of time instances. The indication may e.g., indicate to not calculate the model performance metric in remaining time instances of the first number ‘X’ of time instances in case an early report has been, or is to be, transmitted by the UE 220. The indication is received in any one out of the received configuration, the updated configuration, or separately from either of the received configuration and the updated configuration.

[0175] Additionally, or alternatively, the configuration indicates a set of ranges of one or more performance metrics.

[0176] Additionally, or alternatively, the report indicates a specific range out of the configured set of ranges. The specific range may comprise the calculated performance metric.

[0177] Additionally, or alternatively, the set of ranges is any one out of uniformly distributed between a minimum value and a maximum value, or non-uniformly distributed between a minimum value and a maximum value.

[0178] Additionally, or alternatively, the report comprises an index indicating a specific range among the configured set of ranges. The specific range may comprise the model performance metric.

[0179] Additionally, or alternatively, the network node 220 requests the UE 220 to deactivate the model responsive to the received report being an early transmitted report.

[0180] Fig. 6 illustrates Inference-accuracy based real-time monitoring.

[0181] As described in 3GPP RAN1#114 “Ericsson-R1 -2306928 Discussion on general aspects of Al ML framework". One aspect of UE / NW-sided models is real-time monitoring of AI / ML models, where the NW can activate a model in a certain time window based on the monitoring results in a previous time-window. This is a resource consuming scheme where extra resources for monitoring should be minimized if possible.

[0182] According to examples of embodiments herein, the UE can trigger transmission of an early report, where the NW have preconfigured UL resources for sending control information, in any of the time-instances 1->7 in Fig. 6. The UE can in this embodiment be configured with a value of the required accuracy of being activated in the subsequent time window (e.g., require 6 / 7 to be activated, then UE can trigger an early report if it has observed 2 errors. In case the UE observes 2 errors in time instance 1-2, it will hence save 5 measurements.

[0183] Fig. 7 shows details of a network node 210 according to one or more embodiments of the present disclosure.

[0184] The network node 210 may be in the form of a selection of any of a gNB, a network node, a desktop computer, server, laptop, mobile device, a smartphone, a tablet computer, a smart watch etc. The network node 210 may comprise processing circuitry 912. The network node 210 may optionally comprise a communications interface 904 for wired and / or wireless communication. Further, the network node 210 may further comprise at least one optional antenna (not shown in figure). The antenna may be coupled to a transceiver of the communications interface 904 and is configured to transmit and / or emit and / or receive wireless signals, e.g., in a wireless communication system.

[0185] In one example, the processing circuitry 912 may be any of a selection of processor and / or a central processing unit and / or processor modules and / or multiple processors configured to cooperate with each-other. Further, the network node 210 may further comprise a memory 915. The memory 915 may contain instructions executable by the processing circuitry 912, that when executed causes the processing circuitry 912 to perform any of the methods and / or method steps described herein.

[0186] The communications interface 904, e.g., the wireless transceiver and / or a wired / wireless communications network adapter is configured to send and / or receive data values or parameters as a signal. In an embodiment, the communications interface 904 communicates directly between nodes or via a communications network.

[0187] In one or more embodiments the network node 900 may further comprise an input device or interface 917, configured to receive input or indications from a user and output / send a userinput signal indicative of the user input or indications to the processing circuitry 912. In other words, the input device / interface 917 receives input or indications from the user and translates this into data that the processing circuitry 912 can interpret. In one or more embodiments the network node 210 may further comprise a display 918 configured to receive a display signal indicative of rendered objects, such as text or graphical user input objects, from the processing circuitry 912 and to display the received signal as objects, such as text or graphical user input objects.

[0188] In one embodiment the display 918 is integrated with the user input device / interface 917 and is configured to receive a display signal indicative of rendered objects, such as text or graphical user input objects, from the processing circuitry 912 and to display the received signal as objects, such as text or graphical user input objects, and / or configured to receive input or indications from a user and send a user-input signal indicative of the user input or indications to the processing circuitry 912.

[0189] In one or more embodiments the network node 210 may further comprise one or more sensors (not shown).

[0190] In embodiments, the processing circuitry 912 is communicatively coupled to the memory 915 and / or the communications interface 904 and / or the input device 917 and / or the display 918.

[0191] In embodiments, the communications interface and / or transceiver 904 communicates using wired and / or wireless communication techniques.

[0192] In embodiments, the one or more memory 915 may comprise a selection of a hard RAM, disk drive, a floppy disk drive, a magnetic tape drive, an optical disk drive, a CD or DVD drive (R or RW), or other removable or fixed media drive.

[0193] In a further embodiment, the network node 210 may further comprise and / or be coupled to one or more additional sensors (not shown) configured to receive and / or obtain and / or measure physical properties pertaining to the network node or the environment of the network node and send one or more sensor signals indicative of the physical properties to the processing circuitry 912.

[0194] It is to be understood that a network node comprises any suitable combination of hardware and / or software needed to perform the tasks, features, functions, and methods disclosed herein. Moreover, while the components of the network node are depicted as single boxes located within a larger box, or nested within multiple boxes, in practice, a network node may comprise multiple different physical components that make up a single illustrated component (e.g., memory 915 may comprise multiple separate hard drives as well as multiple RAM modules).

[0195] Similarly, the network node 210 may be composed of multiple physically separate components, which may each have their own respective components. The communications interface 904 may also include multiple sets of various illustrated components for different wireless technologies, such as, for example, Global System for Mobile Communications GSM, Wideband Code-Division Multiple Access, WCDMA, Long- Term Evolution, LTE, New Radio, NR, Wireless Fidelity, Wi-Fi, or Bluetooth wireless technologies. These wireless technologies may be integrated into the same or different chip or set of chips and other components within the network node 210.

[0196] Processing circuitry 912 is configured to perform any determining, calculating, or similar operations (e.g., certain obtaining operations) described herein as being provided by a network 900. These operations performed by processing circuitry 912 may include processing information obtained by processing circuitry 912 by, for example, converting the obtained information into other information, comparing the obtained information, or converted information to information stored in the network node, and / or performing one or more operations based on the obtained information or converted information, and as a result of said processing making a determination.

[0197] Processing circuitry 912 may comprise a combination of one or more of a microprocessor, controller, microcontroller, central processing unit, digital signal processor, application-specific integrated circuit, field programmable gate array, or any other suitable computing device, resource, or combination of hardware, software and / or encoded logic operable to provide, either alone or in conjunction with other network node 210 components, such as device readable medium, computer 210 functionality. For example, processing circuitry 912 may execute instructions stored in device readable medium 915 or in memory within processing circuitry 912. Such functionality may include providing any of the various wireless features, functions, or benefits discussed herein. In some embodiments, processing circuitry 912 may include a system on a chip.

[0198] In some embodiments, processing circuitry 912 may include one or more of radio frequency, RF, transceiver circuitry and baseband processing circuitry. In some embodiments, RF transceiver circuitry and baseband processing circuitry may be on separate chips or sets of chips, boards, or units, such as radio units and digital units. In alternative embodiments, part or all of RF transceiver circuitry and baseband processing circuitry may be on the same chip or set of chips, boards, or units.

[0199] In certain embodiments, some or all the functionality described herein as being provided by a network node 210 may be performed by the processing circuitry 912 executing instructions stored on device readable medium 915 or memory within processing circuitry 912. In alternative embodiments, some or all the functionalities may be provided by processing circuitry 912 without executing instructions stored on a separate or discrete device readable medium, such as in a hard-wired manner. In any of those embodiments, whether executing instructions stored on a device readable storage medium or not, processing circuitry 912 can be configured to perform the described functionality. The benefits provided by such functionality are not limited to processing circuitry 912 alone or to other components of network node but are enjoyed by network node 210 and / or by end users.

[0200] Device readable medium or memory 915 may comprise any form of volatile or non-volatile computer readable memory including, without limitation, persistent storage, solid-state memory, remotely mounted memory, magnetic media, optical media, random access memory (RAM), read-only memory (ROM), mass storage media (for example, a hard disk), removable storage media (for example, a flash drive, a Compact Disk (CD) or a Digital Video Disk (DVD)), and / or any other volatile or non-volatile, non-transitory device readable and / or computerexecutable memory devices that store information, data, and / or instructions that may be used by processing circuitry 912. Device readable medium 915 may store any suitable instructions, data, or information, including a computer program, software, an application including one or more of logic, rules, code, tables, etc. and / or other instructions capable of being executed by processing circuitry 912 and, utilized by network node 210. Device readable medium may be used to store any calculations made by processing circuitry 912 and / or any data received via interface 904. In some embodiments, processing circuitry 912 and device readable medium 915 may be considered to be integrated.

[0201] The communications interface 904 is used in the wired or wireless communication of signaling and / or data between network node 210 and other nodes. Interface 904 may comprise port(s) / terminal(s) to send and receive data, for example to and from network node 210 over a wired connection. Interface 904 also includes radio front end circuitry that may be coupled to, or in certain embodiments a part of, an antenna. Radio front end circuitry may comprise filters and amplifiers. Radio front end circuitry may be connected to the antenna and / or processing circuitry 912.

[0202] Examples of a network node 210 include, but are not limited to an gNB, a gateway, a smart phone, a mobile phone, a cell phone, a voice over I P (Vol P) phone, a wireless local loop phone, a tablet computer, a desktop computer, a personal digital assistant (PDA), a wireless cameras, a gaming console or device, a music storage device, a playback appliance, a wearable terminal device, a wireless endpoint, a mobile station, a tablet, a laptop, a laptop-embedded equipment (LEE), a laptop-mounted equipment (LME), a smart device, a wireless customer-premise equipment (CPE), a vehicle-mounted wireless terminal device, a drone etc.

[0203] The communication interface 904 may encompass wired and / or wireless networks such as a local-area network (LAN), a wide-area network (WAN), a computer network, a wireless network, a telecommunications network, another like network or any combination thereof. The communication interface may be configured to include a receiver and a transmitter interface used to communicate with one or more other devices over a communication network according to one or more communication protocols, such as Ethernet, TCP / IP, SONET, ATM, optical, electrical, and the like). The transmitter and receiver interface may share circuit components, software, or firmware, or alternatively may be implemented separately.

[0204] In embodiments, the UE 220 may comprise all of the features described in relation to Fig. 7, or a subset of the features described in relation to Fig. 7.

[0205] A more detailed background of beam prediction is provided below.

[0206] One of the key features of New Radio, NR, compared to previous generation of wireless networks, is the ability to operate in higher frequencies (e.g., above 10 GHz). The available large transmission bandwidths in these frequency ranges can potentially provide large data rates. However, as carrier frequency increases, both pathloss and penetration loss increase.

[0207] To maintain the coverage levels at the same level, highly directional antenna beams are required to focus the radio transmitter energy in a particular direction on the receiver. However, large radio antenna arrays, at both receiver and transmitter sides, are needed to create such highly directional beams.

[0208] To reduce hardware costs, large antenna arrays for high frequencies use time-domain analog beamforming. The core idea of analog beamforming is to share a single radio frequency, RF, chain between many (or, potentially, all) of the antenna elements of the large antenna array. A limitation of analog beamforming is that it is only possible to transmit radio energy using one beam (in one direction) at any given time.

[0209] The above limitation requires the network (NW) and user equipment (UE) to preform beam management procedures to establish and maintain suitable transmitter (Tx) / receiver (Rx) beam-pairs. For example, beam management procedures can be used by a transmitter to sweep a geographic area by transmitting reference signals on different candidate beams, during non-overlapping time intervals, and using a predetermined pattern of the sweep. By measuring the quality of these reference signals at the receiver side, the best transmit and receive beams can be identified.

[0210] NR Beam management procedures

[0211] Beam management procedures in NR are defined by a set of L1 / L2 procedures that establish and maintain a suitable beam pairs for both transmitting and receiving data. A beam management procedure can include the following sub procedures: beam determination, beam measurements, beam reporting, and beam sweeping. In case of downlink transmission from the NW to the UE, P1 / P2 / P3 beam management procedures can be performed, e.g., according to the NR SI technical report, to overcome the challenges of establishing and maintaining the TX / RX beam pairs when, for example, a UE moves or some blockage in the environment requires changing the beams.

[0212] Although these scenarios are not directly mentioned in the 3rd Generation Partnership Project ,3GPP, specifications, there are relevant procedures defined which enables the realization of these scenarios. The procedures P1-P3 are further described below.

[0213] P1 : The P1 procedure is used to enable UE measurement on different transmission / reception point (TRP) Tx beams to support selection of TRP Tx beams / UE Rx beam(s). During initial access, for example, the gNB transmits SS / PBCH block Synchronization Signal Block, SSB, beams in different directions to cover the whole cell. The UE measures signal quality on corresponding SSB signals to detect and select an appropriate Synchronization Signal Block, SSB, beam, this is shown in Fig. 8.

[0214] Fig. 8 illustrates SSB beam selection as part of Initial access procedure according to the P1 scenario.

[0215] Random access is then transmitted on the Random Access Channel, RACH, resources indicated by the selected SSB. The corresponding beam will be used by both the UE and the network to communicate until connected mode beam management is active. The network infers which SSB beam was chosen by the UE without any explicit signaling.

[0216] For beamforming at the Transmission / Reception Point, TRP, it typically includes an intra / inter- TRP Tx beam sweep from a set of different beams. For beamforming at the UE, a UE Rx beam sweep from a set of different beams is typically included.

[0217] P2: The P2 procedure is used to enable UE measurement on different TRP Tx beams to possibly change inter / intra-TRP Tx beam(s). The network can use the SSB beam as an indication of which (narrow) CSI-RS beams to try; that is, the selected SSB beam can be used to define a candidate set of narrow CSI-RS beams for beam management. Once Channel State Information Reference Symbol, CSI-RS, is transmitted, the UE measures the Reference Signal Strength Indicator, RS

[0218] RP, and reports the result to the network. If the network receives a CSI-RSRP report from the UE where a new CSI-RS beam is better than the old used to transmit the Physical Downlink Control Channel / Physical Downlink Shared Channel PDCCH / PDSCH, the network updates the serving beam for the UE accordingly, and possibly also modifies the candidate set of CSI- RS beams. The network can also instruct the UE to perform measurements on SSBs. If the network receives a report from the UE where a new SSB beam is better than the previous best SSB beam, a corresponding update of the candidate set of CSI-RS beams for the UE may be motivated.

[0219] The P2 procedure is performed on a possibly smaller set of beams for beam refinement than in P1. Note that P2 can be a special case of P1. For example, in connected mode the network node, e.g., gNB, configures the UE with different CSI-RSs and transmits each CSI-RS on a corresponding beam. UE then measures the quality of each CSI-RS beam on its current RX beam and send feedback about the quality of the measured beams. Thereafter, based on this feedback, gNB will decide and possibly indicates to the UE which beam will be used in future transmissions. This is shown in Fig. 9.

[0220] Fig. 9 illustrates CSI-RS Tx beam selection in Downlink according to a P2 scenario.

[0221] P3: is used to enable UE measurement on the same TRP Tx beam to change UE Rx beam in the case UE uses beamforming. Once in connected mode, the UE is configured with a set of reference signals. Based on measurements, the UE determines which Rx beam is suitable to receive each reference signal in the set. The network then indicates which reference signals are associated with the beam that will be used to transmit PDCCH / PDSCH, and the UE uses this information to adjust its Rx beam when receiving PDCCH / PDSCH.

[0222] In connected mode, P3 can be used by the UE to find the best Rx beam for corresponding Tx beam. In this case the network node / gNB keeps one CSI-RS Tx beam at a time, and UE performs the sweeping and measurements on its own Rx beams for that specific Tx beam. UE then finds the best corresponding Rx beam based on the measurements and will use it in future for reception when gNB indicates the use of that Tx beam.

[0223] Fig. 10 illustrates UE Rx beam selection for corresponding CSI-RS Tx beam in DL according to the P3 scenario.

[0224] Beam measurement and reporting in NR

[0225] For beam management, a UE can be configured to report RSRP or / and Signal to Interference plus Noise Ratio, SINR, for each one of up to four beams, either on CSI-RS or SSB. UE measurement reports can be sent either over PUCCH or PUSCH to the network node, e.g., gNB.

[0226] Reference signal configurations in NR

[0227] CSI-RS:

[0228] A CSI-RS is transmitted over each transmit (Tx) antenna port at the network node and for different antenna ports. The CSI-RS are multiplexed in time, frequency, and code domain such that the channel between each Tx antenna port at the network node and each receive antenna port at a UE can be measured by the UE. The time-frequency resource used for transmitting CSI-RS is referred to as a CSI-RS resource.

[0229] In NR, the CSI-RS for beam management is defined as a 1- or 2-port CSI-RS resource in a CSI-RS resource set where the filed repetition is present. The following three types of CSI-RS transmissions are supported:

[0230] Periodic CSI-RS: CSI-RS is transmitted periodically in certain slots. This CSI-RS transmission is semi-statically configured using RRC signaling with parameters such as CSI-RS resource, periodicity, and slot offset.

[0231] Semi-Persistent CSI-RS: Similar to periodic CSI-RS, resources for semi-persistent CSI-RS transmissions are semi-statically configured using RRC signaling with parameters such as periodicity and slot offset. However, unlike periodic CSI-RS, dynamic signaling is needed to activate and deactivate the CSI-RS transmission.

[0232] Aperiodic CSI-RS: This is a one-shot CSI-RS transmission that can happen in any slot. Here, one-shot means that CSI-RS transmission only happens once per trigger. The CSI-RS resources (i.e., the RE locations which consist of subcarrier locations and OFDM symbol locations) for aperiodic CSI-RS are semi-statically configured. The transmission of aperiodic CSI-RS is triggered by dynamic signaling through PDCCH using the CSI request field in UL DCI, in the same DCI where the UL resources for the measurement report are scheduled. Multiple aperiodic CSI-RS resources can be included in a CSI-RS resource set and the triggering of aperiodic CSI-RS is on a resource set basis.

[0233] SSB:

[0234] In NR, an SSB consists of a pair of synchronization signals (SSs), physical broadcast channel (PBCH), and DMRS for PBCH. An SSB is mapped to 4 consecutive OFDM symbols in the time domain and 240 contiguous subcarriers (20 RBs) in the frequency domain.

[0235] NR supports beamforming and beam-sweeping for SSB transmission, by enabling a cell to transmit multiple SSBs in different narrow-beams multiplexed in time. The transmission of these SSBs is confined to a half frame time interval (5 ms). It is also possible to configure a cell to transmit multiple SSBs in a single wide-beam with multiple repetitions. The design of beamforming parameters for each of the SSBs within a half frame is up to network implementation. The SSBs within a half frame are broadcasted periodically from each cell. The periodicity of the half frames with SS / PBCH blocks is referred to as SSB periodicity, which is indicated by SIB1.

[0236] The maximum number of SSBs within a half frame, denoted by L, depends on the frequency band, and the time locations for these L candidate SSBs within a half frame depends on the SCS of the SSBs. The L candidate SSBs within a half frame are indexed in an ascending order in time from 0 to L-1. By successfully detecting Physical Broadcast Channel, PBCH, and its associated Downlink Demodulation Reference Signal, DMRS, a UE knows the SSB index. A cell does not necessarily transmit SS / PBCH blocks in all L candidate locations in a half frame, and the resource of the un-used candidate positions can be used for the transmission of data or control signaling instead. It is up to network implementation to decide which candidate time locations to select for SSB transmission within a half frame, and which beam to use for each SSB transmission.

[0237] Measurement resource configurations in NR

[0238] A UE can be configured with the following:

[0239] N>1 CSI reporting settings (CSI-ReportConfig) and

[0240] M>1 resource settings (CSI-ResourceConfig).

[0241] Each CSI reporting setting is linked to one or more resource setting for channel and / or interference measurement. The CSI framework is modular in the sense that several CSI reporting settings may be associated with the same Resource Setting.

[0242] The measurement resource configurations for beam management are provided to the UE by RRC information element (IE) (CSI-ResourceConfigs). One CSI-ResourceConfig contains several NZP-CSI-RS-ResourceSets and / or CSI-SSB-ResourceSets.

[0243] A UE can be configured to measure CSI-RSs using the RRC IE NZP-CSI-RS-ResourceSet. A NZP CSI-RS resource set contains the configurations of Ks >1 CSI-RS resources. Each CSI- RS resource configuration resource includes at least the following: mapping to REs, the number of antenna ports, and time-domain behavior.

[0244] Up to 64 CSI-RS resources can be grouped together in an NZP-CSI-RS-ResourceSet.

[0245] A UE can be configured to measure SSBs using the RRC IE CSI-SSB-ResourceSet. Resource sets comprising SSB resources are defined in a similar manner to the CSI-RS resources defined above.

[0246] In the case of aperiodic CSI-RS and / or aperiodic CSI reporting, the network node configures the UE with ScCSI triggering states. Each triggering state contains the aperiodic CSI report setting to be triggered along with the associated aperiodic CSI-RS resource sets. Periodic and semi-persistent resource settings can only comprise a single resource set (i.e., S=1). Aperiodic resource settings can have many resources sets (S>= 1 ) , because one out of the S resource sets defined in the resource setting is indicated by the aperiodic triggering state that triggers a CSI report.

[0247] Measurement Reporting

[0248] Three types of CSI reporting are supported in NR as follows:

[0249] Periodic CSI Reporting on Physical Uplink Control Channel, PUCCH,: CSI is reported periodically by a UE. Parameters such as periodicity and slot offset are configured semi- statically by higher layer RRC signaling from the network node to the UE.

[0250] Semi-Persistent CSI Reporting on Physical Uplink Control Channel or PUCCH: similar to periodic CSI reporting, semi-persistent CSI reporting has a periodicity and slot offset which may be semi-statically configured. However, a dynamic trigger from network node to UE may be needed to allow the UE to begin semi-persistent CSI reporting. A dynamic trigger from network node to UE is needed to request the UE to stop the semi-persistent CSI reporting.

[0251] Aperiodic CSI Reporting on Physical Uplink Control Channel, PUSCH. This type of CSI reporting involves a single-shot (i.e., one time) CSI report by a UE which is dynamically triggered by the network node using DCI. Some of the parameters related to the configuration of the aperiodic CSI report is semi-statically configured by RRC but the triggering is dynamic.

[0252] In each CSI reporting setting, the content and time-domain behavior of the report is defined, along with the linkage to the associated Resource Settings.

[0253] The CSI-ReportConfig IE comprise the following configurations: reportConfigType

[0254] Defines the time-domain behavior (periodic CSI reporting, semi-persistent CSI reporting, or aperiodic CSI reporting) along with the periodicity and slot offset of the report for periodic CSI reporting. reportQuantity

[0255] Defines the reported CSI parameters -- the CSI content; for example, the PM I, CQI, Rl, LI (layer indicator), CRI (CSI-RS resource index) and L1-RSRP. Only certain combinations are possible ; for example, ‘cri-RI-PMI-CQI’ is one possible value and ‘cri-RSRP’ is another) and each value of reportQuantity could be said to correspond to a certain CSI mode. codebookConfig Defines the codebook used for PM I reporting, along with possible codebook subset restriction (CBSR). NR supported the following two types of PMI codebooks: Type I CSI and Type II CSI. Additionally, the Type I and Type II codebooks each have two different variants: regular and port selection. reportFrequencyConfiguration

[0256] Define the frequency granularity of PMI and CQI (wideband or sub band), if reported, along with the CSI reporting band, which is a subset of sub bands of the bandwidth part (BWP) which the CSI corresponds to

[0257] Measurement restriction in time domain (ON / OFF) for channel and interference respectively

[0258] For beam management, a UE can be configured to report L1-RSRP for up to four different CSI-RS / SSB resource indicators. The reported RSRP value corresponding to the first (best) CRI / SSBRI requires 7 bits, using absolute values, while the others require 4 bits using encoding relative to the first. In NR release 16, the report of L1-SINR for beam management has already been supported.

[0259] Agreements in 3GPP

[0260] During the 3GPP meeting RAN1#109-e it was agreed to study AI / ML based spatial beam prediction, the core idea of which is as follows: Predict the “best” beam (or beams) from a Set A of beams using measurement results from another Set B of beams.

[0261] Set A and Set B of beams have not been defined yet (left for future study); however, the following two examples illustrate some scenarios that will likely be studied in Release 18:

[0262] Set B is a subset of a Set A. For example, Set A is a set of 8 SSB / CSI-RS beams shown in Fig. 11 (both light and dark circles). The UE measures Set B (the 4 beams indicated by dark circles). The AI / ML model should predict the best beam (or beams) in Set A using only measurements from Set B.

[0263] Fig. 11 illustrates an example where Set B is a subset of Set A. The figure illustrates a grid-of- beam type radiation pattern: Each row (resp. column) depicts a certain zenith (resp. azimuth) angle from the antenna array. Set A has 8 beams and Set B has 4 beams (indicated by dark circles).

[0264] Set A and Set B correspond to two different sets of beams. For example, Set A is a set of 30 narrow CSI-RS beams, and Set B is a set of 8 wide SSB beams. The UE measures beams in Set B and the AI / ML model should predict the best beam(s) from Set A.

[0265] Fig. 12: Illustrates an example where Set A is a set of narrow beams and Set B is a set of wide beams. The spatial beam prediction can be performed in the gNB or the UE - the study item will cover both scenarios.

[0266] During the 3GPP meeting RAN1#110, it was agreed to study AI / ML model training both at the NW and UE side. Which side that performs the training is expected to impact how data collection is performed, where another agreement is to study the aspect of data collection for beam management. Moreover, it was agreed to study the aspect of model monitoring and the standard impact on AI / ML model inference (e.g., reporting of predicted values).

[0267] Third Generation Partnership Project, 3GPP, study item technical report

[0268] The following text is captured in the TR 38.843 regarding performance monitoring.

[0269] Performance monitoring:

[0270] For the performance monitoring of BM-Casel and BM-Case2:

[0271] - Performance metric(s) with the following alternatives:

[0272] - Alt.1 : Beam prediction accuracy related KPIs, e.g., Top-K / 1 beam prediction accuracy

[0273] - Alt.2: Link quality related KPIs, e.g., throughput, Ll-RSRP, Ll-SINR, hypothetical BLER

[0274] - Alt.3 : Performance metric based on input / output data distribution of AI / ML

[0275] - Alt.4: The Ll-RSRP difference evaluated by comparing measured RSRP and predicted RSRP

[0276] - Benchmark / reference for the performance comparison, including:

[0277] - Alt.1 : The best beam(s) obtained by measuring beams of a set indicated by gNB (e.g., Beams from Set A)

[0278] - Alt.4: Measurements of the predicted best beam(s) corresponding to model output (e.g., Comparison between actual Ll-RSRP and predicted RSRP of predicted Top-l / K Beams)

[0279] - Signalling / configuration / measurement / report for model monitoring, e.g., signalling aspects related to assistance information (if supported), Reference signals

[0280] For BM-Casel and BM-Case2 with a UE-side AI / ML model:

[0281] - Typel performance monitoring:

[0282] - Configuration / Signalling from gNB to UE for measurement and / or reporting

[0283] - UE may have different operations

[0284] - Optionl : UE sends reporting to NW (e.g., for the calculation of performance metric at NW) - Option2: UE calculates performance metric(s), either reports it to NW or reports an event to NW based on the performance metric(s)

[0285] - Indication from NW for UE to do LCM operations

[0286] - Note: At least the performance and reporting overhead of model monitoring mechanism should be considered

[0287] - Type2 performance monitoring (UE-side performance monitoring):

[0288] - Indication / request / report from UE to gNB for performance monitoring

[0289] - Note: The indication / request / report may be not needed in some case(s)

[0290] - Configuration / Signalling from gNB to UE for performance monitoring measurement and / or reporting

[0291] - UE calculates performance metric(s), either reports it to NW or reports an event to NW based on the performance metric(s)

[0292] - If it is for UE-side model monitoring, UE makes decision(s) of model selection / activation / deactivation / switching / fallback operation

[0293] - Indication from NW to UE to do LCM operation

[0294] - UE reporting of beam measurement(s) based on a set of beams indicated by gNB

[0295] - Signalling, e.g., RRC-based, Ll-based

[0296] - Note: Performance and UE complexity, power consumption should be considered

[0297] Mechanism that facilitates the UE to detect whether the functionality / model is suitable or no longer suitable

[0298] Table 7.2.3-1 summarizes applicability of various alternatives for performance metric(s) of AI / ML model monitoring for BM-Casel and BM-Case2.

[0299] Fig. 13 illustrates a table with alternatives for Performance metric(s) of AI / ML model monitoring for BM-Case 1 and BM-Case 2.

[0300] Abbreviations used herein.

[0301] 3GPP 3rd Generation Partnership Project

[0302] 5G Fifth Generation

[0303] ACK Acknowledgement Al Artificial Intelligence

[0304] AoA Angle of Arrival

[0305] CORESET Control Resource Set

[0306] CSI Channel State Information

[0307] CSI-RS CSI Reference Signal

[0308] DCI Downlink Control Information

[0309] DoA Direction of Arrival

[0310] DL Downlink

[0311] DMRS Downlink Demodulation Reference Signals

[0312] FDD Frequency-Division Duplex

[0313] FR2 Frequency Range 2

[0314] HARQ Hybrid Automatic Repeat Request

[0315] ID identity gNB gNodeB

[0316] MAC Medium Access Control

[0317] MAC-CE MAC Control Element

[0318] ML Machine Learning

[0319] NR New Radio

[0320] NW Network

[0321] OFDM Orthogonal Frequency Division Multiplexing

[0322] PBCH Physical Broadcast Channel

[0323] PCI Physical Cell Identity

[0324] PDCCH Physical Downlink Control Channel

[0325] PDSCH Physical Downlink Shared Channel

[0326] PRB Physical Resource Block

[0327] QCL Quasi co-located

[0328] RB Resource Block

[0329] RRC Radio Resource Control RSRP Reference Signal Strength Indicator

[0330] RSRQ Reference Signal Received Quality

[0331] RSSI Received Signal Strength Indicator

[0332] SCS Subcarrier Spacing

[0333] SINR Signal to Interference plus Noise Ratio

[0334] SSB Synchronization Signal Block

[0335] RL Reinforcement Learning

[0336] RS Reference Signal

[0337] Rx Receiver

[0338] TB Transport Block

[0339] TDD Time-Division Duplex

[0340] TCI Transmission configuration indication

[0341] TRP Transmission / Reception Point

[0342] Tx Transmitter

[0343] UE User Equipment

[0344] UL Uplink

[0345] Finally, it should be understood that the embodiments herein are not limited to the embodiments described above, but also relates to and incorporates all embodiments within the scope of the appended independent claims.

Claims

CLAIMS1. A method (400) performed by a user equipment, UE (220), in a radio network (100), the method comprising: calculating (320, 410) a model performance metric based on a received configuration, wherein the model performance metric reflects a performance of a beam prediction model, transmitting (340, 360, 420) a report indicating the calculated model performance metric.

2. The method according to claim 1 , wherein the report is transmitted periodically.

3. The method according to any of the preceding claims, wherein the method further comprises: evaluating (320, 430) if the calculated model performance metric meets one or more criteria for early transmission of the report, and, transmitting (340, 440) the report if the calculated model performance metric meets the one or more criteria.

4. The method according to claim 3, wherein the received configuration is indicative of instructions to the UE (220) for calculating the model performance metric based on a first number ‘X’ of time instances, and wherein the evaluation is based on the model performance metric calculated based on a second number of time instances smaller than the first number ‘X’ of time instances.

5. The method according to any of claims 3-4, wherein the method further comprises receiving (350, 450) an updated configuration.

6. The method according to claim any of claims 3-5, wherein the method comprises receiving an indication to not calculate a model performance metric in remaining time instances of the first number ‘X’ of time instances, wherein the indication is received in any one out of:- the received configuration,- the updated configuration, or- separately from either of the received configuration and the updated configuration.

7. The method according to any of claims 5-6, wherein the received configuration and / or the updated configuration is received via downlink control information, DCI, signaling and / or via radio resource control, RRC, signaling.

8. The method according to any of the preceding claims, wherein the UE (220) is configured with a set of ranges of one or more model performance metrics, and wherein the report indicates a specific range out of the configured set of ranges, and the specific range comprises the calculated performance metric.

9. The method according to claim 8, wherein the ranges in the configured set of ranges are any one out of:- uniformly distributed between a minimum value and a maximum value, or- non-uniformly distributed between a minimum value and a maximum value.

10. The method according to any of the preceding claims, wherein the report is transmitted via radio resource control, RRC, signaling and / or Uplink Control Information, UCI, signaling and / or Physical Uplink Shared Channel, PUSCH signaling and / or Dedicated Scheduling Request signaling.

11. The method according to any of the preceding claims, wherein the received configuration is indicative of a selection of any of: an indication if early transmission of the report is allowed or not, definition of criteria upon which the UE may trigger early transmission of the report, an indication of the time / frequency resources that can be used for transmitting the report, a list of uniform and / or non-uniform model performance ranges, a threshold value indicative of a tolerated prediction error of the model, a minimum number of measurement samples and / or time instances the UE shall use before evaluating the calculated model performance, an indication on whether the UE (220) should abort ongoing performance monitoring if the calculating model performance metric is below the threshold value,12. The method according to any of the preceding claims, wherein the calculated model performance metric is based on a distance in time between predictions not meeting an evaluation criterion and / or or frequency of predictions not meeting the evaluation criterion.

13. The method according to claim 12, wherein the calculated model performance metric is further based on a severity classification of predictions not meeting the evaluation criterion.

14. The method according to any of the preceding claims, wherein the transmitted report comprises a result of a function converting the calculated model performance metric to an index.

15. The method according to claim 14, wherein the index indicates a specific range among the configured set of ranges, and the specific range comprises the calculated model performance metric.

16. The method according to any of claims 14-15, wherein the function uses time stamps of measurement samples to weight the measurement samples when calculating the model performance metric.

17. The method according to any of the preceding claims, further comprising receiving a request from the network node (210) to deactivate the model responsive to the transmitted report being an early transmitted report.

18. A user device, UE, in a radio network (100), the user device comprising: a processor, and a memory, said memory containing instructions executable by said processor, whereby said user device is operative to perform the method according to any of claims 1-17.

19. A method (500) performed by a network node (210), in a radio network (100), the method comprising: generating (310, 510) a configuration for antenna beam prediction performance monitoring, transmitting (315, 520) a signal (S1) comprising the configuration to a UE (220).

20. The method according to claim 19, further comprising receiving (340, 360) a report indicative of a model performance metric of a beam prediction model.

21. The method according to claim 20, wherein the transmitted configuration is indicative of instructions to the UE (220) to calculate a model performance metric based on a first number ‘X’ of time instances, and wherein the report is an early transmitted report where the model performance metric of the beam prediction model is based on a second number of time instances smaller than the first number ‘X’ of time instances.

22. The method according to claim any of claims 21 , wherein the method comprises sending an indication to not calculate a performance metric in remaining time instances of the first number ‘X’ of time instances, wherein the indication is sent in any one out of:- the transmitted configuration,- an updated configuration, or- separately from either of the transmitted configuration and the updated configuration.

23. The method according to any of claims 20-22, wherein configuration indicates a set of ranges of one or more performance metrics, and wherein the report indicates a specific range out of the configured set of ranges, and the specific range comprises the calculated performance metric.

24. The method according to claim 23, wherein the set of ranges is any one out of:- uniformly distributed between a minimum value and a maximum value, or- non-uniformly distributed between a minimum value and a maximum value.

25. The method according to any of claims 20-24, wherein the report comprises an index indicating a specific range among the configured set of ranges, and the specific range comprises the model performance metric.

26. The method according to any of claims 19-25, wherein the method further comprises transmitting (350) an updated configuration.

27. The method according to any of claims 19-26, wherein the transmitted configuration and / or the updated configuration is transmitted via downlink control information, DCI, signaling and / or via radio resource control, RRC, signaling.

28. The method according to any of claims 19-27, wherein the report is received via radio resource control, RRC, signaling and / or Uplink Control Information, UCI, signaling and / or Physical Uplink Shared Channel, PUSCH signaling and / or Dedicated Scheduling Request signaling.

29. The method according to any of claims 19-28, wherein the transmitted configuration is indicative of a selection of any of: an indication if early transmission of the report is allowed or not, definition of criteria upon which the UE may trigger early transmission of the report,an indication of the time / frequency resources that can be used for transmitting the report, a list of uniform and / or non-uniform model performance ranges, a threshold value indicative of a tolerated prediction error of the model, a minimum number of measurement samples and / or time instances the UE shall use before evaluating the calculated model performance, an indication on whether the UE (220) should abort ongoing performance monitoring if the calculating model performance metric is below the threshold value,30. The method according to any of claims 20-29, further comprising requesting the UE (220) to deactivate the model responsive to the received report being an early transmitted report.

31. A node (210) in a radio network (100), the node comprising: a processor, and a memory, said memory containing instructions executable by said processor, whereby said node is operative to perform the method according to any of claims 19-30.

Images