Efficient real-time monitoring of user equipment-sided models using validity of monitoring results

Abstract

A method (1000) by a user equipment, UE (712), for monitoring of UE-sided models using validity of monitoring results includes transmitting (1002), to a network node (710), model performance feedback associated with a performance of a model during a first monitoring occasion. The UE receives (1004) configuration information associated with a validity of a model monitoring result associated with the first monitoring occasion. The configuration information is for performing at least one of activating the model, deactivating the model, and initiating a second monitoring occasion.

Description

[0001] EFFICIENT REAL-TIME MONITORING OF USER EQUIPMENT-SIDED MODELS USING

[0002] VALIDITY OF MONITORING RESULTS

[0003] TECHNICAL FIELD

[0004] The present disclosure relates, in general, to wireless communications and, more particularly, systems and methods for efficient real-time monitoring of User Equipment-sided models using validity of monitoring results.

[0005] BACKGROUND

[0006] A Release 18 study item relating to monitoring of Artificial Intelligence (Al) / Machine Learning (ML) models for Beam Management (BM) discloses that, in order to obtain a monitoring ground truth data sample, a User Equipment (UE) should be configured to measure / report a set of beams that consist of both set A and set B beams. For UE-sided models, the set B beam measurements are used as the model input to generate the model output such as, for example, top- K predicted beams, while the set A beam measurements are used for obtaining the ground truth labels. Different ground-truth labels may be needed depending on how the inference accuracy metric(s) is / are defined for model performance monitoring.

[0007] For Al models, which provide predicted Layer 1-Reference Signal Received Power (Ll- RSRP) value(s) of top- 1 / K predicted beam as the model output, for obtaining a single monitoring data sample, a UE should be configured to at least measure / report the Ll-RSRP of the same beam(s). Note that, if a UE is configured to measure / report a set of beams consisting of both set A and set B beams as discussed for the case where Al models predict top- 1 / K beams only, the collected data can also be used for monitoring the models that predict Ll-RSRP values. For example, the same framework / configuration can be reused for multiple models.

[0008] Technical Report 38.843 V18.0.0 discloses the following with respect to performance monitoring:

[0009] For the performance monitoring of BM-Casel and BM-Case2:

[0010] • Performance metric(s) with the following alternatives: o Alt.l: Beam prediction accuracy related Key Performance Indicators (KPIs), e.g., Top-K / l beam prediction accuracy o Alt.2: Link quality related KPIs, e.g., throughput, Ll-RSRP, Layer 1- Signal Interference to Noise Ratio (Ll-SINR), hypothetical Block Error Rate (BLER) o Alt.3: Performance metric based on input / output data distribution of AI / ML o Alt.4: The Ll-RSRP difference evaluated by comparing measured RSRP and predicted RSRP

[0011] • Benchmark / reference for the performance comparison, including: o Alt.1 : The best beam(s) obtained by measuring beams of a set indicated by gNodeB (gNB) (e.g., Beams from Set A) o Alt.4: Measurements of the predicted best beam(s) corresponding to model output (e.g., Comparison between actual Ll-RSRP and predicted RSRP of predicted Top- 1 / K Beams)

[0012] • Signalling / configuration / measurement / report for model monitoring, e.g., signalling aspects related to assistance information (if supported), Reference signals

[0013] For BM-Casel and BM-Case2 with a UE-side AI / ML model:

[0014] • Typel performance monitoring: o Configuration / Signalling from gNB to UE for measurement and / or reporting o UE may have different operations

[0015] ■ Optionl : UE sends reporting to Network (NW) (e.g., for the calculation of performance metric at NW)

[0016] ■ Option2: UE calculates performance metric(s), either reports it to NW or reports an event to NW based on the performance metric(s) o Indication from NW for UE to do Life Cycle Management (LCM) operations o Note: At least the performance and reporting overhead of model monitoring mechanism should be considered

[0017] • Type2 performance monitoring (UE-side performance monitoring): o Indication / request / report from UE to gNB for performance monitoring

[0018] ■ Note: The indication / request / report may be not needed in some case(s) o Configuration / Signalling from gNB to UE for performance monitoring o If it is for UE-side model monitoring, UE makes decision(s) of model selection / activation / deactivation / switching / fallback operation o UE reporting of beam measurement(s) based on a set of beams indicated by gNB o Signalling, e.g., Radio Resource Control (RRC)-based, Ll-based o Note: Performance and UE complexity, power consumption should be considered

[0019] There is an ongoing 3rdGeneration Partnership Project (3GPP) discussion on how to monitor a UE-sided model. The performance of a model / functionality might be very different over the area where it is claimed to be valid. For example, based on the UE capability report, a model / functionality is valid in a certain cell or a number of cells, or might be very different over time even when the UE is in the same location. The UE might claim that a model may achieve 90% accuracy over such wide area or for a time duration.

[0020] FIGURE 1 illustrates an example with regions where a model is not performing accurately. As depicted in FIGURE 1, the example includes three UEs that experience a local area (e.g., channel coherence time interval) part of a wide area. The performance of a model / functionality might be very different over the area where it is claimed to be valid, for example, based on the UE capability report, a model / functionality is valid in a certain cell or a number of cells. The UE might claim that a model may achieve 90% accuracy over such wide area. However, the accuracy may be due to achieving, for example:

[0021] Local area 1,2=100% accuracy

[0022] Local area 3 = 70%

[0023] A model monitored over the wide area is a slow process and, in the example depicted in FIGURE 1, will reach the desired accuracy requirement if samples are received uniformly. However, the performance in some parts of the local areas might be very bad due to, for example, temporary blockage that cannot be learned by collecting more data. Thus, the local area might not be present for subsequent devices in the same region.

[0024] One solution for detecting a possible low accuracy region is to perform real-time monitoring. However, such procedures typically correspond to some extra overhead in measurement or processing. It would, thus, be kept at minimum level. The method for model accuracy might also need to have a lot of samples to reach a stable monitoring result.

[0025] FIGURES 2A and 2B illustrate T-distributed Stochastic Neighbor Embedding (T-SNE) visualization of input regions showing model performance. Specifically, FIGURE 2A depicts a random forest, and FIGURE 2B depicts a feedforward neural network. In the depicted example of beam prediction, models are trained to predict the best beam among thirty-two possible narrow beams, using eight wide-beams as input. The monitoring of such procedure can be done by NW sweeping all thirty -two beams plus eight wide beams, and the UE compares the measured ground truth with the actual best predicted value. To evaluate such predictor, a simulation scenario is conducted based on the 3GPP agreed scenario. To illustrate how certain regions of the input data are performing inaccurately, the T-SNE method is used to represent the high-dimensional model input, and each sample in the plot is depicted to indicate whether the prediction model was accurate or not where solid black is used to show erroneous predictions.

[0026] T-SNE is a dimensionality reduction technique that is widely used for visualizing highdimensional data in a lower-dimensional space. The main goal of T-SNE is to capture the underlying structure and relationships between data points by mapping them to a lowerdimensional space while preserving the local and global structure of the data as much as possible. It is particularly useful for visualizing complex and non-linear patterns in the data.

[0027] The results are presented in FIGURES 2A and 2B when using a random forest and neural network, both predictors have in general high accuracy, but as can be seen it shows weak performance in certain regions. FIGURES 2A and 2B can be used to draw the following observations:

[0028] 1. T-SNE indicates the similarity in model input. Thus, a similar input will provide a similar inaccurate or accurate result.

[0029] 2. Until the model input has changed substantially in the bad performing region, the prediction will be inaccurate, in particular for the random forest model.

[0030] 3. The selection of model impacts how sensitive the UE-side model is to change of model input.

[0031] There currently exist certain challenge(s), however. For example, there is a cost associated to a real-time monitoring operation of UE-sided models. One clear cost is that the network has to transmit extra reference signals so that the UE can measure the reference signals and determine a ground-truth value, which is to be used for comparing with the predicted value from the AI / ML model. This is illustrated in FIGURE 3, where there is a dedicated time-slot for the purpose of calculating a model monitoring metric, for example the inference accuracy. Next, such metric is used to activate / deactivate a model until next monitoring occasion.

[0032] In general, the benefits / challenges with the different monitoring alternatives are summarized in TABLE 1 : TABLE 1 : Real and non-real time monitoring challenges / benefits

[0033] Additionally, the UE needs to collect a lot of samples to detect whether the model is not performing according to its training step. For example, a model claiming a 90% accuracy, the UE would at least need to collect several samples to understand whether the model is working properly. After establishing such success / failure of a potential model monitoring result, it is not clear for how long such results are valid. In case the network needs to configure such monitoring too frequently, the gains of using the UE-sided model can diminish. This is because the monitoring occasion typically involves performing measurement to obtain the ground truth, the estimation of which would otherwise be predicted by the model.

[0034] Another cost can comprise of processing overhead at the UE, in terms of activating advanced models that can estimate the confidence of the predictions or methods that can monitor the distribution change of input and / or output.

[0035] One solution is to initiate monitoring when the performance in terms of bitrate degradation; however, it is not straightforward to identify that a degraded UE performance is due to model inaccuracy, rather than some other error sources such as interference.

[0036] As shown in FIGURES 2A and 2B, in case the model input is not changing, the result of the monitoring operation will be the same. For example, a lot of successful monitoring procedures, or many failed monitoring procedures. FIGURES 2A and 2B also indicate how the selected model will create different regions of varying size with highly accurate, or erroneous, or mix of erroneous / accurate data. SUMMARY

[0037] Certain aspects of the disclosure and their embodiments may provide solutions to these or other challenges. For example, methods and systems are provided for estimating the validity period of a model monitoring result, where the monitoring results is used to activate / deactivate an ML model in the UE, and when to perform such monitoring operation again based on the estimated validity period.

[0038] According to certain embodiments, a method by a UE for monitoring of UE-sided models using validity of monitoring results includes transmitting, to a network node, model performance feedback associated with a performance of a model during a first monitoring occasion. The UE receives configuration information associated with a validity of a model monitoring result associated with the first monitoring occasion. The configuration information is for performing at least one of: activating the model, deactivating the model, and initiating a second monitoring occasion.

[0039] According to certain embodiments, a UE for monitoring UE-sided models using validity of monitoring results is configured to transmit, to a network node, model performance feedback associated with a performance of a model during a first monitoring occasion. The UE is configured to receive configuration information associated with a validity of a model monitoring result associated with the first monitoring occasion. The configuration information is for performing at least one of: activating the model, deactivating the model, and initiating a second monitoring occasion.

[0040] According to certain embodiments, a method by a network node for monitoring of UE- sided models using validity of monitoring results includes receiving, from a UE, model performance feedback associated with a performance of a model during a first monitoring occasion. Based on the model performance feedback, the network node determines a validity of a model monitoring result associated with the first monitoring occasion, Based on the validity of the model monitoring result associated with the first monitoring occasion, the network node transmits model configuration information for performing at least one of activating the model, deactivating the model, and initiating a second monitoring occasion. According to certain embodiments, a network node for monitoring UE-sided models using validity of monitoring results is configured to receive, from a UE, model performance feedback associated with a performance of a model during a first monitoring occasion. Based on the model performance feedback, the network node is configured to determine a validity of a model monitoring result associated with the first monitoring occasion. Based on the validity of the model monitoring result associated with the first monitoring occasion, the network node is configured to transmit model configuration information for performing at least one of activating the model, deactivating the model, and initiating a second monitoring occasion.

[0041] Certain embodiments may provide one or more of the following technical advantage(s). For example, certain embodiments may provide a technical advantage of enabling an efficient procedure for real-time monitoring of UE-sided models. As another example, certain embodiments may provide a technical advantage of performing model monitoring less frequently, leading to reduced overhead in monitoring in comparison to a periodical monitoring of a model.

[0042] As another example, in comparison to previous systems and techniques that activate model monitoring based on UE service level performance degradation, certain embodiments may provide a technical advantage of providing a method to proactively test a model performance prior to UE service level performance degradation is observed.

[0043] Other advantages may be readily apparent to one having skill in the art. Certain embodiments may have none, some, or all of the recited advantages.

[0044] BRIEF DESCRIPTION OF THE DRAWINGS

[0045] For a more complete understanding of the disclosed embodiments and their features and advantages, reference is now made to the following description, taken in conjunction with the accompanying drawings, in which:

[0046] FIGURE 1 illustrates an example with regions where a model is not performing accurately;

[0047] FIGURES 2A and 2B illustrate T-distributed Stochastic Neighbor Embedding (T-SNE) visualization of input regions showing model performance;

[0048] FIGURE 3 illustrates a dedicated time-slot being used for the purpose of calculating a model monitoring metric such as inference accuracy;

[0049] FIGURE 4 illustrates a flowchart and signaling diagram for estimating a validity time of a model monitoring result, according to certain embodiments;

[0050] FIGURE 5 illustrates an example mapping to TSNE region for a BM use case, according to certain embodiments;

[0051] FIGURE 6 illustrates an example of indirect real-time monitoring via prediction uncertainty, according to certain embodiments;

[0052] FIGURE 7 illustrates another example T-SNE visualization indicating the probability of correctly classified strongest beam in the same T-SNE dimension, according to certain embodiments;

[0053] FIGURE 8 illustrates an adaptive model monitoring period, according to certain embodiments; FIGURE 9 illustrates an adaptive model monitoring period, according to certain embodiments;

[0054] FIGURE 10 illustrates an example communication system, according to certain embodiments;

[0055] FIGURE 11 illustrates an example UE, according to certain embodiments;

[0056] FIGURE 12 illustrates an example network node, according to certain embodiments;

[0057] FIGURE 13 illustrates an example method by a UE for monitoring of UE-sided models using validity of monitoring results, according to certain embodiments; and

[0058] FIGURE 14 illustrates an example method by a network node for monitoring of UE-sided models using validity of monitoring results, according to certain embodiments.

[0059] DETAILED DESCRIPTION

[0060] Some of the embodiments contemplated herein will now be described more fully with reference to the accompanying drawings. Embodiments are provided by way of example to convey the scope of the subject matter to those skilled in the art.

[0061] As used herein, ‘node’ can be a network node or a UE.

[0062] In some embodiments, generic terminology such as “radio network node” or simply “network node” is used. Examples of network nodes are NodeB, base station (BS), multi-standard radio (MSR) radio node such as MSR BS, eNodeB (eNB), gNodeB (gNB), Master eNB (MeNB), Secondary eNB (SeNB), integrated access backhaul (IAB) node, network controller, radio network controller (RNC), base station controller (BSC), relay, donor node controlling relay, base transceiver station (BTS), Central Unit (e.g., in a gNB), Distributed Unit (e.g., in a gNB), Baseband Unit, Centralized Baseband, Cloud-Radio Access Network (C-RAN), access point (AP), transmission points, transmission nodes, Remote Radio Unit (RRU), Remote Radio Head (RRH), nodes in distributed antenna system (DAS), core network node (e.g., Mobile Switching Center (MSC), Mobility Management Entity (MME), etc.), Operations & Maintenance (O&M), Operations Support System (OSS), Self Organizing Network (SON), positioning node (e.g., E- SMLC), etc.

[0063] Another example of a node is user equipment (UE), which is a non-limiting term and refers to any type of wireless device communicating with a network node and / or with another UE in a cellular or mobile communication system. Examples of UE are target device, device-to-device (D2D) UE, vehicular-to-vehicular (V2V), machine type-UE, machine-type communications (MTC) UE or UE capable of machine-to-machine (M2M) communication, Personal Digital Assistant (PDA), Tablet, mobile terminals, smart phone, laptop embedded equipment (LEE), laptop mounted equipment (LME), Unified Serial Bus (USB) dongles, etc.

[0064] The term radio access technology (RAT) may refer to any RAT such as, for example, Universal Terrestrial Radio Access Network (UTRA), Evolved Universal Terrestrial Radio Access Network (E-UTRA), narrow band internet of things (NB-IoT), WiFi, Bluetooth, next generation RAT, New Radio (NR), 4G, 5G, etc. Any of the equipment denoted by the terms node, network node or radio network node may be capable of supporting a single or multiple RATs.

[0065] The term signal or radio signal used herein can be any physical signal or physical channel. Examples of downlink (DL) physical signals are reference signal (RS) such as Primary Synchronization Signal (PSS), Secondary Synchronization Signal (SSS), Channel State Information-Reference Signal (CSI-RS), Demodulation Reference Signal (DMRS), signals in SS / PBCH block (SSB), discovery reference signal (DRS), Cell-Specific reference Signal (CRS), Positioning Reference Signal (PRS), etc. RS may be periodic. For example, RS occasion carrying one or more RSs may occur with certain periodicity such as, for example, 20 ms, 40 ms, etc. The RS may also be aperiodic. Each SSB carries NR-PSS, NR-SSS and NR-PBCH in four successive symbols. One or multiple SSBs are transmitted in one SSB burst that is repeated with certain periodicity such as, for example, 5 ms, 10 ms, 20 ms, 40 ms, 80 ms, and 160 ms. The UE is configured with information about SSB on cells of certain carrier frequency by one or more SS / PBCH block measurement timing configuration (SMTC) configurations. The SMTC configuration comprising parameters such as SMTC periodicity, SMTC occasion length in time or duration, SMTC time offset with regard to reference time (e.g., serving cell’s SFN), etc. Therefore, SMTC occasion may also occur with certain periodicity such as, for example 5 ms, 10 ms, 20 ms, 40 ms, 80 ms, and 160 ms. Examples of uplink (UL) physical signals are reference signal such as Sounding Reference Signal (SRS), DMRS, etc. The term physical channel refers to any channel carrying higher layer information such as, for example, data, control, etc. Examples of physical channels are Physical Broadcast Channel (PBCH), Physical Uplink Shared Channel (PUSCH), Physical Uplink Control Channel (PUCCH), Physical Downlink Control Channel (PDCCH), Physical Downlink Shared Chanel (PDSCH), shortened PUCCH (sPUCCH), shortened PDSCH (sPDSCH), shortened PUCCH (sPUCCH), shortened PUSCH (sPUSCH), MTC PDCCH (MPDCCH), Narrowband PBCH (NPBCH), Narrowband PDCCH (NPDCCH), Narrowband PDSCH (NPDSCH), Narrowband PUSCH (NPUSCH), and Enhanced-PDCCH (E-PDCCH), etc.

[0066] The term time resource used herein may correspond to any type of physical resource or radio resource expressed in terms of length of time. Examples of time resources are: symbol, time slot, subframe, radio frame, Transmission Time Interval (TTI), interleaving time, slot, sub-slot, mini-slot, system frame number (SFN) cycle, hyper-SFN (H-SFN) cycle, etc.

[0067] According to certain embodiments, method(s) are provided for estimating how long a model monitoring result is valid, configuring an AI / ML model to use (or stay deactivated) during its validity, and configuring next monitoring occasion based on such estimate.

[0068] The validity period can correspond to how long the model / functionality is expected to be performing consistently. As used herein, performing consistently may refer to a model that is performing in a constantly inaccurate manner or providing consistently accurate predictions. In general, the validity period refers to how frequently the model is to be monitored to determine the performance consistency in the UE local area. The validity period is determined by:

[0069] • UE model sensitivity (how a change of model input affects model output),

[0070] • UE expected change in its model input over a certain time-period,

[0071] • Cost of model monitoring (e.g. RS-resource overhead), and / or

[0072] • UE model monitoring metric (how accurate is the model in current region).

[0073] According to certain embodiments, a method enabling the network to understand each of the above aspects, when configuring a monitoring procedure at the UE, includes one or more of the steps depicted in FIGURE 4, which illustrates a flowchart and signaling diagram 50 for estimating a validity time of a model monitoring result, according to certain embodiments. As depicted, the steps are performed by and the signals are communicated between a UE 60 and a network node 70, such as, for example, a gNB. An objective of the method 50 is for real-time monitoring, motivated by the need of turning models / functionalities off in areas where it is not performing well.

[0074] Specifically, in the depicted example, the method 50 includes, one or more of the UE 60 and the gNB 70 initiating a monitoring procedure, at step 100. In particular embodiments, either the network 70 (i.e., gNB) or the UE 60 may initiate the monitoring procedure. The decision on whether to initiate the procedure can be performed based on:

[0075] UE service level requirements (e.g., UE 60 is configured with UltraReliable Low-Latency Communication (URLLC) type of traffic and cannot afford AI / ML model errors that might lead to packet drops), a UE transition from idle to connected mode, UE performance degrades, any of the events described herein,

[0076] - NW performance degrades, a UE updates / finetunes the model, and / or based on UE battery level information such as, for example, UEs or NW only initiating monitoring procedure when UEs have high battery level.

[0077] At step 110, the gNB 70 sends a configuration to the UE 60 for evaluating predictions. At step 120a, the UE 60 sends model performance feedback to the gNB 70. At step 120b, the gNB 70 determines a model monitoring success / failure metric based on the received information.

[0078] In a particular embodiment, the performance can be evaluated by configuring a UE 60 to measure and determine the ground-truth based on the selected performance metric(s), in addition to using the ML-model to generate predictions. The performance feedback can comprise the difference between the model generated prediction and the ground truth.

[0079] In particular embodiments, taking an example of BM use case, one or more of the following agreed performance metric(s) may be considered:

[0080] • Alt.l: Beam prediction accuracy related KPIs, e.g., Top-K / 1 beam prediction accuracy

[0081] • Alt.2: Link quality related KPIs, e.g., throughput, Ll-RSRP, Ll-SINR, hypothetical BLER

[0082] • Alt.3: Performance metric based on input / output data distribution of AI / ML

[0083] • Alt.4: The Ll-RSRP difference evaluated by comparing measured RSRP and predicted RSRP

[0084] In another particular embodiment, taking an example of CSI use case, one or more of the following agreed performance metric(s) may be considered:

[0085] • CSI prediction accuracy related KPIs, e.g. SGCS

[0086] At step 130, the validity of the monitoring is determined. At an optional step 130a, the gNB 70 requests information to determine how long the monitoring result is valid, and the UE 60 transmi ts / indicates the monitoring validity information. At step 130b, the gNB 70 determines validity based on: model sensitivity, UE input change, cost associated to the monitoring, monitoring success / failure metric, and / or historical information. Optionally, the validity is additionally or alternatively determined based on the UE assistance information provided at step 130b, in a particular embodiment.

[0087] At step 140, the gNB 70 configures activation / deactivation of a model based on the monitoring results.

[0088] At an optional step 150, the gNB 70 indicates validity and configures corresponding event to be used in step 100.

[0089] At step 160, the method returns to step 100 and the method continues based on the validity information. Some aspects of the steps, features and operations depicted in FIGURE 4 are described in more detail below.

[0090] Determining Monitoring Validity (Step 130)

[0091] Estimating Model Sensitivity

[0092] In a particular embodiment, the UE understands the consistency in predictions by measuring the output variation and associated accuracy / loss impact by performing perturbations of input. The UE 60 may, for example, use the training data to understand such sensitivity, for example consider the BM use case described above. The input data were subject to constant change in the inputted eight RSRP values. Table 2 summarizes the results of the simulations for the random forest model:

[0093] Table 2

[0094] Table 2 shows the impact of noise to the input to each of the eight RSRP measurements for the training data, indicating how much model output varies for a given perturbation in input.

[0095] The evaluations indicate that a change according to +ldB deviation in the input have limited impact on the prediction output. The results are 95% similar as to when no input perturbation is considered. In this scenario, the UE-assisted signaling in step 130a), can comprise of the difference in model output for a given perturbation of the input. The perturbation can, in one embodiment, be NW configured. The NW configures the UE with some rules on how to measure the model sensitivity. For example, the NW can instruct the UE to change the RSRP measurements with ±x dB, where the exact perturbation of each RSRP measurement might follow some agreed rules / distributions between UE and NW or some rules / distributions indicated from NW to UE , and UE reports how such perturbation changes the output value. For example, if top- K beams changes or not. This information can be part of the UE capability reporting (e.g., as part of the UE capability reporting associated to the AI / ML based feature when the UE connects to the NW), or RRC signaling (e.g., when requested by the NW or triggered via UE-sided model sensitivity check), or UCI signaling (e.g., reported together with the predicted CSI / beam report for the AI / ML based feature, or / and reported as a new type of UCI when configured / triggered by the NW). In a related embodiment, the UE can apply the Gaussian noise, or constant value, to the latest input data sample and measure how sensitive the output is to a perturbation. For example, the UE can check if the output statistics is very different when applying a certain perturbation in input compared with no perturbation. Based on this, the UE can get an understanding on how much perturbation is needed to change the model output for the current UE location. The UE-assisted signaling in step 130a can, for example, indicate how much change in its measured RSRP that is needed for changing its output. For example, on average, the RSRP needs to be changed with ±x dB for its input.

[0096] In a particular embodiment, for a given / selected model, the model sensitivity depends on the perturbation of input data. When the model sensitivity is reported from the UE, this could be represented as an indicator. The selection of the indicator depends on the UE capability and / or some agreed rules between UE and NW or some rules indicated from NW to UE:

[0097] • For a UE that has high capability, an indicator could contain a value between 1-10, where 10 can indicate a highly sensitive model, where a slight change in input generates a new model output.

[0098] • For a UE that has low capability, an indicator could only contain 0 / False or 1 / True, where 1 / True means that the model is very sensitive for small change of the input and 0 / False means that the model is robust to the small change of the input.

[0099] In another particular embodiment, the model sensitivity depends on the selected model. The UE indicates a value of 1-10 if its prediction model peacefully degrades such as, for example, if the model is having areas in its input where it constantly performs badly, as is exemplified in FIGURE 2B described above. In FIGURE 2B, the random forest method indicates a large region of weak performances, while the erroneous samples are more scattered for the Neural Network (NN) method.

[0100] In another particular embodiment, the model sensitivity depends on the fluctuation of the prediction error. The UE indicates the information of low model sensitivity to the network if the fluctuation of the prediction error is low. The UE indicates the information of middle model sensitivity to the network if the fluctuation of the prediction error is middle, or the UE indicates the information of high model sensitivity to the network if the fluctuation of the prediction error is high. In a particular embodiment, for an AI / ML based feature, the standard specification defines a set of perturbation configurations of the input (e.g., the perturbation values / ranges of model input, Gaussian noise or Constant value, etc.) and the rule(s) on how to calculate / derive model output sensitivity for a given perturbation value / range. Based on the defined rule(s), a UE derives its model output sensitivity value / level for one or more defined model input perturbation value / range of this AI / ML based feature and reports to its model output sensitivity information for the feature to the network. In a particular embodiment, the UE can report the model sensitivity information via UE capability reporting signaling (e.g., as part of the UE capability reporting associated to the AI / ML based feature when the UE connects to the NW), RRC signaling (e.g., when requested by the NW or triggered via UE-sided model sensitivity check), or UCI signaling (e.g., reported together with the predicted CSI / beam report for the AI / ML based feature, or / and reported as a new type of UCI when configured / triggered by the NW).

[0101] As an example, in a particular embodiment, the reported model output sensitivity information contains a list of sensitivity values / ranges, where each sensitivity value / range maps to a specific perturbation value / range of model input.

[0102] As another example, in a particular embodiment, the reported model output sensitivity information only contains a subset of the sensitivity values / ranges for a subset of perturbation values / ranges of model input, where the sensitivity values / ranges contained in the subset are below or above a certain threshold. The threshold can be configured by the NW or defined in the specification for the feature.

[0103] In a particular embodiment, for instance, in UE capability reporting, the UE only reports model sensitivity information for the input perturbation cases where the AI / ML model output difference are below 10%, or the UE only reports model sensitivity values / ranges for the input perturbation cases where the AI / ML model output are above 90% similar as to when no input perturbation is considered. This reported information can be used as assistance information for the NW to consider when configuring / reconfiguring / activating the AI / ML model / feature or / and determining the configuration for the model performance monitoring.

[0104] In a particular embodiment, as another example, in RRC signaling or UCI signaling, the UE only reports model sensitivity information for the input perturbation cases where the AI / ML model output difference is above 10%, or when the AI / ML model output are below 90% similar as to when input perturbation is considered. This reported information can be used as assistance information for the NW to consider when deactivating the AI / ML model / feature or / and determining the configuration for model performance monitoring. In a further particular embodiment, the standard specification also defines a set of model output sensitivity ranges / levels as requirements for the AI / ML based feature (e.g., model output difference ranges as 0~3%, 3~5%, 5~8%, 8-10%). For a given model input perturbation value / range, the UE maps the derived model output sensitivity value / level to one of the sensitivity ranges / levels defined for this AI / ML based feature in the specification. The UE reports the mapped model output sensitivity range(s) / level(s) for one or more model input perturbation value(s) / range(s) to the NW. The UE can also obtain the corresponding model input perturbation range(s) for one or more defined model output sensitivity range / value, and report the tolerant model input perturbation range(s) to the NW.

[0105] Estimating Model Input Change

[0106] The validity is impacted on how the UE input to its model is expected to change. For example, a high-speed UE will have quicker change in its RSRP measurements than a slow- moving UE, and also will have a quicker change in its CSI measurements than a slow-moving UE. The mobility states in NR can be used to estimate how much the model input are likely to change in the future at the NW (stationary, slow, medium or high mobility). The mobility states also can be UE speed information from network to UE, such as, for example, low-speed UE, middle-speed UE, or high-speed UE. The NW can also use other type of information of the UE mobility such as the doppler estimated from the uplink.

[0107] The condition of LOS and / or the transition from outdoor to indoor or vice versa would also lead to the model input change, which significantly affects the monitoring validity in step 130a depicted in FIGURE 4.

[0108] Regarding indicating monitoring validity information from UE to network in step 130a, UE could conclude the model validity from itself, aim to deactivate the model if the validity is not good as expected, and send the info to NW for asking configuration to fall back. While NW will eventually determine the validity of the monitoring without considering the UE assistance information. It is noted that, for UE-side model monitoring, UE could make the decision(s) of model selection, activation, deactivation, switching, and fallback operation.

[0109] Regarding UE assistance information in step 130b, the UE can for example use the autocorrelation of previous measurements that have been used as an input to the model, to report when a new measurement is uncorrelated with previous. The UE can, for example, report when the correlation is below a certain value in step 130b. Where the value can in, one embodiment, be a NW configured value. In a particular embodiment, the model input change is reported from the UE. For example, an expected change of the current location will change the model input. Or in general how much model output changes with a slight change of input. This could be represented as a value between 1-10, where 10 can indicate a high expected change in model input. In another example, this could be represented as a distance between the current input data distribution and the training data distribution, or a distance between the current input data distribution within a time window T1 and the previous input data distribution within a time window T2.

[0110] Note that in some use cases, when UE can measure the inputs also when model is deactivated. The need for estimating the change of model input may not be mandatory. This could be used in conjunction with an event that triggers a monitoring operation as described below.

[0111] Model Monitoring Metric (Determine Success / Failure Metric (Step 120b))

[0112] For the step 120a, UE could either indicate the model performance feedback to help the NW determine model monitoring metrics or directly indicate model monitoring success / failure metric. Then, the step 120b could be skipped.

[0113] The monitoring can be seen as “successful” if the predictions where same as, or within a certain range of the ground-truth. The monitoring result can also be based on the uncertainty reported by the UE, for example, if the model produces very uncertain results, the NW can select to determine that the model is not successful. In another example, the network can use the received uncertainty information from UE for the decision of UE scheduling, precoding design and / or positioning strategy.

[0114] The metric can comprise a loss-value, for example, using the mean squared error or logloss in case of two or more samples. A high-loss value indicates that the model is very inaccurate given the current input, and for the model to be valid again there needs to be a high change of the model input (depending on the model sensitivity). A low-loss value indicates that the model is accurate given the current input, and for the model to be invalid there needs to be a high change of the model input (depending on the model sensitivity). In general, if the model output is not very sensitive to the model input changes, then, there is no need to configure very frequent model performance monitoring that is based on the comparing of model output and ground-truth, since the monitoring results will likely not change that much during a relative long time period.

[0115] The UE reports the monitoring results in terms of loss values, BLER of decoding PDCCH (i.e., 1%, 10%, or other values) or the accuracy in case enough samples are obtained. Note that the monitoring metric will be having a source of uncertainty based on how many samples that were collected. FIGURE 5 illustrates an example mapping to TSNE region for a BM use case 200, according to certain embodiments. In case the UE collects multiple samples, the different regions where the UE is in its input space will affect the monitoring results, whereas shown in FIGURE 5, the UEs in region 205 will have large number of failures in comparison to the region 210.

[0116] In another example, the monitoring comprises the probability of a confident result, for example, as exemplified in the BM use case described above. This can be used to decide for each report whether to use a model or not. In a particular embodiment, the NW can decide when the UE should perform the extra overhead in estimating and signaling such uncertainty.

[0117] FIGURE 6 illustrates an example 300 of indirect real-time monitoring via prediction uncertainty, according to certain embodiments. Using an ML model that is capable of also providing a confidence of a prediction can be used to get an indirect estimate of the estimated accuracy (uncertainty), which can be estimated for each sample such as, for example, in case of a SoftMax output layer in a classifier neural network. This would be an indirect method for UE to estimate the performance of its model. Moreover, the network can use such “uncertainty” to deactivate the model in the local region in case the model uncertainty is too high and not useful at the network. Note that the uncertainty is similar to the error value specified for OTDOA for positioning as shown below: error-Value

[0118] This field specifies the target device's best estimate of the uncertainty of the OTDOA (or TOA) measurement.

[0119] The encoding on five bits is as follows:

[0120] '00000' 0 to (R* 1-1) metres

[0121] '00001 ' R*1 to (R*2-l) metres

[0122] '00010' R*2 to (R*3-l) metres

[0123] '11111' R*31 metres or more; where R is the resolution defined by error-Resolution field.

[0124] E.g., R=20 m corresponds to 0-19 m, 20-39 m, ... ,620+ m.

[0125] The network could use the predicted uncertainty of a beam, CSI or positioning model to determine whether to perform fallback to another model / method or other purposes for performance improvement. FIGURE 6 shows how such uncertainty could be provided during each UE inference, where the provision of such confidence could cost extra resources at the UE / network, for example in terms of computational resources.

[0126] The random forest model can also output the probability of a correctly classified sample. FIGURE 7 illustrates another example T-SNE visualization 400 indicating the probability of correctly classified strongest beam in the same T-SNE dimension, according to certain embodiments. FIGURE 7 shows how the low-confidence samples corresponds to the inaccurate predictions as compared with FIGURE 2B.

[0127] Model Monitoring Cost

[0128] The cost may, for example, include extra reference signals transmitted for the UE to collect a ground-truth sample to be used for comparing the predicted value. Or, in a particular embodiment, the cost may also include processing overhead at the UE, in terms of activating advanced models that can estimate the confidence of the predictions, or methods that can monitor the distribution change of input / output.

[0129] In a particular embodiment, the cost may be indicated from the UE to the network node (in step 130a of FIGURE 4). For example, the UE indicates the energy consumed for the monitoring procedure, in a particular embodiment. The energy may be related to the processing cost of estimating the uncertainty of the prediction, or the energy consumed for measurements and processing for calculating the inference accuracy. As another example, the UE indicates the maximum number of measurements it can perform within a time window (e.g., per slot, per radio frame, per 10ms) for model monitoring, in a particular embodiment.

[0130] When to Configure a New UE Monitoring Procedure

[0131] The determination of the method for triggering a new monitoring procedure can be assisted from the UE. For example, the UE can recommend or request any of the validity types below with a recommended value. That is, UE can indicate the recommended method for when it estimates that the monitoring procedure to have a different outcome.

[0132] The method for determining the validity for when model is expected to perform consistently can be represented as:

[0133] Timer: In a particular embodiment, the NW can have a time estimate on the validity of the monitoring. For example, the NW can set the time based on: Model-sensitivity, monitoring metric, and input information:

[0134] In general, the cost of monitoring will vary and be use case dependent. In particular embodiments, the following simple rules can be used:

[0135] - High model sensitivity -> more frequent monitoring, and / or

[0136] - High expected change in model inputs -> more frequent monitoring, and / or

[0137] - Model monitoring metrics does not indicate a large error nor high accuracy, for example in case predictions are 50% accurate -> more frequent monitoring, and / or

[0138] - High cost, e.g. requires lot of RS-transmissions to establish a metric -Mess frequent monitoring.

[0139] For example, a high sensitivity and high expected input change will lead to more frequent monitoring operation. The expected model input change for a beam prediction model can be based on UE mobility, for example. In case the UE is moving quickly, its radio environment will change which can cause a new model performance in the new UE location. Thus, a high-speed UE will have shorter time to the next monitoring occasion than a fast-moving UE. The expected change in UE model input can also be based on UE CSI information. For example, if the UE CSI is varying quickly, it indicates a low channel coherence time, which can motivate a shorter timespan between the monitoring occasions. Also, in case the monitoring metric indicates a strong success or failure, it will result in less frequent monitoring.

[0140] Historical Information of Monitoring Performance

[0141] In another particular embodiment, only historical information of previous monitoring results is used. For example, the number of monitoring success in the last N monitoring occasions. For example, the NW can increase the time until next monitoring occasion if the previous one or more monitoring occasions were a success. Where success can be defined as whether the predictions where same as, or within a certain range of the ground-truth. One implementation is having a timer that is increased if the model is performing well.

[0142] In a related embodiment, information of other UEs is used, for example the typical period from a failed monitoring result until a successful monitoring result occur. Event Triggered: In step 150 of FIGURE 4, the NW indicates to the UE the validity of the monitoring result and configures a corresponding event based on the validity. For example, in a particular embodiment, the NW indicates when the UE should indicate a new monitoring process. As another example, in a particular embodiment, the NW configures the UE with an event that initiates the monitoring process, the event can include one or more of:

[0143] - indication of that the UE geolocation should change with a certain threshold value, indication of that the UE radio measurements should change with a certain threshold value (e.g., UE should observe a certain change of the SSB measurements), and

[0144] - indication that the UE non-radio measurements should change with a certain value (e.g. change of UE speed).

[0145] Location:

[0146] In a particular embodiment, the validity can be defined as the UE geolocation, where a model monitoring result is assumed to be valid until a UE moves a certain range outside of its location. UE location can be estimated with Global Navigation Satellite System (GNSS) or 3GPP-based procedures such as Observed Time Difference of Arrival (OTDOA). Note that UE location can also be whether it is indoor or outdoor. For example, the model monitoring result are valid until the UE transitions from outdoor to indoor (or vice versa). UE location can also be denoted as an area ID or / and zone ID associated to the AI / ML based feature.

[0147] Radio-Measurements:

[0148] The validity can be defined as the UE radio location, where a model monitoring result is assumed to be valid until a UE measures a certain number of reference signals with a threshold difference to a previous result. For example, the Euclidian distance for a set of SSB measurements should be larger than a certain threshold value. The threshold can be determined based on the historical information such as, for example, what threshold that provided a transition from monitoring success to failure (or vice versa). Non-Radio Measurements:

[0149] The validity can be defined as the UE non-radio measurements, where a model monitoring result is assumed to be valid until, for example, the UE Inertial Measurement Unit (IMU) senses a change in UE speed / orientation.

[0150] Configure a Model / Functionality Deactivation / Activation

[0151] In step 140 of FIGURE 4, the NW can configure a UE fallback in case the monitoring results were incorrect and use such decision until next monitoring instance as determined by the validity information. For example, the UE does not provide beam predictions until it has performed another monitoring procedure as determined by the monitoring validity.

[0152] In another embodiment, the NW can configure a UE fallback in case the monitoring performance from UE to network is lower than one threshold for a given time.

[0153] Note that the UE might deactivate the model and request for fallback in step 120a of FIGURE 4. Then, steps after 120a are all skipped unless the model is activated again.

[0154] In another embodiment, the NW can configure that UE falls back to a given solution in case the monitoring performance of the UE is lower than one threshold for a given time.

[0155] Adaptive Period of Model Monitoring

[0156] In a particular embodiment, the model monitoring period is updated adaptively based on the model monitoring result(s).

[0157] In a particular embodiment, for example, in each model monitoring instance, a result is taken as output of the model monitoring mechanism, where the result may take one value out of a list of possible values. The list of possible values may be denoted as: {'normal', 'neutral', 'fail'}.

[0158] • If the result is 'normal' (i.e. , the AI / ML model is valid and is working properly as expected), then the model monitoring period is increased, i.e., the model monitoring procedure is performed less frequently.

[0159] • If the result is 'fail' (i.e., the AI / ML model is invalid and is not working as expected), then model monitoring period is decreased, i.e., the model monitoring procedure is performed more frequently.

[0160] • If the result is 'neutral' (i.e., the AI / ML model is not working very well but not yet failed either), the model monitoring period can stay the same, i.e., the model monitoring procedure is performed with the same frequency as the previous period. To use the model monitoring result in calculations as an example, the numerical value of the result can be set as: 1 for 'normal', -1 for 'fail', 0 for 'neutral'. Note that this example is used to illustrate the principle, and other values can be used to achieve similar effect.

[0161] The adaptation can be realized by the exemplary procedure below.

[0162] • Step 1. Set k=0. Initialize the start of model monitoring period to t(0) = t0, the duration of model monitoring period to T(0) = To;

[0163] • Step 2. For Uth period, the time period to collect data for the purpose of model monitoring is [t(k), t(k + 1)] , where t(k + 1) = t(k) + T(k). During the Uth period, the gNB and UE can cooperate to arrive at the desired model monitoring accuracy. For example, for beam management, the gNB sends additional DL reference signals to the UE so that the UE can measure and determine the ground truth for top- T beam(s). The model monitoring outcome of the Uth period is recorded as result D(k), which can take value l,-l,0 as explained above.

[0164] • Step 3: For (k+ l)-th period, calculate the duration of model monitoring period as

[0165] Here and f2are scaling factors and take positive values. Values Tminand Tmaxare the minimum and maximum duration of a model monitoring period, respectively.

[0166] • Step 4. Set k=k+l, and go back to Step 2.

[0167] In the procedure above, parameter values To, f , f2, Tminand Tmaxcan be given a predefined value, or be configured by the gNB, in particular embodiments. The parameter values can be determined as a function of various factors, such as model sensitivity, UE speed, generalization capability of the deployed model, etc. For example, for a model with high sensitivity, and Tmaxmay be set to smaller values, so that the model monitoring duration does not grow too fast and does not become too sparse to capture model degradation. The increase factor ^(used when the model monitoring procedure decides that the model functions well) and decrease factor f2(used when the model monitoring procedure decides that the model functions poorly) can be set differently. For instance, the parameter for duration increase is set to allow faster duration increase (e.g., set to 1 to allow doubling at a time), while f2for duration increase is set to have slower duration decrease (e.g., set to 0.9 to allow 10% decrease at a time). The adaptive procedure is illustrated in FIGURE 8 and FIGURE 9. Specifically, FIGURE 8 illustrates an adaptive model monitoring period 500, where the period duration is increased when the model monitoring decision is ‘normal,’ according to certain embodiments.

[0168] In FIGURE 8, an example is shown to illustrate the scenario where the AI / ML model is working well, and each model monitoring result of 'normal' allows the subsequent model monitoring period duration to increase till the maximum value Tmaxis reached. It is noted that the duration cannot be increased above the parameter Tmax, i.e., even if the model functions well, the model monitoring should be done at least once every Tmaxto guard against potential model deterioration.

[0169] FIGURE 9 illustrates an adaptive model monitoring period 600, where the period duration is decreased when model monitoring decision is ‘fail,’ according to certain embodiments. Specifically, the example illustrates the scenario where the AI / ML model is degrading, and each model monitoring result of 'fail' causes the subsequent model monitoring period duration to decrease till the minimum value Tminis reached. It is noted that the duration cannot be decreased below the parameter Tmin, i.e., even if the model functions unsatisfactorily, the model monitoring should be done at most once every Tmin. Tmincan be set according to the largest overhead or largest cost acceptable, where the overhead or cost (e.g., extra RS transmission, extra signal processing) is due to the model monitoring function.

[0170] The example in FIGURE 9 also illustrates that the model monitoring decision of 'fail' does not necessarily immediately trigger model deactivation. The deactivation decision may be taken after N consecutive model monitoring result of 'fail', where N is a positive integer. For example, if N=3, then the deployed model is deactivated if the model monitoring procedures gives a result of 'fail' 3 times in a row; in contrast, if the result sequence is {'fail', 'normal', 'fail'} or {'normal', 'fail', 'fail'} or {'neutral', 'neutral', 'fail'}, ... , then the deployed model is not deactivated yet since the model performance may be unstable temporarily.

[0171] It is noted that many variations to the above procedure are possible, without deviating from the basic design principle of adaptively adjust the model monitoring period based on the model monitoring result(s).

[0172] In one variation, an equation different from (1) is used. For instance, equation (2) below can be used to realize linear increase and decrease, instead of exponential increase and decrease in (1). Other equation than (1) and (2) are possible as well. For example, use exponential increase when D(k)>=0 and use linear decrease when D(k)<0; or use linear increase when D(k)>=0 and use exponential decrease when D(k)<0.

[0173] In another variation, the adaptation is not based on a single model monitoring result as in (1) and (2), but based on multiple model monitoring results in previous periods. For instance, equation (1) can be modified to (3) below, so that the previous two monitoring results (D(k) and D(k-l)) are taken into account in the adaptation for period T(k+1). Note that in this case, initialization in Step 1 should be updated to have proper calculation for all k values, e.g., set D(- l) = 0.

[0174] FIGURE 10 shows an example of a communication system 700 in accordance with some embodiments. In the example, the communication system 700 includes a telecommunication network 702 that includes an access network 704, such as a radio access network (RAN), and a core network 706, which includes one or more core network nodes 708. The access network 704 includes one or more access network nodes, such as network nodes 710a and 710b (one or more of which may be generally referred to as network nodes 710), or any other similar 3rdGeneration Partnership Project (3GPP) access node or non-3GPP access point. The network nodes 710 facilitate direct or indirect connection of user equipment (UE), such as by connecting UEs 712a, 712b, 712c, and 712d (one or more of which may be generally referred to as UEs 712) to the core network 706 over one or more wireless connections.

[0175] Example wireless communications over a wireless connection include transmitting and / or receiving wireless signals using electromagnetic waves, radio waves, infrared waves, and / or other types of signals suitable for conveying information without the use of wires, cables, or other material conductors. Moreover, in different embodiments, the communication system 700 may include any number of wired or wireless networks, network nodes, UEs, and / or any other components or systems that may facilitate or participate in the communication of data and / or signals whether via wired or wireless connections. The communication system 700 may include and / or interface with any type of communication, telecommunication, data, cellular, radio network, and / or other similar type of system.

[0176] The UEs 712 may be any of a wide variety of communication devices, including wireless devices arranged, configured, and / or operable to communicate wirelessly with the network nodes 710 and other communication devices. Similarly, the network nodes 710 are arranged, capable, configured, and / or operable to communicate directly or indirectly with the UEs 712 and / or with other network nodes or equipment in the telecommunication network 702 to enable and / or provide network access, such as wireless network access, and / or to perform other functions, such as administration in the telecommunication network 702.

[0177] In the depicted example, the core network 706 connects the network nodes 710 to one or more hosts, such as host 716. These connections may be direct or indirect via one or more intermediary networks or devices. In other examples, network nodes may be directly coupled to hosts. The core network 706 includes one more core network nodes (e.g., core network node 708) that are structured with hardware and software components. Features of these components may be substantially similar to those described with respect to the UEs, network nodes, and / or hosts, such that the descriptions thereof are generally applicable to the corresponding components of the core network node 708. Example core network nodes include functions of one or more of a Mobile Switching Center (MSC), Mobility Management Entity (MME), Home Subscriber Server (HSS), Access and Mobility Management Function (AMF), Session Management Function (SMF), Authentication Server Function (AUSF), Subscription Identifier De-concealing function (SIDF), Unified Data Management (UDM), Security Edge Protection Proxy (SEPP), Network Exposure Function (NEF), and / or a User Plane Function (UPF).

[0178] The host 716 may be under the ownership or control of a service provider other than an operator or provider of the access network 704 and / or the telecommunication network 702, and may be operated by the service provider or on behalf of the service provider. The host 716 may host a variety of applications to provide one or more service. Examples of such applications include live and pre-recorded audio / video content, data collection services such as retrieving and compiling data on various ambient conditions detected by a plurality of UEs, analytics functionality, social media, functions for controlling or otherwise interacting with remote devices, functions for an alarm and surveillance center, or any other such function performed by a server.

[0179] As a whole, the communication system 700 of FIGURE 10 enables connectivity between the UEs, network nodes, and hosts. In that sense, the communication system may be configured to operate according to predefined rules or procedures, such as specific standards that include, but are not limited to: Global System for Mobile Communications (GSM); Universal Mobile Telecommunications System (UMTS); Long Term Evolution (LTE), and / or other suitable 2G, 3G, 4G, 5G standards, or any applicable future generation standard (e.g., 6G); wireless local area network (WLAN) standards, such as the Institute of Electrical and Electronics Engineers (IEEE) 802.11 standards (WiFi); and / or any other appropriate wireless communication standard, such as the Worldwide Interoperability for Microwave Access (WiMax), Bluetooth, Z-Wave, Near Field Communication (NFC) ZigBee, LiFi, and / or any low-power wide-area network (LPWAN) standards such as LoRa and Sigfox.

[0180] In some examples, the telecommunication network 702 is a cellular network that implements 3GPP standardized features. Accordingly, the telecommunications network 702 may support network slicing to provide different logical networks to different devices that are connected to the telecommunication network 702. For example, the telecommunications network 702 may provide Ultra Reliable Low Latency Communication (URLLC) services to some UEs, while providing Enhanced Mobile Broadband (eMBB) services to other UEs, and / or Massive Machine Type Communication (mMTC) / Massive loT services to yet further UEs.

[0181] In some examples, the UEs 712 are configured to transmit and / or receive information without direct human interaction. For instance, a UE may be designed to transmit information to the access network 704 on a predetermined schedule, when triggered by an internal or external event, or in response to requests from the access network 704. Additionally, a UE may be configured for operating in single- or multi-RAT or multi-standard mode. For example, a UE may operate with any one or combination of Wi-Fi, NR (New Radio) and LTE, i.e. being configured for multi-radio dual connectivity (MR-DC), such as E-UTRAN (Evolved-UMTS Terrestrial Radio Access Network) New Radio - Dual Connectivity (EN-DC).

[0182] In the example, the hub 714 communicates with the access network 704 to facilitate indirect communication between one or more UEs (e.g., UE 712c and / or 712d) and network nodes (e.g., network node 710b). In some examples, the hub 714 may be a controller, router, content source and analytics, or any of the other communication devices described herein regarding UEs. For example, the hub 714 may be a broadband router enabling access to the core network 706 for the UEs. As another example, the hub 714 may be a controller that sends commands or instructions to one or more actuators in the UEs. Commands or instructions may be received from the UEs, network nodes 710, or by executable code, script, process, or other instructions in the hub 714. As another example, the hub 714 may be a data collector that acts as temporary storage for UE data and, in some embodiments, may perform analysis or other processing of the data. As another example, the hub 714 may be a content source. For example, for a UE that is a VR headset, display, loudspeaker or other media delivery device, the hub 714 may retrieve VR assets, video, audio, or other media or data related to sensory information via a network node, which the hub 714 then provides to the UE either directly, after performing local processing, and / or after adding additional local content. In still another example, the hub 714 acts as a proxy server or orchestrator for the UEs, in particular in if one or more of the UEs are low energy loT devices. The hub 714 may have a constant / persistent or intermittent connection to the network node 710b. The hub 714 may also allow for a different communication scheme and / or schedule between the hub 714 and UEs (e.g., UE 712c and / or 712d), and between the hub 714 and the core network 706. In other examples, the hub 714 is connected to the core network 706 and / or one or more UEs via a wired connection. Moreover, the hub 714 may be configured to connect to an M2M service provider over the access network 704 and / or to another UE over a direct connection. In some scenarios, UEs may establish a wireless connection with the network nodes 710 while still connected via the hub 714 via a wired or wireless connection. In some embodiments, the hub 714 may be a dedicated hub - that is, a hub whose primary function is to route communications to / from the UEs from / to the network node 710b. In other embodiments, the hub 714 may be a nondedicated hub - that is, a device which is capable of operating to route communications between the UEs and network node 710b, but which is additionally capable of operating as a communication start and / or end point for certain data channels.

[0183] FIGURE 11 shows a UE 800 in accordance with some embodiments. As used herein, a UE refers to a device capable, configured, arranged and / or operable to communicate wirelessly with network nodes and / or other UEs. Examples of a UE include, but are not limited to, a smart phone, mobile phone, cell phone, voice over IP (VoIP) phone, wireless local loop phone, desktop computer, personal digital assistant (PDA), wireless cameras, gaming console or device, music storage device, playback appliance, wearable terminal device, wireless endpoint, mobile station, tablet, laptop, laptop-embedded equipment (LEE), laptop-mounted equipment (LME), smart device, wireless customer-premise equipment (CPE), vehicle-mounted or vehicle embedded / integrated wireless device, etc. Other examples include any UE identified by the 3rd Generation Partnership Project (3GPP), including a narrow band internet of things (NB-IoT) UE, a machine type communication (MTC) UE, and / or an enhanced MTC (eMTC) UE.

[0184] A UE may support device-to-device (D2D) communication, for example by implementing a 3GPP standard for sidelink communication, Dedicated Short-Range Communication (DSRC), vehicle-to-vehicle (V2V), vehicle-to-infrastructure (V2I), or vehicle-to-everything (V2X). In other examples, a UE may not necessarily have a user in the sense of a human user who owns and / or operates the relevant device. Instead, a UE may represent a device that is intended for sale to, or operation by, a human user but which may not, or which may not initially, be associated with a specific human user (e.g., a smart sprinkler controller). Alternatively, a UE may represent a device that is not intended for sale to, or operation by, an end user but which may be associated with or operated for the benefit of a user (e.g., a smart power meter). The UE 800 includes processing circuitry 802 that is operatively coupled via a bus 804 to an input / output interface 806, a power source 808, a memory 810, a communication interface 812, and / or any other component, or any combination thereof. Certain UEs may utilize all or a subset of the components shown in FIGURE 11. The level of integration between the components may vary from one UE to another UE. Further, certain UEs may contain multiple instances of a component, such as multiple processors, memories, transceivers, transmitters, receivers, etc.

[0185] The processing circuitry 802 is configured to process instructions and data and may be configured to implement any sequential state machine operative to execute instructions stored as machine-readable computer programs in the memory 810. The processing circuitry 802 may be implemented as one or more hardware-implemented state machines (e.g., in discrete logic, field- programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), etc.); programmable logic together with appropriate firmware; one or more stored computer programs, general-purpose processors, such as a microprocessor or digital signal processor (DSP), together with appropriate software; or any combination of the above. For example, the processing circuitry 802 may include multiple central processing units (CPUs).

[0186] In the example, the input / output interface 806 may be configured to provide an interface or interfaces to an input device, output device, or one or more input and / or output devices. Examples of an output device include a speaker, a sound card, a video card, a display, a monitor, a printer, an actuator, an emitter, a smartcard, another output device, or any combination thereof. An input device may allow a user to capture information into the UE 800. Examples of an input device include a touch-sensitive or presence-sensitive display, a camera (e.g., a digital camera, a digital video camera, a web camera, etc.), a microphone, a sensor, a mouse, a trackball, a directional pad, a trackpad, a scroll wheel, a smartcard, and the like. The presence-sensitive display may include a capacitive or resistive touch sensor to sense input from a user. A sensor may be, for instance, an accelerometer, a gyroscope, a tilt sensor, a force sensor, a magnetometer, an optical sensor, a proximity sensor, a biometric sensor, etc., or any combination thereof. An output device may use the same type of interface port as an input device. For example, a Universal Serial Bus (USB) port may be used to provide an input device and an output device.

[0187] In some embodiments, the power source 808 is structured as a battery or battery pack. Other types of power sources, such as an external power source (e.g., an electricity outlet), photovoltaic device, or power cell, may be used. The power source 808 may further include power circuitry for delivering power from the power source 808 itself, and / or an external power source, to the various parts of the UE 800 via input circuitry or an interface such as an electrical power cable. Delivering power may be, for example, for charging of the power source 808. Power circuitry may perform any formatting, converting, or other modification to the power from the power source 808 to make the power suitable for the respective components of the UE 800 to which power is supplied.

[0188] The memory 810 may be or be configured to include memory such as random access memory (RAM), read-only memory (ROM), programmable read-only memory (PROM), erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), magnetic disks, optical disks, hard disks, removable cartridges, flash drives, and so forth. In one example, the memory 810 includes one or more application programs 814, such as an operating system, web browser application, a widget, gadget engine, or other application, and corresponding data 816. The memory 810 may store, for use by the UE 800, any of a variety of various operating systems or combinations of operating systems.

[0189] The memory 810 may be configured to include a number of physical drive units, such as redundant array of independent disks (RAID), flash memory, USB flash drive, external hard disk drive, thumb drive, pen drive, key drive, high-density digital versatile disc (HD-DVD) optical disc drive, internal hard disk drive, Blu-Ray optical disc drive, holographic digital data storage (HDDS) optical disc drive, external mini-dual in-line memory module (DIMM), synchronous dynamic random access memory (SDRAM), external micro-DIMM SDRAM, smartcard memory such as tamper resistant module in the form of a universal integrated circuit card (UICC) including one or more subscriber identity modules (SIMs), such as a USIM and / or ISIM, other memory, or any combination thereof. The UICC may for example be an embedded UICC (eUICC), integrated UICC (iUICC) or a removable UICC commonly known as ‘SIM card.’ The memory 810 may allow the UE 800 to access instructions, application programs and the like, stored on transitory or non-transitory memory media, to off-load data, or to upload data. An article of manufacture, such as one utilizing a communication system may be tangibly embodied as or in the memory 810, which may be or comprise a device-readable storage medium.

[0190] The processing circuitry 802 may be configured to communicate with an access network or other network using the communication interface 812. The communication interface 812 may comprise one or more communication subsystems and may include or be communicatively coupled to an antenna 822. The communication interface 812 may include one or more transceivers used to communicate, such as by communicating with one or more remote transceivers of another device capable of wireless communication (e.g., another UE or a network node in an access network). Each transceiver may include a transmitter 818 and / or a receiver 820 appropriate to provide network communications (e.g., optical, electrical, frequency allocations, and so forth). Moreover, the transmitter 818 and receiver 820 may be coupled to one or more antennas (e.g., antenna 822) and may share circuit components, software or firmware, or alternatively be implemented separately.

[0191] In the illustrated embodiment, communication functions of the communication interface 812 may include cellular communication, Wi-Fi communication, LPWAN communication, data communication, voice communication, multimedia communication, short-range communications such as Bluetooth, near-field communication, location-based communication such as the use of the global positioning system (GPS) to determine a location, another like communication function, or any combination thereof. Communications may be implemented in according to one or more communication protocols and / or standards, such as IEEE 802.11, Code Division Multiplexing Access (CDMA), Wideband Code Division Multiple Access (WCDMA), GSM, LTE, New Radio (NR), UMTS, WiMax, Ethernet, transmission control protocol / intemet protocol (TCP / IP), synchronous optical networking (SONET), Asynchronous Transfer Mode (ATM), QUIC, Hypertext Transfer Protocol (HTTP), and so forth.

[0192] Regardless of the type of sensor, a UE may provide an output of data captured by its sensors, through its communication interface 812, via a wireless connection to a network node. Data captured by sensors of a UE can be communicated through a wireless connection to a network node via another UE. The output may be periodic (e.g., once every 15 minutes if it reports the sensed temperature), random (e.g., to even out the load from reporting from several sensors), in response to a triggering event (e.g., when moisture is detected an alert is sent), in response to a request (e.g., a user initiated request), or a continuous stream (e.g., a live video feed of a patient).

[0193] As another example, a UE comprises an actuator, a motor, or a switch, related to a communication interface configured to receive wireless input from a network node via a wireless connection. In response to the received wireless input the states of the actuator, the motor, or the switch may change. For example, the UE may comprise a motor that adjusts the control surfaces or rotors of a drone in flight according to the received input or to a robotic arm performing a medical procedure according to the received input.

[0194] A UE, when in the form of an Internet of Things (loT) device, may be a device for use in one or more application domains, these domains comprising, but not limited to, city wearable technology, extended industrial application and healthcare. Non-limiting examples of such an loT device are a device which is or which is embedded in: a connected refrigerator or freezer, a TV, a connected lighting device, an electricity meter, a robot vacuum cleaner, a voice controlled smart speaker, a home security camera, amotion detector, a thermostat, asmoke detector, adoor / window sensor, a flood / moisture sensor, an electrical door lock, a connected doorbell, an air conditioning system like a heat pump, an autonomous vehicle, a surveillance system, a weather monitoring device, a vehicle parking monitoring device, an electric vehicle charging station, a smart watch, a fitness tracker, a head-mounted display for Augmented Reality (AR) or Virtual Reality (VR), a wearable for tactile augmentation or sensory enhancement, a water sprinkler, an animal- or itemtracking device, a sensor for monitoring a plant or animal, an industrial robot, an Unmanned Aerial Vehicle (UAV), and any kind of medical device, like a heart rate monitor or a remote controlled surgical robot. A UE in the form of an loT device comprises circuitry and / or software in dependence of the intended application of the loT device in addition to other components as described in relation to the UE 800 shown in FIGURE 11.

[0195] As yet another specific example, in an loT scenario, a UE may represent a machine or other device that performs monitoring and / or measurements, and transmits the results of such monitoring and / or measurements to another UE and / or a network node. The UE may in this case be an M2M device, which may in a 3GPP context be referred to as an MTC device. As one particular example, the UE may implement the 3GPP NB-IoT standard. In other scenarios, a UE may represent a vehicle, such as a car, a bus, a truck, a ship and an airplane, or other equipment that is capable of monitoring and / or reporting on its operational status or other functions associated with its operation.

[0196] In practice, any number of UEs may be used together with respect to a single use case. For example, a first UE might be or be integrated in a drone and provide the drone’s speed information (obtained through a speed sensor) to a second UE that is a remote controller operating the drone. When the user makes changes from the remote controller, the first UE may adjust the throttle on the drone (e.g. by controlling an actuator) to increase or decrease the drone’s speed. The first and / or the second UE can also include more than one of the functionalities described above. For example, a UE might comprise the sensor and the actuator, and handle communication of data for both the speed sensor and the actuators.

[0197] FIGURE 12 shows a network node 900 in accordance with some embodiments. As used herein, network node refers to equipment capable, configured, arranged and / or operable to communicate directly or indirectly with a UE and / or with other network nodes or equipment, in a telecommunication network. Examples of network nodes include, but are not limited to, access points (APs) (e.g., radio access points), base stations (BSs) (e.g., radio base stations, Node Bs, evolved Node Bs (eNBs) and NR NodeBs (gNBs)).

[0198] Base stations may be categorized based on the amount of coverage they provide (or, stated differently, their transmit power level) and so, depending on the provided amount of coverage, may be referred to as femto base stations, pico base stations, micro base stations, or macro base stations. A base station may be a relay node or a relay donor node controlling a relay. A network node may also include one or more (or all) parts of a distributed radio base station such as centralized digital units and / or remote radio units (RRUs), sometimes referred to as Remote Radio Heads (RRHs). Such remote radio units may or may not be integrated with an antenna as an antenna integrated radio. Parts of a distributed radio base station may also be referred to as nodes in a distributed antenna system (DAS).

[0199] Other examples of network nodes include multiple transmission point (multi-TRP) 5G access nodes, multi-standard radio (MSR) equipment such as MSR BSs, network controllers such as radio network controllers (RNCs) or base station controllers (BSCs), base transceiver stations (BTSs), transmission points, transmission nodes, multi-cell / multicast coordination entities (MCEs), Operation and Maintenance (O&M) nodes, Operations Support System (OSS) nodes, Self-Organizing Network (SON) nodes, positioning nodes (e.g., Evolved Serving Mobile Location Centers (E-SMLCs)), and / or Minimization of Drive Tests (MDTs).

[0200] The network node 900 includes a processing circuitry 902, a memory 904, a communication interface 906, and a power source 908. The network node 900 may be composed of multiple physically separate components (e.g., aNodeB component and a RNC component, or a BTS component and a BSC component, etc.), which may each have their own respective components. In certain scenarios in which the network node 900 comprises multiple separate components (e.g., BTS and BSC components), one or more of the separate components may be shared among several network nodes. For example, a single RNC may control multiple NodeBs. In such a scenario, each unique NodeB and RNC pair, may in some instances be considered a single separate network node. In some embodiments, the network node 900 may be configured to support multiple radio access technologies (RATs). In such embodiments, some components may be duplicated (e.g., separate memory 904 for different RATs) and some components may be reused (e.g., a same antenna 910 may be shared by different RATs). The network node 900 may also include multiple sets of the various illustrated components for different wireless technologies integrated into network node 900, for example GSM, WCDMA, LTE, NR, WiFi, Zigbee, Z-wave, LoRaWAN, Radio Frequency Identification (RFID) or Bluetooth wireless technologies. These wireless technologies may be integrated into the same or different chip or set of chips and other components within network node 900.

[0201] The processing circuitry 902 may comprise a combination of one or more of a microprocessor, controller, microcontroller, central processing unit, digital signal processor, application-specific integrated circuit, field programmable gate array, or any other suitable computing device, resource, or combination of hardware, software and / or encoded logic operable to provide, either alone or in conjunction with other network node 900 components, such as the memory 904, to provide network node 900 functionality.

[0202] In some embodiments, the processing circuitry 902 includes a system on a chip (SOC). In some embodiments, the processing circuitry 902 includes one or more of radio frequency (RF) transceiver circuitry 912 and baseband processing circuitry 914. In some embodiments, the radio frequency (RF) transceiver circuitry 912 and the baseband processing circuitry 914 may be on separate chips (or sets of chips), boards, or units, such as radio units and digital units. In alternative embodiments, part or all of RF transceiver circuitry 912 and baseband processing circuitry 914 may be on the same chip or set of chips, boards, or units.

[0203] The memory 904 may comprise any form of volatile or non-volatile computer-readable memory including, without limitation, persistent storage, solid-state memory, remotely mounted memory, magnetic media, optical media, random access memory (RAM), read-only memory (ROM), mass storage media (for example, a hard disk), removable storage media (for example, a flash drive, a Compact Disk (CD) or a Digital Video Disk (DVD)), and / or any other volatile or non-volatile, non-transitory device-readable and / or computer-executable memory devices that store information, data, and / or instructions that may be used by the processing circuitry 902. The memory 904 may store any suitable instructions, data, or information, including a computer program, software, an application including one or more of logic, rules, code, tables, and / or other instructions capable of being executed by the processing circuitry 902 and utilized by the network node 900. The memory 904 may be used to store any calculations made by the processing circuitry 902 and / or any data received via the communication interface 906. In some embodiments, the processing circuitry 902 and memory 904 is integrated.

[0204] The communication interface 906 is used in wired or wireless communication of signaling and / or data between anetwork node, access network, and / or UE. As illustrated, the communication interface 906 comprises port(s) / terminal(s) 916 to send and receive data, for example to and from a network over a wired connection. The communication interface 906 also includes radio frontend circuitry 918 that may be coupled to, or in certain embodiments a part of, the antenna 910. Radio front-end circuitry 918 comprises filters 920 and amplifiers 922. The radio front-end circuitry 918 may be connected to an antenna 910 and processing circuitry 902. The radio frontend circuitry may be configured to condition signals communicated between antenna 910 and processing circuitry 902. The radio front-end circuitry 918 may receive digital data that is to be sent out to other network nodes or UEs via a wireless connection. The radio front-end circuitry 918 may convert the digital data into a radio signal having the appropriate channel and bandwidth parameters using a combination of filters 920 and / or amplifiers 922. The radio signal may then be transmited via the antenna 910. Similarly, when receiving data, the antenna 910 may collect radio signals which are then converted into digital data by the radio front-end circuitry 918. The digital data may be passed to the processing circuitry 902. In other embodiments, the communication interface may comprise different components and / or different combinations of components.

[0205] In certain alternative embodiments, the network node 900 does not include separate radio front-end circuitry 918, instead, the processing circuitry 902 includes radio front-end circuitry and is connected to the antenna 910. Similarly, in some embodiments, all or some of the RF transceiver circuitry 912 is part of the communication interface 906. In still other embodiments, the communication interface 906 includes one or more ports or terminals 916, the radio front-end circuitry 918, and the RF transceiver circuitry 912, as part of a radio unit (not shown), and the communication interface 906 communicates with the baseband processing circuitry 914, which is part of a digital unit (not shown).

[0206] The antenna 910 may include one or more antennas, or antenna arrays, configured to send and / or receive wireless signals. The antenna 910 may be coupled to the radio front-end circuitry 918 and may be any type of antenna capable of transmiting and receiving data and / or signals wirelessly. In certain embodiments, the antenna 910 is separate from the network node 900 and connectable to the network node 900 through an interface or port.

[0207] The antenna 910, communication interface 906, and / or the processing circuitry 902 may be configured to perform any receiving operations and / or certain obtaining operations described herein as being performed by the network node. Any information, data and / or signals may be received from a UE, another network node and / or any other network equipment. Similarly, the antenna 910, the communication interface 906, and / or the processing circuitry 902 may be configured to perform any transmiting operations described herein as being performed by the network node. Any information, data and / or signals may be transmited to a UE, another network node and / or any other network equipment.

[0208] The power source 908 provides power to the various components of network node 900 in a form suitable for the respective components (e.g., at a voltage and current level needed for each respective component). The power source 908 may further comprise, or be coupled to, power management circuitry to supply the components of the network node 900 with power for performing the functionality described herein. For example, the network node 900 may be connectable to an external power source (e.g., the power grid, an electricity outlet) via an input circuitry or interface such as an electrical cable, whereby the external power source supplies power to power circuitry of the power source 908. As a further example, the power source 908 may comprise a source of power in the form of a battery or batery pack which is connected to, or integrated in, power circuitry. The battery may provide backup power should the external power source fail.

[0209] Embodiments of the network node 900 may include additional components beyond those shown in FIGURE 12 for providing certain aspects of the network node’s functionality, including any of the functionality described herein and / or any functionality necessary to support the subject matter described herein. For example, the network node 900 may include user interface equipment to allow input of information into the network node 900 and to allow output of information from the network node 900. This may allow a user to perform diagnostic, maintenance, repair, and other administrative functions for the network node 900.

[0210] FIGURE 13 illustrates an example method 1000 by a UE 712 for monitoring of UE-sided models using validity of monitoring results, according to certain embodiments. In the illustrated embodiment, the method begins at step 1002 when the UE 712 transmits, to a network node 710, model performance feedback associated with a performance of a model during a first monitoring occasion. At step 1004, the UE 712 receives configuration information associated with a validity of a model monitoring result associated with the first monitoring occasion. The configuration information is for performing at least one of: activating the model; deactivating the model; and initiating a second monitoring occasion.

[0211] In a particular embodiment, the UE 712 receives, from the network node 710, an indication to initiate the first monitoring occasion and initiates the first monitoring occasion based on the indication.

[0212] In a particular embodiment, the UE 712 initiates the second monitoring occasion based on a timer, and the timer is part of the configuration information. A timer duration is based on the validity of a model monitoring result associated with the first monitoring occasion.

[0213] In a further particular embodiment, duration of the timer is determined based on the UE mobility, where the mobility can comprise of the UE speed, or the change in the UE radio environment.

[0214] In a particular embodiment, the configuration information comprises information associated with an event. The UE 712 detects an occurrence of the at least one event and, based on the occurrence of the at least one event, initiates the second monitoring occasion.

[0215] In a further particular embodiment, detection of the event is associated with a change in at least one radio measurement that is greater than or less than a threshold.

[0216] In a particular embodiment, the UE 712 receives, from the network node 710, information for evaluating performance of the model during the first monitoring occasion. In a particular embodiment, the model performance feedback transmitted to the network node comprises at least one success metric indicating a level of success of the performance of the model or at least one failure metric indicating a level of failure of the performance of the model.

[0217] In a further particular embodiment, the at least one success metric or failure metric is based on a difference between a predicted value and a ground-truth value.

[0218] In a further particular embodiment, the UE 712 transmits information to the network node for determining a validity of the model monitoring result of the model and / or the first modeling occasion, wherein the information comprises at least one of: an indication of a model sensitivity; an indication of UE input change; a cost associated to the first monitoring occasion; at least one success metric; at least one failure metric; and historical information.

[0219] In a particular embodiment, the UE 712 receives, from the network node 710, a request for assistance information. The UE 712 transmits, to the network node 710, the assistance information for determining by the network node the validity of the model monitoring result associated with the first monitoring occasion.

[0220] In a further particular embodiment, the configuration information configures the UE to deactivate the model when the validity of the model monitoring result associated with the first monitoring occasion is below a threshold amount.

[0221] In a particular embodiment, the UE 712 deactivates the model by performing a fallback to a legacy procedure.

[0222] In a particular embodiment, activating the model results in the UE 712 receiving AI / ML inference results at least until the second monitoring occasion.

[0223] In a further particular embodiment, a model monitoring period comprises a duration of time from the first monitoring occasion to the second monitoring occasion, and wherein the configuration information comprises: an indication of the model monitoring period, an indication of an increase in the model monitoring period, an indication of a decrease in the model monitoring period, or an indication that the model monitoring period can stay the same.

[0224] FIGURE 14 illustrates an example method 1100 by a network node 710 for monitoring of UE-sided models using validity of monitoring results, according to certain embodiments. In the illustrated embodiment, the method 1100 begins at step 1102 when the network node 710 receives, from a UE 712, model performance feedback associated with a performance of a model during a first monitoring occasion. Based on the model performance feedback, the network node 710 determines a validity of a model monitoring result associated with the first monitoring occasion, at step 1104. Based on the validity of the model monitoring result associated with the first monitoring occasion, the network node 710 transmits model configuration information to the UE 712 for performing at least one of: activating the model; deactivating the model; and initiating a second monitoring occasion.

[0225] In a particular embodiment, the network node 712 transmits, to the UE 710, an indication to initiate the first monitoring occasion.

[0226] In a particular embodiment, the model configuration information comprises information associated with a timer for initiating the second monitoring occasion.

[0227] In a further particular embodiment, a duration of the timer is determined based on a speed of a UE.

[0228] In a particular embodiment, the configuration information comprises information associated with an event, and the first monitoring occasion is initiated based on a detection of the event.

[0229] In a further particular embodiment, detection of the event is associated with a change in at least one radio measurement that is greater than or less than a second threshold.

[0230] In a particular embodiment, the network node 710 transmits, to the UE 712, information for evaluating performance of the model during the first monitoring occasion.

[0231] In a particular embodiment, the network node 710 determines, based on the model performance feedback, at least one success metric or failure metric. The validity of the first monitoring occasion is determined based on the at least one success metric or failure metric.

[0232] In a particular embodiment, the model performance feedback that is received from the UE includes a success metric indicating a level of success of the performance of the model or a failure metric indicating a level of failure of the performance of the model.

[0233] In a further particular embodiment, the at least one success metric or failure metric is based on a difference between a predicted value and a ground-truth value.

[0234] In a particular embodiment, determining the validity of the first monitoring occasion is based on at least one of: model sensitivity; UE input change; cost associated to the first monitoring occasion; at least one success metric; at least one failure metric; and historical information.

[0235] In a particular embodiment, the network node 712 transmits, to the UE 710, a request for assistance information associated and receives the assistance information from the UE. The validity of the model monitoring result associated with the first model monitoring procedure is determined based on the assistance information from the UE.

[0236] In a particular embodiment, the configuration information indicates at least one event for performing the second modeling occasion. In a particular embodiment, the configuration information configures the UE 712 to deactivate the model when the validity of the model monitoring result associated with the first monitoring occasion is below a threshold amount.

[0237] In a further particular embodiment, when deactivating the model, the UE 712 performs a fallback to a legacy procedure. In a further particular embodiment, when activating the model, the UE 712 receives AI / ML inference results at least until the second monitoring occasion.

[0238] In a particular embodiment, a model monitoring period includes a duration of time from the first monitoring occasion to the second monitoring occasion, and wherein the configuration information comprises: an indication of the model monitoring period, an indication of an increase in the model monitoring period, an indication of a decrease in the model monitoring period, or an indication that the model monitoring period can stay the same.

[0239] Although the computing devices described herein (e.g., UEs, network nodes, hosts) may include the illustrated combination of hardware components, other embodiments may comprise computing devices with different combinations of components. It is to be understood that these computing devices may comprise any suitable combination of hardware and / or software needed to perform the tasks, features, functions and methods disclosed herein. Determining, calculating, obtaining or similar operations described herein may be performed by processing circuitry, which may process information by, for example, converting the obtained information into other information, comparing the obtained information or converted information to information stored in the network node, and / or performing one or more operations based on the obtained information or converted information, and as a result of said processing making a determination. Moreover, while components are depicted as single boxes located within a larger box, or nested within multiple boxes, in practice, computing devices may comprise multiple different physical components that make up a single illustrated component, and functionality may be partitioned between separate components. For example, a communication interface may be configured to include any of the components described herein, and / or the functionality of the components may be partitioned between the processing circuitry and the communication interface. In another example, non-computationally intensive functions of any of such components may be implemented in software or firmware and computationally intensive functions may be implemented in hardware.

[0240] In certain embodiments, some or all of the functionality described herein may be provided by processing circuitry executing instructions stored on in memory, which in certain embodiments may be a computer program product in the form of a non-transitory computer-readable storage medium. In alternative embodiments, some or all of the functionality may be provided by the processing circuitry without executing instructions stored on a separate or discrete device-readable storage medium, such as in a hard-wired manner. In any of those particular embodiments, whether executing instructions stored on a non-transitory computer-readable storage medium or not, the processing circuitry can be configured to perform the described functionality. The benefits provided by such functionality are not limited to the processing circuitry alone or to other components of the computing device, but are enjoyed by the computing device as a whole, and / or by end users and a wireless network generally.

[0241] EXAMPLE EMBODIMENTS

[0242] Group A Example Embodiments

[0243] Example Embodiment Al. A method by a user equipment for monitoring of UE-sided models using validity of monitoring results, the method comprising: any of the user equipment steps, features, or functions described above, either alone or in combination with other steps, features, or functions described above.

[0244] Example Embodiment A2. The method of the previous embodiment, further comprising one or more additional user equipment steps, features or functions described above.

[0245] Example Embodiment A3. The method of any of the previous embodiments, further comprising: providing user data; and forwarding the user data to a host computer via the transmission to the network node.

[0246] Group B Example Embodiments

[0247] Example Embodiment Bl. A method performed by a network node for monitoring of UE- sided models using validity of monitoring results, the method comprising: any of the network node steps, features, or functions described above, either alone or in combination with other steps, features, or functions described above.

[0248] Example Embodiment B2. The method of the previous embodiment, further comprising one or more additional network node steps, features or functions described above.

[0249] Example Embodiment B3. The method of any of the previous embodiments, further comprising: obtaining user data; and forwarding the user data to a host or a user equipment.

[0250] Group C Example Embodiments

[0251] Example Embodiment Cl. A method by a user equipment (UE) for monitoring of UE- sided models using validity of monitoring results, the method comprising: transmitting (120a), to a network node, model performance feedback associated with a performance of a model during a model monitoring procedure and / or model monitoring occasion; based on a validity of the model and / or the model monitoring procedure, receiving model configuration information (140) for performing at least one of: activating the model; deactivating the model; and initiating a subsequent model monitoring procedure in a subsequent monitoring occasion.

[0252] Example Embodiment C2. The method of Example Embodiment Cl, comprising initiating (100) the model monitoring procedure.

[0253] Example Embodiment C3. The method of Example Embodiment C2, comprising receiving, from the network node, an indication to initiate the model monitoring procedure, and wherein the model monitoring procedure is initiated based on the indication.

[0254] Example Embodiment C4. The method of Example Embodiment C2, wherein the model monitoring procedure is initiated based on a timer.

[0255] Example Embodiment C5. The method of Example Embodiment C4, wherein a duration of the timer is determined based on one or more of: a model sensitivity level; an expected frequency of change in model inputs; UE mobility; a speed of a UE; a rate of change of CSI; a required accuracy level for the model monitoring metrics; a required error level; historical information associated with at least one previous model monitoring procedure and / or previous model monitoring occasion; and a cost for model monitoring.

[0256] Example Embodiment C6. The method of any one of Example Embodiments C2 to C5, wherein the model monitoring procedure is initiated based on a detection of an event.

[0257] Example Embodiment C7. The method of Example Embodiment C6, wherein detection of the event is associated with at least one of: a location of the UE; at least one radio measurement associated with the UE; and at least one non-radio measurement associated with the UE.

[0258] Example Embodiment C8. The method of any one of Example Embodiments C7 to C8, wherein detection of the event is associated with at least one of: a change in a UE geolocation that is greater than or less than a first threshold; a change in a radio measurement is greater than or less than a second threshold; a change in a non-radio measurement is greater than or less than a third threshold.

[0259] Example Embodiment C9. The method of any one of Example Embodiments Cl to C8, comprising receiving (110), from the network node, information for evaluating performance of the model during the model monitoring occasion.

[0260] Example Embodiment CIO. The method of any one of Example Embodiments Cl to C9, comprising determining (120b), based on the model performance feedback, at least one success metric or failure metric.

[0261] Example Embodiment C 11. The method of Example Embodiment CIO, wherein the model performance feedback transmitted to the network node comprises the at least one success metric or failure metric.

[0262] Example Embodiment Cl 2. The method of any one of Example Embodiments CIO to Cl 1, wherein the at least one success metric indicates a level of success of the performance of the model and / or the at least one failure metric indicates a level of failure of the performance of the model.

[0263] Example Embodiment Cl 3. The method of any one of Example Embodiments CIO to Cl 1, wherein the at least one success metric or failure metric is based on a difference between a predicted value and a ground-truth value.

[0264] Example Embodiment C14.The method of any one of Example Embodiments Cl to C13, comprising transmitting information to the network node for determining (130) a validity of the model and / or the model monitoring procedure, wherein the information comprises at least one of: an indication of a model sensitivity; an indication of UE input change; a cost associated to the monitoring; at least one success metric; at least one failure metric; and historical information.

[0265] Example Embodiment Cl 5. The method of any one of Example Embodiments Cl to Cl 4, comprising: receiving, from the network node, a request for assistance information associated; transmitting, to the network node, the assistance information for determining by the network node a validity of the model and / or the model monitoring procedure.

[0266] Example Embodiment C16.The method of any one of Example Embodiments C14 to C15, comprising: receiving, from the network node, configuration information determined based on the validity of the model and / or the model monitoring procedure, wherein the configuration information indicates at least one event for performing the modeling monitoring procedure in another modeling occasion or for initiating a subsequent modeling monitoring procedure in another modeling occasion.

[0267] Example Embodiment Cl 7. The method of Example Embodiment Cl 6, comprising; detecting an occurrence of the at least one event; and based on the occurrence of the at least one event, performing the modeling monitoring procedure in another modeling occasion or initiating a subsequent modeling monitoring procedure in another modeling occasion.

[0268] Example Embodiment C18.The method of any one of Example Embodiment C16 to C17, wherein the at least one event is associated with at least one of: a change in a UE geolocation that is greater than or less than a first threshold; a change in a radio measurement is greater than or less than a second threshold; a change in a non-radio measurement is greater than or less tha a third threshold.

[0269] Example Embodiment Cl 9. The method of any one of Example Embodiments Cl 6 to Cl 88, wherein the configuration information configures the UE to revert to a fallback procedure when the validity of the model and / or the model monitoring procedure is below a threshold amount.

[0270] Example Embodiment C20.The method of any one of Example Embodiments C 16 to C 19, wherein the configuration information comprises an indication of a model monitoring period.

[0271] Example Embodiment C21.The method of any one of Example Embodiments C16 to C20, wherein the configuration information comprises: an indication of an increase in a model monitoring period, an indication of a decrease in the model monitoring period, or an indication that the model monitoring period can stay the same.

[0272] Example Embodiment C22. The method of Example Embodiments Cl to C21, further comprising: providing user data; and forwarding the user data to a host via the transmission to the network node.

[0273] Example Embodiment C23.A user equipment comprising processing circuitry configured to perform any of the methods of Example Embodiments Cl to C22.

[0274] Example Embodiment C24. A user equipment configured to perform any of the methods of Example Embodiments Cl to C22.

[0275] Example Embodiment C25.A wireless device comprising processing circuitry configured to perform any of the methods of Example Embodiments Cl to C22.

[0276] Example Embodiment C26. A computer program comprising instructions which when executed on a computer perform any of the methods of Example Embodiments Cl to C22.

[0277] Example Embodiment C27. A computer program product comprising computer program, the computer program comprising instructions which when executed on a computer perform any of the methods of Example Embodiments Cl to C22.

[0278] Example Embodiment C28. A non-transitory computer readable medium storing instructions which when executed by a computer perform any of the methods of Example Embodiments Cl to C22.

[0279] Group D Example Embodiments

[0280] Example Embodiment DI. A method by a network node for monitoring of UE-sided models using validity of monitoring results, the method comprising: receiving (120a), from a UE, model performance feedback associated with a performance of a model during a model monitoring procedure and / or model monitoring occasion; based on the model performance feedback, determining (130) a validity of the model and / or the model monitoring procedure; and based on the validity of the model and / or the model monitoring procedure, transmitting model configuration information (140) for performing at least one of: activating the model; deactivating the model; and initiating a subsequent model monitoring procedure in a subsequent monitoring occasion. Example Embodiment D2. The method of Example Embodiment DI, comprising initiating (100) the model monitoring procedure.

[0281] Example Embodiment D3. The method of Example Embodiment D2, wherein the model monitoring procedure is initiated based on a timer.

[0282] Example Embodiment D4. The method of Example Embodiment D3, wherein a duration of the timer is determined based on one or more of: a model sensitivity level; an expected frequency of change in model inputs; UE mobility; a speed of a UE; a rate of change of CSI; a required accuracy level for the model monitoring metrics; a required error level; historical information associated with at least one previous model monitoring procedure and / or previous model monitoring occasion; and a cost for model monitoring.

[0283] Example Embodiment D5. The method of any one of Example Embodiments D2 to D4, wherein the model monitoring procedure is initiated based on a detection of an event.

[0284] Example Embodiment D6. The method of Example Embodiment D5, wherein detection of the event is associated with at least one of: a location of the UE; at least one radio measurement associated with the UE; and at least one non-radio measurement associated with the UE.

[0285] Example Embodiment D7. The method of any one of Example Embodiments D5 to D6, wherein detection of the event is associated with at least one of: a change in a UE geolocation that is greater than or less than a first threshold; a change in a radio measurement is greater than or less than a second threshold; a change in a non-radio measurement is greater than or less than a third threshold.

[0286] Example Embodiment D8. The method of any one of Example Embodiments DI to D7, comprising transmitting (110), to the UE, information for evaluating performance of the model during the model monitoring occasion.

[0287] Example Embodiment D9. The method of any one of Example Embodiments DI to D8, comprising determining (120b), based on the model performance feedback, at least one success metric or failure metric, wherein the validity of the model and / or the model monitoring procedure is determined based on the at least one success metric or failure metric.

[0288] Example Embodiment D10. The method of any one of Example Embodiments DI to D8, wherein the model performance feedback received from the UE comprises at least one success metric or failure metric.

[0289] Example Embodiment Dl l. The method of any one of Example Embodiments D9 to D10, wherein the at least one success metric indicates a level of success of the performance of the model and / or the at least one failure metric indicates a level of failure of the performance of the model.

[0290] Example Embodiment DI 2. The method of any one of Example Embodiments D9 to DI 1, wherein the at least one success metric or failure metric is based on a difference between a predicted value and a ground-truth value.

[0291] Example Embodiment DI 3. The method of any one of Example Embodiments DI to DI 2, wherein determining (130) the validity of the model and / or the model monitoring procedure is based on at least one of: model sensitivity; UE input change; cost associated to the monitoring; at least one success metric; at least one failure metric; and historical information.

[0292] Example Embodiment DI 4. The method of any one of Example Embodiments DI to DI 3, comprising: transmitting, to the UE, a request for assistance information associated; receiving, from the UE, the assistance information, and wherein the validity of the model and / or the model monitoring procedure is determined based on the assistance information from the UE.

[0293] Example Embodiment DI 5. The method of any one of Example Embodiments DI to DI 4, comprising: based on the validity of the model and / or the model monitoring procedure, transmitting configuration information to the UE, wherein the configuration information indicates at least one event for performing the modeling monitoring procedure in another modeling occasion or for initiating a subsequent modeling monitoring procedure in another modeling occasion.

[0294] Example Embodiment DI 6. The method of Example Embodiment DI 5, wherein the at least one event is associated with at least one of: a change in a UE geolocation that is greater than or less than a first threshold; a change in a radio measurement is greater than or less than a second threshold; a change in a non-radio measurement is greater than or less than a third threshold.

[0295] Example Embodiment D17. The method of any one of Example Embodiments D15 to D16, wherein the configuration information configures the UE to revert to a fallback procedure when the validity of the model and / or the model monitoring procedure is below a threshold amount.

[0296] Example Embodiment DI 8. The method of any one of Example Embodiments D15 to D17, wherein the configuration information comprises an indication of a model monitoring period.

[0297] Example Embodiment D19. The method of any one of Example Embodiments D15 to D17, wherein the configuration information comprises: an indication of an increase in a model monitoring period, an indication of a decrease in the model monitoring period, or an indication that the model monitoring period can stay the same.

[0298] Example Embodiment D20. The method of any of the previous Example Embodiments, further comprising: obtaining user data; and forwarding the user data to a host or a user equipment.

[0299] Example Embodiment D21. A network node comprising processing circuitry configured to perform any of the methods of Example Embodiments DI to D20.

[0300] Example Embodiment D22. A network node configured to perform any of the methods of Example Embodiments DI to D20. Example Embodiment D23. A computer program comprising instructions which when executed on a computer perform any of the methods of Example Embodiments DI to D20.

[0301] Example Embodiment D24. A computer program product comprising computer program, the computer program comprising instructions which when executed on a computer perform any of the methods of Example Embodiments DI to D20.

[0302] Example Embodiment D25. A non-transitory computer readable medium storing instructions which when executed by a computer perform any of the methods of Example Embodiments DI to D20.

[0303] Group E Example Embodiments

[0304] Example Embodiment El. A user equipment for monitoring of UE-sided models using validity of monitoring results, the UE comprising: processing circuitry configured to perform any of the steps of any of the Group A and C Example Embodiments; and power supply circuitry configured to supply power to the processing circuitry.

[0305] Example Embodiment E2. A network node for monitoring of UE-sided models using validity of monitoring results, the network node comprising: processing circuitry configured to perform any of the steps of any of the Group B and D Example Embodiments; power supply circuitry configured to supply power to the processing circuitry.

[0306] Example Embodiment E3. A user equipment (UE) for monitoring of UE-sided models using validity of monitoring results, the UE comprising: an antenna configured to send and receive wireless signals; radio front-end circuitry connected to the antenna and to processing circuitry, and configured to condition signals communicated between the antenna and the processing circuitry; the processing circuitry being configured to perform any of the steps of any of the Group A and C Example Embodiments; an input interface connected to the processing circuitry and configured to allow input of information into the UE to be processed by the processing circuitry; an output interface connected to the processing circuitry and configured to output information from the UE that has been processed by the processing circuitry; and a battery connected to the processing circuitry and configured to supply power to the UE.

[0307] Example Embodiment E4. A host configured to operate in a communication system to provide an over-the-top (OTT) service, the host comprising: processing circuitry configured to provide user data; and a network interface configured to initiate transmission of the user data to a cellular network for transmission to a user equipment (UE), wherein the UE comprises a communication interface and processing circuitry, the communication interface and processing circuitry of the UE being configured to perform any of the steps of any of the Group A and C Example Embodiments to receive the user data from the host.

[0308] Example Embodiment E5. The host of the previous Example Embodiment, wherein the cellular network further includes a network node configured to communicate with the UE to transmit the user data to the UE from the host.

[0309] Example Embodiment E6. The host of the previous 2 Example Embodiments, wherein: the processing circuitry of the host is configured to execute a host application, thereby providing the user data; and the host application is configured to interact with a client application executing on the UE, the client application being associated with the host application.

[0310] Example Embodiment E7. A method implemented by a host operating in a communication system that further includes a network node and a user equipment (UE), the method comprising: providing user data for the UE; and initiating a transmission carrying the user data to the UE via a cellular network comprising the network node, wherein the UE performs any of the operations of any of the Group A embodiments to receive the user data from the host.

[0311] Example Embodiment E8. The method of the previous Example Embodiment, further comprising: at the host, executing a host application associated with a client application executing on the UE to receive the user data from the UE.

[0312] Example Embodiment E9. The method of the previous Example Embodiment, further comprising: at the host, transmitting input data to the client application executing on the UE, the input data being provided by executing the host application, wherein the user data is provided by the client application in response to the input data from the host application.

[0313] Example Embodiment E10. A host configured to operate in a communication system to provide an over-the-top (OTT) service, the host comprising: processing circuitry configured to provide user data; and a network interface configured to initiate transmission of the user data to a cellular network for transmission to a user equipment (UE), wherein the UE comprises a communication interface and processing circuitry, the communication interface and processing circuitry of the UE being configured to perform any of the steps of any of the Group A and C Example Embodiments to transmit the user data to the host.

[0314] Example Embodiment Ell. The host of the previous Example Embodiment, wherein the cellular network further includes a network node configured to communicate with the UE to transmit the user data from the UE to the host.

[0315] Example Embodiment El 2. The host of the previous 2 Example Embodiments, wherein: the processing circuitry of the host is configured to execute a host application, thereby providing the user data; and the host application is configured to interact with a client application executing on the UE, the client application being associated with the host application. Example Embodiment El 3. A method implemented by a host configured to operate in a communication system that further includes a network node and a user equipment (UE), the method comprising: at the host, receiving user data transmitted to the host via the network node by the UE, wherein the UE performs any of the steps of any of the Group A and C Example Embodiments to transmit the user data to the host.

[0316] Example Embodiment E14. The method of the previous Example Embodiment, further comprising: at the host, executing a host application associated with a client application executing on the UE to receive the user data from the UE.

[0317] Example Embodiment El 5. The method of the previous Example Embodiment, further comprising: at the host, transmitting input data to the client application executing on the UE, the input data being provided by executing the host application, wherein the user data is provided by the client application in response to the input data from the host application.

[0318] Example Embodiment E16. A host configured to operate in a communication system to provide an over-the-top (OTT) service, the host comprising: processing circuitry configured to provide user data; and a network interface configured to initiate transmission of the user data to a network node in a cellular network for transmission to a user equipment (UE), the network node having a communication interface and processing circuitry, the processing circuitry of the network node configured to perform any of the operations of any of the Group B and D Example Embodiments to transmit the user data from the host to the UE.

[0319] Example Embodiment El 7. The host of the previous Example Embodiment, wherein: the processing circuitry of the host is configured to execute a host application that provides the user data; and the UE comprises processing circuitry configured to execute a client application associated with the host application to receive the transmission of user data from the host.

[0320] Example Embodiment El 8. A method implemented in a host configured to operate in a communication system that further includes a network node and a user equipment (UE), the method comprising: providing user data for the UE; and initiating a transmission carrying the user data to the UE via a cellular network comprising the network node, wherein the network node performs any of the operations of any of the Group B and D Example Embodiments to transmit the user data from the host to the UE.

[0321] Example Embodiment E19. The method of the previous Example Embodiment, further comprising, at the network node, transmitting the user data provided by the host for the UE.

[0322] Example Embodiment E20. The method of any of the previous 2 Example Embodiments, wherein the user data is provided at the host by executing a host application that interacts with a client application executing on the UE, the client application being associated with the host application.

[0323] Example Embodiment E21. A communication system configured to provide an over-the- top service, the communication system comprising: a host comprising: processing circuitry configured to provide user data for a user equipment (UE), the user data being associated with the over-the-top service; and a network interface configured to initiate transmission of the user data toward a cellular network node for transmission to the UE, the network node having a communication interface and processing circuitry, the processing circuitry of the network node configured to perform any of the operations of any of the Group B and D Example Embodiments to transmit the user data from the host to the UE.

[0324] Example Embodiment E22. The communication system of the previous Example Embodiment, further comprising: the network node; and / or the user equipment.

[0325] Example Embodiment E23. A host configured to operate in a communication system to provide an over-the-top (OTT) service, the host comprising: processing circuitry configured to initiate receipt of user data; and a network interface configured to receive the user data from a network node in a cellular network, the network node having a communication interface and processing circuitry, the processing circuitry of the network node configured to perform any of the operations of any of the Group B and D Example Embodiments to receive the user data from a user equipment (UE) for the host.

[0326] Example Embodiment E24. The host of the previous 2 Example Embodiments, wherein: the processing circuitry of the host is configured to execute a host application, thereby providing the user data; and the host application is configured to interact with a client application executing on the UE, the client application being associated with the host application.

[0327] Example Embodiment E25.The host of the any of the previous 2 Example Embodiments, wherein the initiating receipt of the user data comprises requesting the user data.

[0328] Example Embodiment E26. A method implemented by a host configured to operate in a communication system that further includes a network node and a user equipment (UE), the method comprising: at the host, initiating receipt of user data from the UE, the user data originating from a transmission which the network node has received from the UE, wherein the network node performs any of the steps of any of the Group B and D Example Embodiments to receive the user data from the UE for the host.

[0329] Example Embodiment E27. The method of the previous Example Embodiment, further comprising at the network node, transmitting the received user data to the host.

Claims

CLAIMS1. A method by a user equipment, UE, for monitoring of UE-sided models using validity of monitoring results, the method comprising: transmitting (120a), to a network node, model performance feedback associated with a performance of a model during a first monitoring occasion; receiving configuration information (140) associated with a validity of a model monitoring result associated with the first monitoring occasion, the configuration information for performing at least one of: activating the model; deactivating the model; and initiating a second monitoring occasion.

2. The method of Claim 1, comprising: receiving, from the network node, an indication to initiate the first monitoring occasion, and initiating the first monitoring occasion based on the indication.

3. The method of any one of Claim 1 to 2, comprising: initiating the second monitoring occasion based on a timer, where the timer is part of the configuration information, and the timer duration is based on the validity of a model monitoring result associated with the first monitoring occasion.

4. The method of Claim 3, wherein a duration of the timer is determined based on the UE mobility, where the mobility can comprise of the UE speed, or the change in the UE radio environment.

5. The method of any one of Claims 1 to 4, wherein the configuration information comprises information associated with an event, and the method comprises: detecting an occurrence of the at least one event, and based on the occurrence of the at least one event, initiating the second monitoring occasion.

6. The method of Claim 5, wherein detection of the event is associated with a change in at least one radio measurement that is greater than or less than a threshold.

7. The method of any one of Claims 1 to 6, comprising receiving (110), from the network node, information for evaluating performance of the model during the first monitoring occasion.

8. The method of any one of Claims 1 to 7, wherein the model performance feedback transmitted to the network node comprises at least one success metric indicating a level of successof the performance of the model or at least one failure metric indicating a level of failure of the performance of the model.

9. The method of Claim 8, wherein the at least one success metric or failure metric is based on a difference between a predicted value and a ground-truth value.

10. The method of any one of Claims 1 to 9, comprising transmitting information to the network node for determining (130) a validity of the model monitoring result of the model and / or the first modeling occasion, wherein the information comprises at least one of: an indication of a model sensitivity; an indication of UE input change; a cost associated to the first monitoring occasion; at least one success metric; at least one failure metric; and historical information.

11. The method of any one of Claims 1 to 10, comprising: receiving, from the network node, a request for assistance information; transmitting, to the network node, the assistance information for determining by the network node the validity of the model monitoring result associated with the first monitoring occasion.

12. The method of any one of Claims 1 to 11 , wherein the configuration information configures the UE to deactivate the model when the validity of the model monitoring result associated with the first monitoring occasion is below a threshold amount.

13. The method of any one of Claims 1 to 12, wherein deactivating the model comprises performing a fallback to a legacy procedure.

14. The method of any one of Claims 1 to 12, wherein activating the model comprises receiving AI / ML inference results at least until the second monitoring occasion.

15. The method of any one of Claims 1 to 14, wherein a model monitoring period comprises a duration of time from the first monitoring occasion to the second monitoring occasion, and wherein the configuration information comprises: an indication of the model monitoring period, an indication of an increase in the model monitoring period, an indication of a decrease in the model monitoring period, or an indication that the model monitoring period can stay the same.

16. A method by a network node for monitoring of User Equipment-sided models using validity of monitoring results, the method comprising: receiving (120a), from a User Equipment, UE, model performance feedback associated with a performance of a model during a first monitoring occasion; based on the model performance feedback, determining (130) a validity of a model monitoring result associated with the first monitoring occasion; and based on the validity of the model monitoring result associated with the first monitoring occasion, transmitting model configuration information (140) for performing at least one of: activating the model; deactivating the model; and initiating a second monitoring occasion.

17. The method of Claim 16, comprising transmitting, to the UE, an indication to initiate the first monitoring occasion.

18. The method of any one of Claims 16 to 17, wherein the model configuration information comprises information associated with a timer for initiating the second monitoring occasion.

19. The method of Claim 18, wherein a duration of the timer is determined based on a speed of a UE.

20. The method of any one of Claims 16 to 19, wherein the configuration information comprises information associated with an event, and wherein the first monitoring occasion is initiated based on a detection of the event.

21. The method of Claim 20, wherein detection of the event is associated with a change in at least one radio measurement that is greater than or less than a second threshold.

22. The method of any one of Claims 16 to 21, comprising transmitting (110), to the UE, information for evaluating performance of the model during the first monitoring occasion.

23. The method of any one of Claims 16 to 22, comprising determining (120b), based on the model performance feedback, at least one success metric or failure metric, wherein the validity of the first monitoring occasion is determined based on the at least one success metric or failure metric.

24. The method of any one of Claims 16 to 22, wherein the model performance feedback received from the UE comprises a success metric indicating a level of success of the performance of the model or a failure metric indicating a level of failure of the performance of the model.

25. The method of any one of Claims 23 to 24, wherein the at least one success metric or failure metric is based on a difference between a predicted value and a ground-truth value.

26. The method of any one of Claims 16 to 25, wherein determining (130) the validity of the first monitoring occasion is based on at least one of: model sensitivity;UE input change; cost associated to the first monitoring occasion; at least one success metric; at least one failure metric; and historical information.

27. The method of any one of Claims 16 to 26, comprising: transmitting, to the UE, a request for assistance information associated; receiving, from the UE, the assistance information, and wherein the validity of the model monitoring result associated with the first model monitoring procedure is determined based on the assistance information from the UE.

28. The method of any one of Claims 16 to 27, comprising: based on the validity of the model monitoring result associated with the first monitoring occasion, wherein the configuration information indicates at least one event for performing the second modeling occasion.

29. The method of any one of Claims 16 to 28, wherein the configuration information configures the UE to: deactivate the model when the validity of the model monitoring result associated with the first monitoring occasion is below a threshold amount.

30. The method of any one of Claims 16 to 29, wherein: deactivating the model comprises performing a fallback to a legacy procedure, and / or activating the model comprises receiving AI / ML inference results at least until the second monitoring occasion.

31. The method of any one of Claims 16 to 30, wherein a model monitoring period comprises a duration of time from the first monitoring occasion to the second monitoring occasion, and wherein the configuration information comprises: an indication of the model monitoring period,an indication of an increase in the model monitoring period, an indication of a decrease in the model monitoring period, or an indication that the model monitoring period can stay the same.

32. A User Equipment, UE, for monitoring of UE-sided models using validity of monitoring results, the UE configured to: transmit (120a), to a network node, model performance feedback associated with a performance of a model during a first monitoring occasion; receive configuration information (140) associated with a validity of a model monitoring result associated with the first monitoring occasion, the configuration information for performing at least one of: activating the model; deactivating the model; and initiating a second monitoring occasion.

33. The UE configured to perform any of the methods of Claims 2 to 15.

34. A network node for monitoring of User Equipment-sided models using validity of monitoring results, the network node configured to: receive (120a), from a User Equipment, UE, model performance feedback associated with a performance of a model during a first monitoring occasion; based on the model performance feedback, determine (130) a validity of a model monitoring result associated with the first model monitoring occasion; and based on the validity of the model monitoring result associated with the first monitoring occasion, transmit model configuration information (140) for performing at least one of: activating the model; deactivating the model; and initiating a second monitoring occasion.

35. The network node configured to perform any of the methods of Claims 17 to 31.

Images