External model-monitoring of ai / ML model using line-of-sight link information

Abstract

Systems and methods are described in which a network node monitors the performance of an AI / ML model. Certain embodiments may comprise e.g., sending a request to a second wireless network node for a measurement report; receiving a measurement report from the second wireless network node, where the measurement report contains one or more time- stamped reference measurements; and performing model monitoring of the AI / ML model by checking the accuracy of the model output at time T using the reference measurement at time T, where time T is according to the time stamp. In such a method, in addition to the reference measurement, the first wireless network node may also receive reference assistance information on the reference measurement, for example, a LOS / NLOS indicator.

Description

EXTERNAL MODEL-MONITORING OF AI / ML MODEL USING LINE-OF-SIGHT LINK INFORMATIONTECHNICAL FIELD

[0001] The present disclosure generally relates to systems and methods for monitoring performance of AI / ML models in a wireless network node.BACKGROUND

[0002] Artificial Intelligence (Al) and Machine Learning (ML) have been investigated as promising tools to optimize the design of air-interface in wireless communication networks in both academia and industry. Example use cases include using autoencoders for channel state information (CSI) compression to reduce the feedback overhead and improve channel prediction accuracy; using deep neural networks for classifying line of sight (LOS) and nonline of sight (NLOS) conditions to enhance the positioning accuracy; and using reinforcement learning for beam selection at the network side and / or the user equipment (UE) side to reduce the signaling overhead and beam alignment latency; using deep reinforcement learning to leam an optimal precoding policy for complex multiple input multiple output (MIMO) precoding problems.

[0003] In 3rdgeneration partnership project (3GPP) new radio (NR) standardization work, the release 18 study item on AI / ML for NR air interface has been completed. This study item explores the benefits of augmenting the air-interface with features enabling improved support of AI / ML based algorithms for enhanced performance and / or reduced complexity / overhead. Through studying a few selected use cases (CSI feedback, beam management and positioning), this study item aims at laying the foundation for future air-interface use cases leveraging AI / ML techniques.

[0004] Building an AI / ML model includes several development steps where the actual training of the Al model is just one step in a training pipeline. An important part in AI / ML developing is the AI / ML model lifecycle management. This is illustrated in Figure 1. As shown in that figure, Al model lifecycle management typically consists of a training (re-training) pipeline, a deployment stage to make the trained (or re-trained) AI / ML model part of the inference pipeline, an inference pipeline, and a drift detection stage that reports any drifts in model operation. In this lifecycle management, the training (re-training) pipeline can include data ingestion (gathering raw (training) data from a data storage), a step that controls thevalidity of gathered data after data ingestion, data pre-processing (feature engineering applied to gathered data, e.g., data normalization and / or data transformation required to input data to the AI / ML model), model training (the steps where the model is obtained using the training dataset), model evaluation (benchmarking the performance to some baseline), potentially continuing the steps of model training and model evaluation iteratively until an acceptable level of performance is achieved, and model registration (registering the AI / ML model, including any corresponding AI / ML-meta data that provides information on how the AI / ML model was developed, and possibly AI / ML evaluation performance outcomes). Similarly, the inference pipeline can include data ingestion (gathering raw (inference) data from a data storage), data pre-processing (typically identical to corresponding pre-processing that occurs in the training pipeline), model operational (using the trained and deployed model in an operational mode), and data modelling and distribution (validating that the inference data are from a distribution that aligns well with the training data, as well as monitoring model outputs for detecting any performance, or operational, drifts).

[0005] One important AI / ML physical layer (PHY) use case is the positioning of a target user equipment (UE). Two approaches that have been shown to be effective in obtaining a target UE’s location are direct AI / ML positioning, and AI / ML assisted positioning. In direct AI / ML assisted positioning, the AI / ML model outputs the UE location. Direct AI / ML positioning typically refers to radio fingerprinting, where channel observation is used as the input of the AI / ML model. AI / ML assisted positioning is where the AI / ML model output is new measurement and / or enhancement of existing measurement. The model output can be, for example, LOS / NLOS identification, timing and / or angle measurement, and likelihood or reliability of the measurement. The model input in this case would also be channel observations.

[0006] There are various cases associated with applying direct and assisted AI / ML positioning to a NR wireless communication network. One case (case 1) is UE-based positioning with UE-side model, direct AI / ML or AI / ML assisted positioning. Another case (case 2a) is UE-assisted / LMF-based positioning with UE-side model, AI / ML assisted positioning. Another case (case 2b) is UE-assisted / location management function (LMF)-based positioning with LMF-side model, direct AI / ML positioning. Another case (case 3a) is next generation (NG) - radio access network (RAN) node assisted positioning with gNB (base station in NR)-side model, AI / ML assisted positioning. Another case (case 3b) is NG-RAN node assisted positioning with LMF-side model, direct AI / ML positioning.

[0007] For radio signal based positioning methods, conventional methods rely on a sufficient number of line-of-sight (LoS) links, typically at least three to five LOS links depending on the positioning method, and whether vertical position is estimated in addition to horizontal position.

[0008] In a cluttered environment, there is often a low probability of line-of-sight for a radio link between a UE and a transmission-reception point (TRP). For example, for InF-DH (Indoor Factory with Dense clutter and High base station height (transmitter or receiver elevated above the clutter)) environment, Table 1 shows the LoS probabilities of a radio link between TRP and UE when assuming different InF-DH clutter parameter settings. It is observed that the LoS probability ranges from 44.9% in a mildly cluttered environment to only 0.8% in a heavily cluttered environment.Table 1. LoS probabilities of a radio link between TRP and UE when assuming different InF- DH clutter parameter settings.

[0009] Thus, conventional positioning methods struggle to locate a target UE in a heavily cluttered environment. Evaluations show that the 90%-tile positioning accuracy of conventional positioning methods is more than 15 meters in an InF-DH {60%, 6m, 2m} environment, due to the unavailability of sufficient LoS links. This motivates the application of AI / ML based positioning in such challenging deployment environments.

[0010] Evaluations show that an AI / ML model can be trained to deliver 90%-tile positioning accuracy below 1 meter. While a well-functioning model can accurately determine the target UE's location in model inference, it has also been observed that the model performance can be sensitive to environment changes. Therefore model monitoring is important in the life-cycle management of AI / ML models for positioning.

[0011] 3GPP positioning solutions are articulated between three node types: a wireless device, or UE (user equipment), which is the device to be located; one or more gNB, or network nodes; and a LMF (location management function), which is a core network function responsible for collecting measurements and providing assistance information to other nodes.

[0012] A detailed account of the different methods supported between these nodes can be found in technical specification (TS) 38.305. Overall, all positioning methods involve the LMF sending assistance information to one node consisting of information on how to receive signals from the other node. For network based positioning, the LMF also collects measurement reports from the UE and / or the gNB. For example, for uplink positioning, the LMF collects the configuration for the sounding reference signal from the gNB which is connected to the UE (referred to as the UE’s serving cell). Then the LMF forwards that information to other gNB, to which the UE does not have a connection (referred to as non-serving cell).

[0013] In AI / ML, model monitoring is performed by a node hosting an AI / ML model by comparing the model output to some kind of reference. For positioning, such reference can be a measurement made on the same set of links where the AI / ML model is computing the positioning. For example, for a UE based model using a Al model based on receiving the DL (downlink) PRS (positioning reference signal) from a TRP (transmission reception point), the UE could receive reports from UL (uplink) measurement made by the gNB on the same link, and compare metrics that should be similar or the same, such as reference signal received power (RSRP), LOS state, etc.SUMMARY

[0014] One embodiment under the present disclosure comprises a method performed by a first wireless network node for performance monitoring of an AI / ML model. The method comprises: requesting, from a second wireless network node, a measurement report; receiving, from the second wireless network node, the measurement report, the measurement report containing one or more time-stamped reference measurements; and checking the accuracy of the AI / ML model by comparing an output of the AI / ML model at a time T with a reference measurement of the one or more time-stamped reference measurements at the time T.

[0015] Another embodiment under the present disclosure comprises a method performed by a second wireless network node for assisting performance monitoring of an AI / ML model at a first wireless network node. The method comprises: receiving, from a first wireless network node, a request for a measurement report; and transmitting, to the first wireless network node, the measurement report, the measurement report containing one or more time-stampedreference measurements, such that the first wireless network node can compare an output of the AI / ML model at a time T with a reference measurement of the one or more time-stamped reference measurements at the time T.

[0016] Another embodiment under the present disclosure comprises a first wireless network node for performance monitoring of an AI / ML model. The first wireless network node comprises: processing circuitry; and a memory. The memory contains instructions whereby the processing circuitry is operable to perform the steps of: requesting, from a second wireless network node, a measurement report; receiving, from the second wireless network node, the measurement report, the measurement report containing one or more time-stamped reference measurements; and checking the accuracy of the AI / ML model by comparing an output of the AI / ML model at a time T with a reference measurement of the one or more time-stamped reference measurements at the time T.

[0017] Another embodiment under the present disclosure comprises a second wireless network node for assisting performance monitoring of an AI / ML model at a first wireless network node. The network node comprises: processing circuitry; and a memory. The memory contains instructions whereby the processing circuitry is operable to perform the steps of: receiving, from a first wireless network node, a request for a measurement report; and transmitting, to the first wireless network node, the measurement report, the measurement report containing one or more time-stamped reference measurements, such that the first wireless network node can compare an output of the AI / ML model at a time T with a reference measurement of the one or more time-stamped reference measurements at the time T.

[0018] This summary is provided to introduce a selection of concepts in a simplified form that are further described below in the detailed description. This summary is not intended to identify key features or essential features of the claimed subject matter, nor is it intended to be used as an indication of the scope of the claimed subject matter.BRIEF DESCRIPTION OF THE DRAWINGS

[0019] For a more complete understanding of the present disclosure, reference is now made to the following descriptions taken in conjunction with the accompanying drawings, in which:

[0020] Fig. 1 illustrates a flow chart of a reinforcement learning embodiment under the present disclosure;

[0021] Fig. 2 illustrates a flow-chart of a method embodiment under the present disclosure;

[0022] Fig. 3 illustrates a flow-chart of a method embodiment under the present disclosure;

[0023] Fig. 4 illustrates a possible embodiment of a message from an LMF to a UE reporting a list of resources;

[0024] Fig. 5 illustrates a flow-chart of a method embodiment under the present disclosure;

[0025] Fig. 6 illustrates a flow-chart of a method embodiment under the present disclosure;

[0026] Fig. 7 shows a schematic of a communication system embodiment under the present disclosure;

[0027] Fig. 8 shows a schematic of a user equipment embodiment under the present disclosure;

[0028] Fig. 9 shows a schematic of a network node embodiment under the present disclosure; and

[0029] Fig. 10 shows a schematic of a virtualization environment embodiment under the present disclosure.DETAILED DESCRIPTION

[0030] Before describing various embodiments of the present disclosure in detail, it is to be understood that this disclosure is not limited to the parameters of the particularly exemplified systems, methods, apparatus, products, processes, and / or kits, which may, of course, vary. Thus, while certain embodiments of the present disclosure will be described in detail, with reference to specific configurations, parameters, components, elements, etc., the descriptions are illustrative and are not to be construed as limiting the scope of the claimed embodiments. In addition, the terminology used herein is for the purpose of describing the embodiments and is not necessarily intended to limit the scope of the claimed embodiments. Some of the embodiments contemplated herein will now be described more fully with reference to the accompanying drawings. Embodiments are provided by way of example to convey the scope of the subject matter to those skilled in the art.

[0031] As described above, there currently exist certain challenges. The current framework in positioning does not support monitoring. Additionally, if LOS state is included as part of measurement reporting between gNB / TRP, UE and LMF for comparing a measurement report to model output, it may result in cumbersome signaling and thus high overhead.

[0032] Certain aspects of the disclosure and their embodiments may provide solutions to these or other challenges. For example, aspects of this disclosure allow anode (gNB, UE, LMF, in the example of NR positioning) to receive a time-stamped measurement, or a list of time- stamped measurements, for the purpose of monitoring the model output. The time-stampedmeasurements provide information to check the accuracy of the model output during model deployment.

[0033] The disclosed technology may be used to provide a method which is performed by a first wireless network node to monitor the performance of an AI / ML model. Such a method may comprise: sending a request to a second wireless network node for measurement report; receiving a measurement report from the second wireless network node, where the measurement report contains one or more time-stamped reference measurements; and performing model monitoring of the AI / ML model by checking the accuracy of the model output at time T using the reference measurement at time T, where time T is according to the time stamp. In such a method, in addition to the reference measurement, the first wireless network node may also receive reference assistance information on the reference measurement, for example, a LOS / NLOS indicator. Further, in some embodiments, the first wireless network node is a UE, and the second wireless network node is an LMF. In other embodiments, the first wireless network node is an LMF, and the second wireless network node is a gNB or a UE.

[0034] In some aspects, implementing the disclosed technology can include setting up the condition over which a first wireless network node request a second wireless network node to send a time-stamped measurement report to support model monitoring at the first node; defining measurement report formats, using either a single measurement reported for a given timestamp, or a list of time-stamped measurements; and setting the duration of the procedure and reporting.

[0035] In cases where model monitoring is performed on the UE-side, the UE can receive reference measurements that can be used to monitor the AI / ML model, potentially in combination with reference assistance information related to one or more of the reference measurements. An example of a method which may be performed by a UE in this context is provided in Figure 2. Figure 2 shows a method 200 performed by a user equipment for monitoring performance of an AI / ML model. Step 210 is sending a request to a network node for a measurement report. Step 220 is receiving the measurement report (which may contain one or more time-stamped reference measurements) from the network node. Step 230 is performing model monitoring of the artificial intelligence model. As shown in Figure 2, this may be performed by step 240, which is checking the accuracy of a model output using a first reference measurement. The model output may be a model output at a first time, and the first reference measurement may be a reference measurement from the measurement report which is time-stamped with the first time.

[0036] In cases where model monitoring is performed on the network-side, a network node can receive reference measurements that can be used to monitor the AI / ML model, potentially in combination with reference assistance information related to one or more of the reference measurements. An example of a method which may be performed by a network node in this context is provided in Figure 3. Figure 3 shows a method 300 performed by a network node for monitoring performance of an AI / ML model. Step 310 is sending a request to a UE for a measurement report. Step 320 is receiving the measurement report (which may contain one or more time-stamped reference measurements) from the network node. Once steps 310 and 320 have been completed, the same model monitoring as described in the context of Figure 2 may be performed, though in this case it would be performed by the network node rather than the UE. More specifically, method 300 includes step 330 of performing model monitoring of the artificial intelligence and, as was the case when such monitoring was performed by the UE, the monitoring may be performed by step 340, which is checking the accuracy of a model output using a first reference measurement. In this context, as in the method 200 of Figure 2, the model output may be a model output at a first time, and the first reference measurement may be a reference measurement from the measurement report which is time-stamped with the first time.

[0037] Certain embodiments may provide one or more of the following technical advantages. Embodiments may allow an efficient mechanism to validate model output by comparison to a monitoring measurement. Certain embodiments may be implemented in a manner which has low signaling overhead.

[0038] Descriptions of some embodiments focus on a case of a UE operating an AI / ML model, with assistance data from the LMF and reference signal from a TRP / gNB. Such descriptions can be generalized for the case of a gNB / TRP operating an AI / ML model with assistance from the LMF and reference signals transmitted from the UE, or LMF-based model receiving measurements from any of the other two nodes (e.g., UE and gNB).

[0039] For the case of an AI / ML model deployed at the UE, the UE can operate an AI / ML model based on a reference signal measurement at the input, e.g. DL-PRS (downlink positioning reference signal). The model output can, for example, be a time measurement, or even a location estimate. Due to the UE mobility, model output will change over time. Therefore, the model output is typically time stamped. In the TRP / gNB side, the TRP / gNB collects measurement on the uplink, where the uplink is associated with the downlink measurement used by the UE. For example, for the purpose of monitoring, the UE may be configured to transmit a signal, for example the UL SRS for positioning, toward the gNB. In this context, it may be noted that the 3GPP positioning framework allows the UE transmissionto be “aligned” with a DL signal, such as single sideband modulation (SSB) or DL PRS. This may be used for a reliable reciprocity between UL and DL measurements.

[0040] In certain described embodiments, the UE can request the LMF to provide reference measurement at time t, which can be used to check the accuracy of model output at time instance t. In one example, the reference measurement is the expected model output Yref at time t, Yref(t). Thus, the accuracy of model output Y at time t, Y(t), can be checked by calculating the difference (or distance) between them: d(t)=|Y(t) -Yref(t)|. If the LMF sends a list of K reference values {Yref(to), Yref(ti), ... , Yref(tK-i} to the UE, then the UE can calculate the differences over time, {d(to), d(ti), ... , d(tic-i)}. By tracking the statistics of d(t) over a period of time, the UE can determine if its AI / ML model has drifted or not (i.e., performing model monitoring). In another example, the reference measurement is not the expected model output Yref, but can be used to derive Yref. Then the accuracy of model output Y at time t, Y(t), can be checked by calculating the difference (or distance) between them: d(t)=|Y(t) -Yref(t)|. For example, the model output Y(t) is a relative time-of-arrival (RTOA) of DL-PRS from TRP j to UE at time t, while the reference measurement is the estimated UE location at time t. Then RTOArefj at time t (i.e., Yrci(t)) can be derived from the UE location at time t. Thus, the accuracy of model output Y at time t, Y(t), can be checked by calculating the difference between them: d(t)=|Y(t) -Yref(t)|. By tracking the statistics of d(t) over a period of time, the UE can perform model monitoring. In another example, the reference measurement Zref is not the expected model output, but the model output Y can be used to derive measurement Z corresponding to Zref. Then the accuracy of model output Y at time t, Y(t), can be checked by calculating the difference (or distance) between Z(t) and Zrci(t): d(t)=|Z(t) -Zref(t)|. For example, the model output is a vector of RTOA of DL-PRS from N TRPs, Y=[RTOAo, RTOAi, . . . , RTOAN-I], while the reference measurement is the estimated UE location at time t, Zref. From the model output, the UE location Z can be derived. Then for time t, the difference between Z and Zref can be calculated as the distance between them, d(t)=||Z(t) -Zref(t)||. By tracking the statistics of d(t) over a period of time, the UE can perform model monitoring. In another example, the reference measurement Zref and the model output Y can be used to derive a third measurement Qref and the corresponding Q. Then the accuracy of model output Y at time t, Y(t), can be checked by calculating the difference (or distance) between Q(t) and Qrci(t): d(t)=|Q(t) -Qref(t)|. For example, the model output is a relative time-of-arrival (RTOA) of DL- PRS from TRP j to UE at time t, while the reference measurement Zref is the estimated UE location at time t. Then RTOAj at time t can be used to derive the distance Dj between the UE and the TRP j , while the reference measurement Zref can be used to derive the reference distanceDrefj between the UE and the TRP j. Thus, the accuracy of model output Y at time t, Y(t), can be checked by calculating the difference between them: d(t)=| Dj (t) - Drefj(t)| . By tracking the statistics of d(t) over a period of time, the UE can perform model monitoring.

[0041] For the positioning use case, it is often difficult to come up with a reliable reference measurement for checking model output accuracy, after the model is deployed. Thus it is often useful to receive reference assistance information on the reference measurement, for example, LOS / NLOS indicator. The LOS / NLOS indicator can be used by the UE to identity the reference measurement to use for model monitoring, for example, only using model output and reference measurement associated with a LOS link to calculate the statistics of the model output accuracy d(t). Specifically, LMF may provide information as to which of the DL PRS reference signals received are estimated to have been LOS with the UE, and at what time. This information request may be for previously received measurements, or for the upcoming measurement following the request. The reference assistance information (e.g., LOS / NLOS indicator) and the reference measurement may be provided in a same measurement report, in which case only one time stamp needs to be provided for the measurement report. Alternatively, The reference assistance information (e.g., LOS / NLOS indicator) and the reference measurement may be provided in two different reports, in which case the reference assistance information and the reference measurement may be individually time-stamped.

[0042] In some embodiments, a request may be to report when a signal is LOS, rather than a request to report line of sigh status of a signal at a given time. When this approach is used, a considerable amount of overhead can be saved, because the report does not include signals that are not line of sight. For example, if a DL PRS was received for 10 occasions, but only experienced line-of-sight transmission once, then the report may only include the timestamp for that occasion.

[0043] In some embodiments, the model inference node running the AI / ML model (a UE, LMF, TRP / gNB) sends a request to an assisting node for assistance to support model monitoring. In certain examples to demonstrate such assistance requests, it is assumed that the UE is the model inference node that runs the AI / ML model, and UE sends a request to the assisting node LMF for information to assist with UE's model monitoring. It is understood by those skilled in the art that a similar request can be made by the gNB (the model inference node) to the LMF (the assisting node), or the LMF (the model inference node) to assisting node (either the UE or gNB).

[0044] When making an assistance request to support model monitoring, the assistance request may contain which reference signal(s) is / are requested to be monitored. This can beexpressed either as a single resource or a list of resource, for example, information elements identifying which DL PRS resource ID, DL PRS resource set ID, and TRP ID should be monitored. In some cases, if only the TRP ID is mentioned, all resources under this TRP for which the UE has received assistance data are requested monitored. Alternatively, in some cases, if resource set ID and TRP ID are used, all resources under the resource set are requested to be monitored. As another alternative, in some cases, if no IDs are included it is up to the assisting node to decide which resources to include in the monitoring assistance information reports.

[0045] An assistance request made to support model monitoring may also include how the model inference node wants the assisting node to transmit the monitoring assistance information. This may be one or a time-span list of time stamps for which the UE requests reference measurement information (e.g., RTOA) and its reference assistance information (e.g., LOS / NLOS indicator). In this type of scenario, a list of time stamps may correspond to previous measurements / model inferences that the UE wants to compare the model output toward. An assistance request may also, or alternatively, be a request for periodic updates, where the periodicity is configured in the request and is interrupted by a “stop message” from the device running the model. As another potential approach, the request may optionally include a request for periodic updates, to be provided for a fixed amount of time provided in the request, after which the requested node will stop providing updates. Additionally, in some cases a request may optionally include a confidence threshold over which LOS / NLOS indicator can be considered reliable.

[0046] In some embodiments, for each occasion to report the monitoring assistance information to the device hosting the AI / ML model, the assisting node includes a list of LOS resources and associated (optional) available measurements according to the request, including time stamps.

[0047] Certain embodiments can include LPP (LTE Positioning Protocol). Figure 4 illustrates a possible embodiment of a message from an LMF to a UE reporting a list of resources. As shown in Figure 4, the LMF can report a list of resources that are LOS by including for a given DL PRS resource ID, a list of time stamps for which the UE is estimated to be LOS.

[0048] In some examples, the reference measurement (shown in Figure 4 as nr-reference- measurement-r!9) may have one of the following types, which are measured at a time-stamped time instance: Reference timing measurement, for example, DL-RTOA, UL-RTOA, reference signal time difference (RSTD), RxTxTimeDiff; Reference distance measurement, for example,distance between TRP j and the UE; Reference angle measurement, for example, angle of arrival (Ao A), angle of departure (AoD); and / or reference location of the target UE.

[0049] In some examples, reference assistance information (shown in Figure 4 as nr- reference-measurement-assistance-19) may be one of the following types, which correspond to the time-stamped time instance: LOS / NLOS indicator, which provides information on whether the link is line-of-sight or non-line-of-sight at the time-stamped instance (this indicator can take a hard value (e.g., 0 and 1; 'TRUE' and ‘FALSE') or a soft value (e.g., a value between 0 and 1 providing the probability that the link is line-of-sight)); LOS indicator, which identifies that a link is line-of-sight at the time-stamped instance (the value can be 'TRUE' or 'FALSE', with 'TRUE' indicating line-of-sight); confidence on the reference measurement value, which provides information on how reliable the corresponding reference measurement value is (for example, higher confidence / reliability can be indicated if the reference measurement value is obtained by using measurements of 8 TRPs, as compared to 2 TRPs); accuracy or error range of the reference measurement value (for example, an error range of Vdeita can be provided for the reference measurement value V, to indicate that the true reference measurement value is in the range of [V -Vdeita, V+V delta]); source of reference measurement, which provides information on the source that has given the reference measurement (for example, if the reference measurement is the reference location of the target UE, then the location source may be: A global navigation satellite system (GNSS), wireless local area network (WLAN), BT (Bluetooth), TBS (terrestrial beacon system), sensor, UL time difference of arrival (TDOA), DL TDOA, etc.); and / or validity time of the reference measurement value, which provides information on the duration that the reference measurement value can be considered valid, with reference to the time-stamped time instance.

[0050] A possible method embodiment under the present disclosure is shown in Figure 5. Method 900 comprises a method performed by a first wireless network node for performance monitoring of an AI / ML model. Step 910 is requesting, from a second wireless network node, a measurement report. Step 920 is receiving, from the second wireless network node, the measurement report, the measurement report containing one or more time-stamped reference measurements. Step 930 is checking the accuracy of the AI / ML model by comparing an output of the AI / ML model at a time T with a reference measurement of the one or more time-stamped reference measurements at the time T. Method 900 can comprise a variety of additional, alternative, and / or optional steps.

[0051] Another possible method embodiment under the present disclosure is shown in Figure 6. Method 1100 comprises a method performed by a second wireless network node forassisting performance monitoring of an AI / ML model at a first wireless network node. Step 1110 is receiving, from a first wireless network node, a request for a measurement report. Step 1120 is transmitting, to the first wireless network node, the measurement report, the measurement report containing one or more time-stamped reference measurements, such that the first wireless network node can compare an output of the AI / ML model at a time T with a reference measurement of the one or more time-stamped reference measurements at the time T. Method 1100 can comprise a variety of additional, alternative, and / or optional steps.

[0052] Figure 7 shows an example of a communication system 3100 in accordance with some embodiments. In the example, the communication system 3100 includes a telecommunication network 3102 that includes an access network 3104, such as a radio access network (RAN), and a core network 3106, which includes one or more core network nodes 3108. The access network 3104 includes one or more access network nodes, such as network nodes 3110a and 3110b (one or more of which may be generally referred to as network nodes 3110), or any other similar 3rd Generation Partnership Project (3GPP) access nodes or non- 3GPP access points. Moreover, as will be appreciated by those of skill in the art, a network node is not necessarily limited to an implementation in which a radio portion and a baseband portion are supplied and integrated by a single vendor. Thus, it will be understood that network nodes include disaggregated implementations or portions thereof. For example, in some embodiments, the telecommunication network 3102 includes one or more Open-RAN (ORAN) network nodes. An ORAN network node is a node in the telecommunication network 3102 that supports an ORAN specification (e.g., a specification published by the O-RAN Alliance, or any similar organization) and may operate alone or together with other nodes to implement one or more functionalities of any node in the telecommunication network 3102, including one or more network nodes 3110 and / or core network nodes 3108.

[0053] Examples of an ORAN network node include an open radio unit (O-RU), an open distributed unit (O-DU), an open central unit (O-CU), including an O-CU control plane (O- CU-CP) or an O-CU user plane (O-CU-UP), a RAN intelligent controller (near-real time or non-real time) hosting software or software plug-ins, such as a near-real time control application (e.g., xApp) or anon-real time control application (e.g., rApp), or any combination thereof (the adjective “open” designating support of an ORAN specification). The network node may support a specification by, for example, supporting an interface defined by the ORAN specification, such as an Al, Fl, Wl, El, E2, X2, Xn interface, an open fronthaul user plane interface, or an open fronthaul management plane interface. Moreover, an ORAN access node may be a logical node in a physical node. Furthermore, an ORAN network node may beimplemented in a virtualization environment (described further below) in which one or more network functions are virtualized. For example, the virtualization environment may include an O-Cloud computing platform orchestrated by a Service Management and Orchestration Framework via an 0-2 interface defined by the 0-RAN Alliance or comparable technologies. The network nodes 3110 facilitate direct or indirect connection of user equipment (UE), such as by connecting UEs 3112a, 3112b, 3112c, and 3112d (one or more of which may be generally referred to as UEs 3112) to the core network 3106 over one or more wireless connections.

[0054] Example wireless communications over a wireless connection include transmitting and / or receiving wireless signals using electromagnetic waves, radio waves, infrared waves, and / or other types of signals suitable for conveying information without the use of wires, cables, or other material conductors. Moreover, in different embodiments, the communication system 3100 may include any number of wired or wireless networks, network nodes, UEs, and / or any other components or systems that may facilitate or participate in the communication of data and / or signals whether via wired or wireless connections. The communication system 3100 may include and / or interface with any type of communication, telecommunication, data, cellular, radio network, and / or other similar type of system.

[0055] The UEs 3112 may be any of a wide variety of communication devices, including wireless devices arranged, configured, and / or operable to communicate wirelessly with the network nodes 3110 and other communication devices. Similarly, the network nodes 3110 are arranged, capable, configured, and / or operable to communicate directly or indirectly with the UEs 3112 and / or with other network nodes or equipment in the telecommunication network 3102 to enable and / or provide network access, such as wireless network access, and / or to perform other functions, such as administration in the telecommunication network 3102.

[0056] In the depicted example, the core network 3106 connects the network nodes 3110 to one or more host computing systems, such as host 3116. These connections may be direct or indirect via one or more intermediary networks or devices. In other examples, network nodes may be directly coupled to hosts. The core network 3106 includes one more core network nodes (e.g., core network node 3108) that are structured with hardware and software components. Features of these components may be substantially similar to those described with respect to the UEs, network nodes, and / or hosts, such that the descriptions thereof are generally applicable to the corresponding components of the core network node 3108. Example core network nodes include functions of one or more of a Mobile Switching Center (MSC), Mobility Management Entity (MME), Home Subscriber Server (HSS), Access and Mobility Management Function (AMF), Session Management Function (SMF), Authentication ServerFunction (AUSF), Subscription Identifier De-concealing function (SIDF), Unified Data Management (UDM), Security Edge Protection Proxy (SEPP), Network Exposure Function (NEF), and / or a User Plane Function (UPF).

[0057] The host 3116 may be under the ownership or control of a service provider other than an operator or provider of the access network 3104 and / or the telecommunication network 3102. The host 3116 may host a variety of applications to provide one or more service. Examples of such applications include live and pre-recorded audio / video content, data collection services such as retrieving and compiling data on various ambient conditions detected by a plurality of UEs, analytics functionality, social media, functions for controlling or otherwise interacting with remote devices, functions for an alarm and surveillance center, or any other such function performed by a server.

[0058] As a whole, the communication system 3100 of Figure 7 enables connectivity between the UEs, network nodes, and hosts. In that sense, the communication system may be configured to operate according to predefined rules or procedures, such as specific standards that include, but are not limited to: Global System for Mobile Communications (GSM); Universal Mobile Telecommunications System (UMTS); Long Term Evolution (LTE), and / or other suitable 2G, 3G, 4G, 5G standards, or any applicable future generation standard (e.g., 6G); wireless local area network (WLAN) standards, such as the Institute of Electrical and Electronics Engineers (IEEE) 802.11 standards (WiFi); and / or any other appropriate wireless communication standard, such as the Worldwide Interoperability for Micro wave Access (WiMax), Bluetooth, Z-Wave, Near Field Communication (NFC) ZigBee, LiFi, and / or any low-power wide-area network (LPWAN) standards such as LoRa and Sigfox.

[0059] In some examples, the telecommunication network 3102 is a cellular network that implements 3GPP standardized features. Accordingly, the telecommunications network 3102 may support network slicing to provide different logical networks to different devices that are connected to the telecommunication network 3102. For example, the telecommunications network 3102 may provide Ultra Reliable Low Latency Communication (URLLC) services to some UEs, while providing Enhanced Mobile Broadband (eMBB) services to other UEs, and / or Massive Machine Type Communication (mMTC) / Massive loT services to yet further UEs.

[0060] In some examples, the UEs 3112 are configured to transmit and / or receive information without direct human interaction. For instance, a UE may be designed to transmit information to the access network 3104 on a predetermined schedule, when triggered by an internal or external event, or in response to requests from the access network 3104. Additionally, a UE may be configured for operating in single- or multi-RAT or multi-standardmode. For example, a UE may operate with any one or combination of Wi-Fi, NR (New Radio) and LTE, i.e. being configured for multi -radio dual connectivity (MR-DC), such as E-UTRAN (Evolved-UMTS Terrestrial Radio Access Network) New Radio - Dual Connectivity (EN- DC).

[0061] In the example, the hub 3114 communicates with the access network 3104 to facilitate indirect communication between one or more UEs (e. g. , UE 3112c and / or 3112d) and network nodes (e.g., network node 3110b). In some examples, the hub 3114 may be a controller, router, content source and analytics, or any of the other communication devices described herein regarding UEs. For example, the hub 3114 may be a broadband router enabling access to the core network 3106 for the UEs. As another example, the hub 3114 may be a controller that sends commands or instructions to one or more actuators in the UEs. Commands or instructions may be received from the UEs, network nodes 3110, or by executable code, script, process, or other instructions in the hub 3114. As another example, the hub 3114 may be a data collector that acts as temporary storage for UE data and, in some embodiments, may perform analysis or other processing of the data. As another example, the hub 3114 may be a content source. For example, for a UE that is a VR device, display, loudspeaker, or other media delivery device, the hub 3114 may retrieve VR assets, video, audio, or other media or data related to sensory information via a network node, which the hub 3114 then provides to the UE either directly, after performing local processing, and / or after adding additional local content. In still another example, the hub 3114 acts as a proxy server or orchestrator for the UEs, in particular if one or more of the UEs are low energy loT devices.

[0062] The hub 3114 may have a constant / persistent or intermittent connection to the network node 3110b. The hub 3114 may also allow for a different communication scheme and / or schedule between the hub 3114 and UEs (e.g., UE 3112c and / or 3112d), and between the hub 3114 and the core network 3106. In other examples, the hub 3114 is connected to the core network 3106 and / or one or more UEs via a wired connection. Moreover, the hub 3114 may be configured to connect to an M2M service provider over the access network 3104 and / or to another UE over a direct connection. In some scenarios, UEs may establish a wireless connection with the network nodes 3110 while still connected via the hub 3114 via a wired or wireless connection. In some embodiments, the hub 3114 may be a dedicated hub - that is, a hub whose primary function is to route communications to / from the UEs from / to the network node 3110b. In other embodiments, the hub 3114 may be a non-dedicated hub - that is, a device which is capable of operating to route communications between the UEs and network node3110b, but which is additionally capable of operating as a communication start and / or end point for certain data channels.

[0063] Figure 8 shows a UE 3200 in accordance with some embodiments. The UE 3200 presents additional details of some embodiments of the UE 3112 of Figure 7. As used herein, a UE refers to a device capable, configured, arranged and / or operable to communicate wirelessly with network nodes and / or other UEs. Examples of a UE include, but are not limited to, a smart phone, mobile phone, cell phone, voice over IP (VoIP) phone, wireless local loop phone, desktop computer, personal digital assistant (PDA), wireless cameras, gaming console or device, music storage / playback device, wearable terminal device, wireless endpoint, mobile station, tablet, laptop, laptop-embedded equipment (LEE), laptop-mounted equipment (LME), an Augmented Reality (AR) or Virtual Reality (VR) device, wireless customer-premise equipment (CPE), vehicle, vehicle-mounted or vehicle embedded / integrated wireless device, etc. Other examples include any UE identified by the 3rd Generation Partnership Project (3 GPP), including a narrow band internet of things (NB-IoT) UE, a machine type communication (MTC) UE, and / or an enhanced MTC (eMTC) UE.

[0064] A UE may support device-to-device (D2D) communication, for example by implementing a 3GPP standard for sidelink communication, Dedicated Short-Range Communication (DSRC), vehicle-to-vehicle (V2V), vehicle-to-infrastructure (V2I), or vehicle- to-everything (V2X). In other examples, a UE may not necessarily have a user in the sense of a human user who owns and / or operates the relevant device. Instead, a UE may represent a device that is intended for sale to, or operation by, a human user but which may not, or which may not initially, be associated with a specific human user (e.g., a smart sprinkler controller). Alternatively, a UE may represent a device that is not intended for sale to, or operation by, an end user but which may be associated with or operated for the benefit of a user (e.g., a smart power meter).

[0065] The UE 3200 includes processing circuitry 3202 that is operatively coupled via a bus 3204 to an input / output interface 3206, a power source 3208, a memory 3210, a communication interface 3212, and / or any other component, or any combination thereof. Certain UEs may utilize all or a subset of the components shown in Figure 8. The level of integration between the components may vary from one UE to another UE. Further, certain UEs may contain multiple instances of a component, such as multiple processors, memories, transceivers, transmitters, receivers, etc.

[0066] The processing circuitry 3202 is configured to process instructions and data and may be configured to implement any sequential state machine operative to execute instructionsstored as machine-readable computer programs in the memory 3210. The processing circuitry 3202 may be implemented as one or more hardware-implemented state machines (e.g., in discrete logic, field-programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), etc.); programmable logic together with appropriate firmware; one or more stored computer programs, general-purpose processors, such as a microprocessor or digital signal processor (DSP), together with appropriate software; or any combination of the above. For example, the processing circuitry 3202 may include multiple central processing units (CPUs).

[0067] In the example, the input / output interface 3206 may be configured to provide an interface or interfaces to an input device, output device, or one or more input and / or output devices. Examples of an output device include a speaker, a sound card, a video card, a display, a monitor, a printer, an actuator, an emitter, a smartcard, another output device, or any combination thereof. An input device may allow a user to capture information into the UE 3200. Examples of an input device include a touch-sensitive or presence-sensitive display, a camera (e.g., a digital camera, a digital video camera, a web camera, etc.), a microphone, a sensor, a mouse, a trackball, a directional pad, a trackpad, a scroll wheel, a smartcard, and the like. The presence-sensitive display may include a capacitive or resistive touch sensor to sense input from a user. A sensor may be, for instance, an accelerometer, a gyroscope, a tilt sensor, a force sensor, a magnetometer, an optical sensor, a proximity sensor, a biometric sensor, etc., or any combination thereof. An output device may use the same type of interface port as an input device. For example, a Universal Serial Bus (USB) port may be used to provide an input device and an output device.

[0068] In some embodiments, the power source 3208 is structured as a battery or battery pack. Other types of power sources, such as an external power source (e.g., an electricity outlet), photovoltaic device, or power cell, may be used. The power source 3208 may further include power circuitry for delivering power from the power source 3208 itself, and / or an external power source, to the various parts of the UE 3200 via input circuitry or an interface such as an electrical power cable. Delivering power may be, for example, for charging of the power source 3208. Power circuitry may perform any formatting, converting, or other modification to the power from the power source 3208 to make the power suitable for the respective components of the UE 3200 to which power is supplied.

[0069] The memory 3210 may be or be configured to include memory such as random access memory (RAM), read-only memory (ROM), programmable read-only memory (PROM), erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), magnetic disks, optical disks, hard disks,removable cartridges, flash drives, and so forth. In one example, the memory 3210 includes one or more application programs 3214, such as an operating system, web browser application, a widget, gadget engine, or other application, and corresponding data 3216. The memory 3210 may store, for use by the UE 3200, any of a variety of various operating systems or combinations of operating systems.

[0070] The memory 3210 may be configured to include a number of physical drive units, such as redundant array of independent disks (RAID), flash memory, USB flash drive, external hard disk drive, thumb drive, pen drive, key drive, high-density digital versatile disc (HD- DVD) optical disc drive, internal hard disk drive, Blu-Ray optical disc drive, holographic digital data storage (HDDS) optical disc drive, external mini-dual in-line memory module (DIMM), synchronous dynamic random access memory (SDRAM), external micro-DIMM SDRAM, smartcard memory such as tamper resistant module in the form of a universal integrated circuit card (UICC) including one or more subscriber identity modules (SIMs), such as a USIM and / or ISIM, other memory, or any combination thereof. The UICC may for example be an embedded UICC (eUICC), integrated UICC (iUICC) or a removable UICC commonly known as ‘SIM card.’ The memory 3210 may allow the UE 3200 to access instructions, application programs and the like, stored on transitory or non-transitory memory media, to off-load data, or to upload data. An article of manufacture, such as one utilizing a communication system may be tangibly embodied as or in the memory 3210, which may be or comprise a device-readable storage medium.

[0071] The processing circuitry 3202 may be configured to communicate with an access network or other network using the communication interface 3212. The communication interface 3212 may comprise one or more communication subsystems and may include or be communicatively coupled to an antenna 3222. The communication interface 3212 may include one or more transceivers used to communicate, such as by communicating with one or more remote transceivers of another device capable of wireless communication (e.g., another UE or a network node in an access network). Each transceiver may include a transmitter 3218 and / or a receiver 3220 appropriate to provide network communications (e.g., optical, electrical, frequency allocations, and so forth). Moreover, the transmitter 3218 and receiver 3220 may be coupled to one or more antennas (e.g., antenna 3222) and may share circuit components, software or firmware, or alternatively be implemented separately.

[0072] In the illustrated embodiment, communication functions of the communication interface 3212 may include cellular communication, Wi-Fi communication, LPWAN communication, data communication, voice communication, multimedia communication,short-range communications such as Bluetooth, near-field communication, location-based communication such as the use of the global positioning system (GPS) to determine a location, another like communication function, or any combination thereof. Communications may be implemented in according to one or more communication protocols and / or standards, such as IEEE 802.11, Code Division Multiplexing Access (CDMA), Wideband Code Division Multiple Access (WCDMA), GSM, LTE, New Radio (NR), UMTS, WiMax, Ethernet, transmission control protocol / intemet protocol (TCP / IP), synchronous optical networking (SONET), Asynchronous Transfer Mode (ATM), QUIC, Hypertext Transfer Protocol (HTTP), and so forth.

[0073] Regardless of the type of sensor, a UE may provide an output of data captured by its sensors, through its communication interface 3212, via a wireless connection to a network node. Data captured by sensors of a UE can be communicated through a wireless connection to a network node via another UE. The output may be periodic (e.g., once every 15 minutes if it reports the sensed temperature), random (e.g., to even out the load from reporting from several sensors), in response to a triggering event (e.g., when moisture is detected an alert is sent), in response to a request (e.g., a user initiated request), or a continuous stream (e.g., a live video feed of a patient).

[0074] As another example, a UE comprises an actuator, a motor, or a switch, related to a communication interface configured to receive wireless input from a network node via a wireless connection. In response to the received wireless input the states of the actuator, the motor, or the switch may change. For example, the UE may comprise a motor that adjusts the control surfaces or rotors of a drone in flight according to the received input or to a robotic arm performing a medical procedure according to the received input.

[0075] A UE, when in the form of an Internet of Things (loT) device, may be a device for use in one or more application domains, these domains comprising, but not limited to, city wearable technology, extended industrial application and healthcare. Non-limiting examples of such an loT device are a device which is or which is embedded in: a connected refrigerator or freezer, a TV, a connected lighting device, an electricity meter, a robot vacuum cleaner, a voice controlled smart speaker, a home security camera, a motion detector, a thermostat, a smoke detector, a door / window sensor, a flood / moisture sensor, an electrical door lock, a connected doorbell, an air conditioning system like a heat pump, an autonomous vehicle, a surveillance system, a weather monitoring device, a vehicle parking monitoring device, an electric vehicle charging station, a smart watch, a fitness tracker, a wearable for tactile augmentation or sensory enhancement, a water sprinkler, an animal- or item-tracking device, asensor for monitoring a plant or animal, an industrial robot, an Unmanned Aerial Vehicle (UAV), and any kind of medical device, like a heart rate monitor or a remote controlled surgical robot. A UE in the form of an loT device comprises circuitry and / or software in dependence of the intended application of the loT device in addition to other components as described in relation to the UE 3200 shown in Figure 8.

[0076] As yet another specific example, in an loT scenario, a UE may represent a machine or other device that performs monitoring and / or measurements, and transmits the results of such monitoring and / or measurements to another UE and / or a network node. The UE may in this case be an M2M device, which may in a 3GPP context be referred to as an MTC device. As one particular example, the UE may implement the 3GPP NB-IoT standard. In other scenarios, a UE may represent a vehicle, such as a car, a bus, a truck, a ship and an airplane, or other equipment that is capable of monitoring and / or reporting on its operational status or other functions associated with its operation.

[0077] In practice, any number of UEs may be used together with respect to a single use case. For example, a first UE might be or be integrated in a drone and provide the drone’s speed information (obtained through a speed sensor) to a second UE that is a remote controller operating the drone. When the user makes changes from the remote controller, the first UE may adjust the throttle on the drone (e.g. by controlling an actuator) to increase or decrease the drone’s speed. The first and / or the second UE can also include more than one of the functionalities described above. For example, a UE might comprise the sensor and the actuator, and handle communication of data for both the speed sensor and the actuators.

[0078] Figure 9 shows a network node 3300 in accordance with some embodiments. As used herein, network node refers to equipment capable, configured, arranged and / or operable to communicate directly or indirectly with a UE and / or with other network nodes or equipment, in a telecommunication network. Examples of network nodes include, but are not limited to, access points (APs) (e.g., radio access points), base stations (BSs) (e.g., radio base stations, Node Bs, evolved Node Bs (eNBs) and NRNodeBs (gNBs)), O-RAN nodes or components of an O-RAN node (e.g., O-RU, O-DU, O-CU).

[0079] Base stations may be categorized based on the amount of coverage they provide (or, stated differently, their transmit power level) and so, depending on the provided amount of coverage, may be referred to as femto base stations, pico base stations, micro base stations, or macro base stations. A base station may be a relay node or a relay donor node controlling a relay. A network node may also include one or more (or all) parts of a distributed radio base station such as centralized digital units, distributed units (e.g., in an O-RAN access node)and / or remote radio units (RRUs), sometimes referred to as Remote Radio Heads (RRHs). Such remote radio units may or may not be integrated with an antenna as an antenna integrated radio. Parts of a distributed radio base station may also be referred to as nodes in a distributed antenna system (DAS).

[0080] Other examples of network nodes include multiple transmission point (multi-TRP) 5G access nodes, multi-standard radio (MSR) equipment such as MSR BSs, network controllers such as radio network controllers (RNCs) or base station controllers (BSCs), base transceiver stations (BTSs), transmission points, transmission nodes, multi-cell / multicast coordination entities (MCEs), Operation and Maintenance (O&M) nodes, Operations Support System (OSS) nodes, Self-Organizing Network (SON) nodes, positioning nodes (e.g., Evolved Serving Mobile Location Centers (E-SMLCs)), and / or Minimization of Drive Tests (MDTs).

[0081] The network node 3300 includes a processing circuitry 3302, a memory 3304, a communication interface 3306, and a power source 3308. The network node 3300 may be composed of multiple physically separate components (e.g., a NodeB component and a RNC component, or a BTS component and a BSC component, etc.), which may each have their own respective components. In certain scenarios in which the network node 3300 comprises multiple separate components (e.g., BTS and BSC components), one or more of the separate components may be shared among several network nodes. For example, a single RNC may control multiple NodeBs. In such a scenario, each unique NodeB and RNC pair, may in some instances be considered a single separate network node. In some embodiments, the network node 3300 may be configured to support multiple radio access technologies (RATs). In such embodiments, some components may be duplicated (e.g., separate memory 3304 for different RATs) and some components may be reused (e.g., a same antenna 3310 may be shared by different RATs). The network node 3300 may also include multiple sets of the various illustrated components for different wireless technologies integrated into network node 3300, for example GSM, WCDMA, LTE, NR, WiFi, Zigbee, Z-wave, LoRaWAN, Radio Frequency Identification (RFID) or Bluetooth wireless technologies. These wireless technologies may be integrated into the same or different chip or set of chips and other components within network node 3300.

[0082] The processing circuitry 3302 may comprise a combination of one or more of a microprocessor, controller, microcontroller, central processing unit, digital signal processor, application-specific integrated circuit, field programmable gate array, or any other suitable computing device, resource, or combination of hardware, software and / or encoded logicoperable to provide, either alone or in conjunction with other network node 3300 components, such as the memory 3304, to provide network node 3300 functionality.

[0083] In some embodiments, the processing circuitry 3302 includes a system on a chip (SOC). In some embodiments, the processing circuitry 3302 includes one or more of radio frequency (RF) transceiver circuitry 3312 and baseband processing circuitry 3314. In some embodiments, the radio frequency (RF) transceiver circuitry 3312 and the baseband processing circuitry 3314 may be on separate chips (or sets of chips), boards, or units, such as radio units and digital units. In alternative embodiments, part or all of RF transceiver circuitry 3312 and baseband processing circuitry 3314 may be on the same chip or set of chips, boards, or units.

[0084] The memory 3304 may comprise any form of volatile or non-volatile computer- readable memory including, without limitation, persistent storage, solid-state memory, remotely mounted memory, magnetic media, optical media, random access memory (RAM), read-only memory (ROM), mass storage media (for example, a hard disk), removable storage media (for example, a flash drive, a Compact Disk (CD) or a Digital Video Disk (DVD)), and / or any other volatile or non-volatile, non-transitory device-readable and / or computerexecutable memory devices that store information, data, and / or instructions that may be used by the processing circuitry 3302. The memory 3304 may store any suitable instructions, data, or information, including a computer program, software, an application including one or more of logic, rules, code, tables, and / or other instructions capable of being executed by the processing circuitry 3302 and utilized by the network node 3300. The memory 3304 may be used to store any calculations made by the processing circuitry 3302 and / or any data received via the communication interface 3306. In some embodiments, the processing circuitry 3302 and memory 3304 is integrated.

[0085] The communication interface 3306 is used in wired or wireless communication of signaling and / or data between a network node, access network, and / or UE. As illustrated, the communication interface 3306 comprises port(s) / terminal(s) 3316 to send and receive data, for example to and from a network over a wired connection. The communication interface 3306 also includes radio front-end circuitry 3318 that may be coupled to, or in certain embodiments a part of, the antenna 3310. Radio front-end circuitry 3318 comprises filters 3320 and amplifiers 3322. The radio front-end circuitry 3318 may be connected to an antenna 3310 and processing circuitry 3302. The radio front-end circuitry may be configured to condition signals communicated between antenna 3310 and processing circuitry 3302. The radio front-end circuitry 3318 may receive digital data that is to be sent out to other network nodes or UEs via a wireless connection. The radio front-end circuitry 3318 may convert the digital data into aradio signal having the appropriate channel and bandwidth parameters using a combination of filters 3320 and / or amplifiers 3322. The radio signal may then be transmitted via the antenna 3310. Similarly, when receiving data, the antenna 3310 may collect radio signals which are then converted into digital data by the radio front-end circuitry 3318. The digital data may be passed to the processing circuitry 3302. In other embodiments, the communication interface may comprise different components and / or different combinations of components.

[0086] In certain alternative embodiments, the network node 3300 does not include separate radio front-end circuitry 3318, instead, the processing circuitry 3302 includes radio front-end circuitry and is connected to the antenna 3310. Similarly, in some embodiments, all or some of the RF transceiver circuitry 3312 is part of the communication interface 3306. In still other embodiments, the communication interface 3306 includes one or more ports or terminals 3316, the radio front-end circuitry 3318, and the RF transceiver circuitry 3312, as part of a radio unit (not shown), and the communication interface 3306 communicates with the baseband processing circuitry 3314, which is part of a digital unit (not shown).

[0087] The antenna 3310 may include one or more antennas, or antenna arrays, configured to send and / or receive wireless signals. The antenna 3310 may be coupled to the radio frontend circuitry 3318 and may be any type of antenna capable of transmitting and receiving data and / or signals wirelessly. In certain embodiments, the antenna 3310 is separate from the network node 3300 and connectable to the network node 3300 through an interface or port.

[0088] The antenna 3310, communication interface 3306, and / or the processing circuitry 3302 may be configured to perform any receiving operations and / or certain obtaining operations described herein as being performed by the network node. Any information, data and / or signals may be received from a UE, another network node and / or any other network equipment. Similarly, the antenna 3310, the communication interface 3306, and / or the processing circuitry 3302 may be configured to perform any transmitting operations described herein as being performed by the network node. Any information, data and / or signals may be transmitted to a UE, another network node and / or any other network equipment.

[0089] The power source 3308 provides power to the various components of network node 3300 in a form suitable for the respective components (e.g., at a voltage and current level needed for each respective component). The power source 3308 may further comprise, or be coupled to, power management circuitry to supply the components of the network node 3300 with power for performing the functionality described herein. For example, the network node 3300 may be connectable to an external power source (e.g., the power grid, an electricity outlet) via an input circuitry or interface such as an electrical cable, whereby the external power sourcesupplies power to power circuitry of the power source 3308. As a further example, the power source 3308 may comprise a source of power in the form of a battery or battery pack which is connected to, or integrated in, power circuitry. The battery may provide backup power should the external power source fail.

[0090] Embodiments of the network node 3300 may include additional components beyond those shown in Figure 9 for providing certain aspects of the network node’s functionality, including any of the functionality described herein and / or any functionality necessary to support the subject matter described herein. For example, the network node 3300 may include user interface equipment to allow input of information into the network node 3300 and to allow output of information from the network node 3300. This may allow a user to perform diagnostic, maintenance, repair, and other administrative functions for the network node 3300. In some embodiments providing a core network node, such as core network node 2108 of Figure 7, some components, such as the radio front-end circuitry 3318 and the RF transceiver circuitry 3312 may be omitted.

[0091] Figure 10 is a block diagram illustrating a virtualization environment 3400 in which functions implemented by some embodiments may be virtualized. In the present context, virtualizing means creating virtual versions of apparatuses or devices which may include virtualizing hardware platforms, storage devices and networking resources. As used herein, virtualization can be applied to any device described herein, or components thereof, and relates to an implementation in which at least a portion of the functionality is implemented as one or more virtual components. Some or all of the functions described herein may be implemented as virtual components executed by one or more virtual machines (VMs) implemented in one or more virtual environments 3400 hosted by one or more of hardware nodes, such as a hardware computing device that operates as a network node, UE, core network node, or host. Further, in embodiments in which the virtual node does not require radio connectivity (e.g., a core network node or host), then the node may be entirely virtualized. In some embodiments, the virtualization environment 3400 includes components defined by the O-RAN Alliance, such as an O-Cloud environment orchestrated by a Service Management and Orchestration Framework via an O-2 interface. Virtualization may facilitate distributed implementations of a network node, UE, core network node, or host.

[0092] Applications 3402 (which may alternatively be called software instances, virtual appliances, network functions, virtual nodes, virtual network functions, etc.) are run in the virtualization environment 3400 to implement some of the features, functions, and / or benefits of some of the embodiments disclosed herein.

[0093] Hardware 3404 includes processing circuitry, memory that stores software and / or instructions executable by hardware processing circuitry, and / or other hardware devices as described herein, such as a network interface, input / output interface, and so forth. Software may be executed by the processing circuitry to instantiate one or more virtualization layers 3406 (also referred to as hypervisors or virtual machine monitors (VMMs)), provide VMs 3408a and 3408b (one or more of which may be generally referred to as VMs 3408), and / or perform any of the functions, features and / or benefits described in relation with some embodiments described herein. The virtualization layer 3406 may present a virtual operating platform that appears like networking hardware to the VMs 3408.

[0094] The VMs 3408 comprise virtual processing, virtual memory, virtual networking or interface and virtual storage, and may be run by a corresponding virtualization layer 3406. Different embodiments of the instance of a virtual appliance 3402 may be implemented on one or more of VMs 3408, and the implementations may be made in different ways. Virtualization of the hardware is in some contexts referred to as network function virtualization (NFV). NFV may be used to consolidate many network equipment types onto industry standard high volume server hardware, physical switches, and physical storage, which can be located in data centers, and customer premise equipment.

[0095] In the context of NFV, a VM 3408 may be a software implementation of a physical machine that runs programs as if they were executing on a physical, non-virtualized machine. Each of the VMs 3408, and that part of hardware 3404 that executes that VM, be it hardware dedicated to that VM and / or hardware shared by that VM with others of the VMs, forms separate virtual network elements. Still in the context of NFV, a virtual network function is responsible for handling specific network functions that run in one or more VMs 3408 on top of the hardware 3404 and corresponds to the application 3402.

[0096] Hardware 3404 may be implemented in a standalone network node with generic or specific components. Hardware 3404 may implement some functions via virtualization. Alternatively, hardware 3404 may be part of a larger cluster of hardware (e.g. such as in a data center or CPE) where many hardware nodes work together and are managed via management and orchestration 3410, which, among others, oversees lifecycle management of applications 3402. In some embodiments, hardware 3404 is coupled to one or more radio units that each include one or more transmitters and one or more receivers that may be coupled to one or more antennas. Radio units may communicate directly with other hardware nodes via one or more appropriate network interfaces and may be used in combination with the virtual components to provide a virtual node with radio capabilities, such as a radio access node or a base station. Insome embodiments, some signaling can be provided with the use of a control system 3412 which may alternatively be used for communication between hardware nodes and radio units.

[0097] Although the computing devices described herein (e.g., UEs, network nodes) may include the illustrated combination of hardware components, other embodiments may comprise computing devices with different combinations of components. It is to be understood that these computing devices may comprise any suitable combination of hardware and / or software needed to perform the tasks, features, functions and methods disclosed herein. Determining, calculating, obtaining or similar operations described herein may be performed by processing circuitry, which may process information by, for example, converting the obtained information into other information, comparing the obtained information or converted information to information stored in the network node, and / or performing one or more operations based on the obtained information or converted information, and as a result of said processing making a determination. Moreover, while components are depicted as single boxes located within a larger box, or nested within multiple boxes, in practice, computing devices may comprise multiple different physical components that make up a single illustrated component, and functionality may be partitioned between separate components. For example, a communication interface may be configured to include any of the components described herein, and / or the functionality of the components may be partitioned between the processing circuitry and the communication interface. In another example, non-computationally intensive functions of any of such components may be implemented in software or firmware and computationally intensive functions may be implemented in hardware.

[0098] In certain embodiments, some or all of the functionality described herein may be provided by processing circuitry executing instructions stored on in memory, which in certain embodiments may be a computer program product in the form of a non-transitory computer- readable storage medium. In alternative embodiments, some or all of the functionality may be provided by the processing circuitry without executing instructions stored on a separate or discrete device-readable storage medium, such as in a hard-wired manner. In any of those particular embodiments, whether executing instructions stored on a non-transitory computer- readable storage medium or not, the processing circuitry can be configured to perform the described functionality. The benefits provided by such functionality are not limited to the processing circuitry alone or to other components of the computing device, but are enjoyed by the computing device as a whole, and / or by end users and a wireless network generally.

[0099] Some descriptions of certain embodiments use the terms UE or a wireless device interchangeably. The UE herein can be any type of wireless device capable of communicatingwith a network node or another UE over radio signals. The UE may also be a radio communication device, target device, device to device (D2D) UE, machine type UE or UE capable of machine to machine communication (M2M), low-cost and / or low-complexity UE, a sensor equipped with UE, tablet, mobile terminals, smart phone, laptop embedded equipment (LEE), laptop mounted equipment (LME), universal serial bus (USB) dongles, Customer Premises Equipment (CPE), an Internet of Things (loT) device, or a Narrowband loT (NB- IOT) device, etc.

[0100] Some descriptions of certain embodiments use the generic term “anchor node” is used. Anchor nodes are used as reference points for determining the location of a target UE. In general, the anchor nodes for positioning can be a variety of nodes in the wireless network. For positioning using the radio link between the target UE and a radio network node, the anchor node can be any kind of a radio network node which may comprise any of base station, radio base station, base transceiver station, Node B, evolved Node B (eNB), Next-Generation Node B (gNodeB or gNB), NG-RAN node, Transmission Point (TP), Transmission-Reception Point (TRP), Multi-cell / multicast Coordination Entity (MCE), relay node, access point (AP), Antenna Reference Point (ARP), radio access point, Remote Radio Unit (RRU), Remote Radio Head (RRH). For positioning using sidelink between two UEs, the anchor node is a UE or a wireless device.

[0101] For ease of discussion, the methods are described using the radio links between a UE and an anchor node. TRP is used as a representative example of the anchor node, where the TRP is connected to a gNB. It is understood by those skilled in the art that the same methodology can be easily applied to many other wireless communication scenarios, e.g., sidelink-based positioning.

[0102] When describing certain embodiments, the term measurements refers to quantities reported from one node to the other based on some measured resource, and thus can include time / frequency / power measurements, and can also include derived link information (for example, LOS / NLOS indicator) based on the time / frequency / power measurements.

Claims

CLAIMSWhat is claimed is:

1. A method (900) performed by a first wireless network node (3300) for performance monitoring of an artificial intelligence / machine learning, AI / ML, model, the method comprising: requesting (910), from a second wireless network node (3300), a measurement report; receiving (920), from the second wireless network node, the measurement report, the measurement report containing one or more time-stamped reference measurements; and checking the accuracy (930) of the AI / ML model by comparing an output of the AI / ML model at a time T with a reference measurement of the one or more time-stamped reference measurements at the time T.

2. The method of claim 1, wherein the measurement report includes line of sight information.

3. The method of claim 1 or 2, wherein requesting a measurement report includes requesting at least one of: line of sight information; a signal for which line of sight information is provided; a Positioning Reference Signal, PRS, Identification number.

4. The method of any of claims 1 to 3, wherein the second wireless network node comprises at least one of: a Location Management Function, LMF ; base station, BS; gNodeB, gNB; a user equipment, UE.

5. The method of any of claims 1 to 4, wherein the first wireless network node comprises at least one of: a Location Management Function, LMF ; base station, BS; gNodeB, gNB; a user equipment, UE.

6. The method of any of claims 1 to 5, wherein the second wireless network node is a Location Management Function, LMF, and the first wireless network node is at least one of: a user equipment, UE; a base station, BS.

7. The method of any of claims 1 to 5, wherein the first wireless network node is a Location Management Function, LMF, and the second wireless network node is at least one of: a user equipment, UE; a base station, BS.

8. The method of any of claims 1 to 7, wherein the one or more time-stamped reference measurements comprise at least one of: a downlink-relative time of arrival, DL-RTOA; an uplink-relative time of arrival, UL-RTOA; a reference signal timing difference, RSTD; reception transmission time difference, RxTxTimeDiff; a reference distance measurement; a distance between a transmission reception point, TRP, and a user equipment, UE; a line of sight / non-line of sight, LOS / NLOS, indicator; a line of sight / non-line of sight, LOS / NLOS, indicator with a hard value; a line of sight / non-line of sight, LOS / NLOS, indicator with a soft value; a source or location of the one or more time-stamped reference measurements; a validity time of the one or more time-stamped reference measurements.

9. The method of any of claims 1 to 8, wherein the requesting the measurement report comprises at least one of: one or more time stamps for which the measurement report is requested; a request for periodic updates to be provided until interrupted by a stop message; a request for periodic updates to be provided for a fixed amount of time; a confidence threshold over which the measurement report can be considered reliable.

10. A method (1100) performed by a second wireless network node (3300) for assisting performance monitoring of an artificial intelligence / machine learning, AI / ML, model at a first wireless network node (3300), the method comprising: receiving (1110), from a first wireless network node, a request for a measurement report; and transmitting (1120), to the first wireless network node, the measurement report, the measurement report containing one or more time-stamped reference measurements, such that the first wireless network node can compare an output of the AI / ML model at a time T with a reference measurement of the one or more time-stamped reference measurements at the time T.

11. The method of claim 10, wherein the measurement report includes line of sight information.

12. The method of claim 10 or 11, wherein the request for the measurement report includes requesting at least one of: line of sight information; a signal for which line of sight informationis provided; a Positioning Reference Signal, PRS, Identification number.

13. The method of any of claims 10 to 12, wherein the second wireless network node comprises at least one of: a Location Management Function, LMF; base station, BS; gNodeB, gNB; a user equipment, UE.

14. The method of any of claims 10 to 13, wherein the first wireless network node comprises at least one of: a Location Management Function, LMF ; base station, BS; gNodeB, gNB; a user equipment, UE.

15. The method of any of claims 10 to 14, wherein the second wireless network node is a Location Management Function, LMF, and the first wireless network node is at least one of: a user equipment, UE; a base station, BS.

16. The method of any of claims 10 to 15, wherein the first wireless network node is a Location Management Function, LMF, and the second wireless network node is at least one of: a user equipment, UE; a base station, BS.

17. The method of any of claims 10 to 16, wherein the one or more time-stamped reference measurements comprise at least one of: a downlink-relative time of arrival, DL-RTOA; an uplink-relative time of arrival, UL-RTOA; a reference signal timing difference, RSTD; reception transmission time difference, RxTxTimeDiff; a reference distance measurement; a distance between a transmission reception point, TRP, and a user equipment, UE; a line of sight / non-line of sight, LOS / NLOS, indicator; a line of sight / non-line of sight, LOS / NLOS, indicator with a hard value; a line of sight / non-line of sight, LOS / NLOS, indicator with a soft value; a source or location of the one or more time-stamped reference measurements; a validity time of the one or more time-stamped reference measurements.

18. The method of any of claims 10 to 17, wherein the request for the measurement report comprises at least one of: one or more time stamps for which the measurement report is requested; a request for periodic updates to be provided until interrupted by a stop message; a request for periodic updates to be provided for a fixed amount of time; a confidence threshold over which the measurement report can be considered reliable.

19. A first wireless network node (3300) for performance monitoring of an artificial intelligence / machine learning, AI / ML, model, the first wireless network node comprising: processing circuitry (3302) configured to perform any of the steps of any of claims 1 to 9; power supply circuitry (3308) configured to supply power to the processing circuitry.

20. A second wireless network node (3300) for assisting performance monitoring of an artificial intelligence / machine learning, AI / ML, model at a first wireless network node (3300), the network node comprising: processing circuitry (3302) configured to perform any of the steps of any of claims 9 to 18; power supply circuitry (3308) configured to supply power to the processing circuitry.

21. A first wireless network node (3300) for performance monitoring of an artificial intelligence / machine learning, AI / ML, model, the first wireless network node comprising: processing circuitry (3302); and a memory (3304) containing instructions whereby the processing circuitry is operable to perform the steps of: requesting (910), from a second wireless network node (3300), a measurement report; receiving (920), from the second wireless network node, the measurement report, the measurement report containing one or more time-stamped reference measurements; and checking the accuracy of the AI / ML model by comparing (930) an output of the AI / ML model at a time T with a reference measurement of the one or more time-stamped reference measurements at the time T.

22. A second wireless network node (3300) for assisting performance monitoring of an artificial intelligence / machine learning, AI / ML, model at a first wireless network node (3300), the network node comprising: processing circuitry (3302); and a memory (3304) containing instructions whereby the processing circuitry is operable to perform the steps of:receiving (1110), from a first wireless network node, a request for a measurement report; and transmitting (1120), to the first wireless network node, the measurement report, the measurement report containing one or more time-stamped reference measurements, such that the first wireless network node can compare an output of the AI / ML model at a time T with a reference measurement of the one or more time-stamped reference measurements at the time T.

Images