Methods and apparatuses for ai model performance assessment

Abstract

A method of operating a first apparatus, the first apparatus being either a node of a core network or a server. The method comprises determining that the first apparatus is to perform a performance assessment of an artificial intelligence, AI, model used by a second apparatus to perform a task, wherein the second apparatus is either a communications device or an infrastructure equipment of a radio access network, transmitting, to the second apparatus, a request to perform measurements and to report the performed measurements to the first apparatus, receiving, from the second apparatus, an indication of the performed measurements, performing a performance metric calculation by comparing an output of the AI model when the performed measurements are used as an input to the AI model with reference information, and transmitting, to the second apparatus based on the calculated performance metric, an indication of a validity status of the AI model.

Description

[0001] METHODS, FIRST APPARATUS, AND SECOND APPARATUS

[0002] BACKGROUND

[0003] Field of Disclosure

[0004] The present disclosure relates to wireless communications, and particularly to first and second apparatus and methods of operating such first and second apparatus for the purposes of Al model monitoring.

[0005] The present applications claims the Paris Convention priority from United Kingdom patent application number GB2412107.1, filed on 16 August 2024, the contents of which are hereby incorporated by reference.

[0006] Description of Related Art

[0007] The “background” description provided herein is for the purpose of generally presenting the context of the disclosure. Work of the presently named inventors, to the extent it is described in this background section, as well as aspects of the description which may not otherwise qualify as prior art at the time of filing, are neither expressly or impliedly admitted as prior art against the present invention.

[0008] Previous generation mobile telecommunication systems, such as those based on the 3GPP defined UMTS and Long Term Evolution (LTE) architecture, are able to support a wider range of services than simple voice and messaging services offered by previous generations of mobile telecommunication systems. For example, with the improved radio interface and enhanced data rates provided by LTE systems, a user is able to enjoy high data rate applications such as mobile video streaming and mobile video conferencing that would previously only have been available via a fixed line data connection. The demand to deploy such networks is therefore strong and the coverage area of these networks, i.e. geographic locations where access to the networks is possible, is expected to continue to increase rapidly.

[0009] Current and future wireless communications networks are expected to routinely and efficiently support communications with an ever-increasing range of devices associated with a wider range of data traffic profiles and types than existing systems are optimised to support. For example, it is expected future wireless communications networks will be expected to efficiently support communications with devices including reduced complexity devices, machine type communication (MTC) devices, high resolution video displays, virtual reality headsets, Extended Reality (XR) and so on. Some of these different types of devices may be deployed in very large numbers, for example low complexity devices for supporting the “The Internet of Things”, and may typically be associated with the transmissions of relatively small amounts of data with relatively high latency tolerance. Other types of device, for example supporting high-definition video streaming, may be associated with transmissions of relatively large amounts of data with relatively low latency tolerance. Other types of device, for example used for autonomous vehicle communications and for other critical applications, may be characterised by data that should be transmitted through the network with low latency and high reliability. A single device type might also be associated with different traffic profiles / characteristics depending on the application(s) it is running. For example, different considerations may apply for efficiently supporting data exchange with a smartphone when it is running a video streaming application (high downlink data) as compared to when it is running an Internet browsing application (sporadic uplink and downlink data) or being used for voice communications by an emergency responder in an emergency scenario (data subject to stringent reliability and latency requirements).

[0010] In view of this there is expected to be a desire for current wireless communications networks, for example those which may be referred to as 5G or new radio (NR) systems / new radio access technology (RAT) systems, or indeed future 6G wireless communications, as well as future iterations / releases of existing systems, to efficiently support connectivity for a wide range of devices associated with different applications and different characteristic data traffic profiles and requirements.

[0011] One example of a new service is referred to as Ultra Reliable Low Latency Communications (URLLC) services which, as its name suggests, requires that a data unit or packet be communicated with a high reliability and with a low communications delay. Another example of a new service is extended Reality (XR), which may be provided by various user equipment such as wearable devices. XR combines real- world and virtual environments, incorporating aspects such as augmented reality (AR), mixed reality (MR), and virtual reality (VR), and thus requires high quality and minimised interaction delay. Services such as URLLC and XR therefore represent a challenging example for both LTE type communications systems and 5G / NR communications systems, as well as future generation communications systems.

[0012] 5G NR has continuously evolved and the current work plan includes 5G-NR-advanced in which some further enhancements are expected, especially to support new use-cases / scenarios with higher requirements. One of these aspects includes the use of artificial intelligence (Al) or machine learning (ML) to model parameters which can be used to perform certain tasks. The desire to support these new use-cases and scenarios gives rise to new challenges for efficiently handling communications in wireless communications systems that need to be addressed.

[0013] SUMMARY OF THE DISCLOSURE

[0014] The present disclosure can help address or mitigate at least some of the issues discussed above.

[0015] Embodiments of the present technique can provide a method of operating a first apparatus, the first apparatus being either a node of a core network or a server. The method comprises determining that the first apparatus is to perform a performance assessment of an artificial intelligence, Al, model used by a second apparatus to perform a task, wherein the second apparatus is either a communications device or an infrastructure equipment of a radio access network, transmitting, to the second apparatus, a request to perform measurements and to report the performed measurements to the first apparatus, receiving, from the second apparatus, an indication of the performed measurements, performing a performance metric calculation by comparing an output of the Al model when the performed measurements are used as an input to the Al model with reference information, and transmitting, to the second apparatus based on the calculated performance metric, an indication of a validity status of the Al model.

[0016] Such embodiments of the present technique, which, in addition to methods of operating a first apparatus (which may be a core network apparatus such as a location management function (LMF)), relate to methods of operating a second apparatus (e.g. a communications device or an infrastructure equipment) of wireless communications networks, to such first and second apparatus, to wireless communications systems, to computer programs, and to computer-readable storage mediums, can allow for the more efficient and effective operation of communications devices and infrastructure equipment at which AI / ML models are utilised to perform tasks.

[0017] Respective aspects and features of the present disclosure are defined in the appended claims.

[0018] It is to be understood that both the foregoing general description and the following detailed description are exemplary, but are not restrictive, of the present technology. The described embodiments, together with further advantages, will be best understood by reference to the following detailed description taken in conjunction with the accompanying drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0019] A more complete appreciation of the disclosure and many of the attendant advantages thereof will be readily obtained as the same becomes better understood by reference to the following detailed description when considered in connection with the accompanying drawings wherein like reference numerals designate identical or corresponding parts throughout the several views, and wherein:

[0020] Figure 1 schematically represents some aspects of an LTE-type wireless telecommunication system which may be configured to operate in accordance with certain embodiments of the present disclosure;

[0021] Figure 2 schematically represents some aspects of a new radio access technology (RAT) wireless telecommunications system which may be configured to operate in accordance with certain embodiments of the present disclosure;

[0022] Figure 3 is a schematic block diagram of an example infrastructure equipment and communications device which may be configured to operate in accordance with certain embodiments of the present disclosure;

[0023] Figure 4 schematically illustrates a life cycle management (LCM) architecture for an artificial intelligence (Al) model;

[0024] Figure 5 schematically illustrates a deployment of Al / machine learning (ML) positioning;

[0025] Figure 6 illustrates an example of base station-assisted UL based positioning;

[0026] Figure 7 shows a part schematic, part message flow diagram representation of an example wireless communications system comprising a first apparatus and a second apparatus in accordance with embodiments of the present technique;

[0027] Figure 8 shows a flow chart illustrating an example operation of the first apparatus in accordance with embodiments of the present technique;

[0028] Figure 9 an example of how a method of monitoring an AI / ML model can be performed in the network when the AI / ML model is deployed at a base station in accordance with arrangements of embodiments of the present technique;

[0029] Figure 10 illustrates an example of model selection and replacement in accordance with arrangements of embodiments of the present technique;

[0030] Figure 11 shows a part schematic, part message flow diagram representation of a first example process of communications in a communications system in accordance with embodiments of the present technique; Figure 12 shows a part schematic, part message flow diagram representation of a second example process of communications in a communications system in accordance with embodiments of the present technique; and

[0031] Figure 13 shows a flow diagram illustrating a third example process of communications in a communications system in accordance with embodiments of the present technique.

[0032] DETAILED DESCRIPTION OF THE EMBODIMENTS

[0033] Long Term Evolution Advanced Radio Access Technology (4G)

[0034] Figure 1 provides a schematic diagram illustrating some basic functionality of a mobile telecommunications network / system 6 operating generally in accordance with LTE principles, but which may also support other radio access technologies, and which may be adapted to implement embodiments of the disclosure as described herein. Various elements of Figure 1 and certain aspects of their respective modes of operation are well-known and defined in the relevant standards administered by the 3GPP (RTM) body, and also described in many books on the subject, for example, Holma H. and Toskala A [1], It will be appreciated that operational aspects of the telecommunications networks discussed herein which are not specifically described (for example in relation to specific communication protocols and physical channels for communicating between different elements) may be implemented in accordance with any known techniques, for example according to the relevant standards and known proposed modifications and additions to the relevant standards. The network 6 includes a plurality of base stations 1 connected to a core network 2. Each base station provides a coverage area 3 (i.e. a cell) within which data can be communicated to and from communications devices 4. Although each base station 1 is shown in Figure 1 as a single entity, the skilled person will appreciate that some of the functions of the base station may be carried out by disparate, inter-connected elements, such as antennas (or antennae), remote radio heads, amplifiers, etc. Collectively, one or more base stations may form a radio access network.

[0035] Data is transmitted from base stations 1 to communications devices 4 within their respective coverage areas 3 via a radio downlink (DL). Data is transmitted from communications devices 4 to the base stations 1 via a radio uplink (UL). The core network 2 routes data to and from the communications devices 4 via the respective base stations 1 and provides functions such as authentication, mobility management, charging and so on. Communications devices may also be referred to as mobile stations, user equipment (UEs), user terminals, mobile radios, mobile terminals, terminal devices, wireless transmit and receive units (WTRUs), and so forth. Services provided by the core network 2 may include connectivity to the internet or to external telephony services. The core network 2 may further track the location of the communications devices 4 so that it can efficiently contact (i.e. page) the communications devices 4 for transmitting downlink data towards the communications devices 4.

[0036] Base stations, which are an example of network infrastructure equipment, may also be referred to as transceiver stations, nodeBs, e-nodeBs, eNB, g-nodeBs, gNB and so forth. In this regard different terminology is often associated with different generations of wireless telecommunications systems for elements providing broadly comparable functionality. However, certain embodiments of the disclosure may be equally implemented in different generations of wireless telecommunications systems, and for simplicity certain terminology may be used regardless of the underlying network architecture. That is to say, the use of a specific term in relation to certain example implementations is not intended to indicate these implementations are limited to a certain generation of network that may be most associated with that particular terminology.

[0037] New Radio Access Technology (5G)

[0038] Systems incorporating NR technology are expected to support different services (or types of services), which may be characterised by different requirements for latency, data rate and / or reliability. For example, Enhanced Mobile Broadband (eMBB) services are characterised by high capacity with a requirement to support up to 20 Gb / s. The requirements for Ultra Reliable and Low Latency Communications (URLLC) services are for one transmission of a 32 byte packet to be transmitted from the radio protocol layer 2 / 3 SDU ingress point to the radio protocol layer 2 / 3 SDU egress point of the radio interface within 1 ms with a reliability of 1 - 10"5(99.999 %) or higher (99.9999%) [2],

[0039] Massive Machine Type Communications (mMTC) is another example of a service which may be supported by NR-based communications networks. In addition, systems may be expected to support further enhancements related to Industrial Internet of Things (IIoT) in order to support services with new requirements of high availability, high reliability, low latency, and in some cases, high-accuracy positioning.

[0040] An example configuration of a wireless communications network which uses some of the terminology proposed for and used in NR and 5G is shown in Figure 2. In Figure 2 a plurality of transmission and reception points (TRPs) 10 are connected to distributed control units (DUs) 41, 42 by a connection interface represented as a line 16. Each of the TRPs 10 is arranged to transmit and receive signals via a wireless access interface within a radio frequency bandwidth available to the wireless communications network. Thus, within a range for performing radio communications via the wireless access interface, each of the TRPs 10, forms a cell of the wireless communications network as represented by a circle 12. As such, wireless communications devices 14 which are within a radio communications range provided by the cells 12 can transmit and receive signals to and from the TRPs 10 via the wireless access interface. Each of the distributed units 41, 42 are connected to a central unit (CU) 40 (which may be referred to as a controlling node) via an interface 46. The central unit 40 is then connected to the core network 20 which may contain all other functions required to transmit data for communicating to and from the wireless communications devices and the core network 20 may be connected to other networks 25.

[0041] The elements of the wireless access network shown in Figure 2 may operate in a similar way to corresponding elements of an LTE network as described with regard to the example of Figure 1. It will be appreciated that operational aspects of the telecommunications network represented in Figure 2, and of other networks discussed herein in accordance with embodiments of the disclosure, which are not specifically described (for example in relation to specific communication protocols and physical channels for communicating between different elements) may be implemented in accordance with any known techniques, for example according to currently used approaches for implementing such operational aspects of wireless telecommunications systems, e.g. in accordance with the relevant standards.

[0042] The TRPs 10 of Figure 2 may in part have a corresponding functionality to a base station or eNodeB of an LTE network or gNodeB of an NR network. Similarly, the communications devices 14 may have a functionality corresponding to the UE devices 4 known for operation with an LTE network. It will be appreciated therefore that operational aspects of a new RAT network (for example in relation to specific communication protocols and physical channels for communicating between different elements) may be different to those known from LTE or other known mobile telecommunications standards. However, it will also be appreciated that each of the core network component, base stations and communications devices of a new RAT network will be functionally similar to, respectively, the core network component, base stations and communications devices of an LTE wireless communications network.

[0043] In terms of broad top-level functionality, the core network 20 connected to the new RAT telecommunications system represented in Figure 2 may be broadly considered to correspond with the core network 2 represented in Figure 1, and the respective central units 40 and their associated distributed units / TRPs 10 may be broadly considered to provide functionality corresponding to the base stations 1 of Figure 1. The term network infrastructure equipment / access node may be used to encompass these elements and more conventional base station type elements of wireless telecommunications systems. Depending on the application at hand the responsibility for scheduling transmissions which are scheduled on the radio interface between the respective distributed units and the communications devices may lie with the controlling node / central unit and / or the distributed units / TRPs. A communications device 14 is represented in Figure 2 within the coverage area of the first communication cell 12. This communications device 14 may thus exchange signalling with the first central unit 40 in the first communication cell 12 via one of the distributed units / TRPs 10 associated with the first communication cell 12.

[0044] It will further be appreciated that Figure 2 represents merely one example of a proposed architecture for a new RAT based telecommunications system in which approaches in accordance with the principles described herein may be adopted, and the functionality disclosed herein may also be applied in respect of wireless telecommunications systems having different architectures.

[0045] Thus, certain embodiments of the disclosure as discussed herein may be implemented in wireless telecommunication systems / networks according to various different architectures, such as the example architectures shown in Figures 1 and 2. It will thus be appreciated the specific wireless telecommunications architecture in any given implementation is not of primary significance to the principles described herein. In this regard, certain embodiments of the disclosure may be described generally in the context of communications between network infrastructure equipment / access nodes and a communications device, wherein the specific nature of the network infrastructure equipment / access node and the communications device will depend on the network infrastructure for the implementation at hand. For example, in some scenarios the network infrastructure equipment / access node may comprise a base station, such as an LTE-type base station 1 as shown in Figure 1 which is adapted to provide functionality in accordance with the principles described herein, and in other examples the network infrastructure equipment may comprise a control unit / controlling node 40 and / or a TRP10 of the kind shown in Figure 2 which is adapted to provide functionality in accordance with the principles described herein.

[0046] A more detailed diagram of some of the components of the network shown in Figure 2 is provided by Figure 3. In Figure 3, a TRP10 as shown in Figure 2 comprises, as a simplified representation, a wireless transmitter 30, a wireless receiver 32 and a controller or controlling processor 34 which may operate to control the transmitter 30 and the wireless receiver 32 to transmit and receive radio signals to one or more UEs 14 within a cell 12 formed by the TRP10. As shown in Figure 3, an example UE 14 is shown to include a corresponding transmitter 49, a receiver 48 and a controller 44 which is configured to control the transmitter 49 and the receiver 48 to transmit signals representing uplink data to the wireless communications network via the wireless access interface formed by the TRP10 and to receive downlink data as signals transmitted by the transmitter 30 and received by the receiver 48 in accordance with the conventional operation.

[0047] The transmitters 30, 49 and the receivers 32, 48 (as well as other transmitters, receivers and transceivers described in relation to examples and embodiments of the present disclosure) may include radio frequency filters and amplifiers as well as signal processing components and devices in order to transmit and receive radio signals in accordance for example with the 5G / NR standard. The controllers 34, 44 (as well as other controllers described in relation to examples and embodiments of the present disclosure) may be, for example, a microprocessor, a CPU, or a dedicated chipset, etc., configured to carry out instructions which are stored on a computer readable medium, such as a non-volatile memory. The processing steps described herein may be carried out by, for example, a microprocessor in conjunction with a random access memory, operating according to instructions stored on a computer readable medium. The transmitters, the receivers and the controllers are schematically shown in Figure 3 as separate elements for ease of representation. However, it will be appreciated that the functionality of these elements can be provided in various different ways, for example using one or more suitably programmed programmable computer(s), or one or more suitably configured application-specific integrated circuit(s) / circuitry / chip(s) / chipset(s). As will be appreciated the infrastructure equipment / TRP / base station as well as the UE / communications device will in general comprise various other elements associated with its operating functionality.

[0048] As shown in Figure 3, the TRP10 also includes a network interface 50 which connects to the DU 42 via a physical interface 16. The network interface 50 therefore provides a communication link for data and signalling traffic from the TRP 10 via the DU 42 and the CU 40 to the core network 20.

[0049] The interface 46 between the DU 42 and the CU 40 is known as the F 1 interface which can be a physical or a logical interface. The Fl interface 46 between CU and DU may operate in accordance with specifications 3GPP TS 38.470 and 3GPP TS 38.473, and may be formed from a fibre optic or other wired or wireless high bandwidth connection. In one example the connection 16 from the TRP 10 to the DU 42 is via fibre optic. The connection between a TRP10 and the core network 20 can be generally referred to as a backhaul, which comprises the interface 16 from the network interface 50 of the TRP10 to the DU 42 and the Fl interface 46 from the DU 42 to the CU 40. The core network 20 is connected to the CU 40 via the N2 (also called NG-C) interface for carrying control data and via the N3 (also called NG- U) interface for carrying user data.

[0050] The core network 20 may comprise core network functions such as a location management function (UMF) for managing a position of communications devices in the wireless communications network. In addition, the core network 20 may comprise one or more network functions for Artificial Intelligence / Machine beaming (AI / MU) model management, training, and storage.

[0051] Although reference is made to 5G networks, the discussions in this specification apply equally to 6G networks (and beyond) where there is expected to be significantly higher throughput, lower latency and higher reliability utilising sub-THz frequencies. In the case of 6G, base stations, which are an example of network infrastructure equipment, may also referred to as 6G NB (6G Node B), 6G RAN node, and so forth. In 6G, the core network 20 may be one or more network functions.

[0052] Artificial Intelligence (Al)

[0053] In existing techniques (also called “legacy techniques”), the position of a UE can be determined by a UE or gNB based on positioning measurements made by the gNB or UE. Positioning measurements may be performed by a UE on downlink signals such as positioning reference signals (PRS), or performed by a gNB on uplink signals such as sounding reference signals (SRS). Positioning measurements may comprise one or more of time of arrival (TOA), angle of arrival (AOA), angle of departure (AOD), a round trip time (RTT), reference signal received power (RSRP), and any other measurement used in determining the position of a UE. A UE may determine its position for example by receiving plurality of downlink signals, each from a different gNB, measuring a time of arrival and / or angle of the downlink signals and, based on the measurements, determining the position of the UE (e.g. multilateration).

[0054] However, the precision of the position determined by existing techniques is limited. In order to address technical problems in determining a precise location / position estimation of a communications device, the use of Artificial Intelligence / Machine Learning (AI / ML) models for positioning was firstly studied in Release- 18 of the 3GPP standards and further developed in later release(s).

[0055] AI / ML-based positioning tends to outperform other positioning techniques because AI / ML models are configured to collect and process large amounts of positioning measurements. By performing model training (based on input data), AI / ML models learn and establish an understanding of the environment associated with the positioning measurements. AI / ML models may then be deployed, for example at a UE or base-station (such as a gNB), in order to generate output, such as improved (i.e. more accurate) positioning measurements or a position estimate. This process / operation is also known as AI / ML model inference. For example, the output positioning measurements may be improved in the sense that they are more likely to result in a more accurate position estimate. For example, a UE may receive a DL-PRS from a gNB and make positioning measurements on it. However, the UE may not know that the DL-PRS reflected off an object before it reached the UE. The AI / ML model may be configured to correct for such reflections. The AI / ML model can produce output on refined line of sight (LOS) / non-LOS (NLOS) component identification. Additional detail on how AI / ML models can be used to generate more accurate positioning measurements, or generate a position estimate, can be found in [3], the contents of which are hereby incorporated by reference in their entirety. As examples, AI / ML models may utilise one or more of: supervised learning, generative Al, autoencoding, and reinforcement learning. These are explained in detail in the forthcoming paragraphs.

[0056] Supervised Learning

[0057] AI / ML models may implement a supervised machine learning model.

[0058] The supervised learning model is trained using labelled training data to learn a function that maps inputs (typically provided as feature vectors) to outputs (i.e. labels). The labelled training data comprises pairs of inputs and corresponding output labels. The output labels are typically provided by an operator to indicate the desired output for each input. The supervised learning model processes the training data to produce an inferred function that can be used to map new (i.e. unseen) inputs to a label.

[0059] The input data (during training and / or inference) may comprise various types of data, such as numerical values, images, video, text, or audio. Raw input data may be pre-processed to obtain an appropriate feature vector used as input to the model - for example, features of an image or text input may be extracted to obtain a corresponding feature vector. It will be appreciated that the type of input data and techniques for pre-processing of the data (if required) may be selected based on the specific task the supervised learning model is used for.

[0060] Once prepared, the labelled training data set is used to train the supervised learning model. During training the model adjusts its internal parameters (e.g. weights) so as to optimize (e.g. minimize) an error function, aiming to minimize the discrepancy between the model’s predicted outputs and the labels provided as part of the training data. In some cases, the error function may include a regularization penalty to reduce overfitting of the model to the training data set.

[0061] The supervised learning model may use one or more machine learning algorithms in order to learn a mapping between its inputs and outputs. Example suitable learning algorithms include linear regression, logistic regression, artificial neural networks, decision trees, support vector machines (SVM), random forests, and the K-nearest neighbour algorithm.

[0062] For positioning AI / ML models, the input to the supervised learning model may comprise positioning measurements (for example, measurements of one or more PRSs, or one or more SRSs, such as time of arrival and angle of arrival) and the output of the supervised learning model may comprise a position estimate of the communications device or improved positioning measurements.

[0063] Once trained, the supervised learning model may be used for inference - i.e. for predicting outputs for previously unseen input data. The supervised learning model may perform classification and / or regression tasks. In a classification task, the supervised learning model predicts discrete class labels for input data, and / or assigns the input data into predetermined categories. In a regression task, the supervised learning model predicts labels that are continuous values.

[0064] For positioning AI / ML models, the supervised learning model is used to predict an output comprising improved positioning measurements or a position estimate based on an input comprising positioning measurements.

[0065] In some cases, limited amounts of labelled data may be available for training of the model (e.g. because labelling of the data is expensive or impractical). In such cases, the supervised learning model may be extended to further use unlabelled data and / or to generate labelled data. Considering using unlabelled data, the training data may comprise both labelled and unlabelled training data, and semi-supervised learning may be used to learn a mapping between the model’s inputs and outputs. For example, a graph-based method such as Laplacian regularization may be used to extend a SVM algorithm to Laplacian SVM in order to perform semi-supervised learning on the partially labelled training data.

[0066] Considering generating labelled data, an active learning model may be used in which the model actively queries an information source (such as a user, or operator) to label data points with the desired outputs. Labels are typically requested for only a subset of the training data set thus reducing the amount of labelling required as compared to fully supervised learning. The model may choose the examples for which labels are requested - for example, the model may request labels for data points that would most change the current model, or that would most reduce the model's generalization error. Semi-supervised learning algorithms may then be used to train the model based on the partially labelled data set.

[0067] Generative Al

[0068] AI / ML models may implement generative artificial intelligence (Al).

[0069] A generative Al system learns patterns and structures in its input training data, in order to then generate new output data which exhibits similar characteristics to the training data. For positioning AI / ML models, the input training data may comprise positioning measurements and the output training data may comprise a position estimate of the communications device or improved positioning measurements.

[0070] The generative Al system may generate output data based on an input prompt. The prompt may comprise various types of data, such as images, video, text, or audio. The prompt may be of the same or different datatype to the model’s training and / or output data.

[0071] The generative AI / ML model may comprise a generative model trained to learn a probability distribution of the input training data, and generate new output data based on this learned distribution. For example, for a set of data instances / observable variables (X) and a set of labels / target variables (Y) in the training data set, the generative model may learn a joint probability distribution of data instances and labels p(X,Y), and / or a probability distribution of the data instances p(X) (for example where no labels are available).

[0072] Example suitable generative models for learning a probability distribution of the input training data include Variational Autoencoder (VAEs), transformer-based models, diffusion models (e.g. de-noising diffusion probabilistic models (DDPMs)), Reinforcement Learning (RL), and Generative Adversarial Networks (GANs). The choice of generative model may depend on the specific task performed by the generative AI / ML.

[0073] The generative model may comprise one or more artificial neural networks. For example, a Variational Autoencoder (VAE) may comprise a pair of neural networks acting as an encoder and a decoder to and from a reduced (i.e. latent space) representation of the training data respectively, and a Generative Adversarial Network (GAN) may comprise a first ‘generator’ neural network that generates new data and a second ‘discriminator’ neural network that learns to discriminate between generated data and real data. The one or more constituent neural networks of the generative model may be trained together or separately.

[0074] During training the generative model may adjust its internal parameters (e.g. neural network weights) so as to optimize (e.g. minimize) a loss / error function, aiming to minimize discrepancy between the generated output data and desired output data. It will be appreciated that the specific loss function, and algorithm used to optimize the function may vary depending on the nature of the generative model, and its intended application. For example, a mean squared error loss function may be used for an image generation task, and a cross-entropy loss function may be used for a text generation task. These loss functions may be optimized using various existing optimization algorithms, such as gradient descent.

[0075] Once trained, the generative model may be used to generate new output data based on an input prompt. The input prompt may be provided by a user, or by an appropriate device (e.g. using an application programming interface (API)). Thus, the generative Al / ML model allows generating new based on only a prompt and without requiring detailed instructions for doing so.

[0076] Autoencoders

[0077] AI / ML models may implement autoencoding.

[0078] An autoencoder is a type of an unsupervised machine learning model that uses one or more artificial neural networks to learn an efficient representation of unlabelled input data. The autoencoder may be used to encode various types of data, such as images, video, text, audio, or positioning measurements.

[0079] The autoencoder may comprise an encoder neural network that encodes input data into a reduced representation (also called a “latent space”), and a decoder neural network that aims to recreate the input data from the encoded reduced representation. The latent space is typically of a lower-dimension than the input data - thus, the latent space generated by the encoder typically provides a more efficient, compressed representation of the input data that requires less memory storage than the original input data.

[0080] The encoder neural network may comprise one or more layers that transform input data into a reduced representation. The encoder neural network receives input data, and the final layer of the encoder neural network outputs a reduced representation of the input data, i.e. a latent space (also termed a “bottleneck layer”).

[0081] The decoder neural network comprises one or more layers that transform data from the latent space into output data of the same dimensionality as the data input to the encoder. The decoder aims to reconstruct the data originally input to the encoder neural network from the latent space representation of the data.

[0082] The encoder and / or decoder neural networks typically comprise a plurality of hidden layers. For example, an encoder may comprise a plurality of hidden layers that progressively extract further reduced representations of the input data. Using deeper neural networks (i.e. with a higher number of hidden layers) for the encoder and / or the decoder may improve performance of the autoencoder, and in some cases may reduce the amount of training data that is required.

[0083] The encoder and decoder neural networks are typically trained together. During training the autoencoder may adjust its internal parameters (e.g. weights and biases of the encoder and decoder neural networks) so as to optimize (e.g. minimize) a loss / error function, aiming to minimize discrepancy between the data input to the encoder and the output reconstructed data generated by the decoder. It will be appreciated that the specific loss function, and algorithm used to optimize the function may vary depending on the nature of the autoencoder model, and its intended application. In an example, a mean squared error loss function optimized using gradient descent may be used. In some cases, a sparse autoencoder may be used in order to promote sparsity of the latent representation (as compared to the input) and to prevent the autoencoder from learning the identity function - for example, a sparse autoencoder may be implemented by modifying the loss function to include a sparsity regularization penalty. In some cases, the autoencoder may be a Variational Autoencoder (VAE). The VAE is a specific type of auto-encoder in which a probability model is imposed on the encoded representation by the training process (in that deviations from the probability model are penalised by the training process). The VAE may be used for generative artificial intelligence applications to generate new output data which exhibits similar characteristics to the input encoded data by sampling from the learned latent space.

[0084] For positioning AI / ML models, the input data may comprise positioning measurements and the output data may comprise a position estimate of the communications device or improved positioning measurements.

[0085] Reinforcement Learning

[0086] AI / ML models may implement reinforcement learning (RL).

[0087] Reinforcement learning is a type of machine learning directed to training an artificial intelligence agent to take actions in an environment that maximize the notion of a cumulative reward. During reinforcement learning, the agent interacts with the environment, and learns from the results of its actions, thus allowing the agent to progressively improve its decision-making.

[0088] An RL model typically comprises an action-reward feedback loop. The feedback loop comprises: an environment, state, agent, policy, action, and reward. The environment is the system with which the agent interacts and in which the agent operates - for example, the environment may be a virtual environment of a video game. The state represents the current conditions in the environment. The agent receives the state as an input and takes an action which may affect the environment and change the state of the environment. The agent takes the action based on its policy which is a mapping from states of the environment to actions of the agent. The policy may be deterministic or stochastic. The reward represents feedback from the environment to the action taken by the agent. The reward provides an indication (typically in the form of a numerical value) of the desirability of the result of the agent’s action. The reward may comprise positive signals to reward desirable behaviour of the agent and / or negative signals to penalize undesirable behaviour of the agent.

[0089] Through multiple iterations of action-reward feedback loop, the agent aims to maximise the total cumulative reward it receives, thus learning how to take optimal actions in the environment. The reinforcement learning process thus allows the agent to learn an optimal policy that maximizes the cumulative reward. The cumulative award may be estimated using a value function which estimates the expected return starting from a given state or from a given state and action. Using the cumulative reward in the reinforcement learning process allows the agent to consider long-term effects of its policy.

[0090] A reinforcement learning algorithm may be used to refine the agent’s policy and the value function over iterations of the action-reward feedback loop. The learning algorithm may rely on a model of the environment (e.g. based on Markov Decision Processes (MDPs)) or be model-free. Example suitable model-free reinforcement learning algorithms include Q-leaming, State-Action-Reward-State-Action (SARSA), Deep Q-Networks (DQNs), or Deep Deterministic Policy Gradient (DDPG).

[0091] It will be appreciated that the agent will typically engage in both exploration and exploitation of the environment in which it operates. In exploration, the agent takes typically random actions to gather information about the environment and identify potentially desirable actions (i.e. actions that maximise cumulative reward). In exploitation, the agent takes actions that are expected to maximise reward (e.g. by selecting the action based on the agent’s latest policy). Various techniques may be used to control the proportion of explorative and exploitative actions taken by the agent - for example, a predetermined probability of taking an explorative action in a given iteration of the feedback loop may be set (and optionally reduced over time to allow the agent to shifts more towards exploitation over time to maximise cumulative reward in view of diminishing returns for further exploration).

[0092] In some cases, the RL model may be configured to learn from feedback provided by a user. Utilising user feedback in this way may allow the agent to improve its choice of actions and better align with user preferences. For example, reinforcement learning from human feedback (RLHF) techniques may be used. RLHF includes training a reward model based on user feedback and using this model for determining the reward in the reinforcement learning process described above. The user feedback may be received in various forms depending on the specific reinforcement learning problem being solved - for example, the feedback may be received in the form of a user ranking of instances of the agent’s actions. RLHF thus allows incorporating user feedback into the reinforcement learning process. RLHF approaches may be advantageous where it is easier for a user than for an algorithm to assess the quality of the machine learning model’s output (e.g. for generative artificial intelligence RL models).

[0093] For positioning AI / ML models, the input data may comprise positioning measurements and the output data may comprise a position estimate of the communications device or improved positioning measurements.

[0094] Although a number of types of AI / ML models have been described above in connection with positioning, the above types of AI / ML can also be used in beam management or channel state information (CSI) feedback enhancement tasks, or indeed other tasks known to a person skilled in the art.

[0095] For beam management AI / ML models, input data may comprise one or more of an Ll-RSRP and its associated beam / resource ID. The output data may be one or more of a predicted beam, narrow beam, improved beam prediction or a refined RSRP.

[0096] For CSI AI / ML models (e.g. CSI measurement and reporting enhancement, CSI compression or CSI prediction), input data may comprise one or more of a signal to interference and noise ratio (SINR) estimation, a Modulation and Coding (MCS) index, Channel Quality Indicator (CQI) index, or a Precoding matrix index (PMI). The output data may be one or more of a refined or predicted MCS, CQI index, PMI index, RI index, or a new type of output. In some examples of CSI compression AI / ML models, input data may be a full CSI compression report and output data may be a reduced size CSI compression report.

[0097] Life Cycle Management (LCM) Architecture for an AI / ML Model

[0098] 3GPP has identified a general AI / ML framework for the NR air interface to facilitate different radio frequency applications, including positioning applications, but also for beam management and CSI feedback enhancements applications. The aim of the framework is to cover a general architecture addressing the whole Al model life cycle, Life Cycle Management (LCM), including such steps as data collection, model training, etc. Here, LCM for AI / ML for the NR air interface is described in the below paragraphs. As those skilled in the art would appreciate, the following description is also applicable to 6G and other wireless communications systems.

[0099] An example of a life cycle management (LCM) architecture for an artificial intelligence model is schematically illustrated in Figure 4 which has been reproduced from [3], the contents of which are hereby incorporated by reference in their entirety. As shown in Figure 4, LCM architecture comprises a data collection function 402, an inference function 404, a management function 406, a model training function 408 and a model storage function 410.

[0100] The data collection function 402 is configured to provide training data to the model training function 408, to provide monitoring data to the management function 406 and to provide inference data to the inference function 404. The data collection function 402 is a process / function of collecting data by the network nodes, management entity, or UE for the purpose of AI / ML model training, data analytics and inference.

[0101] The inference function 404 is configured to provide an inference output to the management function 406, to receive a management instruction from the management function 406 and to receive a model from the model storage function 410. The inference function 404 is a process / function of using a trained AI / ML model to produce a set of outputs based on a set of inputs.

[0102] The management function is configured to provide performance feedback and / a retraining request to the model training function 408, and to provide a model delivery request to the model storage unit 410.

[0103] The model training function 408 is configured to provide an updated model to the model storage unit 410. The model training function 408 is a process / function to train an AI / ML Model (e.g., by learning the input / output relationship) in a data driven manner and obtain the trained AI / ML Model for inference.

[0104] As shown in figure 4, the management function 406 represents the core of the LCM architecture. The function of the management function 406 is to monitor model performance at different entities (such as UEs or gNBs) and request delivery of updated models. Model management comprises two procedures - model switching and model updating.

[0105] Existing positioning procedures in wireless communications networks, such as 5G NR networks, involve communications between a UE, a gNB and a LMF. The AI / ML model may be deployed at the UE, gNB or LMF. An AI / ML model for positioning may be a direct AI / ML positioning model or an AI / ML assisted positioning model. Both direct AI / ML positioning models and AI / ML assisted positioning models use positioning measurements (such as measurements based on received PRS or SRS) as input data. However, the output data for direct AI / ML positioning models is an estimate of a position of the UE (such as a co-ordinate) whereas the output data for AI / ML assisted positioning models is a more accurate positioning measurement or a new set of positioning measurements. Examples of existing positioning techniques using a direct and assisted AI / ML positioning models - which are described in [3] - are illustrated in Figure 5.

[0106] Segment (A) of Figure 5 (referred to as Case 1 in [3]) illustrates an example of UE-based positioning with a UE-side direct AI / ML model. In this example, one or more gNBs transmit PRS to the UE. The UE performs positioning measurements on the PRS and uses a direct AI / ML model to generate an estimate of a position of the UE based on the positioning measurements. The UE forwards the position estimate to the LMF.

[0107] Segment (B) of Figure 5 (referred to as Case 2a in [3]) illustrates an example of UE-assisted / LMF -based positioning with a UE-side AI / ML assisted positioning model. In this example, one or more gNBs transmit PRS to the UE. The UE performs positioning measurements on the PRS and uses an AI / ML assisted positioning model to generate improved positioning measurements of the PRS. The UE forwards the improved measurements of the PRS to the LMF. The LMF generates a position estimate of the UE based on the improved measurements of the PRS. Segment (C) of Figure 5 (referred to as Case 2b in [3]) illustrates an example of LMF -based positioning with an LMF-side direct AI / ML model. In this example, one or more gNBs transmit PRS to the UE. The UE performs positioning measurements on the PRS and transmits the positioning measurements of the PRS to the LMF. The LMF then uses the direct AI / ML model generate a position estimate for the UE based on the positioning measurements of the PRS.

[0108] Segment (D) of Figure 5 (referred to as Case 3a in [3]) illustrates an example of NG-RAN node assisted positioning with a gNB-side AI / ML assisted positioning model. In this example, the UE transmits SRS to one or more gNBs. The gNB performs positioning measurements on the SRS and uses the AI / ML assisted positioning model to generate more improved positioning measurements of the SRS. The gNB then transmits the improved positioning measurements of the SRS to the LMF. The LMF generates a position estimate of the UE based on the improved positioning measurements of the SRS.

[0109] Segment (E) of Figure 5 (referred to as Case 3b in [3]) illustrates NG-RAN node assisted positioning with an LMF-side direct AI / ML model. In this example, the UE transmits SRS to one or more gNBs. The gNB performs positioning measurements on the SRS and transmits the positioning measurements of the SRS to the LMF. The LMF uses the direct AI / ML model to generate a position estimate of the UE based on the received positioning measurements of the SRS.

[0110] An AI / ML positioning model may be deployed for a particular area, while another AI / ML positioning model may be deployed for another area. In this way, AI / ML positioning models can be adapted to be effective in the areas in which they are deployed. For example, a particular AI / ML model may be trained under particular environmental conditions because these environmental conditions will be similar to the environmental conditions in which the AI / ML model will be deployed. Having multiple AI / ML models, each adapted for different areas, also facilitates model switching and selection, thus enabling the provision of high accuracy positioning across a diverse range of areas.

[0111] However, the environmental conditions of an area in which an AI / ML positioning model is deployed may change over time. For example, the position (e.g. layout or constellation) of communications devices relative to passive environmental objects (such as a table or a wall) in the area may change over time. Since the passive environmental objects may reflect / scatter signals transmitted by the communications devices, the change in position of the communications devices relative to passive environmental objects means that different signals will be reflected / scattered in different ways over time. This may affect channel conditions and propagation characteristics such as line-of-sight (LOS) between the transmitter and the receiver in the area. Accordingly, an AI / ML model adapted for a particular area may become outdated over time. If the AI / ML model becomes outdated, then the accuracy of positioning measurements or positioning estimates generated by the AI / ML model decreases to unacceptable levels. The decrease in performance of an AI / ML model over time may be due to data drift and / or model drift.

[0112] Data drift occurs when the distribution of input data (for example, positioning measurements) to the AI / ML model change over time. This happens, for example, when the distribution of the positioning measurements obtained by a UE deviates from the distribution of the positioning measurements used to train the AI / ML model. For example, when the AI / ML model is trained, it may be assumed that the UEs may evenly spread out in the room. However, in practice or in a reality at a given period of time, UEs tend to cluster or stay in sub-areas of the room (e.g. most UEs may be near the centre of the room). This deviation between the assumed UE statistic (distribution of location and positioning measurements) in training phase and the realistic UE statistic might harm the overall performance. Model drift occurs when a relationship between the input data and output data of the AI / ML model changes. This relationship may change due to a change in environmental conditions. For example, a room may have different reflectors such as tables and chairs. The location of these reflectors may not be fixed all the time. Considering a case where the configuration of the tables and chairs in the room is changed but the UE is still using the same AI / ML model for positioning, then the issue of model drift occurs, leading to compromised positioning performance.

[0113] Therefore, the decrease in performance of AI / ML problems over time represents a technical problem.

[0114] To address the issues of data drift and model drift, a model monitoring procedure can be conducted. Such a model monitoring procedure can be carried out semi-persistently, or can be triggered by either an internal node in the core network or an external entity, such as a third party server (e.g., application server). This process involves gathering new signal measurements (referred to herein as reference information) using, for example, a reference UE known as a Positioning Reference Unit (PRU), collecting information with a Ground Truth (GT) label (also referred to as ground truth (GT) information), and calculating Performance Metrics (PM). Subsequently the results of the calculated PM will be reported to the device at which the AI / ML model is deployed (e.g. the UE, which may be the PRU, or the gNB) for further action.

[0115] Here, as those skilled in the art would understand, a PRU functions similarly to a standard UE by transmitting PRS and receiving SRS. However, unlike typical UEs, a PRU has a fixed and / or known location. Initially introduced by 3GPP in Release 17 of NR, the PRU’s primary role was to calibrate timing errors in NR positioning. In release 18 and beyond, the PRU’s capabilities are being further explored within the context of AI / ML positioning, where such a PRU may be used to assist in model training and monitoring.

[0116] Furthermore, the radio system can prepare a model pool of multiple different Al models providing the same functionality, with only one model from the model pool being deployed at the network side at any given time. A model pool may consist of a few selected Al models, such as an active Al model (i.e., the one that is currently being used), and one or more selected Al models which are likely to be used to replace the active one. In the event of model or data drift, the LMF can select a more suitable model from the model pool in view of the updated wireless environment. This new model (along with all required parameters for its deployment at the gNB or UE), or only its model ID (assuming knowledge of such a model within the model pool is known at the gNB or UE, will be then signalled to the gNB / UE.

[0117] These two enhancements described above, relating to model monitoring and model selection, ensure the performance of the deployed model is maintained in a sustainable way.

[0118] Technical Problem

[0119] Here, the following three AI / ML positioning cases described above are considered:

[0120] • Case 1 : UE-based positioning with UE-side direct AI / ML model (as shown in segment (A) of Figure 5);

[0121] • Case 2a: UE-assisted / LMF -based positioning with UE-side model, AI / ML assisted positioning (as shown in segment (B) of Figure 5); and

[0122] • Case 3a: NG-RAN node assisted positioning with gNB-side model, AI / ML assisted positioning (as shown in segment (D) of Figure 5). Figure 6 illustrates an example of the NG-RAN node (i.e. gNB) assisted UL based positioning (Case 3a). The UE(s) 601, 602, 603, including PRUs 601, 602, transmit reference signals (e.g., SRS) for positioning purposes to gNBs 604, 605, 606. The gNBs 604, 605, 606 receive these SRS, and perform positioning measurement, to determine parameters such as relative TOA (RTOA), AOA, and RSRP. In order to improve the performance of the measurement results, each of gNBs 604, 605, 606 may be equipped with AI / ML operation. Hence, gNBs 604, 605, 606 perform AI / ML inference based on the received SRS to produce positioning measurement results to be reported to LMF 607.

[0123] The aim of such a positioning scheme as that illustrated by the example of Figure 6 is to provide more accurate positioning measurement results. The positioning measurements of a particular UE (e.g., UE 603) which are improved by AI / ML at a gNB uses a specific AI / ML model. UE 603 and PRUs 601, 602 may have similar radio channel characteristics. The validity of that AI / ML model needs to be monitored as described above to ensure that the optimal AI / ML model is being used at any given time. This will guarantee that the usage of AI / ML can improve the positioning measurements of that UE.

[0124] As described above, the environments and / or the UE which is performing positioning may change over time. Hence, the obtained AI / ML model for positioning computation may become no longer applicable. In certain conditions / scenarios, the obtained AI / ML model may therefore require some updates, or may require being switched to an entirely different AI / ML model. However, a mechanism for the update / change of AI / ML models involving various nodes and various positioning techniques has, at the date of priority of the present application, yet to have been defined.

[0125] As noted above, the decrease in performance of AI / ML problems over time represents a technical problem. In co-pending United Kingdom patent application number 2402279.0 [4], the contents of which are hereby incorporated by reference, an approach is described where the radio node (e.g. gNB) performs PM computation with assistance from the LMF. As those skilled in the art would appreciate, there are advantages and disadvantages to performing the PM computation at various nodes, and the ability for different nodes (both within the radio access network and core network, as well as externally to both) can complement each other and provides options and flexibility for different deployments. Embodiments of the present technique therefore define further arrangements to those described in [4] that enable the efficient and effective performance of AI / ML models deployed at UEs and gNBs over long periods of time.

[0126] Distributed AI / ML Model Performance Metric Computation for Model Monitoring

[0127] Figure 7 shows a part schematic, part message flow diagram representation of a wireless communications system comprising a first apparatus 710 (e.g. either a node of a core network or a server) and a second apparatus 720 (e.g. a communications device / UE or an infrastructure equipment (e.g. a base station or gNB)) in accordance with at least some embodiments of the present technique. The first apparatus 710 may be an LMF, for example, and is configured to perform AI / ML model monitoring based on performance metric (PM) computation for an AI / ML model deployed at the second apparatus.

[0128] The first apparatus 710 and the second apparatus 720 each comprise a transceiver (or transceiver circuitry) 711, 721 and a controller (or controller circuitry) 712, 722. Each of the controllers 712, 722 may be, for example, a microprocessor, a CPU, or a dedicated chipset, etc. The controllers 712, 722 may also each be equipped with a memory unit (which is not shown in Figure 7). As those skilled in the art would appreciate the wireless communications system of Figure 7 may include a number of other nodes, such as other communications devices and infrastructure equipment, which are not shown in the example of Figure 7 for the purposes of simplicity. As shown in the example of Figure 7, the controller 712 of the first apparatus 710 is configured in combination with the transceiver 711 of the first apparatus to determine 730 that the first apparatus 710 is to perform a performance assessment of an artificial intelligence, Al, model used by (and deployed at) the second apparatus 720 to perform a task, wherein as noted above the second apparatus 720 is either a communications device or an infrastructure equipment of a radio access network, to transmit 740, to the second apparatus 720, a request to perform measurements (e.g. positioning or CSI measurements or beam management-related measurement to support the first apparatus 710 in performing the Al model monitoring) and to report the performed measurements to the first apparatus 710, to receive 750, from the second apparatus 720, an indication of the performed measurements, to perform 760 a performance metric calculation by comparing an output of the Al model when the performed measurements 750 are used as an input to the Al model with reference information, and to transmit 770, to the second apparatus 720 based on the calculated performance metric 760, an indication of a validity status of the Al model.

[0129] Here, where reference is made to the task performed by the second apparatus (i.e. a base station or UE), such a task is generally described in accordance with the below arrangements of embodiments of the present technique as positioning. However, such a task may be any appropriate task performed by the UE or gNB for which utilization of the AI / ML model aids such a task, which includes but is not limited to positioning, beam management, and channel state information (CSI) feedback enhancements.

[0130] Essentially then, embodiments of the present technique, as exemplified by the example wireless communications system of Figure 7 for example, propose that a network node other than the UE or base station - i.e. a core network node, such as the Location Management Function (LMF), Analytics Data Repository Function (ADRF), Data Collection Coordination Function (DCCF), or another network node (outside the core network), such as the application server or a third party server - is able to perform AI / ML model monitoring by performing performance metric (PM) computation of an AI / ML model deployed in a UE or base station. That is, the PM computation is proposed to be distributed to outside the node (base station or UE) where the AI / ML model is deployed and is performing AI / ML inference.

[0131] For simplicity, arrangements of embodiments of the present technique as described below generally refer to the AI / ML model monitoring being performed by a core network node (where generally this is described as being the LMF), but it is not intended that such arrangements are limited to being performed by such an entity, and indeed any core network entity or network node other than the UE or base station may be configured to perform such arrangements.

[0132] In accordance with arrangements of embodiments of the present technique, the core network node is configured to determine the validity of an AI / ML model, e.g. for AI / ML-assisted positioning, deployed in a UE or base station by: a. Being triggered / configured to perform model monitoring; b. Requesting a reference UE (e.g. a PRU) to initiate positioning measurements for model monitoring purposes; c. Receiving monitoring data from the UE or base station at which the Al model is deployed to further calculate the PM; and d. Reporting the monitoring results / output to the UE or base station at which the Al model is deployed.

[0133] In accordance with some arrangements of embodiments of the present technique, model monitoring can be triggered internally by other nodes in the core network, externally by other nodes outside of the core network. In other words, the first apparatus may be configured to receive, from a third apparatus (e.g. a core network or radio access network node or a third party server), a triggering command, wherein the step of determining that the first apparatus is to perform the performance assessment of the Al model is based on receiving the triggering command.

[0134] In accordance with some arrangements of embodiments of the present technique, model monitoring can be activated autonomously with respect to configured time occasions (for example, periodically or relative to a time reference). The input signal which activates the model monitoring may be a signal received from another node in the core network or externally from the core network (e.g. via the application layer). In other words, the first apparatus may be configured to determine that the first apparatus is to perform the performance assessment of the Al model at one or more preconfigured time occasions.

[0135] In accordance with some arrangements of embodiments of the present technique, the received data (also referred to herein as the monitoring data) at the core network node from the UE or base station at which the Al model is deployed may include Ground Truth (GT) information and measurement results (e.g. positioning data or channel state information data).

[0136] Here, in at least some arrangements, the GT information may comprise a position of the reference UE, where it can be either absolute position information in a global coordinate system or a relative distance between the reference UE and the base station. The GT information may alternatively or additionally comprise some statistical positioning measurement / estimation information of the reference UE. The LMF may have collected and stored information from previously received positioning measurements and / or positioning estimation calculations, and can then perform statistical analysis on this previously collected and stored information. In other words, the reference information may be ground truth (GT) information associated with a reference communications device, which may for example be a PRU. Here, this reference communications device may for example be a communications device with a current position that is known to the first apparatus or a communications device with sufficient statistical measurement information that has been obtained by the first apparatus - here, sufficient statistical measurement information means that such information is considered reliable by the LMF and so trusts such information to an extent that it is able to use it as GT information. For example, such sufficient information may have been collected recently, e.g. within a predetermined amount of time, or may be based on reference signals received from multiple devices that all indicate that such measurement information is accurate. Furthermore, here, the GT information may be the known current position of the reference communications device and / or an estimated current position of the reference communications device determined by the first apparatus based on a previous position of the reference communications device and / or one or more other communications devices.

[0137] In some arrangements of embodiments of the present technique, the GT information of the reference UE may already be known by the core network node, and therefore, no extra signalling of the GT information is needed in such arrangements. In other words, the reference information may be known by and stored at the first apparatus.

[0138] Here, the positioning data is to be obtained during an AI / ML monitoring period, and may be received from the UE or base station at which the Al model is deployed, for example, as a measurement report indicating the positioning data and / or as reference signals from which the LMF is able to extract the positioning data. In other words, the received indication of the performed measurements may be positioning data (or, for tasks other than positioning, may be channel estimation data or data regarding one or more beams used by the second apparatus for communication) received by the first apparatus from the second apparatus. This positioning data may comprise one or more of: measurements based on legacy positioning (i.e., without AI / ML), such as RTOA, AOA, RSRP, the received signal waveform, such as in- phase and quadrature (IQ) data of the received reference signal, a channel frequency response (i.e., in the frequency domain) obtained from the received reference signal, a channel impulse response (CIR) (i.e., in time domain) obtained from the received reference signal, and a time-stamp of the measurement. Here, the granularity in the frequency domain may be at sub-carrier level or a set of sub-carriers (e.g., resource block level).

[0139] In accordance with arrangements of embodiments of the present technique, the data input associated with the AI / ML model may include the AI / ML model parameters (e.g., AI / ML model coefficients, formula) or an index of the AI / ML model deployed at and being used by the UE or base station performing AI / ML model inference. The LMF may receive such a data input from the UE or base station at which the AI / ML model is deployed, or may alternatively receive such information from elsewhere, such as from another node in the radio access network or core network, for example. In other words, the first apparatus may be configured to receive, from the second apparatus, information associated with the Al model, wherein the information associated with the Al model comprises one or more parameters of the Al model. Additionally, this data input associated with the AI / ML model can also be accompanied by information associated with a secondary AI / ML model, which may be another AI / ML model that can be potentially be used in the evaluation performed by the LMF.

[0140] In accordance with arrangements of embodiments of the present technique, the core network node (e.g. the LMF) inspects whether the model deployed at the UE or base station, which is referred to herein as Model#X, is valid or not by calculating a PM and evaluate its outcome. The PM computation can be based on a statistical comparison of the ground truth (GT) to a new positioning estimate generated by Model#X using the received / determined positioning data as an input to the model. In other words, the step of transmitting the indication of the validity status of the Al model may be dependent on whether the calculated performance metric satisfies a predetermined condition (i.e. when the statistical comparison between the GT information and the output of the Al model is determined). Here, this statistical analysis may be any appropriate mathematical operation, such as, for example, the absolute difference between the GT and the output of the AI / ML model, or a root-mean-square (RMS) difference between the GT and the output of the AI / ML model, or a Kolmogorov-Smirnov (KS) test performed on the GT and the output of the AI / ML model.

[0141] For example, in some arrangements of embodiments of the present technique, if the PM calculated from the Model#X, which is referred to herein as PM#X, is lower than a threshold, the core network node can consider that the model is still valid. Otherwise, if PM#X is greater than this threshold, the core network will determine and indicate to that UE or base station that the AI / ML model performed at the UE / base station is to be updated.

[0142] In the event that Model#X fails the validity check, arrangements of embodiments of the present technique define various ways in which the core network can seek to remedy this. For example, the core network node can select a new model from a (predefined) model pool with multiple models sharing the same functionality as the Model#X, where such an example model pool and multiple models within that pool may be referred to herein as {Model#X, Model#Y, Model#Z}. Here, the core network node can calculate multiple PMs associated with each of these models respectively, e.g., PM#X, PM#Y, PM#Z. The core network node can then select the best model, which may for example be based on the selected model having the smallest PM (e.g. the smallest difference between the output of the model and the received / derived positioning data). In other words, the first apparatus may be configured to calculate the performance metric by comparing the reference information with an output of each of one or more further Al models when the performed measurements are used as an input to that Al model, wherein the second Al model is one of the further Al models.

[0143] Subsequently, after such a model validity check, the core network node may report one or more of the following:

[0144] • An indication as to whether the AI / ML model is valid or not. For example, this may be indicated as a one-bit flag, where if the flag is 0, it indicates that the AI / ML model is invalid, while if the flag is 1 it indicates that the AI / ML model is still valid. In other words, the indication of the validity status of the Al model may comprise a flag that indicates either that the Al model is valid or that the Al model is invalid; and / or

[0145] • An indication to update or fine-tune the current AI / ML model. In other words, if the calculated performance metric does not satisfy the predetermined condition, the step of transmitting the indication of the validity status of the Al model may comprise transmitting, to the second apparatus, a request to update the Al model; or

[0146] • An indication to change to a new AI / ML model, where such an indication could involve providing the entire set of required parameters of the new AI / ML model, or an index of the new AI / ML model (assuming the UE or base station has a set of pre-defined AI / ML models, such as those defined by a model pool known to both the UE / base station and the LMF). In other words, if the calculated performance metric does not satisfy the predetermined condition, the step of transmitting the indication of the validity status of the Al model may comprise transmitting, to the second apparatus, a request to change the Al model from a first Al model to a second Al model. Here, the request to change the Al model may comprise either an indication of an index of the second Al model or an indication of one or more parameters of the second Al model.

[0147] Figure 8 shows a flow chart illustrating an example of the operation in the first apparatus (e.g. a core network node such as the LMF) in accordance with at least some arrangements of embodiments of the present technique.

[0148] In step S801, the LMF is triggered to perform the PM, for example based on a trigger signal received or in respect of a configured timing requirement being met as described above. In step S802, the LMF obtains information relating to the AI / ML model; for example from the base station or UE where it is deployed. In step S803, the LMF obtains GT information, for example with respect to a PRU or other reference UE(s). In step S804, the LMF obtains the AI / ML model input, for example, positioning or channel state information received from the base station or UE where the model is used. In step S805, the LMF performs a PM computation, firstly by feeding the received model input into the AI / ML model at the LMF to determine an output. In step S806, the LMF compares the received GT information and model output as part of the PM computation. In step S807, the LMF determines whether the PM information meets one or more configured conditions; for example, it is within an acceptable range for the AI / ML model. If so, the method proceeds to step S809, where the current AI / ML model is kept and the LMF indicates this to the base station / UE, but if not, the method proceeds to step S808, where the LMF determines that the AI / ML should be either updated or replaced, and indicates such to the base station / UE where the AI / ML model is deployed.

[0149] In some arrangements of embodiments of the present technique, the reference UE (from or based on which the GT is received or derived) may be a positioning reference unit (PRU), or one or more other selected UEs. The selected UE(s) may for example be a UE for which the LMF has previously collected statistical information and is therefore confident that such a UE is able to be used as the reference UE. In some arrangements of embodiments of the present technique, a maximum duration (i.e. T max) from triggering the PM and providing the PM output to the radio node may be defined. In other words, the first apparatus may be configured to transmit the indication of the validity status of the Al model within a specified time from determining that the first apparatus is to perform the performance assessment of the Al model.

[0150] In accordance with some arrangements of embodiments of the present technique, where the LMF works as the core network node performing model monitoring, the trigger, monitoring data, and monitoring result communicated between the UE / base station at which the Al model is deployed and the LMF may be specified either using the NRPPa protocol when the model is deployed at a base station (i.e. case 3a), or using the LPP protocol when the model is deployed at a UE (i.e. cases 1 or 2a). Such protocols would be familiar to those skilled in the art.

[0151] For example, the positioning data received from the base station at the LMF may be expected to use the NRPPa protocol (or from the UE using the LPP protocol), where the following information element (IE) may be define to carry such data:

[0152] Al ML Positioning Data

[0153] • Received signal waveform (complex signal);

[0154] • Measurement results: o UL RToA; o UL AoA; o UL RSRP; o CFR; o CIR; and

[0155] • Time information / time stamp.

[0156] Furthermore, in accordance with such arrangements of embodiments of the present technique, the AI / ML model being used at the UE / base station (i.e. its parameters) can also be provided to the LMF. This is also carried out using NRPPa protocol when the model is deployed at a base station and using the LPP protocol when the model is deployed at a UE. Here, the following example IE may be used:

[0157] • Option 1 : conveying the active AI / ML model being used: o AIML_model:

[0158] ■ Model_ index (0,1, ...);

[0159] • Option 2: conveying the active AI / ML model index and secondary / altemative model: o AIML_model:

[0160] ■ Model_active_index (0,1, . . . );

[0161] ■ Model_secondary_index (0,1, . . . ) ;

[0162] • Option 3: Conveying the detailed of the current AI / ML model: o AIML model_structure (e.g., size of layers, and number of hidden layers); o AIML_model parameters (for each layer).

[0163] The output from the LMF (as the outcome of PM computation) to the base station / UE may also carried out using the NRPPa / LPP protocol respectively. Here, the following example IE may be used:

[0164] • Option 1 : informing whether valid or not: o AIML model_status: 0 (invalid), 1 (valid); • Option 2: informing fine tuning the current AIML model: o AIML_model_update:

[0165] ■ Element indices;

[0166] ■ Element values;

[0167] • Option 3: informing to switch the AIML model: o AIML_model_switch:

[0168] ■ AIML_new_index (0,1,...).

[0169] To handle the issue of model drift, embodiments of the present technique therefore propose to use the calculation of performance metrics (PM) as a criteria to determine whether a model is valid or not. Figure 9 illustrates an example of how this method can be performed in the network when an AI / ML model is deployed at a base station. Those skilled in the art would appreciate that such a method could be applied in a similar manner when the model is deployed at a UE.

[0170] In the example of Figure 9, the monitoring is carried out for an AI / ML-assisted positioning model 901 which uses a received reference signal waveform 902 or a channel response as an input and generates refined positioning measurements (e.g., RTOA++) as a model output. Here it should be noted that RTOA is an example of the positioning measurement using legacy NR positioning while RTOA++ is the output 905 by utilizing AI / ML positioning. In this specific example, as noted above, the model (i.e. Model#X 901) is currently deployed at a base station. Since the core network node performing the AI / ML monitoring is aware of the deployed model 901 (e.g., based the active model ID associated with the base station) and GT information 903 of a PRU, it can perform monitoring of Model#X on behalf of the base station.

[0171] Here, where the model is deployed at a base station, the core network (e.g. the LMF) first assigns multiple registered reference UEs or PRUs to perform UL SRS transmission to the designated base station. The base station receives these SRS signals and reports 902 the raw received SRS waveform or a channel response (a post-production of the raw received SRS waveform) or positioning measurement for each SRS transmission to the core network. Then the core network node generates a positioning measurement 905 by feeding the received SRS waveform 902 as model input to Model#X 901. If the input is a positioning measurement then the core network node generates an updated positioning measurement 905. The output 905 will be further used to compare with a positioning measurement 904 based on the GT information 903 (e.g. GT PRU location or GT positioning measurement) stored in the core network. The different parameters from the comparison results are then grouped in metrics, e.g., PM#X 906.

[0172] As those skilled in the art would appreciate, where the model is deployed at a base station, the procedure is very similar to that described above. The main differences are that the core network communicates with the UE at which the model is deployed, and receives positioning data (and the parameters of the model if required) from that UE. Here, the positioning data / measurements may be determined by the UE based on PRS received from its serving base station.

[0173] In the event of invalidity, and as described above, the core network may send a one-bit piece of information, for example a flag indicating the invalidity, to the base station. If invalid, the core network may indicate to the base station that it is to update or fine-tune the model, or that the base station is to replace the model entirely with a new AI / ML model.

[0174] An example of how the core network can carry out model selection and replacement in accordance with at least some arrangements of embodiments of the present technique illustrated in Figure 10. This allows the core network to compare between multiple models and choose a new model to be deployed at the base station, among multiple models within a model pool 1001 sharing similar functionality as the currently deployed model. In the particular case of the example of Figure 10, the model pool 1001 comprises three models {Model#X, Model#Y and Model#Z}. By performing PM calculations individually using a same received SRS waveform 1002 and GT information 1003 (e.g. that relates to a PRU), different models may result in different PM values 1004, such as {PM#X, PM#Y and PM#Z}. {PM#X, PM#Y, PM#Z} can be further used as a criteria to identify a new model to be deployed, such as by identifying the model with smallest PM value (e.g. the smallest difference between the output of that model and the GT information).

[0175] Implementing PM calculation in the core network has several technical benefits, compared to its implementation in a base station (as described in [4] for example) or UE. Firstly, this allows third-party UEs (i.e. those authenticated / registered by the application but not by the network operator) to be used as a reference UE, which increases the amount of available positioning data that can be used for model monitoring, thereby improving the monitoring accuracy. Secondly, this method can be transparent to base station / UE, as they do not know the GT information and the calculation of the PM is performed solely in the code network. This ensures confidentiality of data for the application server. Thirdly, this method enables model selection within the core network. Typically, multiple models may serve the same functionality. By receiving positioning data with a corresponding GT label, the core network can search for and identify the most suitable model to be deployed in the base station (or UE) by performing a PM calculation for each model. This ensures optimal performance and efficient model deployment. Finally, implementing PM calculation in the core network enables the burden of model monitoring to be lifted from the UE and base station, meaning that they can engage in communication more efficiently and reduce power consumption.

[0176] Signalling diagrams of examples of the operation described herein illustrating the interaction between a core network or other node (e.g. LMF), a base station (e.g. gNB), and a UE according to variously described arrangements of embodiments of the present technique are shown in Figures 11 and 12. Figure 11 depicts the AI / ML positioning Case 3a where the Al model is deployed at the base station, while Figure 12 depicts the AI / ML positioning Case 1 or Case 2a where the Al model is deployed at the UE.

[0177] In Figure 11, the LMF begins the PM (model monitoring) process, and sends a trigger to the base station to provide positioning information. The base station in turn triggers the UE to transmit SRS towards the base station, which then transmits the positioning information to the LMF based on the received SRS. The base station may also indicate the parameters or index of the AI / ML model to the LMF. The LMF then derives the GT information, for example based on parameters of the UE and / or one or more other reference UEs, and calculates positioning data using the received positioning data from the base station as an input AI / ML model. Finally, the LMF compares this to the GT information to determine validity of the AI / ML model, and if invalid, may indicate to the base station that it is required to update or replace its AI / ML model.

[0178] Similarly, in Figure 12, the LMF begins the PM (model monitoring) process, and sends a trigger to the UE to provide positioning information. The UE determines such positioning information based on PRS received from its serving base station, which it then transmits to the LMF. The UE may also indicate the parameters or index of the AI / ML model to the LMF. The LMF then derives the GT information, for example based on parameters of the UE (which may be a PRU) and / or one or more other reference UEs, and calculates positioning data using the received positioning data from the UE as an input AI / ML model. Finally, the LMF compares this to the GT information to determine validity of the AI / ML model, and if invalid, may indicate to the UE that it is required to update or replace its AI / ML model. In the example of Figure 12, the LMF may be configured to provide the UE with the indication of Al model validity within a specified maximum time T max from determining that the LMF is to perform the AI / ML model monitoring.

[0179] Embodiments of the present technique therefore define a proposed method to be applied for AI / ML monitoring which is generally described above in respect of being for positioning purposes. As noted above however, in principle, such embodiments of the present technique can equally be applied to other AI / ML cases, such as AI / ML for beam management, and AI / ML for CSI feedback enhancements. In these two cases, the LMF may not be the node that performs the AI / ML model monitoring, but instead this may be performed by another computation node in the core network. As those skilled in the art would appreciate, positioning data (i.e., positioning measurements) would also not be used in these cases, but will instead be replaced by channel information data, such as CSI report, RSRP, signal to noise ratio (SNR), CIR measurement, etc.

[0180] Figure 13 shows a flow diagram illustrating an example process of communications in a communications system in accordance with embodiments of the present technique. The process shown by Figure 13 is specifically a method of operating a first apparatus, the first apparatus being either a node of a core network or a server.

[0181] The method begins in step S 1301. The method comprises, in step S1302, determining that the first apparatus is to perform a performance assessment of an artificial intelligence, Al, model used by a second apparatus to perform a task, wherein the second apparatus is either a communications device or an infrastructure equipment of a radio access network. In step S1303, the process comprises transmitting, to the second apparatus, a request to perform measurements and to report the performed measurements to the first apparatus. The process then comprises, in step SI 304, receiving, from the second apparatus, an indication of the performed measurements. In step S1305, the method comprises performing a performance metric calculation by comparing an output of the Al model when the performed measurements are used as an input to the Al model with reference information. Following this, in step S1306, the method comprises transmitting, to the second apparatus based on the calculated performance metric, an indication of a validity status of the Al model. The process ends in step S1307.

[0182] Those skilled in the art would appreciate that the method shown by Figure 13 and the procedures illustrated by Figures 8 to 12 may be adapted in accordance with embodiments of the present technique. For example, other intermediate steps may be included in such methods / procedures, or the steps may be performed in any logical order. Though embodiments of the present technique have been described largely by way of the example communications system shown in Figure 7, it would be clear to those skilled in the art that they could be equally applied to other systems to those described herein, provided that these are within the scope of the claims.

[0183] Those skilled in the art would further appreciate that such infrastructure equipment and / or communications devices as herein defined may be further defined in accordance with the various arrangements and embodiments discussed in the preceding paragraphs. It would be further appreciated by those skilled in the art that such infrastructure equipment and communications devices as herein defined and described may form part of communications systems other than those defined by the present disclosure, provided that these are within the scope of the claims. The following numbered paragraphs provide further example aspects and features of the present technique:

[0184] Paragraph 1. A method of operating a first apparatus, the first apparatus being either a node of a core network or a server, the method comprising determining that the first apparatus is to perform a performance assessment of an artificial intelligence, Al, model used by a second apparatus to perform a task, wherein the second apparatus is either a communications device or an infrastructure equipment of a radio access network, transmitting, to the second apparatus, a request to perform measurements and to report the performed measurements to the first apparatus, receiving, from the second apparatus, an indication of the performed measurements, performing a performance metric calculation by comparing an output of the Al model when the performed measurements are used as an input to the Al model with reference information, and transmitting, to the second apparatus based on the calculated performance metric, an indication of a validity status of the Al model.

[0185] Paragraph 2. A method according to Paragraph 1, comprising receiving, from a third apparatus, a triggering command, wherein the step of determining that the first apparatus is to perform the performance assessment of the Al model is based on receiving the triggering command.

[0186] Paragraph 3. A method according to Paragraph 1 or Paragraph 2, wherein the step of determining that the first apparatus is to perform the performance assessment of the Al model is performed at one or more preconfigured time occasions.

[0187] Paragraph 4. A method according to any of Paragraphs 1 to 3, wherein the reference information is ground truth, GT, information associated with a reference communications device, wherein the reference communications device is either a communications device with a current position that is known to the first apparatus or a communications device with sufficient statistical measurement information that has been obtained by the first apparatus.

[0188] Paragraph 5. A method according to Paragraph 4, wherein the GT information is the known current position of the reference communications device.

[0189] Paragraph 6. A method according to Paragraph 4, wherein the GT information is an estimated current position of the reference communications device determined by the first apparatus based on a previous position of the reference communications device and / or one or more other communications devices. Paragraph 7. A method according to any of Paragraphs 1 to 6, wherein the reference information is stored at the first apparatus.

[0190] Paragraph 8. A method according to any of Paragraphs 1 to 7, wherein the received indication of the performed measurements is positioning data received by the first apparatus from the second apparatus. Paragraph 9. A method according to any of Paragraphs 1 to 7, wherein the received indication of the performed measurements is channel estimation data received by the first apparatus from the second apparatus.

[0191] Paragraph 10. A method according to any of Paragraphs 1 to 9, comprising receiving, from the second apparatus, information associated with the Al model, wherein the information associated with the Al model comprises one or more parameters of the Al model.

[0192] Paragraph 11. A method according to any of Paragraphs 1 to 10, wherein the indication of the validity status of the Al model comprises a flag that indicates either that the Al model is valid or that the Al model is invalid.

[0193] Paragraph 12. A method according to any of Paragraphs 1 to 11, wherein the step of transmitting the indication of the validity status of the Al model is dependent on whether the calculated performance metric satisfies a predetermined condition. Paragraph 13. A method according to Paragraph 12, wherein if the calculated performance metric does not satisfy the predetermined condition, the step of transmitting the indication of the validity status of the Al model comprises transmitting, to the second apparatus, a request to update the Al model.

[0194] Paragraph 14. A method according to Paragraph 12, wherein if the calculated performance metric does not satisfy the predetermined condition, the step of transmitting the indication of the validity status of the Al model comprises transmitting, to the second apparatus, a request to change the Al model from a first Al model to a second Al model.

[0195] Paragraph 15. A method according to Paragraph 14, wherein the request to change the Al model comprises an indication of an index of the second Al model.

[0196] Paragraph 16. A method according to Paragraph 14 or Paragraph 15, wherein the request to change the Al model comprises an indication of one or more parameters of the second Al model.

[0197] Paragraph 17. A method according to any of Paragraphs 14 to 16, wherein the step of performance metric calculation further comprises performing a performance metric calculation by comparing the reference information with an output of each of one or more further Al models when the performed measurements are used as an input to that Al model, wherein the second Al model is one of the further Al models.

[0198] Paragraph 18. A method according to any of Paragraphs 1 to 17, wherein the first apparatus is a location and management function, LMF, node of the core network.

[0199] Paragraph 19 A method according to any of Paragraphs 1 to 18, wherein the first apparatus is configured to transmit the indication of the validity status of the Al model within a specified time from determining that the first apparatus is to perform the performance assessment of the Al model.

[0200] Paragraph 20. A first apparatus, the first apparatus being either a node of a core network or a server, the first apparatus comprising a transceiver configured to transmit and to receive signals, a controller configured in combination with the transceiver to determine that the first apparatus is to perform a performance assessment of an artificial intelligence, Al, model used by a second apparatus to perform a task, wherein the second apparatus is either a communications device or an infrastructure equipment of a radio access network, to transmit, to the second apparatus, a request to perform measurements and to report the performed measurements to the first apparatus, to receive, from the second apparatus, an indication of the performed measurements, to perform a performance metric calculation by comparing an output of the Al model when the performed measurements are used as an input to the Al model with reference information, and to transmit, to the second apparatus based on the calculated performance metric, an indication of a validity status of the Al model.

[0201] Paragraph 21. A method of operating a second apparatus, the second apparatus being either a communications device or an infrastructure equipment of a radio access network and configured to communicate with the other of the communications device or the infrastructure equipment, the method comprising receiving, from a first apparatus, a request to perform measurements and to report the performed measurements to the first apparatus, the first apparatus being either a node of a core network or a server, performing the requested measurements, transmitting, to the first apparatus, an indication of the performed measurements, and receiving, from the first apparatus, an indication of a validity status of an artificial intelligence, Al, model used by the second apparatus to perform a task. Paragraph 22. A second apparatus, the second apparatus being either a communications device or an infrastructure equipment of a radio access network and configured to communicate with the other of the communications device or the infrastructure equipment, the second apparatus comprising a transceiver configured to transmit and to receive signals, a controller configured in combination with the transceiver to receive, from a first apparatus, a request to perform measurements and to report the performed measurements to the first apparatus, the first apparatus being either a node of a core network or a server, to perform the requested measurements, to transmit, to the first apparatus, an indication of the performed measurements, and to receive, from the first apparatus, an indication of a validity status of an artificial intelligence, Al, model used by the second apparatus to perform a task.

[0202] Paragraph 23. A wireless communications system comprising a first apparatus according to Paragraph 20 and a second apparatus according to Paragraph 22.

[0203] Paragraph 24. A computer program comprising instructions which, when loaded onto a computer, cause the computer to perform a method according to any of Paragraphs 1 to 19 or Paragraph 21.

[0204] Paragraph 25. A non-transitory computer-readable storage medium storing a computer program according to Paragraph 24.

[0205] It will be appreciated that the above description for clarity has described embodiments with reference to different functional units, circuitry and / or processors. However, it will be apparent that any suitable distribution of functionality between different functional units, circuitry and / or processors may be used without detracting from the embodiments.

[0206] Described embodiments may be implemented in any suitable form including hardware, software, firmware or any combination of these. Described embodiments may optionally be implemented at least partly as computer software running on one or more data processors and / or digital signal processors. The elements and components of any embodiment may be physically, functionally and logically implemented in any suitable way. Indeed, the functionality may be implemented in a single unit, in a plurality of units or as part of other functional units. As such, the disclosed embodiments may be implemented in a single unit or may be physically and functionally distributed between different units, circuitry and / or processors.

[0207] Although the present disclosure has been described in connection with some embodiments, it is not intended to be limited to the specific form set forth herein. Additionally, although a feature may appear to be described in connection with particular embodiments, one skilled in the art would recognise that various features of the described embodiments may be combined in any manner suitable to implement the technique.

[0208] References

[0209] [1] Holma H. and Toskala A, “LTE for UMTS OFDMA and SC-FDMA based radio access”, John Wiley and Sons, 2009.

[0210] [2] TR 38.913, “3rdGeneration Partnership Project; Technical Specification Group Radio Access Network; Study on Scenarios and Requirements for Next Generation Access Technologies

[0211] (Release 14)”, 3GPP, vl4.3.0, August 2017.

[0212] [3] TR 38.843, “Study on Artificial Intelligence (AI)ZMachine Learning (ML) for NR air interface (Release 18)”, 3GPP, vl8.0.0, December 2023.

[0213] [4] United Kingdom patent application number 2402279.0.

Claims

CLAIMSWhat is claimed is:

1. A method of operating a first apparatus, the first apparatus being either a node of a core network or a server, the method comprising determining that the first apparatus is to perform a performance assessment of an artificial intelligence, Al, model used by a second apparatus to perform a task, wherein the second apparatus is either a communications device or an infrastructure equipment of a radio access network, transmitting, to the second apparatus, a request to perform measurements and to report the performed measurements to the first apparatus, receiving, from the second apparatus, an indication of the performed measurements, performing a performance metric calculation by comparing an output of the Al model when the performed measurements are used as an input to the Al model with reference information, and transmitting, to the second apparatus based on the calculated performance metric, an indication of a validity status of the Al model.

2. A method according to Claim 1, comprising receiving, from a third apparatus, a triggering command, wherein the step of determining that the first apparatus is to perform the performance assessment of the Al model is based on receiving the triggering command.

3. A method according to Claim 1, wherein the step of determining that the first apparatus is to perform the performance assessment of the Al model is performed at one or more preconfigured time occasions.

4. A method according to Claim 1, wherein the reference information is ground truth, GT, information associated with a reference communications device, wherein the reference communications device is either a communications device with a current position that is known to the first apparatus or a communications device with sufficient statistical measurement information that has been obtained by the first apparatus.

5. A method according to Claim 4, wherein the GT information is the known current position of the reference communications device.

6. A method according to Claim 4, wherein the GT information is an estimated current position of the reference communications device determined by the first apparatus based on a previous position of the reference communications device and / or one or more other communications devices.

7. A method according to Claim 1, wherein the reference information is stored at the first apparatus.

8. A method according to Claim 1, wherein the received indication of the performed measurements is positioning data received by the first apparatus from the second apparatus.

9. A method according to Claim 1, wherein the received indication of the performed measurements is channel estimation data received by the first apparatus from the second apparatus.

10. A method according to Claim 1, comprising receiving, from the second apparatus, information associated with the Al model, wherein the information associated with the Al model comprises one or more parameters of the Al model.

11. A method according to Claim 1, wherein the indication of the validity status of the Al model comprises a flag that indicates either that the Al model is valid or that the Al model is invalid.

12. A method according to Claim 1, wherein the step of transmitting the indication of the validity status of the Al model is dependent on whether the calculated performance metric satisfies a predetermined condition.

13. A method according to Claim 12, wherein if the calculated performance metric does not satisfy the predetermined condition, the step of transmitting the indication of the validity status of the Al model comprises transmitting, to the second apparatus, a request to update the Al model.

14. A method according to Claim 12, wherein if the calculated performance metric does not satisfy the predetermined condition, the step of transmitting the indication of the validity status of the Al model comprises transmitting, to the second apparatus, a request to change the Al model from a first Al model to a second Al model.

15. A method according to Claim 14, wherein the request to change the Al model comprises an indication of an index of the second Al model.

16. A method according to Claim 14, wherein the request to change the Al model comprises an indication of one or more parameters of the second Al model.

17. A method according to Claim 14, wherein the step of performance metric calculation further comprises performing a performance metric calculation by comparing the reference information with an output of each of one or more further Al models when the performed measurements are used as an input to that Al model, wherein the second Al model is one of the further Al models.

18. A method according to Claim 1, wherein the first apparatus is a location and management function, LMF, node of the core network.19 A method according to Claim 1, wherein the first apparatus is configured to transmit the indication of the validity status of the Al model within a specified time from determining that the first apparatus is to perform the performance assessment of the Al model.

20. A first apparatus, the first apparatus being either a node of a core network or a server, the first apparatus comprising a transceiver configured to transmit and to receive signals, a controller configured in combination with the transceiver to determine that the first apparatus is to perform a performance assessment of an artificial intelligence, Al, model used by a second apparatus to perform a task, wherein the second apparatus is either a communications device or an infrastructure equipment of a radio access network, to transmit, to the second apparatus, a request to perform measurements and to report the performed measurements to the first apparatus, to receive, from the second apparatus, an indication of the performed measurements,to perform a performance metric calculation by comparing an output of the Al model when the performed measurements are used as an input to the Al model with reference information, and to transmit, to the second apparatus based on the calculated performance metric, an indication of a validity status of the Al model.

21. A method of operating a second apparatus, the second apparatus being either a communications device or an infrastructure equipment of a radio access network and configured to communicate with the other of the communications device or the infrastructure equipment, the method comprising receiving, from a first apparatus, a request to perform measurements and to report the performed measurements to the first apparatus, the first apparatus being either a node of a core network or a server, performing the requested measurements, transmitting, to the first apparatus, an indication of the performed measurements, and receiving, from the first apparatus, an indication of a validity status of an artificial intelligence, Al, model used by the second apparatus to perform a task.

22. A second apparatus, the second apparatus being either a communications device or an infrastructure equipment of a radio access network and configured to communicate with the other of the communications device or the infrastructure equipment, the second apparatus comprising a transceiver configured to transmit and to receive signals, a controller configured in combination with the transceiver to receive, from a first apparatus, a request to perform measurements and to report the performed measurements to the first apparatus, the first apparatus being either a node of a core network or a server, to perform the requested measurements, to transmit, to the first apparatus, an indication of the performed measurements, and to receive, from the first apparatus, an indication of a validity status of an artificial intelligence, Al, model used by the second apparatus to perform a task.

23. A wireless communications system comprising a first apparatus according to Claim 20 and a second apparatus according to Claim 22.

24. A computer program comprising instructions which, when loaded onto a computer, cause the computer to perform a method according to Claim 1 or Claim 21.

25. A non-transitory computer-readable storage medium storing a computer program according to Claim 24.

Images