Data collection quality indication in AI / ML positioning

Abstract

Example embodiments of the present disclosure relate to data collection quality indication in artificial intelligence / machine learning AI / ML positioning. A first device for communication obtains a monitoring metric for an AI / ML model configured for direct positioning or assisted positioning of a terminal device within a communication network, the monitoring metric is determined based on an error between a localization reasoning result of the AI / ML model for the model input and a truth localization tag within the monitoring data set for the model input; determining a monitoring quality criterion for the monitoring data set based on a spatial label analysis (SLA) indicator and a minimum inference error (MIE) indicator of the monitoring data set; and determining an action to be applied to the AI / ML model in accordance with a determination that the monitored quality criterion is not satisfied by the monitored metric.

Description

CROSS-REFERENCE TO RELATED APPLICATIONS

[0001] This application claims priority to and the benefit of Finnish application No. 20245161, filed on February 14, 2024, the contents of which are incorporated herein by reference in their entirety. Technical Field

[0002] Various example embodiments of the present disclosure relate generally to the field of telecommunications, and in particular, to methods, devices, apparatus, and computer-readable storage media for data collection quality indication in artificial intelligence / machine learning (AI / ML) positioning. Background Art

[0003] In the telecommunications industry, artificial intelligence / machine learning (AI / ML) models are already being adopted in telecom systems to improve performance. For example, the 3rd Generation Partnership Project (3GPP) Release 18 identified the first study of artificial intelligence (AI) / machine learning (ML) for New Radio (NR). Consequently, 3GPP Release 19 initiated a work item on AI / ML for NR. The goal is to explore and standardize the benefits of evaluating the characteristics of collected datasets used in AI / ML positioning. It also aims to ensure fair monitoring metrics for ground-truth performance monitoring. Summary of the Invention

[0004] In a first aspect of the present disclosure, a first device is provided. The first device includes at least one processor; and at least one memory storing instructions, which, when executed by the at least one processor, causes the first device to at least: obtain a monitoring metric for an artificial intelligence / machine learning AI / ML model, the AI / ML model being configured for direct positioning or assisted positioning of a terminal device within a communication network, the monitoring metric being determined based on the following error: an error between a positioning inference result of the AI / ML model for a model input and a true positioning label for the model input in a monitoring dataset; determine a monitoring quality criterion for the monitoring dataset based on a spatial label analysis SLA indicator and a minimum inference error (MIE) indicator of the monitoring dataset, the SLA indicator indicating the spatial distribution of the monitoring dataset, and the MIE indicator indicating the inference error obtained by the AI / ML model using the monitoring dataset during a training phase; and determine an action to be applied to the AI / ML model based on determining that the monitoring quality criterion is not satisfied by the monitoring metric.

[0005] In a second aspect of the present disclosure, a method is provided. The method includes: obtaining a monitoring metric for an artificial intelligence / machine learning AI / ML model, the AI / ML model being configured for direct positioning or assisted positioning of a terminal device within a communication network, the monitoring metric being determined based on the following error: an error between a positioning inference result of the AI / ML model for a model input and a true positioning label for the model input in a monitoring dataset; determining a monitoring quality criterion for the monitoring dataset based on a spatial label analysis of an SLA indicator and a minimum inference error (MIE) indicator of the monitoring dataset, the SLA indicator indicating a spatial distribution of the monitoring dataset, the MIE indicator indicating an inference error obtained by the AI / ML model using the monitoring dataset during a training phase; and determining an action to be applied to the AI / ML model based on determining that the monitoring quality criterion is not satisfied by the monitoring metric.

[0006] In a third aspect of the present disclosure, a first apparatus is provided. The first apparatus includes: a component for obtaining a monitoring metric for an artificial intelligence / machine learning AI / ML model, the AI / ML model being configured for direct positioning or assisted positioning of a terminal device within a communication network, the monitoring metric being determined based on the following error: an error between a positioning inference result of the AI / ML model for a model input and a true positioning label for the model input in a monitoring dataset; a component for determining a monitoring quality criterion for the monitoring dataset based on a spatial label analysis of an SLA indicator and a minimum inference error (MIE) indicator of the monitoring dataset, the SLA indicator indicating the spatial distribution of the monitoring dataset, the MIE indicator indicating the inference error obtained by the AI / ML model using the monitoring dataset during a training phase; and a component for determining an action to be applied to the AI / ML model based on determining that the monitoring quality criterion is not satisfied by the monitoring metric.

[0007] In a fourth aspect of the present disclosure, a computer-readable medium is provided, wherein the computer-readable medium includes instructions stored thereon, and the instructions are used to cause a device to at least execute the method according to the second aspect.

[0008] It should be understood that the invention summary is not intended to identify the key features or essential features of the embodiments of the present disclosure, nor is it intended to limit the scope of the present disclosure. Other features of the present disclosure will become readily understood through the following description. BRIEF DESCRIPTION OF THE DRAWINGS

[0009] Some example embodiments will now be described with reference to the accompanying drawings, in which:

[0010] Figure 1 shows an example communication environment in which example embodiments of the present disclosure may be implemented;

[0011] Figure 2A flowchart illustrating a method implemented at a location management function (LMF) device according to some example embodiments of the present disclosure is shown;

[0012] Figures 3A to 3C Examples of different data spatial distributions are shown;

[0013] Figure 4 A flowchart illustrating a performance monitoring method implemented at a first device according to some example embodiments of the present disclosure is shown;

[0014] Figure 5 shows a signaling flow for a performance monitoring process according to some example embodiments of the present disclosure;

[0015] Figure 6 shows a signaling flow for a performance monitoring process according to some example embodiments of the present disclosure;

[0016] Figure 7 a flow chart illustrating a method implemented at a first device according to some example embodiments of the present invention;

[0017] Figure 8 A simplified block diagram of a device suitable for implementing an example embodiment of the present disclosure is shown.

[0018] Throughout the drawings, the same or similar reference numerals refer to the same or similar elements. DETAILED DESCRIPTION

[0019] The principles of the present disclosure will now be described with reference to some example embodiments. It should be understood that these embodiments are described for illustrative purposes only and to help those skilled in the art understand and implement the present disclosure without implying any limitation on the scope of the present disclosure. In addition to the embodiments described below, the embodiments described herein can be implemented in various ways.

[0020] In the following description and claims, unless defined otherwise, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this disclosure belongs.

[0021] References in this disclosure to "one embodiment," "an embodiment," "example embodiment," etc., indicate that the described embodiment may include a particular feature, structure, or characteristic, but not every embodiment necessarily includes the particular feature, structure, or characteristic. Furthermore, these phrases are not necessarily referring to the same embodiment. Furthermore, when a particular feature, structure, or characteristic is described in connection with an embodiment, it is within the knowledge of those skilled in the art to affect such feature, structure, or characteristic in other embodiments, whether or not explicitly described.

[0022] It should be understood that, although the terms "first", "second", etc. may be used herein to describe various elements, these elements should not be limited by these terms. These terms are only used to distinguish one element from another, and they do not limit the order of nouns. For example, without departing from the scope of the example embodiments, the first element may be referred to as the second element, and similarly, the second element may be referred to as the first element. As used herein, the term "and / or" includes any and all combinations of one or more of the listed terms.

[0023] As used herein, “at least one of: ” and “” and similar expressions, where a list of two or more elements is connected by “and” or “or”, means at least any one of the elements, or at least any two or more of the elements, or at least all of the elements.

[0024] As used herein, unless explicitly stated otherwise, performing a step "in response to A" does not mean that the step is performed immediately after "A" occurs, but may include one or more intermediate steps.

[0025] The terms used herein are used only to describe particular embodiments and are not intended to limit the example embodiments. As used herein, the singular forms "a," "an," and "the" are intended to include the plural forms as well, unless the context clearly indicates otherwise. It should also be understood that the terms "comprise," "comprising," "having," "having," "includes," and / or "including," when used herein, specify the presence of stated features, elements, and / or components, etc., but do not preclude the presence or addition of one or more other features, elements, components, and / or combinations thereof.

[0026] As used in this application, the term "circuitry" may refer to one or more or all of the following: (a) hardware circuit implementation only (such as implementation only in analog and / or digital circuitry); and (b) a combination of hardware circuitry and software, such as (if applicable): (i) a combination of analog and / or digital hardware circuitry and software / firmware; and (ii) any portion of a hardware processor (including a digital signal processor) with software, software, and memory that work together to enable a device (such as a mobile phone or server) to perform various functions; and (c) a hardware circuit and / or processor such as a microprocessor or a portion of a microprocessor that requires software (e.g., firmware) to operate, but that software may not be present when it is not required for operation.

[0027] This definition of circuitry applies to all uses of the term in this application, including in any claims. As another example, as used in this application, the term circuitry also covers an implementation of merely a hardware circuit, or a processor (or multiple processors), or a portion of a hardware circuit or processor and its (or their) accompanying software and / or firmware. The term circuitry also covers (e.g., if applicable to a particular claim element) a baseband integrated circuit or processor integrated circuit for a mobile device, or a similar integrated circuit in a server, cellular network device, or other computing or network device.

[0028] As used herein, the term "communication network" refers to a network that complies with any suitable communication standard, such as New Radio (NR), Long Term Evolution (LTE), Advanced LTE (LTE-A), Wideband Code Division Multiple Access (WCDMA), High Speed Packet Access (HSPA), Narrowband Internet of Things (NB-IoT), etc. In addition, the communication between the terminal device and the network device in the communication network can be carried out according to any suitable generation communication protocol, including but not limited to first generation (1G), second generation (2G), 2.5G, 2.75G, third generation (3G), fourth generation (4G), 4.5G, fifth generation (5G), sixth generation (6G) communication protocols and / or any other protocol currently known or developed in the future. The embodiments of the present disclosure can be applied to various communication systems. In view of the rapid development of communications, there are of course future types of communication technologies and systems that can implement the present disclosure. The scope of the present disclosure should not be limited to the above-mentioned systems.

[0029] As used herein, the term "network device" refers to a node in a communication network via which a terminal device accesses the network and receives services from it. A network device may refer to a base station (BS) or an access point (AP), for example, a Node B (Node B or NB), an evolved Node B (e Node B or eNB), a NR NB (also known as gNB), a remote radio unit (RRU), a radio head (RH), a remote radio head (RRH), a relay, an integrated access and backhaul (IAB) node, a low-power node such as a femto, a micro, a non-network (NTN) or a non-network device (such as a satellite network device), a low-earth orbit (LEO) satellite and a geosynchronous earth orbit (GEO) satellite, an aircraft network device, depending on the terminology and technology applied. In some example embodiments, a radio access network (RAN) split architecture includes a centralized unit (CU) and a distributed unit (DU) at an IAB donor node. An IAB node includes a mobile terminal (IAB-MT) portion that behaves like a UE towards a parent node, and the DU portion of the IAB node behaves like a base station towards a next-hop IAB node. In some examples, the network device may include a core network (CN) device. The core network includes one or more core network devices or devices structured with hardware and software components. The features of these components can be substantially similar to those described with respect to the UE, network device and / or host, so that their description is generally applicable to the corresponding components of the core network device. Example core network devices include one or more functions of a location management function (LMF), a mobile switching center (MSC), a mobility management entity (MME), a home subscriber server (HSS), an access and mobility management function (AMF), a session management function (SMF), an authentication server function (AUSF), a subscription identifier dehiding function (SIDF), a unified data management (UDM), a security edge protection proxy (SEPP), a network exposure function (NEF) and / or a user plane function (UPF).

[0030] The term "terminal device" refers to any terminal device capable of wireless communication. As an example and not limitation, a terminal device may also be referred to as a communication device, user equipment (UE), user station (SS), portable user station, mobile station (MS) or access terminal (AT). Terminal devices may include, but are not limited to, mobile phones, cellular phones, smart phones, voice over IP (VoIP) phones, wireless local loop phones, tablet computers, wearable terminal devices, personal digital assistants (PDAs), portable computers, desktop computers, image capture terminal devices (such as digital cameras, game terminal devices, music storage and playback devices), vehicle-mounted wireless terminal devices, wireless endpoints, mobile stations, laptop embedded devices (LEEs), laptop devices (LMEs), USB dongles, smart devices, wireless customer premises equipment (CPEs), Internet of Things (IoT) devices, watches or other wearable devices, head-mounted displays (HMDs), vehicles, drones, medical devices and applications (e.g., remote surgery), industrial devices and applications (e.g., robots and / or other wireless devices operating in the context of industrial and / or automated processing chains), consumer electronic devices, devices operating on commercial and / or industrial wireless networks, etc. The terminal device may also correspond to the mobile terminal (MT) portion of an IAB node (eg, a relay node).In the following description, the terms "terminal device," "communication device," "terminal," "user equipment," and "UE" may be used interchangeably.

[0031] As used herein, the terms "resources," "transmission resources," "resource blocks," "physical resource blocks" (PRBs), "uplink resources," or "downlink resources" may refer to any resources used to perform communications, for example, between a terminal device and a network device, such as resources in the time domain, resources in the frequency domain, resources in the spatial domain, resources in the code domain, or any other combination of time, frequency, space, and / or code domain resources that enable communications. Hereinafter, unless explicitly stated otherwise, resources in the frequency domain and the time domain will be used as examples of transmission resources for describing some example embodiments of the present disclosure. Note that the example embodiments of the present disclosure are equally applicable to other resources in other domains.

[0032] As used herein, the term "model" refers to the association between input and output learned from training data, and therefore a corresponding output can be generated for a given input after training. The generation of the model can be based on machine learning (ML) technology. Machine learning technology can also refer to artificial intelligence (AI) technology. Typically, a machine learning model can be constructed that receives input information and makes predictions based on the input information. For example, a classification model can predict the category of input information in a predetermined set of categories. As used herein, a "model" can also be referred to as a "machine learning model", "learning model", "machine learning network" or "learning network", which are used interchangeably herein.

[0033] To facilitate understanding of the terminology, here are some definitions for the AI / ML term list.

[0034] AI / ML model: A data-driven algorithm that applies AI / ML techniques to generate a set of outputs based on a set of inputs.

[0035] AI / ML model delivery: A general term referring to the delivery of AI / ML models from one entity to another in any manner. Note: Entities can refer to network nodes / functions (e.g., gNB, LMF, etc.), UEs, dedicated servers, etc.

[0036] AI / ML model inference: The process of using a trained AI / ML model to produce a set of outputs based on a set of inputs.

[0037] AI / ML model testing: A sub-process of training to evaluate the performance of the final AI / ML model using a dataset different from that used for model training and validation. Unlike AI / ML model validation, testing does not assume subsequent tuning of the model.

[0038] AI / ML model training: The process of training an AI / ML model in a data-driven manner [by learning the input / output relationship] and obtaining the trained AI / ML model for inference.

[0039] Data Collection: The process of collecting data by network nodes, management entities, or UEs for the purpose of AI / ML model training, data analysis, and inference.

[0040] Function identification: The process / method for identifying AI / ML functions for common understanding between the network and the UE. Note: Information about AI / ML functions can be shared during function identification. Where the AI / ML function resides depends on the specific use case and sub-use case.

[0041] Network-side (AI / ML) model: an AI / ML model whose reasoning is performed entirely at the network.

[0042] Semi-supervised learning: The process of training a model using a mixture of labeled and unlabeled data.

[0043] Supervised learning: The process of training a model from inputs and their corresponding labels.

[0044] UE-side (AI / ML) model: an AI / ML model whose reasoning is performed entirely at the terminal device or UE.

[0045] Unsupervised learning: The process of training a model without labeled data.

[0046] Figure 1 An example communication environment 100 is shown in which example embodiments of the present disclosure may be implemented. It should be understood that the elements shown in the communication environment 100 are intended to represent the primary functionality provided within the system. Figure 1 The blocks shown in refer to specific elements of the communication network that provide these primary functions. However, other network elements may be used to implement some or all of the primary functions represented. Furthermore, it should be understood that not all functions of the communication network are present in Figure 1 Rather, functions that facilitate explanation of the exemplary embodiments are represented. Figure 1 The number of elements shown in is also for illustration purposes only, and any number of elements may be present.

[0047] As shown, communication environment 100 includes multiple communication devices, including terminal device 110 and network device 120, such as radio access network (RAN) equipment. The service area of network device 120 can be called a cell. Terminal device 110 and network device 120 can operate in the RAN. Although one terminal device (e.g., terminal device 110) is shown, more or fewer terminal devices may exist within the service area of the network device, and there may also be more network devices serving terminal devices in communication environment 100.

[0048] Hereinafter, for illustrative purposes, some example embodiments of the terminal device 110 operating as a terminal device and the network device 120 operating as a network device are described. However, in some example embodiments, the operations described in conjunction with the terminal device may be implemented at a network device or other device, and the operations described in conjunction with the network device may be implemented at a terminal device or other device.

[0049] In some exemplary embodiments, if terminal device 110 is a terminal device and network device 120 is a network device, the link from network device 120 to terminal device 110 is called a downlink (DL), and the link from terminal device 110 to network device 120 is called an uplink (UL). In the DL, network device 120 is a transmitting (TX) device (or transmitter), and terminal device 110 is a receiving (RX) device (or receiver). In the UL, terminal device 110 is a TX device (or transmitter), and network device 120 is an RX device (or receiver).

[0050] Communications in the communication environment 100 may be implemented according to any suitable communication protocol, including but not limited to first generation (1G), second generation (2G), third generation (3G), fourth generation (4G), fifth generation (5G), sixth generation (6G), etc. cellular communication protocols, wireless local area network communication protocols such as Institute of Electrical and Electronics Engineers (IEEE) 802.11, etc., and / or any other protocol currently known or developed in the future. In addition, communications may utilize any suitable wireless communication technology, including but not limited to: code division multiple access (CDMA), frequency division multiple access (FDMA), time division multiple access (TDMA), frequency division duplex (FDD), time division duplex (TDD), multiple input multiple output (MIMO), orthogonal frequency division multiple access (OFDM), discrete Fourier transform spread OFDM (DFT-s-OFDM), and / or any other technology currently known or developed in the future.

[0051] In some example embodiments, one or more AI / ML models 105-1, 105-2, ..., 105-M (collectively or individually referred to as AI / ML models 105) may be configured, for example, as functions. The AI / ML model(s) 105 may sometimes be referred to as AI models or ML models. Different AI / ML models 105 may be configured to implement different algorithms within the communication environment 100. In a positioning scenario, the AI / ML model 105 may be configured for direct or assisted positioning of a terminal device within a communication network. Inference, testing, training, and / or validation of the AI / ML model 105 may be performed at the terminal device 110, the network device 120, and / or other entities (such as the LMF 130). The AI / ML model 105 may be delivered from one entity to another in any manner. Delivery of the AI / ML model 105 over the air interface may be in a manner that is opaque to 3GPP signaling, with parameters of a model structure known to the receiving end, or as a new model with parameters. The delivery may include a complete model or a partial model.

[0052] LMF 130 may be an entity that implements location management services. LMF 130 may be a network node in the RAN or core network, or may be an external entity. In some example embodiments, LMF 130 may be able to communicate with network device 120 (e.g., a RAN network device).

[0053] For the purposes of AI / ML model training, data analysis, and inference, a data collection process is typically required to collect data from network devices, management entities, or terminal devices. Data collection for AI / ML models may be referred to as "AI / ML-related data collection" in this disclosure. For specific (multiple) AI / ML models, AI / ML-related data collection may be performed at the terminal device 110, the network device 120, and / or other entities (such as the LMF 130).

[0054] AI / ML positioning has been approved as part of the specification work. The positioning accuracy enhancements related to AI / ML positioning are shown in Table 1 below. Table 1

[0055] Here, the following sub-use cases are prioritized: 1) Case 1 with UE-side AI / ML model for direct positioning; Case 3a with gNB-side AI / ML model for assisted positioning; and Case 3b with LMF-side AI / ML model for direct positioning.

[0056] In addition, the performance monitoring in Table 2 below defines the following requests. Table 2

[0057] As a supplement, some protocols related to monitoring are listed in Table 3 below. Table 3

[0058] Some additional protocols related to ground truth label generation are shown in Table 4 below. Table 4

[0059] In some conventional technical solutions, a solution is provided that provides a monitoring device, a monitoring method, and a computer program for assessing the risk of causing an anomaly or failure. Specifically, in this solution, the monitoring device includes: a data acquisition unit for collecting system status information of a monitoring target system; a machine learning unit in which a normal state is learned in advance based on the system status information collected by the data acquisition unit; and a risk assessment unit for assessing the risk of the monitoring target system based on the current system status information collected by the data acquisition unit and the normal state pre-learned by the machine learning.

[0060] In another embodiment, a method for analyzing big data and monitoring performance based on machine learning is proposed, wherein data analysis management is easy. The method includes the following steps: receiving a front-end framework and a back-end framework; defining settings for a data collection method, a database, table information, field information, and an integrated database; defining methods and parameters for data analysis; analyzing information collected during a specified period according to set rules; defining an input layer, a hidden layer, a learning model, and parameters; performing machine learning; generating a report; collecting data corresponding to set keywords; classifying to match preset values and storing the data in a database; generating analysis information; performing machine learning on the collected data or the classified data; and generating and transmitting a report.

[0061] However, the above solutions do not consider the quality of the data collected for monitoring purposes, especially for the AI / ML based UE positioning case.

[0062] For AI / ML-based positioning, the availability of labeled data is essential, as it utilizes supervised learning methods. Additionally, several indicators are mentioned to ensure optimal inference performance. Specifically, for data collection purposes, dataset size and the distribution of labels (UE locations) that conform to a uniform distribution are indicators of dataset quality. While dataset size has a clear quantitative value based on the number of samples, there is no quantitative reference for evaluating uniform distribution.

[0063] Therefore, the following problems need to be addressed: how to evaluate and ensure the quantitative value of qualitative data collection for efficient ML model training for ML-based UE positioning use cases in real-world conditions, how to ensure ground-truth monitoring using dataset quality indicators, and how to guarantee fair monitoring metrics for ground-truth monitoring.

[0064] According to some exemplary embodiments of the present disclosure, a solution for quantifying the quality of data collection and its related signaling enhancement is provided. In addition, a data set quality indicator (also called a quantitative quality label indicator) is proposed to provide basic information to assist performance monitoring based on true values. In this solution, a first device obtains a monitoring metric for an AI / ML model and determines a monitoring quality criterion for a monitoring data set based on the data set quality indicator, the monitoring quality criterion including a spatial labeling analysis (SLA) indicator and a minimum inference error (MIE) indicator of the monitoring data set. If the monitoring quality criterion is not satisfied by the monitoring metric, the first device determines the action to be applied to the AI / ML model. The solution provides accurate and true value-based performance monitoring using a quantitative quality label indicator. In this regard, the quantitative quality label indicator plays a role in both data collection and performance monitoring.

[0065] The embodiments of the present invention are described in detail below with reference to the accompanying drawings.

[0066] Figure 2 A flow chart of a method 200 for obtaining a quality indication of a dataset for an AI / ML model according to some example embodiments of the present disclosure is shown. In some example embodiments, the method 200 may be implemented offline for a dataset for an AI / ML model. The method 200 may be implemented at any suitable entity, such as a network entity, for example, a LMF. The LMF may be Figure 1 In the following description, for illustrative purposes only, the method 200 is described from the perspective of the LMF. Figure 2 The steps in FIG are described as being performed by the LMF, but these steps may also be performed by other entities, such as the terminal device 110 (e.g., UE or PRU) and the network device 120 (e.g., gNB).

[0067] At box 205, the LMF 130 completes data collection. That is, one or more data sets are collected for AI / ML model training. The data set task of data collection is completed using any suitable conventional method, for example, via a PRU or a terminal device with a known location. Conventional methods also include potential methods for optimizing data collection. Here, the diversity of methods for data collection on the UE side or the network side can be considered, including various aspects of data hybrid data sets and other important aspects. The collected data set may include multiple data samples, each data sample including a model input to the AI / ML model, the model being configured for direct positioning or assisted positioning, and a corresponding true value positioning label for model input. The true value positioning label can be a direct positioning result or an assisted positioning result, depending on the type of AI / ML model.

[0068] The tasks related to collecting information from different sources are completed at block 205. Therefore, information such as the size of the dataset (in terms of the number of samples) is known and always fixed. However, there is no guarantee that the label distribution of the end devices represents a true hypothesis.

[0069] At box 210, in addition to the data size, the LMF 130 also calculates a spatial label analysis (SLA) indicator to quantify the quality of the collected dataset. At box 210, the collected dataset is analyzed to obtain an SLA indicator, for example, a spatial point process, and a quality metric is estimated. The SLA indicator can be calculated to indicate the spatial distribution of the collected dataset. The entire dataset is analyzed to determine whether the label distribution of the terminal devices in the collected dataset follows a uniform distribution, which can be indicated by the SLA indicator. The condition for ensuring high training performance is that the dataset must follow a uniform distribution, and the SLA indicator can be considered as a data quality indicator for a specific dataset. The SLA indicator can also be referred to as a quality indicator or a dataset quality indicator.

[0070] The SLA indicator may be based on a spatial point process and any other method that indicates the level of similarity of the label distribution between two different datasets. For example, "localized" collected measurements are examined as spatial point processes. Correlation tests may be applied to provide an indication of how these measurements are geographically distributed within the region of interest. In some example embodiments, the SLA indicator is determined based at least in part on the similarity between the distribution of the ground truth location labels in the monitoring dataset and a reference distribution. The reference distribution may be a uniform distribution, or any other reference distribution, such as a random distribution or a clustered distribution. More precisely, a spatial point pattern test may be applied to identify the spatial distribution of the measurements, such as Figures 3A to 3C As shown in . Figure 3A An example 301 of a random data space distribution is shown, where anchor points are located without explicit relationships. Figure 3B An example 302 of an aggregated / clustered data spatial distribution is shown, where anchor points are aggregated into clusters. Figure 3C An example 303 of a uniform data space distribution is shown, where the anchor points are randomly distributed in a conventional manner.

[0071] To calculate the SLA indicator, a test of the spatial point pattern is performed to check whether it corresponds to a completely spatially random (CSR) process. The CSR process is defined as implementing a homogeneous Poisson (HP) process. The HP process is considered as a reference model, which is defined as a point process that satisfies the following conditions: 1) the number of points within a spatial region A follows a Poisson distribution characterized by a rate parameter λ|A| (where |A| refers to the area of A and λ is the average number of points per unit area, often called intensity), and 2) the points are independently spread out within the spatial region A. This is equivalent to the number of points in separate regions being independent.

[0072] Therefore, for a given region, conventional methods can be used to test the CSR hypothesis based on the distance d between points. Summary statistics of the function can assess the dispersion and interactions between points within the region of interest. Below, the function J is used to determine the pattern distribution. For the CSR process, function J is equal to 1. Empirically, values greater than 1 indicate homogeneity, while values below 1 suggest clustering.

[0073] Function J is the ratio of the hazard functions defined for functions G and F: Among them, the function G(d) is the nearest neighbor distance distribution function, and the function F(d) is the empty space function.

[0074] Therefore, the SLA indicator (eg, the result of the J function) can be used as a quality indicator of the collected data set.

[0075] At block 215, LMF 130 includes the SLA indicator in the auxiliary information of the collected dataset. It is expected that each dataset contains auxiliary information that contains important information describing important characteristics (e.g., dataset quality indicators). The auxiliary information may sometimes be referred to as metadata information for the dataset. The SLA indicator is included in the corresponding auxiliary information of the collected dataset. To this end, the quantitative value of the SLA metric is stored in the corresponding auxiliary information of the dataset. This auxiliary information is information that can be used between entities without delivering or transferring the dataset itself.

[0076] After the AI / ML model for direct positioning or assisted positioning is trained, the LMF 130 calculates a minimum inference error (MIE) indicator after model training at block 220 and includes the MIE indicator in the auxiliary information of the dataset at block 225. A subset of the collected dataset for testing can be used to calculate the MIE indicator. The MIE indicator indicates the inference error obtained by the AI / ML model using the monitoring dataset during the training phase of the AI / ML model. In some example embodiments, the MIE indicator is determined by monitoring the AI / ML model in the monitoring phase using at least a subset of the monitoring dataset.

[0077] The MIE indicators are mapped to their respective data quality indicators (SLAs) and included in the auxiliary information for the dataset. In other words, mapping information is generated between the MIE indicators and the corresponding SLA indicators to characterize the dataset. Table 5 shows an example of the mapping between the MIE indicators and the corresponding SLA indicators for the available datasets. As can be seen, when the SLA indicator is greater than 1, the MIE indicator is generally low. Table 5

[0078] As mentioned above, Figure 2 It shows the main steps performed by the LMF and how the dataset quality indicators obtained after data collection affect the generation of auxiliary information (SLA and MIE indicators) to be used in the following performance monitoring.

[0079] Figure 4 FIG. 4 is a flow chart showing a performance monitoring method 400 implemented at a first device according to some example embodiments of the present disclosure. For the purpose of discussion, reference will be made to FIG. Figure 1 Method 400 is discussed and may be implemented in Figure 1 It is implemented at the first device in.

[0080] In some exemplary embodiments, the first apparatus may be a terminal device (e.g., UE) or be included in a terminal device (e.g., UE). In some exemplary embodiments, the first apparatus may be or may be included in a RAN network entity or a CN network entity (e.g., Figure 1 in the LMF 130, the network device 130, or any other network element).

[0081] Two approaches have been proposed for performance monitoring of AI / ML models: one that strongly relies on the presence of ground-truth labels and the other that is independent of them. Figure 4 The focus is on the methods for which ground truth is required. In addition, an example LCU-based functional framework is considered as a benchmark to indicate the signaling corresponding to the performance monitoring process.

[0082] like Figure 4 As shown, at block 405, the first device obtains monitoring metrics (MM) for an AI / ML model or an AI / ML function. The AI / ML model or function is configured for direct or assisted positioning of a terminal device within a communication network. One or more AI / ML models may be included in the positioning function. Hereinafter, for ease of discussion, an AI / ML model refers to an AI / ML model, and each time an AI / ML model is referenced, the AI / ML model may also be replaced with an AI / ML function.

[0083] The MM is determined based on the error between the positioning inference results of the AI / ML model for the model input and the ground truth positioning labels for the model input within the monitoring dataset. In some example embodiments, a ground truth performance monitoring process is performed in the first device, and the MM can be calculated on the UE side.

[0084] At block 410, the first device determines a monitoring quality criterion for the monitoring dataset. The monitoring quality criterion is determined based on the SLA indicator and the MIE indicator of the monitoring dataset. As described above, the SLA indicator indicates the spatial distribution of the monitoring dataset, and the MIE indicator indicates the inference error obtained by the AI / ML model using the monitoring dataset during the training phase.

[0085] In some example embodiments, the SLA indicator is determined based at least in part on the similarity between the distribution of ground truth location labels in the monitoring dataset and a reference distribution. Alternatively or additionally, the MIE indicator is determined by monitoring the AI / ML model during a monitoring phase using at least a subset of the monitoring dataset. Specifically, Figure 2 In box 220, after training the AI / ML model, a subset of the collected dataset used for testing is used to calculate the MIE.

[0086] In some example embodiments, the monitoring dataset is collected during the training phase of the AI / ML model and is used for monitoring of the AI / ML model. Note that the dataset may be collected during the training phase of the model and / or when the model has been deployed in the field, so the dataset in this case is the result of data collection for monitoring purposes only. Alternatively or in addition, the SLA indicator and the MIE indicator of the monitoring dataset are indicated in the mapping information between the monitoring dataset, the SLA indicator and the MIE indicator. In further example embodiments, the SLA indicator and the MIE indicator may be included in auxiliary information associated with the monitoring dataset.

[0087] In some example embodiments, the granular monitoring sensitivity (GMS) may be represented by different values. In some example embodiments, the first device may determine the GMS level mapped to the SLA indicator based on a mapping between the SLA indicator and the GMS level. For example, if SLA < 1, then GMS = 8, and if SLA > 1, then GMS = 1.

[0088] In some example embodiments, the first apparatus may determine a monitoring quality criterion for the monitoring data set based on the determined GMS level and the MIE indicator. For example, the monitoring quality criterion may be the product of GMS and MIE, ie, GMS x MIE.

[0089] At block 415, the first device may determine whether the monitoring quality criteria are not met by the MM. If the MM does not meet the monitoring quality criteria (e.g., if the MM is greater than GMS x MIE), the first device may determine an action to be applied to the AI / ML model at block 420. If the MM meets the monitoring quality criteria, the first device may determine not to apply the action to the AI / ML model at block 425.

[0090] In some example embodiments, the MM of the terminal device or the corresponding error location can be compared with the SLA and / or MIE in the auxiliary information based on the monitoring quality criteria. In some example embodiments, if MM > GMS x MIE, the performance monitoring process can determine that the MM does not meet the monitoring quality criteria, and the first device can indicate to the LMF that an action is required on the AI / ML model. And if MM < GMS x MIE, the performance monitoring process can determine that the MM meets the monitoring quality criteria, and the first device can indicate to the LMF that no additional action is required on the AI / ML model.

[0091] In additional example embodiments, if the value of MM obtained during the performance monitoring process is not within the range defined by the MIE, the performance monitoring process can determine that the MM does not meet the monitoring quality criteria, and the first device can indicate that other actions are required on the AI / ML model and / or the positioning function. If the value of MM obtained during the performance monitoring process is within the range defined by the MIE, the performance monitoring process can determine that the MM and the first device meet the monitoring quality criteria and can indicate that no further action is required.

[0092] In some exemplary embodiments, the first device can include the terminal device, and the AI / ML model can be deployed on the terminal device. In this case, the first device can also receive first configuration information from the network device. The first configuration information indicates a positioning function based at least on the capabilities of the terminal device, the AI / ML model, and / or the auxiliary information for the positioning function, and the auxiliary information includes at least auxiliary information associated with the monitoring data set.

[0093] Specifically, performance monitoring of specific functions can be performed. Here, the first device can report a set of conditions to the LMF. The LMF can determine the positioning function based on the set of conditions. That is, based on the set of conditions reported by the terminal device 110, a function is determined, which includes data set auxiliary information for a specific site. For this specific case, the auxiliary information of the data set includes SLA and MIE information generated in the data set. Here, it is desirable that the first device can select the potentially best model fit based on its own criteria.

[0094] Alternatively, the first device can include the terminal device, and the AI / ML model can be deployed at the network device, for example, at the LMF or any other network device (such as a gNB). At this time, the first device can also receive second configuration information from the network device. The second configuration information can indicate the monitoring function of the AI / ML model.

[0095] Specifically, when an AI / ML model is deployed at a network device, the UE reports a set of conditions that the network device will select to determine the monitoring function B. If one of the conditions in the set of conditions indicates that the first device supports reporting radio measurements for inference purposes, the network device may determine the reporting monitoring method in the first device within the scope of function B.

[0096] In some example embodiments, if a condition in the set of conditions indicates that the first device supports a truth-based performance monitoring procedure, the LMF may set a performance monitoring method in the first device within the scope of the positioning functionality.

[0097] In another example embodiment, the first device may transmit a request for a quality indicator of a monitoring dataset to the network device. The first device may then receive the SLA indicator and MIE indicator for the monitoring dataset from the network device. That is, after setting up the performance monitoring method, the first device may request auxiliary information related to the SLA and / or MIE information from the LMF. The auxiliary information indicates the ID of the dataset used for training. The LMF may then extract the SLA / MIE based on the dataset ID and return this information to the terminal device.

[0098] Furthermore, if an action is to be applied to the AI / ML model, the first device may transmit an MM to the network device for the network device to determine the action to be applied to the AI / ML model. Alternatively, the first device may transmit an indication of the action to be applied to the network device. Specifically, for network-side monitoring or LMF-side monitoring, the first device may report an MM. For UE-side monitoring, the first device may report the monitoring decision.

[0099] According to the above-described exemplary embodiments of the present disclosure, the data set quality indicator is used to assist performance monitoring based on a ground truth.

[0100] In order to better understand the exemplary embodiments of the present disclosure, reference will be made to Figure 5 and Figure 6 Some examples are described in method 400 based on the functional framework of the LCE.

[0101] Figure 5 FIG. 5 shows a signaling flow 500 for a performance monitoring process according to some example embodiments of the present disclosure. For the purpose of discussion, reference will be made to FIG. Figure 1 Discuss the signaling process 500. The signaling process 500 may involve Figure 1 In some exemplary embodiments, the terminal device 110 may be a terminal device (eg, a UE) or included in a terminal device (eg, a UE). The LMF 130 may be an LMF or included in an LMF.

[0102] It should be understood that the signaling process 500 may involve more devices or fewer devices, and Figure 5 The number of devices shown in FIG. 1 is for illustrative purposes only and does not imply any limitation.

[0103] exist Figure 5 In the embodiment, it is assumed that the AI / ML model configured for direct UE positioning or assisted UE positioning is deployed at the terminal device 110 as a UE-side model.

[0104] like Figure 5 As shown, the terminal device 110 reports (505) conditions to the LMF 130. The conditions include conditions that support performance monitoring based on truth labels. The LMF 130 receives (510) the conditions and determines a positioning function, for example, function A including data set assistance information for a specific site, scene, or area.

[0105] The LMF 130 then transmits (515) the function A including the dataset assistance information for the specific site to the terminal device 110. The terminal device 110 receives (520) the function A including the dataset assistance information for the specific site and selects (525) a potential best model based on any criteria.

[0106] LMF 130 determines the performance monitoring method as part of the scope of Function A and transmits (530) the performance monitoring method to terminal device 110. Terminal device 110 receives (535) the performance monitoring method and transmits (540) a request for an SLA indicator and a MIE indicator for a particular data set.

[0107] The LMF 130 receives (545) the request and transmits (550) the SLA indicator and the MIE indicator for the particular data set to the terminal device 110. The terminal device 110 receives (555) the SLA indicator and the MIE indicator for the particular data set.

[0108] The terminal device 110 then calculates (560) the MM and determines (565) the GMS based on the SLA indicator. In some examples, if LMF-side monitoring is performed, the terminal device 110 determines (570) whether the MM>GMS x MIE. If the MM>GMS x MIE, the terminal device 110 reports (575) the MM to the LMF 130. The LMF 130 receives (580) the MM and makes (585) a monitoring decision. Alternatively, if UE-side monitoring is performed, the terminal device 110 determines (590) whether the MM>GMS x MIE and makes a monitoring decision. The terminal device 110 then reports (595) the monitoring decision to the LMF 130. The LMF 130 receives (598) the monitoring decision and performs the following actions.

[0109] According to the above exemplary embodiments of the present disclosure, an embodiment of deploying an AI / ML model in a terminal device and calculating MM on the UE side is described, and signaling details are shown. Potential new IEs in the LTE Positioning Protocol (LPP) can be used in steps 515, 520, 530, 535, 540, and 545.

[0110] Figure 6 FIG. 6 shows a signaling flow 600 for a performance monitoring process according to some example embodiments of the present disclosure. For the purpose of discussion, reference will be made to FIG. Figure 1 Discuss the signaling process 600. The signaling process 600 may involve Figure 1 The terminal device 110 and LMF 130 in. Figure 6 We focus on methods that require ground truth. Furthermore, an example LCM-based functional framework is considered as a benchmark to indicate the signaling corresponding to the performance monitoring process. Figure 6 and Figure 5 The difference between Figure 6 In the example, the AI / ML model is deployed at the LMF 130 as an LMF-side model.

[0111] It should be understood that the signaling process 600 may involve more devices or fewer devices, and Figure 6 The number of devices shown is for illustrative purposes only and does not imply any limitation.

[0112] In the case where the AI / ML model is deployed on the network side (e.g., LMF or gNB), the terminal device 110 reports (605) a set of conditions to the LMF 130. The set of conditions includes conditions that support monitoring based on truth values. The LMF 130 receives (610) the set of conditions. The LMF 130 determines a function B for monitoring and transmits (615) the function B for monitoring to the terminal device 110.

[0113] The LMF 130 determines the performance monitoring method as part of the scope of function B. The LMF 130 transmits (625) the performance monitoring method to the terminal device 110 as part of the scope of function B. If a condition of the set of conditions indicates that the first apparatus supports reporting radio measurements for inference purposes, the LMF 130 determines the reporting monitoring method in the terminal device 110 to be within the scope of function B.

[0114] The terminal device 110 receives (630) the performance monitoring method and transmits (635) a request for the SLA and MIE for a specific dataset used for model training. After the performance monitoring method is determined in the network side (e.g., LMF 130), LMF 130 receives (640) the request and retrieves auxiliary information related to information of quality indicators SLA and MIE, which indicates the ID of the specific dataset used to train the AI / ML model. Based on the ID, LMF 130 extracts the SLA and MIE and returns the information to the terminal device 110. LMF 130 transmits (645) the SLA and MIE for the specific dataset to the terminal device 110. In some example embodiments, if the dataset is collected during the monitoring period, the LMF extracts the SLA from the collected dataset and sets the MIE using a common value (e.g., an average value between the MIEs of other available datasets).

[0115] The terminal device 110 receives (650) the SLA and MIE for a particular data set. The terminal device 110 then calculates (655) the MM and determines (660) the GMS based on the SLA. In some cases, if LMF-side monitoring is performed, the terminal device 110 determines (665) whether MM>GMS x MIE. If MM>GMS x MIE, the terminal device 110 reports (670) the MM to the LMF 130. The LMF 130 receives (675) the MM and makes (680) a monitoring decision. Alternatively, if UE-side monitoring is performed, the terminal device 110 determines (685) whether MM>GMS x MIE and makes a monitoring decision. The terminal device 110 reports (690) the monitoring decision to the LMF 130. The LMF 130 receives (698) the monitoring decision.

[0116] According to the exemplary embodiments of the present disclosure, an embodiment of deploying an AI / ML model in a network device (e.g., LMF or gNB) and calculating MM on the UE side is described, and its signaling details are shown. Potential new IEs in the LPP can be used in steps 635, 640, 645, and 650.

[0117] Figure 7 A flow chart of an example method 700 implemented at a first device according to some example embodiments of the present disclosure is shown. For the purpose of discussion, Figure 1 Method 700 is described from the perspective of the terminal device 110.

[0118] At box 710, the terminal device 110 obtains a monitoring metric for an artificial intelligence / machine learning (AI / ML) model that is configured for direct or assisted positioning of the terminal device within the communication network, and the monitoring metric is determined based on the following error: the error between the positioning inference result of the AI / ML model for the model input and the ground truth positioning label for the model input in the monitoring data set.

[0119] At box 720, the terminal device 110 determines a monitoring quality criterion for the monitoring dataset based on a spatial label analysis (SLA) indicator and a minimum inference error (MIE) indicator of the monitoring dataset, where the SLA indicator indicates the spatial distribution of the monitoring dataset and the MIE indicator indicates the inference error obtained by the AI / ML model using the monitoring dataset during the training phase.

[0120] At block 730 , based on determining that the monitoring quality criterion is not satisfied by the monitoring metric, the terminal device 110 determines an action to apply to the AI / ML model.

[0121] In some example embodiments, the method 700 further includes determining that no action is to be applied to the AI / ML model based on determining that the monitoring quality criterion is satisfied by the monitoring metric.

[0122] In some example embodiments, the SLA indicator is determined based at least in part on a similarity between a distribution of ground truth location labels in a monitoring dataset and a reference distribution. In some example embodiments, the MIE indicator is determined by monitoring the AI / ML model in a monitoring phase using at least a subset of the monitoring dataset.

[0123] In some example embodiments, the monitoring dataset is collected during a training phase of the AI / ML model and is used for monitoring the AI / ML model; and / or wherein the SLA indicator and the MIE indicator of the monitoring dataset are indicated in mapping information between the monitoring dataset, the SLA indicator, and the MIE indicator.

[0124] In some example embodiments, the SLA indicator and the MIE indicator are included in assistance information associated with the monitoring data set.

[0125] In some example embodiments, method 700 further includes receiving first configuration information from a network device, the first configuration information indicating a positioning function based at least on capabilities of the terminal device, an AI / ML model, and / or auxiliary information for the positioning function, the auxiliary information including at least auxiliary information associated with a monitoring data set.

[0126] In some example embodiments, the method 700 further includes receiving, from the network device, second configuration information indicating a monitoring function of the AI / ML model.

[0127] In some example embodiments, the method 700 further comprises: transmitting a request for a quality indicator of the monitoring data set to the network device; and receiving the SLA indicator and the MIE indicator of the monitoring data set from the network device.

[0128] In some example embodiments, the network device includes a location management function (LMF).

[0129] In some example embodiments, determining the monitoring quality criterion includes: determining a GMS level mapped to the SLA indicator based on a mapping between the SLA indicator and the granular monitoring sensitivity GMS level; and determining a monitoring quality criterion for the monitoring data set based on the determined GMS level and the MIE indicator.

[0130] In some example embodiments, method 700 further includes transmitting monitoring metrics to the network device based on determining the action to be applied to the AI / ML model, for the network device to determine the action to be applied to the AI / ML model, or transmitting an indication of the action to be applied to the network device.

[0131] In some exemplary embodiments, the first device is or is included in a terminal device, a radio access network (RAN) device, a core network device, or a LMF.

[0132] In some example embodiments, any one of the methods 700 (e.g., terminal device 110, LMF 130, network device 120, or Figure 1 The first device (or any other network entity in the network) may include means for performing the corresponding operations of method 700. The means may be implemented in any suitable form. For example, the means may be implemented in a circuit or a software module. The first device may be implemented as or included in Figure 1 in any device in the .

[0133] In some example embodiments, the first device includes: a component for obtaining a monitoring metric for an artificial intelligence / machine learning AI / ML model, the AI / ML model being configured for direct positioning or assisted positioning of a terminal device within a communication network, the monitoring metric being determined based on the following error: an error between a positioning inference result of the AI / ML model for a model input and a true positioning label for the model input in a monitoring dataset; a component for determining a monitoring quality criterion for the monitoring dataset based on a spatial label analysis SLA indicator and a minimum inference error MIE indicator of the monitoring dataset, the SLA indicator indicating the spatial distribution of the monitoring dataset, and the MIE indicator indicating the inference error obtained by the AI / ML model using the monitoring dataset during a training phase; and a component for determining an action to be applied to the AI / ML model based on determining that the monitoring quality criterion is not satisfied by the monitoring metric.

[0134] In some example embodiments, the first apparatus further comprises means for determining that no action is to be applied to the AI / ML model based on determining that the monitoring quality criterion is satisfied by the monitored metric.

[0135] In some example embodiments, the SLA indicator is determined based at least in part on a similarity between a distribution of ground truth location labels in a monitoring dataset and a reference distribution, and / or the MIE indicator is determined by monitoring the AI / ML model during a monitoring phase using at least a subset of the monitoring dataset.

[0136] In some example embodiments, the monitoring dataset is collected during a training phase of the AI / ML model and is used for monitoring the AI / ML model; and / or wherein the SLA indicator and the MIE indicator of the monitoring dataset are indicated in mapping information between the monitoring dataset, the SLA indicator, and the MIE indicator.

[0137] In some example embodiments, the SLA indicator and the MIE indicator are included in assistance information associated with the monitoring data set.

[0138] In some example embodiments, the first apparatus includes a terminal device, and the AI / ML model is deployed at the terminal device, the first apparatus further comprising: a unit for receiving, from a network device, first configuration information indicating a positioning function based at least on capabilities of the terminal device, the AI / ML model, and / or auxiliary information for the positioning function, the auxiliary information including at least auxiliary information associated with a monitoring data set.

[0139] In some example embodiments, the first apparatus includes a terminal device, and the AI / ML model is deployed at a network device, the first apparatus further comprising: a component for receiving second configuration information from the network device, the second configuration information indicating a monitoring function for the AI / ML model.

[0140] In some example embodiments, the first apparatus further comprises: means for transmitting a request for a quality indicator of the monitoring data set to the network device; and means for receiving the SLA indicator and the MIE indicator of the monitoring data set from the network device.

[0141] In some example embodiments, the network device includes a location management function (LMF).

[0142] In some example embodiments, a device for determining monitoring quality criteria includes: a component for determining a GMS level mapped to an SLA indicator based on a mapping between the SLA indicator and the granular monitoring sensitivity GMS level; and a component for determining a monitoring quality criterion for a monitoring data set based on the determined GMS level and the MIE indicator.

[0143] In some example embodiments, the first apparatus further comprises: components for transmitting monitoring metrics to the network device based on determining the action to be applied to the AI / ML model, for the network device to determine the action to be applied to the AI / ML model, or components for transmitting an indication of the action to be applied to the network device.

[0144] In some exemplary embodiments, the first device is or is included in a terminal device, a radio access network (RAN) device, a core network device, or a LMF.

[0145] In some example embodiments, the first apparatus further comprises means for performing other operations in method 700 or some example embodiments of the first apparatus. In some example embodiments, the apparatus comprises at least one processor, and at least one memory storing instructions that, when executed by the at least one processor, cause the execution of the first apparatus.

[0146] Figure 8 is a simplified block diagram of a device 800 suitable for implementing an example embodiment of the present disclosure. The device 800 may be used to implement a communication device, such as Figure 1 The terminal device 110, the network device 120 or the LMF 130 is shown. As shown in the figure, the device 800 includes one or more processors 810, one or more memories 820 coupled to the processor 810, and one or more communication modules 840 coupled to the processor 810.

[0147] The communication module 840 is configured for bidirectional communication. The communication module 840 has one or more communication interfaces to support communication with one or more other modules or devices. A communication interface may represent any interface necessary for communicating with other network elements. In some example embodiments, the communication module 840 may include at least one antenna.

[0148] As non-limiting examples, processor 810 can be of any type suitable for a local technology network and can include one or more of the following: a general-purpose computer, a special-purpose computer, a microprocessor, a digital signal processor (DSP), and a processor based on a multi-core processor architecture. Device 800 can have multiple processors, such as application-specific integrated circuit chips, which are time-slaved to a clock that synchronizes a master processor.

[0149] The memory 820 may include one or more non-volatile memories and one or more volatile memories. Examples of non-volatile memories include, but are not limited to, read-only memory (ROM) 824, electrically programmable read-only memory (EPROM), flash memory, hard disks, compact disks (CDs), digital video disks (DVDs), optical disks, laser disks, and other magnetic and / or optical storage. Examples of volatile memories include, but are not limited to, random access memory (RAM) 822 and other volatile memories that do not persist during a power outage.

[0150] The computer program 830 includes computer-executable instructions executed by the associated processor 810. The instructions of the program 830 may include instructions for performing the operations / actions of some example embodiments of the present disclosure. The program 830 may be stored in a memory, such as ROM 824. The processor 810 may perform any suitable actions and processes by loading the program 830 into the RAM 822.

[0151] The exemplary embodiments of the present disclosure may be implemented with the aid of a program 830, so that the device 800 may execute the following steps: Figures 2 to 7 Any process of the present disclosure discussed. The example embodiments of the present disclosure may also be implemented by hardware or a combination of software and hardware.

[0152] In some example embodiments, the program 830 may be tangibly embodied in a computer-readable medium that may be included in the device 800 (such as in the memory 820) or other storage device accessible by the device 800. The device 800 may load the program 830 from the computer-readable medium to the RAM 822 for execution. In some example embodiments, the computer-readable medium may include any type of non-transitory storage medium, such as a ROM, EPROM, flash memory, hard disk, CD, DVD, etc. As used herein, the term "non-transitory" is a limitation of the medium itself (i.e., tangible rather than a signal), not a limitation on data storage persistence (e.g., RAM versus ROM).

[0153] In general, various embodiments of the present disclosure may be implemented in hardware or dedicated circuitry, software, logic, or any combination thereof. Some aspects may be implemented in hardware, while other aspects may be implemented in firmware or software that may be executed by a controller, microprocessor, or other computing device. Although various aspects of the embodiments of the present disclosure are shown and described as block diagrams, flow charts, or using some other graphical representation, it should be understood that, as non-limiting examples, the blocks, devices, systems, techniques, or methods described herein may be implemented in hardware, software, firmware, dedicated circuitry or logic, general-purpose hardware or a controller or other computing device, or some combination thereof.

[0154] Some example embodiments of the present disclosure also provide at least one computer program product tangibly stored on a computer-readable medium, such as a non-transitory computer-readable medium. The computer program product includes computer-executable instructions, such as those included in a program module, that are executed in a device on a target physical or virtual processor to perform any of the methods described above. Typically, program modules include routines, programs, libraries, objects, classes, components, data structures, etc. that perform specific tasks or implement specific abstract data types. The functionality of the program modules can be combined or split between program modules as needed in various embodiments. The machine-executable instructions for the program modules can be executed within a local or distributed device. In a distributed device, the program modules can be located in local and remote storage media.

[0155] The program code for performing the method of the present disclosure may be written in any combination of one or more programming languages. The program code may be provided to a processor or controller of a general-purpose computer, a special-purpose computer, or other programmable data processing device so that when the program code is executed by the processor or controller, the functions / operations specified in the flow chart and / or block diagram are implemented. The program code may be executed entirely on the machine, partially on the machine, as a separate software package, partially on the machine and partially on a remote machine, or entirely on a remote machine or server.

[0156] In the context of the present disclosure, computer program codes or related data may be carried by any suitable carrier wave to enable a device, apparatus or processor to perform various processes and operations as described above. Examples of carrier waves include signals, computer-readable media, and the like.

[0157] The computer-readable medium may be a computer-readable signal medium or a computer-readable storage medium. The computer-readable medium may include, but is not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any suitable combination of the foregoing. More specific examples of computer-readable storage media would include an electrical connection having one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disc read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

[0158] In addition, although operations are depicted in a particular order, this should not be understood as requiring that such operations be performed in the particular order shown or in a sequential order, or that all illustrated operations be performed to achieve the desired result. In some cases, multitasking and parallel processing may be advantageous. Similarly, although several specific implementation details are included in the above discussion, these details should not be interpreted as limiting the scope of this disclosure, but rather as describing features that may be specific to a particular embodiment. Unless expressly stated otherwise, certain features described in the context of a separate embodiment may also be implemented in combination in a single embodiment. On the contrary, unless expressly stated otherwise, various features described in the context of a single embodiment may also be implemented in multiple embodiments individually or in any suitable subcombination.

[0159] Although the disclosure has been described in language specific to structural features and / or methodological acts, it should be understood that the disclosure defined in the appended claims is not necessarily limited to the specific features or acts described above. Rather, the specific features and acts described above are disclosed as example forms of implementing the claims.

Claims

1. A first apparatus for communication, comprising: at least one processor; as well as At least one memory stores instructions that, when executed by the at least one processor, cause the first apparatus to at least: Obtaining a monitoring metric for an artificial intelligence / machine learning (AI / ML) model configured for direct or assisted positioning of a terminal device within a communication network, the monitoring metric being determined based on an error between a positioning inference result of the AI / ML model for a model input and a ground truth positioning label for the model input within a monitoring dataset; determining a monitoring quality criterion for the monitoring dataset based on a spatial label analysis SLA indicator and a minimum inference error (MIE) indicator of the monitoring dataset, wherein the SLA indicator indicates a spatial distribution of the monitoring dataset and the MIE indicator indicates an inference error obtained by the AI / ML model using the monitoring dataset during a training phase; as well as Based on determining that a monitoring quality criterion is not satisfied by the monitoring metric, an action to be applied to the AI / ML model is determined.

2. The first device according to claim 1, wherein the first device is further configured to: Based on determining that the monitoring quality criterion is satisfied by the monitoring metric, determining that no action is to be applied to the AI / ML model.

3. The first apparatus of claim 1 , wherein the SLA indicator is determined based at least in part on: a similarity between a distribution of ground truth location labels in the monitoring dataset and a reference distribution, and / or The MIE indicator is determined by monitoring the AI / ML model during a monitoring phase using at least a subset of the monitoring dataset.

4. The first apparatus of claim 1 , wherein the monitoring dataset is collected during a training phase of the AI / ML model and is used for monitoring the AI / ML model; and / or The SLA indicator and the MIE indicator of the monitoring data set are indicated in mapping information between the monitoring data set, the SLA indicator and the MIE indicator. 5 . The first apparatus of claim 1 , wherein the SLA indicator and the MIE indicator are included in assistance information associated with the monitoring data set.

6. The first apparatus according to claim 1, wherein the first apparatus comprises the terminal device, and the AI / ML model is deployed on the terminal device, and the first apparatus is further caused to: First configuration information is received from a network device, where the first configuration information indicates a positioning function based at least on the capabilities of the terminal device, the AI / ML model and / or auxiliary information for the positioning function, the auxiliary information including at least the auxiliary information associated with the monitoring data set.

7. The first device according to claim 6, wherein the first device is further configured to: transmitting a request for a quality indicator of the monitoring data set to the network device; and The SLA indicator and the MIE indicator of the monitoring data set are received from the network device. The first apparatus according to claim 6 , wherein the network device comprises a location management function (LMF).

9. The first apparatus according to claim 1, wherein the first apparatus comprises the terminal device, and the AI / ML model is deployed on a network device, and the first apparatus is further configured to: Second configuration information is received from the network device, where the second configuration information indicates a monitoring function for the AI / ML model.

10. The first device according to claim 1, wherein the first device is caused to: Based on a mapping between the SLA indicator and the granular monitoring sensitivity GMS level, determining the GMS level mapped to the SLA indicator; and The monitoring quality criterion for the monitoring data set is determined based on the determined GMS level and the MIE indicator.

11. The first device according to claim 1 , wherein the first device is further caused to: Based on the actions determined to be applied to the AI / ML model, transmitting the monitoring metrics to a network device for use by the network device in determining an action to be applied to the AI / ML model, or An indication of the action to be applied is transmitted to the network device.

12. The first device according to any one of claims 1 to 5, wherein the first device is or is included in a terminal device, a radio access network device, a core network device, or a LMF.

13. A method for communication, comprising: Obtaining, by a first device, a monitoring metric for an artificial intelligence / machine learning AI / ML model, the AI / ML model being configured for direct or assisted positioning of a terminal device within a communication network, the monitoring metric being determined based on an error between a positioning inference result of the AI / ML model for a model input and a ground truth positioning label for the model input within a monitoring dataset; determining a monitoring quality criterion for the monitoring dataset based on a spatial label analysis SLA indicator and a minimum inference error (MIE) indicator of the monitoring dataset, wherein the SLA indicator indicates a spatial distribution of the monitoring dataset and the MIE indicator indicates an inference error obtained by the AI / ML model using the monitoring dataset during a training phase; as well as Based on determining that a monitoring quality criterion is not satisfied by the monitoring metric, an action to be applied to the AI / ML model is determined.

14. A computer-readable medium comprising instructions stored thereon, which, when executed by at least one processor, cause a first device to perform at least the method of claim 13.

Images