Data evaluation for AI / ML

Abstract

Example embodiments are directed to data evaluation for AI / ML. A method includes receiving, at a first device and from at least one second device, bitmap information indicating respective locations of data samples collected by the at least one second device, the at least one second device configured for data collection for artificial intelligence (AI) / machine learning (ML) services; determining a quality level of data samples collected by the at least one second device based at least in part on the bitmap information; selecting data samples to be included in a data set for the AI / ML service by comparing the quality level with a target quality level of the data set; and sending a request to the at least one second device to request to report the selected data samples to be included in the data set to a target entity.

Description

Technical Field

[0001] Various example embodiments of the present disclosure relate generally to the field of telecommunications, and more particularly, to methods, devices, apparatuses, and computer-readable storage media for data evaluation for artificial intelligence (AI) / machine learning (ML). Background Art

[0002] As communication systems evolve, new techniques have been proposed. For example, it has been proposed that artificial intelligence (AI) / machine learning (ML) can be applied to cellular networks. Data collection is essential for implementing ML solutions for training, validation and testing purposes in cellular networks. For example, for AI / ML-based solutions that handle positioning in 5G New Radio (NR) systems, the network needs to collect positioning-related measurements made by the user equipment (UE). These measurements can be made over the Uu interface between the network and the UE (i.e., uplink and downlink) and over the PC5 interface (i.e., sidelink), for example, for ranging between two UEs. Therefore, it is worth studying the mechanisms for data collection. Summary of the invention

[0003] In a first aspect of the present disclosure, a first device is provided. The first device includes at least one processor; and at least one memory, the at least one memory storing instructions, when the instructions are executed by the at least one processor, the instructions cause the first device to at least perform: receiving bitmap information from at least one second device, the bitmap information indicating the corresponding positions of data samples collected by the at least one second device, the at least one second device being configured for data collection for artificial intelligence (AI) / machine learning (ML) services; determining the quality level of the data samples collected by the at least one second device based at least in part on the bitmap information; selecting data samples to be included in the data set by comparing the quality level with the target quality level of the data set for the AI / ML service; and sending a request to the at least one second device to request reporting the selected data samples to be included in the data set to a target entity.

[0004] In a second aspect of the present disclosure, a second device is provided. The second device includes: at least one processor; and at least one memory, the at least one memory storing instructions, the instructions when executed by the at least one processor, causing the second device to at least perform: sending bitmap information to a first device, the bitmap information indicating corresponding locations of data samples collected by the second device, the second device being configured for data collection for artificial intelligence (AI) / machine learning (ML) services; and receiving a request from the first device to request reporting to a target entity the selected data samples to be included in the data set.

[0005] In a third aspect of the present disclosure, a method is provided. The method includes: receiving bitmap information at a first device and from at least one second device, the bitmap information indicating corresponding locations of data samples collected by the at least one second device, the at least one second device being configured for data collection for an artificial intelligence (AI) / machine learning (ML) service; determining a quality level of the data samples collected by the at least one second device based at least in part on the bitmap information; selecting data samples to be included in the data set by comparing the quality level with a target quality level of the data set for the AI / ML service; and sending a request to the at least one second device to request reporting of the selected data samples to be included in the data set to a target entity.

[0006] In a fourth aspect of the present disclosure, a method is provided. The method includes: sending bitmap information at a second device to a first device, the bitmap information indicating corresponding locations of data samples collected by the second device, the second device being configured for data collection for artificial intelligence (AI) / machine learning (ML) services; and receiving a request from the first device to request reporting to a target entity the selected data samples to be included in the data set.

[0007] In a fifth aspect of the present disclosure, a first device is provided. The first device includes: a component for receiving bitmap information from at least one second device, the bitmap information indicating the corresponding positions of data samples collected by the at least one second device, the at least one second device being configured for data collection for artificial intelligence (AI) / machine learning (ML) services; a component for determining the quality level of the data samples collected by the at least one second device based at least in part on the bitmap information; a component for selecting data samples to be included in the data set by comparing the quality level with a target quality level of the data set for the AI / ML service; and a component for sending a request to the at least one second device to request reporting of the selected data samples to be included in the data set to a target entity.

[0008] In a sixth aspect of the present disclosure, a second apparatus is provided. The second apparatus includes: a component for sending bitmap information to a first device, the bitmap information indicating corresponding locations of data samples collected by the second apparatus, the second apparatus being configured for data collection for artificial intelligence (AI) / machine learning (ML) services; and a component for receiving a request from the first apparatus to request reporting to a target entity the selected data samples to be included in the data set.

[0009] In a seventh aspect of the present disclosure, a computer-readable medium is provided, wherein the computer-readable medium includes instructions stored thereon, and the instructions are used to cause a device to at least execute the method according to the third aspect.

[0010] In an eighth aspect of the present disclosure, a computer-readable medium is provided, wherein the computer-readable medium includes instructions stored thereon, and the instructions are used to cause a device to at least execute the method according to the fourth aspect.

[0011] It should be understood that the invention summary is not intended to identify the key features or essential features of the embodiments of the present disclosure, nor is it intended to limit the scope of the present disclosure. Other features of the present invention will become readily understood through the following description. BRIEF DESCRIPTION OF THE DRAWINGS

[0012] Some example embodiments will now be described with reference to the accompanying drawings, in which:

[0013] Figure 1 An example communication environment is shown in which example embodiments of the present disclosure may be implemented;

[0014] Figure 2 shows a signaling flow of an example communication process according to some embodiments of the present disclosure;

[0015] Figure 3 shows examples of three different sub-data sets according to some embodiments of the present disclosure;

[0016] Figure 4 An example of a bitmap method according to some embodiments of the present disclosure is shown;

[0017] Figure 5 A flowchart showing a method implemented at a first device according to some example embodiments of the present disclosure is shown;

[0018] Figure 6 A flowchart showing a method implemented at a second device according to some example embodiments of the present disclosure is shown;

[0019] Figure 7 shows a simplified block diagram of a device suitable for implementing an example embodiment of the present disclosure; and

[0020] Figure 8 A block diagram of an example computer-readable medium is shown according to some example embodiments of the present disclosure.

[0021] Throughout the drawings, the same or similar reference numerals refer to the same or similar elements. DETAILED DESCRIPTION

[0022] The principle of the present invention will now be described with reference to some exemplary embodiments. It should be understood that these embodiments are described only for the purpose of illustration, and help those skilled in the art to understand and implement the present disclosure, without suggesting any limitation to the scope of the present disclosure. In addition to the embodiments described below, the embodiments described herein can be implemented in various ways.

[0023] In the following description and claims, unless defined otherwise, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this disclosure belongs.

[0024] References in this disclosure to "one embodiment," "an embodiment," "an example embodiment," etc., indicate that the described embodiment may include a particular feature, structure, or characteristic, but not every embodiment necessarily includes the particular feature, structure, or characteristic. Moreover, such phrases do not necessarily refer to the same embodiment. Further, when a particular feature, structure, or characteristic is described in conjunction with an embodiment, it is considered to be within the knowledge of those skilled in the art to affect such feature, structure, or characteristic in conjunction with other embodiments (whether or not explicitly described).

[0025] It should be understood that, although the terms "first", "second", etc. in front of nouns can be used in this article to describe various elements, these elements should not be limited by these terms. These terms are only used to distinguish one element from another element, and they do not limit the order of nouns. For example, without departing from the scope of the exemplary embodiment, the first element can be referred to as the second element, and similarly, the second element can be referred to as the first element. As used herein, the term "and / or" includes any and all combinations of one or more listed terms.

[0026] As used herein, “at least one of: «a list of two or more elements» and “at least one of «a list of two or more elements» and similar expressions, where a list of two or more elements is connected by “and” or “or”, refer to at least any one of the elements, or at least any two or more of the elements, or at least all of the elements.

[0027] As used herein, unless explicitly stated, performing a step “in response to A” does not mean performing the step immediately after “A” occurs, and may include one or more intermediate steps.

[0028] The terms used herein are only used for the purpose of describing specific embodiments and are not intended to limit the exemplary embodiments. As used herein, the singular forms "a", "an", and "the" are intended to include the plural forms as well, unless the context clearly indicates otherwise. It will be further understood that when used herein, the terms "comprises", "comprising", "has", "having", "includes", and / or "including" specify the presence of the features, elements, and / or parts, etc., but do not exclude the presence or addition of one or more other features, elements, parts, and / or combinations thereof.

[0029] As used in this application, the term "circuitry" may refer to one or more or all of the following:

[0030] (a) hardware circuit implementation only (e.g., implementation only in analog and / or digital circuits), and

[0031] (b) a combination of hardware circuitry and software such as (where applicable):

[0032] (i) a combination of analog and / or digital hardware circuitry and software / firmware, and

[0033] (ii) any portion of a hardware processor with software (including a digital signal processor, software, and memory working together to enable a device such as a mobile phone or server to perform various functions); and (c) a hardware circuit and / or processor that requires software (e.g., firmware) for operation, such as a microprocessor or a portion of a microprocessor, but the software may not be present when not required for operation.

[0034] This definition of circuitry applies to all uses of the term in this application, including in any claims. As a further example, as used in this application, the term circuitry also covers an implementation of only a hardware circuit or processor (or multiple processors) or a portion of a hardware circuit or processor and its (or its) accompanying software and / or firmware. For example and if applicable to the elements of a particular claim, the term circuitry also covers a baseband integrated circuit or processor integrated circuit for a mobile device or a similar integrated circuit in a server, cellular network device, or other computing or network device.

[0035] As used herein, the term "communication network" refers to a network that complies with any suitable communication standard, such as New Radio (NR), Long Term Evolution (LTE), Advanced LTE (LTE-A), Wideband Code Division Multiple Access (WCDMA), High Speed ​​Packet Access (HSPA), Narrowband Internet of Things (NB-IoT), etc. In addition, the communication between the terminal device and the network device in the communication network can be performed according to any suitable generation communication protocol, including but not limited to the first generation (1G), second generation (2G), 2.5G, 2.75G, third generation (3G), fourth generation (4G), 4.5G, fifth generation (5G), sixth generation (6G) communication protocol and / or any other protocol currently known or developed in the future. The embodiments of the present disclosure can be applied to various communication systems. In view of the rapid development of communication, there are of course future types of communication technologies and systems that can implement the present disclosure. It should not be considered that the scope of the present disclosure is limited to the above-mentioned system.

[0036] As used herein, the term "network device" refers to a node in a communication network via which a terminal device accesses a network and receives services from it. A network device may refer to a base station (BS) or an access point (AP), such as a NodeB (NodeB or NB), an evolved NodeB (eNodeB or eNB), an NRNB (also known as a gNB), a remote radio unit (RRU), a radio head (RH), a remote radio head (RRH), a repeater, an integrated access and backhaul (IAB) node, a low power node (such as a femto), a micro-net, a non-terrestrial network (NTN) or non-terrestrial network equipment, such as satellite network equipment, a low earth orbit (LEO) satellite and a geosynchronous orbit (GEO) satellite, an aircraft network equipment, etc., depending on the terminology and technology applied. In some example embodiments, a radio access network (RAN) split architecture includes a centralized unit (CU) and a distributed unit (DU) at an IAB donor node. An IAB node includes a mobile terminal (IAB-MT) portion that behaves like a UE toward a parent node, and a DU portion of an IAB node behaves like a base station toward a next-hop IAB node.

[0037] The term "terminal device" refers to any terminal device capable of wireless communication. As an example and not limitation, a terminal device may also be referred to as a communication device, a user equipment (UE), a subscriber station (SS), a portable subscriber station, a mobile station (MS), or an access terminal (AT). The terminal device may include, but is not limited to, a mobile phone, a cellular phone, a smart phone, a voice over IP (VoIP) phone, a wireless local loop phone, a tablet computer, a wearable terminal device, a personal digital assistant (PDA), a portable computer, a desktop computer, an image capture terminal device such as a digital camera, a game terminal device, a music storage and playback device, a vehicle-mounted wireless terminal device, a wireless endpoint, a mobile station, a laptop embedded device (LEE), a laptop mounted device (LME), a USB dongle, a smart device, a wireless customer premises equipment (CPE), an Internet of Things (IoT) device, a watch or other wearable, a head mounted display (HMD), a vehicle, a drone, medical equipment and applications (e.g., remote surgery), industrial equipment and applications (e.g., robots and / or other wireless devices operating in industrial and / or automated processing chain environments), consumer electronic devices, devices operating on commercial and / or industrial wireless networks, etc. The terminal device may also correspond to the mobile terminal (MT) part of an IAB node (eg, a relay node).In the following description, the terms "terminal device", "communication device", "terminal", "user equipment" and "UE" may be used interchangeably.

[0038] As used herein, the terms "resource", "transmission resource", "resource block", "physical resource block" (PRB), "uplink resource" or "downlink resource" may refer to any resource used to perform communication (e.g., communication between a terminal device and a network device), such as a resource in the time domain, a resource in the frequency domain, a resource in the spatial domain, a resource in the code domain, or any other combination of time, frequency, space and / or code domain resources to achieve communication. In the following, unless explicitly stated, resources in the frequency domain and the time domain will be used as examples of transmission resources for describing some example embodiments of the present disclosure. It should be noted that the example embodiments of the present disclosure are equally applicable to other resources in other domains.

[0039] To help understand the terminology, some definitions are provided below for the list of technologies used in AI / ML.

[0040] AI / ML model: A data-driven algorithm that applies AI / ML techniques to generate a set of outputs based on a set of inputs.

[0041] AI / ML model transfer: A general term that refers to the transfer of AI / ML models from one entity to another in any manner. Note: Entities may refer to network nodes / functions (e.g., gNB, Location Management Function (LMF), etc.), UEs, proprietary servers, etc.

[0042] AI / ML model inference: The process of using a trained AI / ML model to produce a set of outputs based on a set of inputs.

[0043] AI / ML model testing: A subprocess of training used to evaluate the performance of the final AI / ML model using a dataset different from the one used for model training and validation. Unlike AI / ML model validation, testing does not assume subsequent adjustments to the model.

[0044] AI / ML model training: The process used to train AI / ML models in a data-driven manner [by learning input / output relationships] and obtain the trained AI / ML models for reasoning.

[0045] AI / ML Model Transfer: The transfer of an AI / ML model over the air interface in a manner that is opaque to 3GPP signaling, with parameters of the model structure known at the receiving end or a new model with parameters. The transfer can contain a complete model or a partial model.

[0046] AI / ML Model Validation: A subprocess of training used to evaluate the quality of an AI / ML model using a dataset different from the one used for model training, which helps select model parameters that generalize beyond the dataset used for model training.

[0047] Data Collection: The process of collecting data by network nodes, management entities or UEs for AI / ML model training, data analysis and inference.

[0048] Federated Learning / Federated Training: A machine learning technique that trains AI / ML models on multiple distributed edge nodes (e.g., UE, gNB), each of which performs local model training using local data samples. The technique requires multiple interactions of the model, but does not exchange local data samples.

[0049] Function identification: The process / method of identifying AI / ML functions for common understanding between the network and UE. NOTE: Information about AI / ML functions can be shared during function identification. Where AI / ML functions depend on specific use cases and sub-use cases.

[0050] Model activation: Enabling an AI / ML model for a specific function.

[0051] Model deactivation: Disabling AI / ML models used for specific features.

[0052] Model download: Model transmission from the network to the UE.

[0053] Model Identification: The process / method of identifying AI / ML models for common understanding between the network (NW) and UE. NOTE: The process / method of model identification may or may not be applicable. NOTE: Information about AI / ML models may be shared during model identification.

[0054] Model monitoring: The process of monitoring the inference performance of an AI / ML model.

[0055] Model parameter update: The process of updating the model parameters of a model.

[0056] Model Selection: The process of selecting the AI / ML model to activate among multiple models for the same AI / ML enabled feature. Note: Model selection can be performed with or without model activation.

[0057] Model switching: Deactivate the currently active AI / ML model and activate a different AI / ML model for a specific function.

[0058] Model updating: The process of updating the model parameters and / or model structure of a model.

[0059] Model upload: Model transmission from UE to the network.

[0060] Network-side (AI / ML) models: AI / ML models whose reasoning is performed entirely at the network.

[0061] Offline field data: Data collected from the field and used for offline training of AI / ML models.

[0062] Offline Training: An AI / ML training process in which a model is trained based on a collected dataset, and in which the trained model is later used or delivered for inference. Note: The definition is intended only as a guide. There may be cases that may not fully meet the definition but can still be classified as offline training by accepted conventions.

[0063] Online field data: Data collected from the field and used for online training of AI / ML models.

[0064] Online Training: AI / ML training process where the model used for inference is trained in (near) real-time (usually continuously) as new training samples arrive. Note: the notion of (near) real-time vs. non-real-time is context dependent and relative to the inference time scale. Note: the definitions are intended only as a guide. There may be cases that may not fit neatly into the definitions but can still be classified as online training by accepted conventions. Note: fine-tuning / retraining can be done via online or offline training. (This note may be removed when we define the term fine-tuning.)

[0065] Reinforcement Learning (RL): The process of training an AI / ML model based on the inputs (aka states) and feedback signals (aka rewards) generated by the model’s outputs (aka actions) in the environment with which the model interacts.

[0066] Semi-supervised learning: The process of training a model using a mixture of labeled and unlabeled data.

[0067] Supervised learning: The process of training a model based on inputs and their corresponding labels.

[0068] Two-Sided (AI / ML) Model: A paired AI / ML model whereby joint reasoning is performed, wherein the joint reasoning includes AI / ML reasoning where reasoning is performed jointly on the UE and the network, i.e., the first part of the reasoning is first performed by the UE and then the remaining part is performed by the gNB, or vice versa.

[0069] UE-side (AI / ML) model: an AI / ML model whose reasoning is performed entirely at the UE.

[0070] Unsupervised learning: The process of training a model without labeled data.

[0071] Proprietary format models: ML models in vendor / device specific proprietary formats from the 3GPP perspective. They are not mutually recognizable across vendors and hide model design information from other vendors when shared. Note: Examples are device specific binary executable formats.

[0072] Open format models: From the 3GPP perspective, ML models in a specified format that are mutually recognizable across vendors and allow interoperability. They are mutually recognizable between vendors and do not hide model design information from other vendors when shared.

[0073] As mentioned above, data collection is essential in cellular networks. The data required to train, validate, test, and infer from AI / ML solutions involves many sources of data collection. In the case of solutions targeting 5G positioning, the network will maintain AI / ML models, however, these models require data collection from the UE or Positioning Reference Unit (PRU).

[0074] The performance impact from the availability of ground truth labels (i.e., some training data may not have ground truth labels) will be studied. The learning algorithms (e.g., supervised learning, semi-supervised learning, unsupervised learning) will be reported by the participating companies, and the data labeling details need to be described when providing the evaluation results. The data labeling details include the meaning of the labels (e.g., UE coordinates; binary identifier for LOS / NLOS; ToA), the percentage of training data without labels (if incomplete labeling is considered in the evaluation), and the imperfection of the ground truth labels (if any).

[0075] Regardless, and if so, how the entities can be used to obtain ground truth labels and / or other training data will be investigated.

[0076] For direct AI / ML localization, the impact of labeling error on localization accuracy is studied. It is important to consider that the ground truth label error in each dimension of the x-axis and y-axis can be modeled as a truncated Gaussian distribution with zero mean and standard deviation of L meters, where the distribution is truncated to the range [-2*L, 2*L]. The value L depends on the source. It is important to consider whether / how to study the impact of labeling error on label-based model monitoring methods. It is important to consider whether / how to study the impact of labeling error on AI / ML-assisted localization.

[0077] For AI / ML-assisted positioning using TOA as a model output, it is necessary to study the impact of labeling error on TOA accuracy and / or positioning accuracy. For example, the ground truth label error of TOA is calculated based on the position error. The position error in each dimension of the x-axis and y-axis can be modeled as a truncated Gaussian distribution with zero mean and a standard deviation of L meters, where the distribution is truncated to the range [-2*L, 2*L]. The value L depends on the source. In addition, other models of labeling error are not excluded. Other timing information such as reference signal time difference (RSTD) is not excluded as model output.

[0078] For AI / ML assisted positioning with LOS / NLOS metrics as model outputs, the impact of labeling errors on LOS / NLOS metric accuracy and / or positioning accuracy needs to be studied. For example, the ground truth labeling error of the LOS / NLOS metric can be modeled as m% LOS labeling error and n% NLOS labeling error. The values ​​of m and n depend on the source. Companies should at least consider hard-valued LOS / NLOS metrics as model outputs.

[0079] The synthetic data set generated according to the statistical channel model in 3GPP Technical Report (TR) 38.901 is used for model training, validation and testing. The data set is generated by a system-level simulator based on the 3GPP simulation method.

[0080] As a starting point, the training, validation, and test datasets come from the same large-scale and small-scale propagation parameter settings. Subsequent evaluations can study the performance when the training and test datasets come from different settings.

[0081] The details of the training data set generation will be reported. For example, the training data set generation includes the size of the training data set, for example, the total number of UEs in the evaluation area used to generate the training data set. For another example, the distribution of UE positions used to generate the training data set can be one of the following options. In one option, the distribution of UE positions used to generate the training data set can be a grid distribution, that is, one training data is collected at the center of a small square, where, for example, the width of the square can be 0.25 / 0.5 / 1.0m. In another option, the distribution of UE positions used to generate the training data set can be a uniform distribution, that is, the UE positions are randomly and uniformly distributed in the evaluation area.

[0082] For AI / ML-assisted positioning, companies report which configuration was applied in their evaluation. One configuration is a single transmit / receive point (TRP) configuration: the input to the ML model is the channel measurement between the target UE and a single TRP, and the output of the ML model is for the same pair of UE and TRP. The other configuration is a multi-TRP configuration: the input to the ML model contains N sets of channel measurements between the target UE and N (N>1) TRPs, and the output of the ML model contains N sets of values, one set for each of the N TRPs.

[0083] Note that for measurements that are relative values ​​between a given TRP and a reference TRP (e.g., RSTD), the TRP in "Single TRP" and "Multiple TRPs" refers only to the given TRP. For single TRP builds, companies report whether they consider the same model for all TRPs or N different models for the TRP.

[0084] When a single TRP construction is used for an AI / ML model, the company reports the AI / ML complexity (model complexity, computational complexity) of at least N TRPs, which are used to determine the location of the target UE considering various constructions.

[0085] Figure 1An example communication environment 100 is shown in which an example embodiment of the present disclosure may be implemented. In the communication environment 100, a high-level framework architecture of a data collection management (DCM) 110 entity is shown at different life cycle management (LCM) steps of a model, including training 102, updating 104, fine-tuning 106, and monitoring 108. In order to evaluate the goodness of data (such as positioning data) for training 102, updating 104, fine-tuning 106, and monitoring 108, a DCM 110 is provided that coordinates the identification, collection, classification, and use of data sets (measurements and their corresponding label information) for positioning.

[0086] In the communication environment 100, the AI / ML positioning entity may indicate bidirectional signaling based on data collection purposes between the LMF 120 and the UE 130, the PRU 140, or the same or other LMF 150. It is to be noted that the PRU 140 is a UE with a known location and is expected to have the ability to consider data collection by default. For convenience, the LMF 120 and the UE 130 will be taken as examples below. In some example embodiments, the DCM 110 may be integrated in the LMF 120, although in other example embodiments, the DCM 110 and the LMF 120 may be separate entities.

[0087] In some example embodiments, during UE entry into the network, the UE 130 may report its capabilities (conditions) on a specific feature or feature group related to AI / ML positioning. If the data collection conditions (to indicate the measurements that can be collected by the specific UE 130 and the specific LCM 120 supported steps) are considered between the reported capabilities, the LMF 120 triggers the data collection method on the specific UE 130.

[0088] In some example embodiments, after a data collection method is triggered for a group of UEs in the same cell ID, the UEs receive a set of settings (eg, PRS settings) to be considered during the data collection process.

[0089] In the following, for the purpose of illustration, some exemplary embodiments are described by taking UE 130 as a terminal device and LMF 120 as a network device as an example. However, in some exemplary embodiments, the operations described in conjunction with the terminal device may be implemented at a network device or other devices, and the operations described in conjunction with the network device may be implemented at a terminal device or other devices.

[0090] The communication in the communication environment 100 may be implemented according to any suitable communication protocol, including but not limited to cellular communication protocols of the first generation (1G), second generation (2G), third generation (3G), fourth generation (4G), fifth generation (5G), sixth generation (6G), etc., wireless local area network communication protocols such as Institute of Electrical and Electronics Engineers (IEEE) 802.11, etc., and / or any other protocol currently known or developed in the future. In addition, the communication may use any suitable wireless communication technology, including but not limited to: code division multiple access (CDMA), frequency division multiple access (FDMA), time division multiple access (TDMA), frequency division duplex (FDD), time division duplex (TDD), multiple input multiple output (MIMO), orthogonal frequency division multiple access (OFDM), discrete Fourier transform spread OFDM (DFT-s-OFDM) and / or any other technology currently known or developed in the future.

[0091] When training supervised learning models to solve specific tasks such as positioning tasks, in order to achieve high positioning accuracy, the UEs from which the training data is collected need to be evenly distributed in the geographical area of ​​interest. Studies have shown that models trained with the same UE density (i.e., the same number of samples per square meter) but with two different UE distributions exhibit a wide range of performance variations. In particular, if the UE distribution is not uniform, the positioning performance degradation becomes severe, with the model accuracy dropping by up to 55% compared to the case where the model is trained with uniformly distributed UEs. Therefore, the premise is to collect training data from UEs that are uniformly distributed in space, i.e., the training data labels generated by the simulation follow a uniform distribution in the spatial region where the positioning model is trained.

[0092] In addition to the distribution behavior of labels, other factors also affect the quality level (or goodness) of the dataset for AI / ML services. Such datasets may sometimes be referred to as training datasets, but it should be appreciated that datasets may be used for training, updating, tuning, monitoring, validating, testing, and / or any other aspect related to AI / ML models.

[0093] It should be noted that for AI / ML models, the “quality”, “quality level” or “goodness” of a dataset refers to the training data attributes that enable accurate model performance, such as accurately positioning an AI / ML positioning model.

[0094] There are example factors that contribute to the goodness of training data. For example, the input features of the training data (e.g., CIR, RSRPP, RSTD, etc.) should have a minimum quality. To ensure this, the input features should be estimated from signals collected in the medium / high SINR range. For another example, if the label is also an estimate, then the minimum label quality should also be ensured.

[0095] In summary, AI / ML positioning model accuracy is limited by the quality of training data (i.e., the size of training data and / or the diversity of training data). The diversity of training data includes the quality of input features and labels, as well as the distribution of labels.

[0096] Therefore, it is beneficial to evaluate the quality of available data before training an AI / ML model. Based on the evaluation results, a decision can be made as to whether the data can be used for training and / or whether the data should be split and used to train multiple AI / ML models.

[0097] According to some example embodiments of the present disclosure, a solution for data evaluation for AI / ML is provided. At least one second device (terminal equipment, such as UE and / or PRU) may be configured for data collection for artificial intelligence (AI) / machine learning (ML) services. A first device (such as LMF or DCM) receives bitmap information from at least one second device, the bitmap information indicating the corresponding positions of data samples collected by at least one second device. The first device determines the quality level of the data samples collected by the at least one second device based at least in part on the bitmap information, and selects the data samples to be included in the data set for the AI / ML service by comparing the quality level with the target quality level of the data set. The first device sends a request to report the selected data samples to be included in the data set to the target entity to at least one second device. The at least one second device may then transmit the collected and selected data samples that have met the target quality level to the entity requiring the data set. The entity may be an LMF, a RAN node (e.g., a gNB), a UE, or any other device that needs to collect data for AI / ML model training, updating, fine-tuning, etc.

[0098] Through the above scheme, before reporting the collected data samples, it is necessary to obtain bitmap information from the device collecting the data to guide the collection and reporting of the data set to achieve the purpose of data collection. This can reduce the signaling overhead between devices and ensure that the quality of the reported data samples meets the target.

[0099] In some example embodiments, a network (NW)-guided procedure for coordinating the collection of positioning data may be proposed, and the goodness of the positioning data may be evaluated (for training, updating, fine-tuning models). Then, based on the continuous evaluation results, a decision may be made on how to collect data at the next collection instance (i.e., the temporal resolution of the collection, the distribution of UEs for which data is being collected, etc.). Based on the continuous evaluation results, a decision may be made on how to use the data after collection for teaching one or more AI / ML models to solve one or more positioning tasks.

[0100] Hereinafter, exemplary embodiments of the present disclosure will be described in detail with reference to the accompanying drawings.

[0101] Figure 2 A signaling flow 200 of an example communication process according to some embodiments of the present disclosure is shown. The signaling flow 200 involves an LMF 210 (which may include a DCM) and one or more terminal devices 220-1, 220-2 (collectively or individually referred to as terminal devices 220). In some example embodiments, the LMF 210 may be replaced with a DCM. In some example embodiments, the terminal device 220 may include one or more UEs, PRUs, and / or other entities that may be configured to collect and report data samples for AI / ML services. Although two terminal devices 220 are shown, it should be appreciated that there may be more or fewer terminal devices.

[0102] exist Figure 2 , terminal device 220-1 sends (218) bitmap information. Terminal device 220-2 sends (222) bitmap information. Accordingly, LMF210 receives (219) bitmap information from terminal device 220-1 and receives (224) bitmap information from terminal device 220-2. Such bitmap information may indicate the respective locations of data samples collected by each terminal device 220. Terminal device 220 may be configured for data collection for artificial intelligence (AI) / machine learning (ML) services. In some example embodiments, the AI / ML services may include AI / ML positioning services, although other AI / ML services in the communication system may also be applicable.

[0103] In some example embodiments, to configure terminal device 220 for data collection, LMF 210 may send (209) a data collection method setting including requirement information to terminal device 220-1. LMF 210 may send (214) a data collection method setting including requirement information to terminal device 220-2. If the reported capabilities are considered data collection conditions (to indicate measurements that can be collected by a particular terminal device and specific LCM steps supported), the data collection method setting is triggered by the LMF on the particular terminal device. Accordingly, terminal device 220-1 receives (212) the data collection method setting from LMF 210, and terminal device 220-2 receives (216) the data collection method setting from LMF 210.

[0104] In some example embodiments, the data collection method settings may include requirement information for data collection, the requirement information indicating at least a target radio quality range. For AI / ML direct positioning, in an embodiment, if available, each terminal device performing data set collection includes a timestamp and ground truth label (2D / 3D position) mapped to a specific measurement (e.g., CIR, PDP, DP, RSRP). Optionally, in another embodiment, each terminal device 220 performing data set collection includes a timestamp and ground truth label of an AIML-assisted positioning task (e.g., TOA; LOS) mapped to a specific measurement (e.g., CIR, PDP, DP, RSRP).

[0105] In some example embodiments, each terminal device 220 may determine (217, 221) a quality sub-level of the collected data samples and select data samples that have satisfied the quality sub-level in the requirement information. In some example embodiments, each terminal device 220 evaluates the quality sub-level for a sub-data set that has been collected locally. In some example embodiments, the quality sub-level (sometimes referred to as a third quality sub-level) determined at the terminal device 220 may include a target radio quality range that is configured for the terminal device 220 to select the data samples indicated in the bitmap information.

[0106] For example, the range of SINR is considered to define the quality sub-level of Q1. Only samples that achieve this quality target are considered in each sub-data set stored locally in each terminal device 220. The terminal device 220 can select the data samples to be indicated in the bitmap information according to the requirement information. For example, the terminal device 220 can select the data samples with the corresponding radio quality within the target radio quality range.

[0107] In some example embodiments, each terminal device 220 sends a bitmap of its location to the DCM located in the LMF 210. Note that only the bitmap information is transmitted without any measurement transmission or data sample transmission.

[0108] In some example embodiments, the bitmap information (or simply bitmap) from the terminal device 220 may include a plurality of bits corresponding to a plurality of location grids within the geographic area, respectively. The bit may be set with a first value or a second value to indicate whether the data sample is collected by the terminal device 220 in the corresponding location grid.

[0109] exist Figure 2In the example, LMF 210 may determine a quality level of data samples collected by terminal device 220 based at least in part on the bitmap information and determine (228) whether the quality level of the data samples collected by terminal device 220 meets a target quality level. LMF 210 may determine an aggregate quality level of data samples from multiple terminal devices 220 (quality levels of sub-data sets at corresponding terminal devices) to evaluate the aggregate quality level of the entire data set.

[0110] In some example embodiments, based on a determination that the quality level fails to meet the target quality level, LMF 210 may transmit a request to update the collected data samples to terminal device 220. LMF 210 may receive updated bitmap information from terminal device 220. The updated bitmap information may indicate corresponding locations of updated data samples collected by terminal device 220. LMF 210 may update the quality level based at least in part on the updated bitmap information. LMF 210 may select data samples to be included in the dataset for the AI / ML service by further comparing the updated quality level with the target quality level.

[0111] In some example embodiments, according to the determination that the quality level satisfies the target quality level, the LMF 210 may select data samples to be included in the dataset for the AI / ML service from among the data samples indicated by the bitmap information.

[0112] That is, LMF 210 may determine the quality level of the data sample based on the bitmap information. LMF 210 then compares the quality level with the target quality level. If the determined quality level does not reach the target quality level, LMF 210 performs (226) iterations between LMF 210 and terminal device 220 until the target quality level is reached.

[0113] exist Figure 2 In the example, the LMF 210 may select ( 230 ) data samples to be included in the dataset for the AI / ML service by comparing the quality level with a target quality level for the dataset.

[0114] exist Figure 2 , LMF 210 may transmit a request (232) to report the selected data sample to terminal device 220-1. LMF 210 may send (236) a request to report the selected data sample to terminal device 220-2. Such data sample may be included in a data set to a target entity. Accordingly, terminal device 220-1 receives (234) the request, and terminal device 220-2 receives (238) the request.

[0115] In some example embodiments, LMF 210 may send confirmation information to terminal device 220. Such confirmation information may be related to the data sample to be included in the data set. The confirmation information includes at least bitmap information for selecting the data sample. In some example embodiments, LMF 210 may transmit the confirmation information to terminal device 220 together with the request.

[0116] In some example embodiments, LMF 210 may receive identification information from terminal device 220. Such identification information may indicate at least one geographic area where data samples are collected. LMF 210 may determine an aggregation area based on the identification information, within which data samples are selected to be included in the data set. LMF 210 may select data samples collected within the aggregation area. That is, during the above process, terminal device 220 may send the identification information together with the bitmap information.

[0117] In some example embodiments, LMF 210 may integrate bitmap information received from several terminal devices 210 and provide feedback to each terminal device 220 based on the target quality level requirements (e.g., samples / m2, location distribution of ground truth labels of terminal devices 220) defined for each specific dataset (e.g., dataset id_1), in other words, LMF 210 evaluates QL and dataset size targets. Here, some interaction between LMF 210 and terminal devices 210 is expected to set different PRSs until the target quality level has been reached.

[0118] These requirements or factors for estimating goodness may vary for each data collection used to implement any particular LCM step (such as model training, monitoring, or updating / fine-tuning). In some example embodiments, the target quality level is determined based on the use of the data set in the AI / ML service. For example, if the purpose of the data set is to train an AI / ML model, the quality level may be set to be higher than the target quality level used to collect the data set for monitoring. The requirements may be, for example, a specific terminal device 210 distribution (e.g., uniformly distributed terminal devices 210) and a signal quality at the receiver side (e.g., SINR), which should be weighted averaged according to the objectives in the LCM step.

[0119] An aggregate and unique target quality level can be determined using the following equation:

[0120] x u" =Ql(Ql # )+Qi(Qi # )+D(D # )

[0121] (Q # )+(Qi # )+(D# )=1

[0122] in:

[0123] x u" : The only factor of the target quality level;

[0124] Ql: minimum label quality of the dataset distribution (also called dataset distribution requirement);

[0125] Q # : The weight of the minimum label quality of the dataset distribution;

[0126] Qi: minimum quality of radio measurements (e.g., SINR range, QCI range, RSRP range, etc.) (also known as radio quality requirement);

[0127] Qi # : Weight of the minimum quality of the radio measurement;

[0128] D: predetermined minimum density (also called density requirement); and

[0129] D # : Weight of the predetermined minimum density.

[0130] For example, in the training step in the LCM framework, Ql # =0.4,Qi # =0.3,Dd # = 0.3; In other instances of monitoring steps in the LCM framework, QI # =0.2,Qi # =0.6,D # =0.2. It should be understood that these values ​​are only examples and other values ​​may be configured.

[0131] In some example embodiments, the target quality level may be determined based on a target data set size required for the data set to be collected. LMF 210 may require terminal device 220 to collect sufficient data samples, or may select sufficient data samples from the collected data samples until the target data size is met.

[0132] In some example embodiments, LMF 210 may determine at least one of the following based at least in part on the bitmap information: a first quality sub-level corresponding to a data set distribution requirement of the data set, a second quality sub-level corresponding to a density requirement of the data set, a third quality sub-level corresponding to a radio quality requirement of the data set, and a fourth quality sub-level corresponding to a target data set size required by the data set. LMF 210 may determine the quality level based on a weighted aggregation of at least one of the first, second, third, and fourth quality sub-levels. The determined quality level may reflect the quality of data samples currently collected on the terminal device 220 side. The aggregation of the quality sub-levels may follow the same manner as determining the target quality level.

[0133] In some example embodiments, the data set distribution requirement may indicate a uniform distribution of data samples. LFM 210 may be caused to determine the overall distribution of data samples collected by terminal device 220 based on the bitmap information. LFM 210 may be caused to determine the first quality sub-grade based on the similarity between the determined distribution and the uniform distribution. It is expected that the data samples are collected according to the uniform distribution. Therefore, if the determined distribution is close to the uniform distribution, the quality sub-grade may be determined as a high grade.

[0134] In some example embodiments, LMF 210 may determine, based on the bitmap information, a total number of data samples collected by terminal device 220 and a total area size of the data samples collected by terminal device 220. LMF 210 may determine, based on the total number of data samples and the total area size, a density of the data samples collected by terminal device 220. LMF 210 may determine a second quality sub-level based on a difference between the determined density and the density requirement.

[0135] In some example embodiments, LMF 210 may determine, based on the bitmap information, a total number of data samples collected by terminal device 220. LMF 210 may determine the fourth quality sublevel by comparing the determined total number of data samples with a target data set size required for the data set.

[0136] In some example embodiments, the quality level may be determined based on a weighted aggregation of at least one of the first, second, and third quality sub-levels. LMF 210 may determine whether the total number of data samples collected by terminal device 220 satisfies a target data set size required by the data set. Based on the determination that the total number of data samples satisfies the target data set size and based on the determination that the quality level satisfies the target quality level, LMF 210 may select data samples to be included in the data set.

[0137] In some example embodiments, the high quality level (or goodness) of the training data may be referenced to cause an accurate positioning AI / ML positioning model to be positioned. Goodness is defined by a set of metrics, which are summarized here and also described in detail in the next section.

[0138] In some example embodiments, "teaching to solve" any operation may refer to changing the results of using an AI / ML model (e.g., training, updating, refining an AI / ML model for a localization task).

[0139] In some example embodiments, the quality level (or goodness) of the positioning training data may be defined as a training data label (where the label may be a 2D position, 3D position, TOA, etc. of the UE). For example, a selected distribution may be followed in a range R (e.g., a uniform distribution Unif(R)), where R characterizes the geographic area for training the AI / ML positioning model. For another example, a minimum label quality Q1 may be included.

[0140] In some example embodiments, the quality level (or goodness) of the positioning training data can be defined as the training data input features (e.g., impulse response (CIR), reference signal received power (RSRP), reference signal received power per path (RSRPP), power delay profile (PDP), delay profile (DP)) with the lowest quality, where the quality Qi depends at least on the SINR standard of the received positioning signal. To ensure Qi, the PRS should be collected in the SINR range characterized by the range Rsinr = [SINRmin, SINRmax].

[0141] In some example embodiments, the goodness of positioning training data may be defined as the training data set size S or density D reaching a predetermined minimum value, where density is described as the number of samples per square meter, such as the number of CIR estimates obtained per square meter.

[0142] Still reference Figure 2 , after the above process, each terminal device 220 transmits ( 240 ) the sub-dataset (ground truth labels and specific measurements) to the target entity.

[0143] In some example embodiments, prior to the above process, terminal device 220-1 sends (202) capability information. Terminal device 220-2 sends (206) capability information. Terminal device 220 reports its capabilities (conditions) regarding a specific feature or feature group related to AI / ML positioning. Accordingly, LMF 210 receives (204) capability information from terminal device 220-1 and receives (208) capability information from terminal device 220-2.

[0144] In some example embodiments, data evaluation for quality aims to guarantee a minimum value of dataset quality to maximize AI / ML performance based on LCM steps (e.g., training, monitoring, fine-tuning). The evaluation is used to append specialized metadata information (e.g., distribution of ground truth labels) and some radio KPIs (e.g., SINR of each link included in the dataset) to the dataset_ID.

[0145] In some example embodiments, the DCM manages the construction of any specific dataset identified by dataset_ID, which is intended to facilitate storage and use at each LCM step. Only bitmap information in the LMF is used, reducing signal overhead impact. The selection of specific ground truth labels and their corresponding measurements enhances the efficient use of protocols for transferring datasets between entities when needed.

[0146] Figure 3 An example of three different sub-datasets according to the present disclosure is shown. Dataset 310 may be aggregated from sub-dataset 320 from UE_A, sub-dataset 330 from UE_B, and sub-dataset 340 from UE_C. Sub-dataset 320 corresponds to a bitmap indicating the location of UE_A, with sub-dataset ID_1. Sub-dataset 330 corresponds to a bitmap indicating the location of UE_B, with sub-dataset ID_2. Sub-dataset 340 corresponds to a bitmap indicating the location of UE_C, with sub-dataset ID_3.

[0147] exist Figure 3 In , each sub-dataset is associated with a unique entity (e.g., UE, PRU and / or LMF) for data collection. The goal is to estimate the UE location (ground truth) mapped to the bitmap. The bitmap is used to enable the UE distribution analysis in the following steps.

[0148] exist Figure 3 In the example, the sub-dataset mentioned is a local ID considered on each specific TRP 305 (with Cell ID_1). It is thus expected that other TRPs (Cell ID) can reuse similar identifications, but always keep them in the local scope. In the opposite direction, the Dataset_ID (e.g., Dataset id_1) is intended to identify a dataset 310 with a regional scope. Several Cell_IDs can be mapped by the same Dataset_ID.

[0149] In some example embodiments, for a bitmap from a terminal device 220 corresponding to a certain geographic area, a grid granularity for a plurality of location grids within the geographic area is determined based on the requirements for the data set. The geographic area may correspond to a cell where the terminal device 220 is located and is identified by a cell ID.

[0150] exist Figure 4 , a representation of a cell-specific bitmap is shown. It shows a bitmap approach for each specific cell ID and a composition between different cell IDs to obtain a desired data set area. The bitmap can be interpreted as a discretized position representation, where the discretization is on a grid with a cell-specific granularity. As shown, bitmap 410, bitmap 420, and bitmap 430 with the same cell ID can be divided at different grid granularities to indicate the location where the terminal device collects data samples. In some example embodiments, a set of cell IDs can constitute a desired data set area, for example, a data set corresponding to an aggregated bitmap 440, where a bit is set to 1 to indicate that data samples are collected in a corresponding position.

[0151] Example Method

[0152] Figure 5 A flowchart of an example method 500 implemented at a first device according to some example embodiments of the present disclosure is shown. For the purpose of discussion, Figure 1 The method 500 is described from the perspective of the DCM 110 or the LMF 120 in FIG. For convenience, the exemplary embodiments of the present disclosure will be described below by taking the LMF 120 as an example.

[0153] At box 510, the LMF 120 receives bitmap information from at least one second device, the bitmap information indicating corresponding locations of data samples collected by the at least one second device, and the at least one second device is configured for data collection for artificial intelligence (AI) / machine learning (ML) services.

[0154] At block 520 , LMF 120 determines a quality level of data samples collected by at least one second device based at least in part on the bitmap information.

[0155] At block 530 , the LMF 120 selects data samples to be included in the dataset for the AI / ML service by comparing the quality level to a target quality level for the dataset.

[0156] At block 540 , the LMF 120 sends a request to at least one second device to report the selected data samples to be included in the data set to the target entity.

[0157] In some example embodiments, method 500 further includes: based on a determination that the quality level does not meet the target quality level, LMF 120 transmitting a request to update the collected data samples to at least one second device; receiving updated bitmap information from the at least one second device, the updated bitmap information indicating corresponding locations of updated data samples collected by the at least one second device; updating the quality level based at least in part on the updated bitmap information; and selecting data samples to be included in the data set for the AI / ML service by further comparing the updated quality level with the target quality level.

[0158] In some example embodiments, the method 500 further includes: based on the determination that the quality level satisfies the target quality level, the LMF 120 selects data samples to be included in the data set for the AI / ML service from the data samples indicated by the bitmap information.

[0159] In some example embodiments, the target quality level is determined based on the use of the dataset in an AI / ML service.

[0160] In some example embodiments, the method 500 further includes: the LMF 120 sending confirmation information related to the data samples to be included in the data set to the at least one second device, the confirmation information including at least bitmap information for selecting the data samples.

[0161] In some example embodiments, the bitmap information from the second device includes a plurality of bits corresponding to a plurality of location grids within the geographic area, respectively, and the bits are set with the first value or the second value to indicate whether the data sample is collected by the second device in the corresponding location grid.

[0162] In some example embodiments, a grid granularity of a plurality of location grids within a geographic area is determined based on requirements for a data set.

[0163] In some example embodiments, method 500 further includes: LMF 120 receiving identification information indicating at least one geographic area where data samples are collected from at least one second device; determining an aggregation area based on the identification information, selecting data samples to be included in the data set within the aggregation area; and selecting the data samples collected within the aggregation area.

[0164] In some example embodiments, method 500 further includes: LMF 120 determining, at least in part based on the bitmap information, at least one of: a first quality sublevel corresponding to a data set distribution requirement of the data set, a second quality sublevel corresponding to a density requirement of the data set, a third quality sublevel corresponding to a radio quality requirement of the data set, and a fourth quality sublevel corresponding to a target data set size required by the data set; and determining the quality level based on a weighted aggregation of at least one of the first, second, third and fourth quality sublevels.

[0165] In some example embodiments, method 500 further includes: the data set distribution requirement indicates a uniform distribution of data samples, and LMF 120 determines an overall distribution of data samples collected by at least one second device based on the bitmap information; and determining the first quality sub-level based on a similarity between the determined distribution and the uniform distribution.

[0166] In some example embodiments, method 500 further includes: LMF 120 determining a total number of data samples collected by at least one second device and a total area size of the data samples collected by at least one second device based on the bitmap information; determining a density of the data samples collected by the at least one second device based on the total number of data samples and the total area size; and determining the second quality sub-level based on a difference between the determined density and the density requirement.

[0167] In some example embodiments, the third quality sublevel comprises a target radio quality range configured for the at least one second device to select the data samples indicated in the bitmap information.

[0168] In some example embodiments, the method 500 further includes: the LMF 120 sending requirement information for data collection to at least one second device, the requirement information indicating at least a target radio quality range.

[0169] In some example embodiments, method 500 further includes: LMF 120 determining a total number of data samples collected by at least one second device based on the bitmap information; and determining the fourth quality sublevel by comparing the determined total number of data samples with a target data set size required for the data set.

[0170] In some example embodiments, a quality level is determined based on a weighted aggregation of at least one of the first, second, and third quality sublevels; and LMF 120 determines whether a total number of data samples collected by at least one second device satisfies a target data set size required by the data set; and data samples to be included in the data set are selected based on a determination that the total number of data samples satisfies the target data set size and based on a determination that the quality level satisfies the target quality level.

[0171] In some example embodiments, the first apparatus comprises a location management function (LMF) and the at least one second apparatus comprises at least one of a terminal device or a positioning reference unit (PRU).

[0172] In some example embodiments, the AI / ML service comprises an AI / ML positioning service.

[0173] Figure 6 A flowchart of an example method 600 implemented at a second device according to some example embodiments of the present disclosure is shown. For the purpose of discussion, the method 600 will be described in detail in FIG. Figure 1 The method 600 is described from the perspective of the UE 130, the PRU 140 or the LMF 150 in the embodiment. For convenience, the exemplary embodiments of the present disclosure will be described below by taking the UE 130 as an example.

[0174] At block 610 , the UE 130 transmits bitmap information to the first device, the bitmap information indicating corresponding locations of data samples collected by the second device, the second device being configured for data collection for an artificial intelligence (AI) / machine learning (ML) service.

[0175] At block 620 , the UE 130 receives a request from the first device to report to a target entity the data samples selected to be included in the data set.

[0176] In some example embodiments, the method 600 further includes: the UE 130 receiving a request to update the collected data samples from the first device; and sending updated bitmap information indicating corresponding positions of the updated data samples collected by the second device to the first device.

[0177] In some example embodiments, the method 600 further includes: the UE 130 receiving, from the first device, confirmation information regarding the data samples to be included in the data set, the confirmation information including at least bitmap information for selecting the data samples.

[0178] In some example embodiments, the bitmap information from the second device includes a plurality of bits corresponding to a plurality of location grids within the geographic area, respectively, and the bits are set with the first value or the second value to indicate whether the data sample is collected by the second device in the corresponding location grid.

[0179] In some example embodiments, a grid granularity of a plurality of location grids within a geographic area is determined based on requirements for a data set.

[0180] In some example embodiments, the method 600 further includes: the UE 130 sending, to the first device, identification information indicating a geographical area in which the data samples are collected.

[0181] In some example embodiments, method 600 further includes: UE 130 receiving requirement information for data collection from the first device, the requirement information at least indicating a target radio quality range corresponding to a radio quality requirement of the data set; and selecting data samples in the bitmap information that need to be indicated in the bitmap information according to the requirement information.

[0182] In some example embodiments, method 600 further includes: UE 130 selecting data samples having corresponding radio qualities within a target radio quality range.

[0183] In some example embodiments, the first apparatus comprises a location management function (LMF) and the at least one second apparatus comprises at least one of a terminal device or a positioning reference unit (PRU).

[0184] In some example embodiments, the AI / ML service comprises an AI / ML positioning service.

[0185] Example devices, equipment, and media

[0186] In some example embodiments, a first device (eg, Figure 1 The DCM 110 or LMF 120 in the method 500 may include means for performing the corresponding operations of the method 500. The means may be implemented in any suitable form. For example, the means may be implemented in a circuit or a software module. The first means may be implemented as or included in Figure 1 in the DCM 110 or the LMF 120.

[0187] In some example embodiments, the first device includes: a component for receiving bitmap information from at least one second device, the bitmap information indicating corresponding locations of data samples collected by the at least one second device, the at least one second device being configured for data collection for an artificial intelligence (AI) / machine learning (ML) service; a component for determining a quality level of the data samples collected by the at least one second device based at least in part on the bitmap information; a component for selecting data samples to be included in the data set by comparing the quality level with a target quality level of the data set for the AI / ML service; and a component for sending a request to the at least one second device to request reporting of the selected data samples to be included in the data set to a target entity.

[0188] In some example embodiments, the first device further includes: a component for sending an update of the collected data samples to the at least one second device if it is determined that the quality level does not meet the target quality level; a component for receiving updated bitmap information from the at least one second device, the updated bitmap information indicating corresponding positions of updated data samples collected by the at least one second device; a component for updating the quality level based at least in part on the updated bitmap information; and a component for selecting data samples to be included in the data set for the AI / ML service by further comparing the updated quality level with the target quality level.

[0189] In some example embodiments, the first apparatus further comprises means for selecting, from among the data samples indicated by the bitmap information, data samples to be included in the data set for the AI / ML service if it is determined that the quality level satisfies the target quality level.

[0190] In some example embodiments, the target quality level is determined based on usage of the dataset in the AI / ML service.

[0191] In some example embodiments, the first device further comprises: means for sending confirmation information related to the data sample to be included in the data set to the at least one second device, the confirmation information comprising at least bitmap information for selecting the data sample.

[0192] In some example embodiments, the bitmap information from the second device includes a plurality of bits corresponding to a plurality of location grids within a geographic area, respectively, and the bits are set with a first value or a second value to indicate whether the data sample is collected by the second device in the corresponding location grid.

[0193] In some example embodiments, a grid granularity of the plurality of location grids within the geographic area is determined based on requirements of the data set.

[0194] In some example embodiments, the first device further includes: a component for receiving identification information indicating at least one geographic area where the data samples are collected from the at least one second device; a component for determining an aggregation area based on the identification information and selecting data samples to be included in the data set within the aggregation area; and a component for selecting data samples collected within the aggregation area.

[0195] In some example embodiments, the first device further comprises: means for determining, based at least in part on the bitmap information, at least one of: a first quality sublevel corresponding to a data set distribution requirement of the data set, a second quality sublevel corresponding to a density requirement of the data set, a third quality sublevel corresponding to a radio quality requirement of the data set, and a fourth quality sublevel corresponding to a target data set size required for the data set; and determining the quality level based on a weighted aggregation of the at least one of the first quality sublevel, the second quality sublevel, the third quality sublevel, and the fourth quality sublevel.

[0196] In some example embodiments, the data set distribution requirement indicates a uniform distribution of data samples, and wherein the first device further includes: a component for determining a total distribution of data samples collected by the at least one second device based on the bitmap information; and a component for determining the first quality sub-level based on a similarity between the determined total distribution and the uniform distribution.

[0197] In some example embodiments, the first device further includes: a component for determining a total number of data samples collected by the at least one second device and a total area size of the data samples collected by the at least one second device based on the bitmap information; a component for determining a density of the data samples collected by the at least one second device based on the total number of data samples and the total area size; and a component for determining the second quality sub-level based on a difference between the determined density and the density requirement.

[0198] In some example embodiments, the third quality sublevel comprises a target radio quality range configured for the at least one second device to select the data samples indicated in the bitmap information.

[0199] In some example embodiments, the first device further comprises: means for sending requirement information for data collection to the at least one second device, the requirement information indicating at least the target radio quality range.

[0200] In some example embodiments, the first device further comprises: means for determining, based on the bitmap information, a total number of data samples collected by the at least one second device; and means for determining the fourth quality sublevel by comparing the determined total number of data samples with the target data set size required for the data set.

[0201] In some example embodiments, the quality level is determined based on a weighted aggregation of at least one of the first quality sublevel, the second quality sublevel, and the third quality sublevel; and wherein the first device further comprises: a component for determining whether the total number of data samples collected by the at least one second device satisfies a target data set size required by the data set; and a component for selecting data samples to be included in the data set if it is determined that the total number of data samples satisfies the target data set size and if it is determined that the quality level satisfies the target quality level.

[0202] In some example embodiments, the first apparatus comprises a location management function (LMF) and the at least one second apparatus comprises at least one of a terminal device or a positioning reference unit (PRU).

[0203] In some example embodiments, the AI / ML service comprises an AI / ML positioning service.

[0204] In some example embodiments, the first apparatus further comprises means for performing method 500 or other operations in some example embodiments of LMF 120 or DCM 110. In some example embodiments, the apparatus comprises at least one processor and at least one memory storing instructions that, when executed by the at least one processor, cause execution of the first apparatus.

[0205] In some example embodiments, a second device (eg, Figure 1 The UE 130 or PRU 140 in the embodiment may include a device for performing the corresponding operation of the method 600. The device may be implemented in any suitable form. For example, the device may be implemented in a circuit or a software module. The second device may be implemented as or included in Figure 1 in the UE 130 or the PRU 140.

[0206] In some example embodiments, the second device includes: a component for sending bitmap information to the first device, the bitmap information indicating corresponding locations of data samples collected by the second device, the second device being configured for data collection for artificial intelligence (AI) / machine learning (ML) services; and a component for receiving a request from the first device to request reporting to a target entity the selected data samples to be included in the data set.

[0207] In some example embodiments, the second device further includes: a component for receiving a request to update the collected data samples from the first device; and a component for sending updated bitmap information to the first device, the updated bitmap information indicating corresponding locations of the updated data samples collected by the second device.

[0208] In some example embodiments, the second device further comprises: means for receiving confirmation information related to the data sample to be included in the data set from the first device, the confirmation information comprising at least bitmap information for selecting the data sample.

[0209] In some example embodiments, the bitmap information from the second device includes a plurality of bits corresponding to a plurality of location grids within a geographic area, respectively, and the bits are set with a first value or a second value to indicate whether the data sample is collected by the second device in the corresponding location grid.

[0210] In some example embodiments, a grid granularity of the plurality of location grids within the geographic area is determined based on requirements of the data set.

[0211] In some example embodiments, the second device further comprises means for transmitting, to the first device, identification information indicating a geographical area where the data sample is collected.

[0212] In some example embodiments, the second device further comprises: a component for receiving requirement information for data collection from the first device, the requirement information indicating at least a target radio quality range corresponding to the radio quality requirement of the data set; and a component for selecting the data samples to be indicated in the bitmap information based on the requirement information.

[0213] In some example embodiments, the second apparatus further comprises means for selecting the data samples having respective radio qualities within the target radio quality range.

[0214] In some example embodiments, the first apparatus comprises a location management function (LMF) and the at least one second apparatus comprises at least one of a terminal device or a positioning reference unit (PRU).

[0215] In some example embodiments, the AI / ML service comprises an AI / ML positioning service.

[0216] In some example embodiments, the second apparatus further comprises means for performing method 600 or other operations of some example embodiments of UE 130 or PRU 140. In some example embodiments, the apparatus comprises at least one processor and at least one memory storing instructions that, when executed by the at least one processor, cause execution of the second apparatus.

[0217] Figure 7 700 is a simplified block diagram of a device 700 suitable for implementing an example embodiment of the present disclosure. The device 700 may be configured to implement a communication device, such as Figure 1 The DCM 110, LMF 120, UE 130, PRU 140 or LMF 150 shown in FIG. As shown in the figure, the device 700 includes one or more processors 710, one or more memories 720 coupled to the processor 710, and one or more communication modules 740 coupled to the processor 710.

[0218] The communication module 740 is used for two-way communication. The communication module 740 has one or more communication interfaces to facilitate communication with one or more other modules or devices. The communication interface may represent any interface necessary to communicate with other network elements. In some example embodiments, the communication module 740 may include at least one antenna.

[0219] Processor 710 may be of any type suitable for the local technology network and may include one or more of the following: a general purpose computer, a special purpose computer, a microprocessor, a digital signal processor (DSP), and a processor based on a multi-core processor architecture, as non-limiting examples. Device 700 may have multiple processors, such as application specific integrated circuit chips that are time slaved to a clock that synchronizes a main processor.

[0220] The memory 720 may include one or more non-volatile memories and one or more volatile memories. Examples of non-volatile memories include, but are not limited to, read-only memory (ROM) 724, electrically programmable read-only memory (EPROM), flash memory, hard disk, compact disk (CD), digital video disk (DVD), optical disk, laser disk, and other magnetic and / or optical memories. Examples of volatile memories include, but are not limited to, random access memory (RAM) 722 and other volatile memories that will not last for the duration of a power outage.

[0221] The computer program 730 includes computer executable instructions executed by the associated processor 710. The instructions of the program 730 may include instructions for performing the operations / actions of some example embodiments of the present disclosure. The program 730 may be stored in a memory (e.g., ROM 724). The processor 710 may perform any suitable actions and processes by loading the program 730 into the RAM 722.

[0222] The exemplary embodiments of the present disclosure may be implemented by a program 730 so that the device 700 may execute the Figures 2 to 6 Any process of the present disclosure discussed. Example embodiments of the present disclosure may also be implemented by hardware or a combination of software and hardware.

[0223] In some example embodiments, the program 730 may be tangibly embodied in a computer-readable medium that may be included in the device 700 (such as in the memory 720) or in other storage devices accessible by the device 700. The device 700 may load the program 730 from the computer-readable medium to the RAM 722 for execution. In some example embodiments, the computer-readable medium may include any type of non-transitory storage medium, such as ROM, EPROM, flash memory, hard disk, CD, DVD, etc. The term "non-transitory" as used herein refers to the limitation of the medium itself (i.e., tangible, not a signal), as opposed to the limitation on the persistence of data storage (e.g., RAM versus ROM).

[0224] Figure 8 An example of a computer readable medium 800 is shown, which may be in the form of a CD, DVD or other optical storage disk. The computer readable medium 800 has a program 730 stored thereon.

[0225] The following examples are provided and described herein.

[0226] Example 1. A first device comprises: at least one processor; and at least one memory, wherein the at least one memory stores instructions, which, when executed by the at least one processor, cause the first device to at least perform: receiving bitmap information from at least one second device, the bitmap information indicating corresponding locations of data samples collected by the at least one second device, the at least one second device being configured for data collection for an artificial intelligence (AI) / machine learning (ML) service; determining a quality level of the data samples collected by the at least one second device based at least in part on the bitmap information; selecting data samples to be included in the data set by comparing the quality level with a target quality level of the data set for the AI / ML service; and sending a request to the at least one second device to request reporting of the selected data samples to be included in the data set to a target entity.

[0227] Example 2. An apparatus according to Example 1, wherein the first apparatus is caused to execute: if it is determined that the quality level does not meet the target quality level, sending a request to update the collected data samples to the at least one second apparatus; receiving updated bitmap information from the at least one second apparatus, the updated bitmap information indicating corresponding positions of updated data samples collected by the at least one second apparatus; updating the quality level based at least in part on the updated bitmap information; and selecting data samples to be included in the data set for the AI / ML service by further comparing the updated quality level with the target quality level.

[0228] Example 3. The apparatus of example 1 or 2, wherein the first apparatus is caused to execute:

[0229] If it is determined that the quality level satisfies the target quality level, data samples to be included in the data set for the AI / ML service are selected from the data samples indicated by the bitmap information.

[0230] Example 4. An apparatus according to any one of Examples 1 to 3, wherein the target quality level is determined based on the use of the data set in the AI / ML service.

[0231] Example 5. An apparatus according to any one of Examples 1 to 4, wherein the first apparatus is further caused to execute: sending confirmation information related to the data sample to be included in the data set to the at least one second apparatus, the confirmation information comprising at least bitmap information for selecting the data sample.

[0232] Example 6. An apparatus according to any one of Examples 1 to 5, wherein the bitmap information from the second device includes a plurality of bits corresponding to a plurality of location grids within a geographic area, respectively, and the bits are set with a first value or a second value to indicate whether the data sample is collected by the second device in the corresponding location grid.

[0233] Example 7. The apparatus of Example 6, wherein a grid granularity of the plurality of location grids within the geographic area is determined based on requirements of the data set.

[0234] Example 8. An apparatus according to any one of Examples 1 to 7, wherein the first apparatus is caused to perform: receiving identification information indicating at least one geographic area where the data samples are collected from the at least one second apparatus; determining an aggregation area based on the identification information, selecting data samples to be included in the data set within the aggregation area; and selecting data samples to be collected within the aggregation area.

[0235] Example 9. An apparatus according to any one of Examples 1 to 8, wherein the first apparatus is caused to perform: determining, based at least in part on the bitmap information, at least one of the following: a first quality sublevel corresponding to a data set distribution requirement of the data set, a second quality sublevel corresponding to a density requirement of the data set, a third quality sublevel corresponding to a radio quality requirement of the data set, and a fourth quality sublevel corresponding to a target data set size required for the data set; and determining the quality level based on a weighted aggregation of the at least one of the first quality sublevel, the second quality sublevel, the third quality sublevel, and the fourth quality sublevel.

[0236] Example 10. An apparatus according to Example 9, wherein the data set distribution requirement indicates a uniform distribution of data samples, and wherein the first apparatus is caused to perform: determining a total distribution of data samples collected by the at least one second apparatus based on the bitmap information; and determining the first quality sub-level based on a similarity between the determined total distribution and the uniform distribution.

[0237] Example 11. An apparatus according to Example 9 or 10, wherein the first apparatus is caused to perform: determining a total number of data samples collected by the at least one second apparatus and a total area size of the data samples collected by the at least one second apparatus based on the bitmap information; determining a density of the data samples collected by the at least one second apparatus based on the total number of data samples and the total area size; and determining the second quality sub-level based on a difference between the determined density and the density requirement.

[0238] Example 12. An apparatus according to any of Examples 8 to 11, wherein the third quality sublevel includes a target radio quality range, and the target radio quality range is configured for the at least one second apparatus to select the data samples indicated in the bitmap information.

[0239] Example 13. The apparatus of Example 12, wherein the first apparatus is further caused to perform: sending requirement information for data collection to the at least one second apparatus, the requirement information indicating at least the target radio quality range.

[0240] Example 14. An apparatus according to any one of Examples 8 to 13, wherein the first apparatus is caused to perform: determining a total number of data samples collected by the at least one second apparatus based on the bitmap information; and determining the fourth quality sublevel by comparing the determined total number of data samples with the target data set size required for the data set.

[0241] Example 15. An apparatus according to any one of Examples 8 to 14, wherein the quality level is determined based on a weighted aggregation of the first quality sublevel, the second quality sublevel, and the at least one of the third quality sublevels; and wherein the first apparatus is further caused to perform: determining whether the total number of data samples collected by the at least one second apparatus satisfies a target dataset size required by the dataset; and if it is determined that the total number of data samples satisfies the target dataset size and if it is determined that the quality level satisfies the target quality level, selecting data samples to be included in the dataset.

[0242] Example 16. An apparatus according to any one of Examples 1 to 15, wherein the first apparatus includes a location management function (LMF) and the at least one second apparatus includes at least one of: a terminal device, or a positioning reference unit (PRU).

[0243] Example 17. A device according to any one of Examples 1 to 16, wherein the AI / ML service includes an AI / ML positioning service.

[0244] Example 18. A second device, comprising: at least one processor; and at least one memory, the at least one memory storing instructions that, when executed by the at least one processor, cause the second device to at least perform: sending bitmap information to a first device, the bitmap information indicating corresponding locations of data samples collected by the second device, the second device being configured for data collection for artificial intelligence (AI) / machine learning (ML) services; and receiving a request from the first device to request reporting to a target entity the selected data samples to be included in the data set.

[0245] Example 19. An apparatus according to Example 18, wherein the second apparatus is further caused to execute: receiving a request to update the collected data samples from the first apparatus; and sending updated bitmap information to the first apparatus, wherein the updated bitmap information indicates corresponding positions of the updated data samples collected by the second apparatus.

[0246] Example 20. An apparatus according to Example 18 or 19, wherein the second apparatus is further caused to execute: receiving confirmation information related to the data sample to be included in the data set from the first apparatus, the confirmation information comprising at least bitmap information for selecting the data sample.

[0247] Example 21. An apparatus according to any one of Examples 18 to 20, wherein the bitmap information from the second device includes a plurality of bits corresponding to a plurality of location grids within a geographic area, respectively, and the bits are set with a first value or a second value to indicate whether the data sample is collected by the second device in the corresponding location grid.

[0248] Example 22. The apparatus of Example 21, wherein a grid granularity of the plurality of location grids within the geographic area is determined based on requirements of the data set.

[0249] Example 23. An apparatus according to any one of Examples 18 to 22, wherein the second apparatus is further caused to execute: transmitting to the first apparatus identification information indicating a geographic area where the data sample is collected.

[0250] Example 24. An apparatus according to any one of Examples 18 to 23, wherein the second apparatus is further caused to perform: receiving requirement information for data collection from the first apparatus, the requirement information at least indicating a target radio quality range corresponding to the radio quality requirements of the data set; and selecting the data samples to be indicated in the bitmap information based on the requirement information.

[0251] Example 25. The apparatus of Example 24, wherein the second apparatus is caused to perform: selecting the data samples having corresponding radio qualities within the target radio quality range.

[0252] Example 26. An apparatus according to any one of Examples 18 to 25, wherein the first apparatus includes a location management function (LMF) and the at least one second apparatus includes at least one of: a terminal device, or a positioning reference unit (PRU).

[0253] Example 27. An apparatus according to any one of Examples 18 to 26, wherein the AI / ML service includes an AI / ML positioning service.

[0254] Example 28. A method comprising:

[0255] Receiving bitmap information at a first device and from at least one second device, the bitmap information indicating corresponding locations of data samples collected by the at least one second device, the at least one second device being configured for data collection for an artificial intelligence (AI) / machine learning (ML) service; determining a quality level of the data samples collected by the at least one second device based at least in part on the bitmap information; selecting data samples to be included in the data set by comparing the quality level with a target quality level of the data set for the AI / ML service; and sending a request to the at least one second device to request reporting of the selected data samples to be included in the data set to a target entity.

[0256] Example 29. A method comprising: sending bitmap information at a second device to a first device, the bitmap information indicating corresponding locations of data samples collected by the second device, the second device being configured for data collection for an artificial intelligence (AI) / machine learning (ML) service; and receiving a request from the first device to request reporting to a target entity selected data samples to be included in the data set.

[0257] Example 30. A first device, comprising: a component for receiving bitmap information from at least one second device, the bitmap information indicating corresponding locations of data samples collected by the at least one second device, the at least one second device being configured for data collection for an artificial intelligence (AI) / machine learning (ML) service; a component for determining a quality level of the data samples collected by the at least one second device based at least in part on the bitmap information; a component for selecting data samples to be included in the data set by comparing the quality level with a target quality level of the data set for the AI / ML service; and a component for sending a request to the at least one second device to request reporting of the selected data samples to be included in the data set to a target entity.

[0258] Example 31. A second device, comprising: a component for sending bitmap information to a first device, the bitmap information indicating corresponding locations of data samples collected by the second device, the second device being configured for data collection for artificial intelligence (AI) / machine learning (ML) services; and a component for receiving a request from the first device to request reporting to a target entity the selected data samples to be included in the data set.

[0259] Example 32. A computer-readable medium comprising instructions stored thereon, the instructions being used to cause an apparatus to perform at least the method according to Example 28 or the method according to Example 29.

[0260] Generally, various embodiments of the present disclosure may be implemented in hardware or dedicated circuits, software, logic, or any combination thereof. Some aspects may be implemented in hardware, and other aspects may be implemented in firmware or software, which may be executed by a controller, microprocessor, or other computing device. Although various aspects of the embodiments of the present disclosure are illustrated and described as block diagrams, flow charts, or using some other graphical representations, it should be understood that, as non-limiting examples, the blocks, devices, systems, techniques, or methods described herein may be implemented in hardware, software, firmware, dedicated circuits or logic, general hardware or controllers or other computing devices, or some combination thereof.

[0261] Some example embodiments of the present disclosure also provide at least one computer program product tangibly stored on a computer-readable medium (such as a non-volatile computer-readable medium). The computer program product includes computer executable instructions, such as those included in a program module, that are executed in a device on a target physical or virtual processor to perform any method as described above. Typically, a program module includes routines, programs, libraries, objects, classes, components, data structures, etc. that perform specific tasks or implement specific abstract data types. The functionality of a program module can be combined or split between program modules as needed in different embodiments. Machine executable instructions for program modules can be executed in local or distributed devices. In distributed devices, program modules can be located in local and remote storage media.

[0262] The program code for performing the method of the present disclosure may be written in any combination of one or more programming languages. The program code may be provided to a processor or controller of a general-purpose computer, a special-purpose computer, or other programmable data processing device so that the program code, when executed by the processor or controller, causes the implementation of the functions / operations specified in the flow chart and / or block diagram. The program code may be executed entirely on the machine, partially on the machine, as an independent software package, partially on the machine, partially on a remote machine, or entirely on a remote machine or server.

[0263] In the context of the present disclosure, computer program codes or related data may be carried by any suitable carrier to enable a device, apparatus or processor to perform the above different processes and operations. Examples of carriers include signals, computer readable media, etc.

[0264] The computer readable medium may be a computer readable signal medium or a computer readable storage medium. The computer readable medium may include, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, devices, or equipment, or any suitable combination of the foregoing. More specific examples of computer readable storage media will include an electrical connection with one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

[0265] Further, although the operations are depicted in a particular order, this should not be construed as requiring the execution of such operations in the particular order shown or in a sequential order, or the execution of all described operations, to achieve the desired result. In some cases, multitasking and parallel processing may be advantageous. Similarly, although several specific implementation details are included in the above discussion, these should not be construed as limitations on the scope of the present disclosure, but rather as descriptions of features that may be specific to a particular embodiment. Unless explicitly stated, certain features described in the context of a separate embodiment may also be implemented in combination in a single embodiment. On the contrary, unless explicitly stated, the different features described in the context of a single embodiment may also be implemented in multiple embodiments individually or in any suitable subcombination.

[0266] Although the disclosure has been described in language specific to structural features and / or methodological actions, it is to be understood that the disclosure defined in the appended claims is not necessarily limited to the specific features or actions described above. Instead, the specific features and actions described above are disclosed as example forms of implementing the claims.

[0267] Abbreviations used herein are listed below together with their unshortened representations. It should be noted that widely established and unique abbreviations may be assumed to be known.

[0268] List of abbreviations

[0269] CIR Channel Impulse Response

[0270] PDP Power Delay Profile

[0271] DP Delay Distribution

[0272] RSRP Reference Signal Received Power

[0273] RSRPP reference signal received power per path

[0274] TOA Time of Arrival

[0275] RSTD Reference Signal Time Difference

[0276] PRU Positioning Reference Unit

[0277] DCM Data Collection Management

[0278] KPI Key Performance Indicator

Claims

1. A second device, comprising: at least one processor; as well as at least one memory storing instructions that, when executed by the at least one processor, cause the second device to at least perform: sending bitmap information to the first device, the bitmap information indicating corresponding locations of data samples collected by the second device, the second device being configured for data collection for an artificial intelligence (AI) / machine learning (ML) service; as well as A request is received from the first device to report selected data samples to be included in the data set to a target entity.

2. The device according to claim 1, wherein: The second device is further caused to execute: receiving a request from the first device to update the collected data samples; and Updated bitmap information is sent to the first device, the updated bitmap information indicating corresponding locations of updated data samples collected by the second device.

3. The device according to any one of claims 1 or 2, wherein: The second device is further caused to execute: Confirmation information related to the data samples to be included in the data set is received from the first device, the confirmation information comprising at least bitmap information for selecting the data samples.

4. The device according to any one of claims 1 to 2, wherein: The bitmap information from the second device includes a plurality of bits corresponding to a plurality of location grids within the geographic area, respectively, and the bits are set with a first value or a second value to indicate whether the data sample is collected by the second device in the corresponding location grid.

5. The device according to claim 4, wherein: A grid granularity of the plurality of location grids within the geographic area is determined based on requirements of the data set.

6. The device according to any one of claims 1 to 2, wherein: The second device is further caused to execute: Identification information indicating the geographic area where the data sample was collected is transmitted to the first device.

7. The device according to any one of claims 1 to 2, wherein: The second device is further caused to execute: receiving requirement information for data collection from the first device, the requirement information indicating at least a target radio quality range corresponding to a radio quality requirement of the data set; as well as The data samples to be indicated in the bitmap information are selected according to the requirement information.

8. The device according to claim 7, wherein: The second device is caused to execute: The data samples having corresponding radio qualities within the target radio quality range are selected.

9. The device according to any one of claims 1 to 2, wherein: The first device comprises a location management function (LMF), and the at least one second device comprises at least one of: a terminal device, or a positioning reference unit (PRU).

10. The device according to any one of claims 1 to 2, wherein: The AI / ML service includes an AI / ML positioning service.

Images