AI / ML model test mechanism

Abstract

Example embodiments of the present disclosure relate to artificial intelligence / machine learning (AI / ML) model testing. And the first device sends test configuration information to the second device. The test configuration information indicates a test mode of an AI / ML model with respect to at least one transmit and receive unit (TRP). The at least one TRP is arranged within the environment based on a test plan for the at least one test channel indicator. The first device receives, from the second device, at least one predicted channel indicator for the at least one TRP. The at least one predicted channel indicator is derived by the second device using an AI / ML model. The first device determines a test result of the AI / ML model based on a comparison between the at least one predicted channel indicator and the at least one test channel indicator. The test result indicates whether the AI / ML model is verified.

Description

CROSS-REFERENCE TO RELATED APPLICATIONS

[0001] The present disclosure claims priority to and the benefit of Indian Provisional Application No. 202341055825 filed on August 21, 2023, the contents of which are incorporated herein by reference in their entirety. Technical Field

[0002] Various example embodiments of the present disclosure relate generally to the field of telecommunications, and in particular, to methods, devices, apparatuses, and computer-readable storage media for artificial intelligence / machine learning (AI / ML) model testing. Background Art

[0003] In the telecommunication industry, AI / ML has been adopted in telecommunication systems to improve performance. The 3rd Generation Partnership Project (3GPP) Release 18 started research on AI / ML for the New Radio (NR) air interface. The goal is to explore the benefits of enhancing the air interface with features that enable improved support for AI / ML-based algorithms to improve performance and / or reduce complexity / overhead. Several use cases are considered to enable the identification of common AI / ML frameworks, including functional requirements of AI / ML architectures that can be used in subsequent projects. It also aims to cover the interoperability and testability aspects of newly defined AI / ML-enabled features in communication systems. Summary of the invention

[0004] In a first aspect of the present disclosure, a first device is provided. The first device includes: at least one processor; and at least one memory storing instructions, which, when executed by the at least one processor, causes the first device to at least perform: sending test configuration information to a second device, the test configuration information indicating a test mode of an artificial intelligence / machine learning AI / ML model relative to at least one transmitting and receiving unit TRP, at least one TRP is arranged in an environment based on a test plan for at least one test channel indicator, and the test channel indicator indicates the probability that the communication channel between the second device and the TRP is a line-of-sight channel or a non-line-of-sight channel; receiving at least one predicted channel indicator for at least one TRP from the second device, at least one predicted channel indicator is derived by the second device using an AI / ML model; and determining a test result of the AI / ML model based on a comparison between the at least one predicted channel indicator and the at least one test channel indicator, the test result indicating whether the AI / ML model is verified.

[0005] In a second aspect of the present disclosure, a second device is provided. The second device includes: at least one processor; and at least one memory storing instructions, which, when executed by the at least one processor, causes the second device to at least perform: receiving test configuration information from a first device, the test configuration information indicating a test mode of an artificial intelligence / machine learning AI / ML model relative to at least one transmitting and receiving unit TRP, at least one TRP being arranged in an environment based on a test plan for at least one test channel indicator, the test channel indicator indicating the probability that a communication channel between the second device and the TRP is a line-of-sight channel or a non-line-of-sight channel; based on the test configuration information, deriving at least one predicted channel indicator for at least one TRP using the AI / ML model; and sending at least one predicted channel indicator for at least one TRP to the first device.

[0006] In a third aspect of the present disclosure, a method is provided. The method includes: at a first device, sending test configuration information to a second device, the test configuration information indicating a test mode of an artificial intelligence / machine learning AI / ML model relative to at least one transmitting and receiving unit TRP, at least one TRP is arranged in an environment based on a test plan for at least one test channel indicator, and the test channel indicator indicates the probability that the communication channel between the second device and the TRP is a line-of-sight channel or a non-line-of-sight channel; receiving at least one predicted channel indicator for at least one TRP from the second device, the at least one predicted channel indicator being derived by the second device using the AI / ML model; and determining a test result of the AI / ML model based on a comparison between the at least one predicted channel indicator and the at least one test channel indicator, the test result indicating whether the AI / ML model is verified.

[0007] In a fourth aspect of the present disclosure, a method is provided. The method includes: receiving test configuration information from a first device at a second device, the test configuration information indicating a test mode of an artificial intelligence / machine learning AI / ML model relative to at least one transmitting and receiving unit TRP, at least one TRP being arranged in an environment based on a test plan for at least one test channel indicator, the test channel indicator indicating a probability that a communication channel between the second device and the TRP is a line-of-sight channel or a non-line-of-sight channel; deriving at least one predicted channel indicator for at least one TRP using the AI / ML model based on the test configuration information; and sending at least one predicted channel indicator for at least one TRP to the first device.

[0008] In a fifth aspect of the present disclosure, a first device is provided. The first device includes: a component for sending test configuration information to a second device, the test configuration information indicating a test mode of an artificial intelligence / machine learning AI / ML model relative to at least one transmitting and receiving unit TRP, at least one TRP is arranged in an environment based on a test plan for at least one test channel indicator, and the test channel indicator indicates the probability that the communication channel between the second device and the TRP is a line-of-sight channel or a non-line-of-sight channel; a component for receiving at least one predicted channel indicator for at least one TRP from the second device, at least one predicted channel indicator is derived by the second device using the AI / ML model; and a component for determining a test result of the AI / ML model based on a comparison between the at least one predicted channel indicator and the at least one test channel indicator, the test result indicating whether the AI / ML model is verified.

[0009] In a sixth aspect of the present disclosure, a second device is provided. The second device includes: a component for receiving test configuration information from a first device, the test configuration information indicating a test mode of an artificial intelligence / machine learning AI / ML model relative to at least one transmitting and receiving unit TRP, at least one TRP is arranged in an environment based on a test plan for at least one test channel indicator, and the test channel indicator indicates the probability that the communication channel between the second device and the TRP is a line-of-sight channel or a non-line-of-sight channel; a component for deriving at least one predicted channel indicator for at least one TRP using the AI / ML model based on the test configuration information; and a component for sending at least one predicted channel indicator for at least one TRP to the first device.

[0010] In a seventh aspect of the present disclosure, a computer-readable medium is provided, wherein the computer-readable medium includes instructions stored thereon for causing a device to at least execute the method according to the third aspect.

[0011] In an eighth aspect of the present disclosure, a computer-readable medium is provided, wherein the computer-readable medium includes instructions stored thereon for causing a device to at least execute the method according to the fourth aspect.

[0012] It should be understood that the invention summary is not intended to identify the key or essential features of the embodiments of the present disclosure, nor is it intended to be used to limit the scope of the present disclosure. Other features of the present disclosure will become readily understood through the following description. BRIEF DESCRIPTION OF THE DRAWINGS

[0013] Some example embodiments will now be described with reference to the accompanying drawings, in which:

[0014] Figure 1 An example communication environment is shown in which example embodiments of the present disclosure may be implemented;

[0015] Figure 2 A flowchart illustrating a signaling flow for AI / ML model testing according to some example embodiments of the present disclosure is shown;

[0016] Figure 3 A flowchart showing an AI / ML model testing process according to some example embodiments of the present disclosure is shown;

[0017] Figure 4A and Figure 4B shows an example test setup for NLOS conditions and LOS conditions according to some example embodiments of the present disclosure;

[0018] Figure 5 A flowchart showing a method implemented at a first device according to some example embodiments of the present disclosure is shown;

[0019] Figure 6 A flowchart showing a method implemented at a second device according to some example embodiments of the present disclosure is shown;

[0020] Figure 7 shows a simplified block diagram of a device suitable for implementing an example embodiment of the present disclosure; and

[0021] Figure 8 A block diagram of an example computer-readable medium is shown according to some example embodiments of the present disclosure.

[0022] Throughout the drawings, the same or similar reference numerals refer to the same or similar elements. DETAILED DESCRIPTION

[0023] The principle of the present disclosure will now be described with reference to some exemplary embodiments. It should be understood that these embodiments are described only to illustrate and help those skilled in the art to understand and implement the present disclosure, and do not imply any limitation on the scope of the present disclosure. The embodiments described herein can be implemented in various ways except the way described below.

[0024] In the following description and claims, unless defined otherwise, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this disclosure belongs.

[0025] References in this disclosure to "one embodiment," "an embodiment," "an example embodiment," etc. indicate that the described embodiment may include a particular feature, structure, or characteristic, but not every embodiment necessarily includes the particular feature, structure, or characteristic. Furthermore, such phrases do not necessarily refer to the same embodiment. Furthermore, when a particular feature, structure, or characteristic is described in conjunction with an embodiment, it is considered within the knowledge of those skilled in the art that other embodiments may be combined to affect such feature, structure, or characteristic, whether or not explicitly described.

[0026] It should be understood that although the terms "first", "second", etc. may be used to describe various elements in this article, these elements should not be limited by these terms. These terms are only used to distinguish one element from another element. For example, a first element may be referred to as a second element, and similarly, a second element may be referred to as a first element without departing from the scope of the exemplary embodiments. As used herein, the term "and / or" includes any and all combinations of one or more of the listed terms.

[0027] As used herein, “at least one of: ” and “at least one of ” and similar expressions, where a list of two or more elements is combined by “and” or “or”, mean at least any one of the elements, or at least any two or more of the elements, or at least all of the elements.

[0028] As used herein, unless explicitly stated, performing a step "in response to A" does not indicate that the step is performed immediately after "A" occurs, and may include one or more intermediate steps.

[0029] The terms used herein are used only to describe specific embodiments and are not intended to limit the example embodiments. As used herein, the singular forms "a", "an", and "the" are also intended to include the plural forms, unless the context clearly indicates otherwise. It will be further understood that the terms "comprise", "including", "having", "having", "including" and / or "comprising", when used herein, specify the presence of the features, elements and / or components, etc., but do not exclude the presence or addition of one or more other features, elements, components and / or combinations thereof.

[0030] The term "circuitry" as used in this application may refer to one or more or all of the following: (a) hardware circuit implementation only (such as implementation in analog and / or digital circuits only) and (b) a combination of hardware circuitry and software such as (where applicable): (i) a combination of analog and / or digital hardware circuits and software / firmware and (ii) any portion of a hardware processor(s) with software (including digital signal processor(s), software and memory(s) that work together to enable a device (such as a mobile phone or server) to perform various functions) and (c) Hardware circuits and / or processor(s), such as microprocessor(s) or portions of microprocessor(s), that require software (e.g., firmware) to operate, but which may not be present when the software is not required to operate.

[0031] The definition of circuitry applies to all uses of the term in this application, including any claims. As yet another example, as used in this application, the term circuitry also covers an implementation of only a hardware circuit or processor (or multiple processors) or a portion of a hardware circuit or processor and its (or their) accompanying software and / or firmware. The term circuitry also covers, for example (and if applicable to a particular claim element), a baseband integrated circuit or processor integrated circuit for a mobile device or a similar integrated circuit in a server, cellular network device, or other computing or network device.

[0032] As used herein, the term "communication network" refers to a network that complies with any suitable communication standard, such as New Radio (NR), Long Term Evolution (LTE), Advanced LTE (LTE-A), Wideband Code Division Multiple Access (WCDMA), High Speed ​​Packet Access (HSPA), Narrowband Internet of Things (NB-IoT), etc. In addition, the communication between the terminal equipment and the network equipment in the communication network can be performed according to any suitable generation communication protocol, including but not limited to the first generation (1G), second generation (2G), 2.5G, 2.75G, third generation (3G), fourth generation (4G), 4.5G, fifth generation (5G), sixth generation (6G) communication protocol and / or any other protocol currently known or developed in the future. The embodiments of the present disclosure can be applied to various communication systems. In view of the rapid development of communication, there will certainly be future types of communication technologies and systems that can implement the present disclosure. The scope of the present disclosure should not be limited to the above-mentioned system. In addition, communications may utilize any appropriate wireless communication technology, including, but not limited to, code division multiple access (CDMA), frequency division multiple access (FDMA), time division multiple access (TDMA), frequency division duplex (FDD), time division duplex (TDD), multiple input multiple output (MIMO), orthogonal frequency division multiple access (OFDM), discrete Fourier transform spread OFDM (DFT-s-OFDM), and / or any other technology currently known or developed in the future.

[0033] As used herein, the term "network device" refers to a node in a communication network, via which a terminal device accesses the network and receives services from it. A network device may refer to a base station (BS) or an access point (AP), such as a Node B (NodeB or NB), an evolved Node B (eNodeB or eNB), a NR NB (also referred to as a gNB), a remote radio unit (RRU), a radio head (RH), a remote radio head (RRH), a relay, an integrated access and backhaul (IAB) node, a low-power node (such as a femto, a micro, a non-terrestrial network (NTN)) or a non-terrestrial network device (such as a satellite network device, a low earth orbit (LEO) satellite and a geosynchronous orbit (GEO) satellite, an aircraft network device), etc., depending on the terminology and technology applied. In some example embodiments, a radio access network (RAN) split architecture includes a centralized unit (CU) and a distributed unit (DU) located at an IAB donor node. An IAB node includes a mobile terminal (IAB-MT) portion that behaves similarly to a UE toward a parent node, and a DU portion of an IAB node is similar to a base station toward a next-hop IAB node.

[0034] The term "terminal device" refers to any terminal device that may be capable of wireless communication. As an example and not limitation, a terminal device may also be referred to as a communication device, a user equipment (UE), a user station (SS), a portable user station, a mobile station (MS), or an access terminal (AT). The terminal device may include, but is not limited to, a mobile phone, a cellular phone, a smart phone, a voice over IP (VoIP) phone, a wireless regional loop phone, a tablet computer, a wearable terminal device, a personal digital assistant (PDA), a portable computer, a desktop computer, an image capture terminal device (such as a digital camera), a game terminal device, a music storage and playback device, a vehicle-mounted wireless terminal device, a wireless endpoint, a mobile station, a laptop embedded device (LEE), a laptop device (LME), a USB dongle, a smart device, a wireless client device (CPE), an Internet of Things (IoT) device, a watch or other wearable device, a head-mounted display (HMD), a vehicle, a drone, medical equipment and applications (e.g., remote surgery), industrial equipment and applications (e.g., robots and / or other wireless devices operating in the context of an industrial and / or automated processing chain), consumer electronic devices, devices operating on commercial and / or industrial wireless networks, etc. The terminal device may also correspond to the mobile terminal (MT) part of an IAB node (eg, a relay node).In the following description, the terms "terminal device", "communication device", "terminal", "user equipment" and "UE" may be used interchangeably.

[0035] As used herein, the terms "resource", "transmission resource", "resource block", "physical resource block" (PRB), "uplink resource" or "downlink resource" may refer to any resource used to perform communication, for example, communication between a terminal device and a network device, such as resources in the time domain, resources in the frequency domain, resources in the spatial domain, resources in the code domain, or any other combination of time, frequency, space and / or code domain resources to achieve communication. In the following, unless explicitly stated, resources in the frequency domain and the time domain will be used as examples of transmission resources for describing some example embodiments of the present disclosure. Note that the example embodiments of the present disclosure are equally applicable to other resources in other domains.

[0036] As used herein, the term "model" is referred to as the association between input and output learned from training data, and thus a corresponding output can be generated for a given input after training. The generation of the model can be based on ML technology. ML technology can also be referred to as AI technology. Generally, an ML model can be constructed that receives input information and makes predictions based on the input information. For example, a classification model can predict the category of input information in a set of predetermined categories. As used herein, a "model" can also be referred to as a "machine learning model", "learning model", "machine learning network" or "learning network", which are used interchangeably herein. Supervised learning refers to the process of training a model from inputs and their corresponding labels. The trained model is then used to infer the output.

[0037] Typically, model lifecycle management can generally include three phases, namely the training phase, the verification phase, and the application phase (also known as the reasoning phase). In the training phase, a given AI / ML model can be iteratively trained (or optimized) using a large amount of training data until the model can reason about the expected output in the training data set or the labeled data set. During training, a set of parameter values ​​of the model is iteratively updated until the training goal is reached. Through the training process, the AI / ML model can be regarded as being able to learn the association between input and output (also known as input-output mapping) from the training data. In the verification phase, the verification input is applied to the trained AI / ML model to test whether the model can provide the correct output in order to determine the performance of the model. Typically, the verification phase can be considered as a step in the training process, or can be omitted in some cases. In the reasoning phase, the resulting AI / ML model can be used to process the real-world model input based on the trained model obtained from the training process and determine the corresponding model output. In some cases, a retraining or update phase may be included in the model lifecycle management so that the model can evolve to have better performance.

[0038] To facilitate understanding of the terminology, here are some definitions for a list of terms used in AI / ML.

[0039] AI / ML Model: A data-driven algorithm that applies AI / ML techniques to generate a set of outputs based on a set of inputs.

[0040] AI / ML model delivery: A general term referring to the delivery of AI / ML models from one entity to another in any manner. Note: An entity may refer to a network node / function (e.g., gNB, Location Management Function (LMF), etc.), UE, dedicated server, etc.

[0041] AI / ML model inference: The process of using a trained AI / ML model to produce a set of outputs based on a set of inputs.

[0042] AI / ML model testing: A subprocess of training to evaluate the performance of the final AI / ML model using a dataset different from that used for model training and validation. Unlike AI / ML model validation, testing does not assume subsequent adjustments to the model.

[0043] AI / ML model training: The process of training an AI / ML model in a data-driven manner (by learning input / output relationships) and obtaining a trained AI / ML model for inference.

[0044] AI / ML model delivery: Delivering AI / ML models, or parameters of a model structure known at the receiving end, or a new model with parameters, over the air interface in a way that is not transparent to 3GPP signaling. The delivery may contain a complete model or a partial model.

[0045] AI / ML model validation: A subprocess of training to evaluate the quality of an AI / ML model using a dataset different from the one used for model training, which helps select model parameters that generalize beyond the dataset used for model training.

[0046] Data Collection: The process of collecting data through network nodes, management entities or UEs for AI / ML model training, data analysis and inference.

[0047] Federated learning / federated training: A machine learning technique that trains AI / ML models across multiple distributed edge nodes (e.g., UE, gNB), with each distributed edge node performing local model training using local data samples. This technique requires multiple interactions of the model but does not require the exchange of local data samples.

[0048] Function Identification: The process / method used to identify a commonly understood AI / ML function between the network and the UE. NOTE: Information about the AI / ML function may be shared during function identification. Where the AI / ML function resides depends on the specific use case and sub-use case.

[0049] Model activation: Enabling an AI / ML model for a specific function.

[0050] Model deactivation: Disabling AI / ML models used for specific features.

[0051] Model download: Model transmission from the network to the UE.

[0052] Model Identification: The process / method for identifying a commonly understood AI / ML model between the network (NW) and the UE. Note: The process / method for model identification may or may not be applicable. Note: Information about the AI / ML model may be shared during model identification.

[0053] Model monitoring: The process of monitoring the inference performance of an AI / ML model.

[0054] Model parameter update: The process of updating the model parameters of a model.

[0055] Model Selection: The process of selecting an AI / ML model to activate among multiple models of the same AI / ML enabled features. Note: Model selection may or may not be performed simultaneously with model activation.

[0056] Model switching: Deactivate the currently active AI / ML model and activate a different AI / ML model for a specific function.

[0057] Model updating: The process of updating the model parameters and / or model structure of a model.

[0058] Model upload: Model transmission from UE to the network.

[0059] Network-side (AI / ML) model: an AI / ML model whose reasoning is performed entirely at the network.

[0060] Offline field data: Data collected from the field and used for offline training of AI / ML models.

[0061] Offline Training: The AI / ML training process where a model is trained based on a collected dataset, and where the trained model is later used or delivered for inference. Note: This definition is used as a guide only. There may be cases that may not fit neatly into this definition, but can still be classified as offline training by generally accepted conventions.

[0062] Online field data: Data collected from the field and used for online training of AI / ML models.

[0063] Online Training: An AI / ML training process where a model used for inference is trained (usually continuously) in (near) real-time as new training samples arrive. NOTE: The notion of (near) real-time vs. non-real-time is context-dependent and relative to the inference timescale. NOTE: This definition serves only as a guide. There may be cases that may not fit neatly into this definition, but can still be classified as online training by commonly accepted conventions. NOTE: Fine-tuning / retraining can be done via online or offline training. (This note may be removed when the term fine-tuning is defined).

[0064] Reinforcement Learning (RL): The process of training an AI / ML model using feedback signals (also called rewards) generated from inputs (also called states) and model outputs (also called actions) in the environment the model is interacting with.

[0065] Semi-supervised learning: The process of training a model using a mixture of labeled and unlabeled data.

[0066] Two-Sided (AI / ML) Model: A paired (multiple) AI / ML model on which joint reasoning is performed, where the joint reasoning includes AI / ML reasoning whose reasoning is performed jointly across the UE and the network, i.e., the first part of the reasoning is first performed by the UE and then the remaining part is performed by the gNB, or vice versa.

[0067] UE-side (AI / ML) model: an AI / ML model whose reasoning is performed entirely at the UE.

[0068] Unsupervised learning: The process of training a model without labeled data.

[0069] Proprietary format models: ML models in vendor-specific / device-specific proprietary formats from the perspective of 3GPP. They are not mutually recognizable between vendors and hide model design information from other vendors when shared. Note: Example is device-specific binary executable format.

[0070] Open format models: From the 3GPP perspective, ML models in a specified format that are mutually recognizable across vendors and allow interoperability. They are mutually recognizable between vendors and do not hide model design information from other vendors when shared.

[0071] Figure 1 An example communication environment 100 is shown in which example embodiments of the present disclosure may be implemented. It should be understood that the elements shown in the communication system 100 are intended to represent the primary functions provided within the system. Figure 1The blocks shown in refer to specific elements in the communication network that provide these primary functions. However, other network elements may be used to implement some or all of the primary functions represented. In addition, it should be understood that not all functions of the communication network are present in Figure 1 Rather, the representation is for convenience in explaining the functionality of the illustrative embodiment. Figure 1 The number of elements shown in is also for illustration purposes only, and any number of elements may be present.

[0072] As shown, the communication environment 100 includes a plurality of communication devices, including a test equipment (TE) 110, one or more devices under test (DUT) 120-1, 120-2, ..., 120-N (collectively or individually referred to as DUT 120), and one or more TRPs 130-1, 130-2 (collectively or individually referred to as TRP 130). The service area of ​​TRP 130 can be referred to as a cell. DUT 120 can perform signal transmission and reception with TRP 130.

[0073] In some example embodiments, one or more AI / ML models 125-1, 125-2, ..., 125-N (collectively or individually referred to as AI / ML models 125) may be used by one or more DUTs 120. AI / ML models 125 may sometimes be referred to as AI models or ML models. Different AI / ML models 125 may be configured to implement the same or different algorithms in the communication environment 100. The AI / ML models 125 used by the DUT 120 may sometimes include models or AI / ML functions.

[0074] In some example embodiments, the AI / ML model 125 is configured for AI / ML-based positioning or AI / ML-enabled positioning. AI / ML-enabled positioning is one of the selected use cases for research projects in the development of communication networks. In some example embodiments, two positioning methods are proposed. The first method is direct AI / ML positioning, where the output of the AI / ML model reasoning is the UE position. There are multiple options for the input of the model, including channel observations, such as channel impulse response (CIR), power delay profile (PDP), reference signal received power (RSRP), reference signal received path power (RSRPP), etc. The second method is AI / ML-assisted positioning, where the output of the AI / ML model reasoning is a new measurement and / or enhancement of the existing measurement, and such a new measurement and / or enhancement can be referred to as an intermediate feature because it will be input to the second function to finally estimate the UE position. These measurements include, for example, line-of-sight (LOS) / non-line-of-sight (NLOS) identification, arrival time (ToA), path phase, reference signal time difference (RSTD), etc.

[0075] In some example embodiments of the present disclosure, TE 110 is configured to test the AI / ML model 125 used by DUT 120. In some example embodiments, TE 110 may be a network entity in a core network, a base station (e.g., gNB, eNB) in a RAN, or may be a terminal device (e.g., UE) or any other device configured for AI / ML testing.

[0076] In some example embodiments, DUT 120 may be a terminal device (e.g., UE) that uses AI / ML model 125 for positioning purposes. In some example embodiments, DUT 120 may also be a network entity in a core network or a base station (e.g., gNB, eNB) in a RAN that is configured to use AI / ML model 125 for positioning purposes.

[0077] Study and provide input on the benefit(s) and potential regulatory impacts for at least the following scenarios of AI / ML-based location accuracy enhancement:

[0078] Use case 1: UE-based positioning using UE-side models, direct AI / ML or AI / ML-assisted positioning;

[0079] Use case 2a: UE-assisted / LMF-based positioning using UE-side model, AI / ML-assisted positioning;

[0080] Use case 2b: Direct AI / ML positioning using UE-assisted / LMF-based positioning with LMF-side model;

[0081] Use case 3a: NG-RAN node assisted positioning using gNB side model, AI / ML assisted positioning;

[0082] Use case 3b: Direct AI / ML positioning using NG-RAN node assisted positioning with LMF side model.

[0083] As can be seen from the above protocol, in some example embodiments, for UE-based positioning, AI / ML model reasoning may include LOS / NLOS indicators. LOS / NLOS reasoning may be based on reference signal measurements, such as measurements based on positioning reference signals (PRS). In some example embodiments, PRS is sent by a transmit and receive point (TRP) in the network.

[0084] In communication networks, there are test mechanisms defined for position estimation and latency based on global navigation satellite system (GNSS) positioning. These test mechanisms are defined for various satellites and rely on GNSS measurements. However, there is no test mechanism defined for verifying the LOS / NLOS characteristics of terminal devices. AI / ML models at terminal devices can be supplied in any way. It is desirable to provide a test mechanism to verify the reasoning of AI / ML models.

[0085] For UE-based AI / ML positioning, LOS / NLOS is selected as an intermediate test metric or key performance indicator (KPI) for performance evaluation of inference. In some cases, it is expected that the terminal vendor does not disclose the details of the AI / ML model deployed on the terminal side for positioning purposes, which can be part of its proprietary algorithm (e.g., black box). The network will not be aware of the UE-based AI / ML model, including its accuracy or performance in positioning. However, the network will want to monitor the performance KPIs of the AI / ML model to initiate life cycle management (LCM) based processes (e.g., model reselection, update, switching, deactivation, etc.).

[0086] Currently, requirements for positioning in communication networks assume that the UE is reporting measurements (e.g., RSRPP measurements, etc.) and that the UE position is then determined in the LMF. Therefore, traditional requirements and tests are not formulated based on the actual position, but rather on the accuracy and latency of the measurements performed and reported by the UE. In the case of UE-based positioning or UE-assisted positioning, there may be intermediate features such as LOS / NLOS indications, which may be used as input for deriving location coordinates at the UE or at the network side (e.g., gNB or LMF). Therefore, traditional requirements cannot be applied to intermediate KPIs and do not define requirements and tests to verify UE-based positioning capabilities, such as LOS / NLOS assessments.

[0087] Some legacy positioning technologies, such as Assisted GNSS (A-GNSS), have performance requirements for position estimates and latency for GNSS-based positioning. However, there is no defined test mechanism for validating LOS / NLOS inference as an intermediate characteristic of the UE for measurements based on Positioning Reference Signals (PRS). The LOS / NLOS indication may be the result of inference of an AI / ML model for a TRP, which may be used as input to an AI / ML-based algorithm at the UE or at the network (e.g., gNB and / or LMF) to derive location coordinates. PRS are sent by multiple TRPs in the network to allow the UE to perform positioning measurements.

[0088] One of the main challenges in testing LOS / NLOS inference of UE-based AI / ML models is that there is no defined process for validating the inference results with ground truth, which is the actual information indicating whether the UE is receiving the PRS signal directly from the TRP (i.e., LOS condition) or whether the PRS signal is received indirectly via multipath (i.e., reflected signals are received by the UE, which is also known as NLOS condition).

[0089] Therefore, use cases for AI / ML-based positioning need to be defined to validate the testing mechanisms for LOS / NLOS reasoning.

[0090] According to an exemplary embodiment of the present disclosure, a solution for verifying and testing UE-based or UE-assisted AI / ML positioning is proposed. It is assumed that an AI / ML model is used on the terminal side to derive LOS / NLOS reasoning. The test of this AI / ML model relies on the transmission of a reference signal from the network (e.g., from one or more TRPs). In addition, one or more TRPs are controlled to be arranged according to a test plan having one or more selected test channel indicators (also referred to as reference channel indicators or true channel indicators). The channel indicator indicates a LOS condition or NLOS condition between the device under test and the corresponding TRP. The device under test is configured in a test mode for the AI / ML model, and therefore one or more predicted channel indicators for one or more TRPs can be reported. By comparing the selected test channel indicator with the predicted channel indicator reported by the device under test, the test result of the AI / ML model can be obtained to determine whether the AI / ML model is verified.

[0091] Example embodiments of the present disclosure are described in detail below with reference to the accompanying drawings.

[0092] Figure 2 FIG. 2 is a flow chart showing a signaling flow 200 for AI / ML model testing according to some example embodiments of the present disclosure. Figure 2 As shown, the signaling flow 200 involves a first device 201 and a group of second devices 202 (including one or more second devices).

[0093] The first device 201 may be referred to as a TE, for example, Figure 1TE 110 in environment 100. The first device 201 is configured to emulate / simulate a real wireless network. In some example embodiments, the first device 201 can be a real network device (e.g., a gNB), or can be a system simulator that is operated as a signal generator, probe, or transmission point for sending certain types of radio signals (e.g., used for SSB transmission). In some example embodiments, the first device 201 may also include a channel emulator and / or an attenuator to simulate the propagation of radio signals. Different propagation conditions allow different LOS / NLOS conditions to be established. As will be discussed below, a reference LOS / NLOS probability can be selected for the test setup.

[0094] The one or more second devices 202 may include Figure 1 The DUT in the environment 100 may be one or more of a terminal device, a network entity, or a base station. In an embodiment of the present disclosure, the second device 202 may be able to implement AI / ML assisted positioning. The second device 202 utilizes an AI / ML model to implement AI / ML assisted positioning. The output of the AI / ML model is a channel indicator for a communication channel between the second device 202 and a network device (e.g., a TRP or a base station). The channel indicator indicates the probability that the communication channel between the second device 202 and the TRP is a LOS channel or a NOLS channel. Such a channel indicator may also be referred to as a LOS / NLOS indicator. The input of the AI / ML model may include measurement results of one or more reference signals sent from the TRP.

[0095] In an exemplary embodiment of the present disclosure, the second device 202 may be tested in an environment (also referred to as a test environment or test area). The environment is an area where the testing of the AI / ML model is performed. In some exemplary embodiments, the environment may be an anechoic chamber or a larger / outdoor area with multiple TRPs transmitted with LOS toward the second device 202 and multiple TRPs transmitted with NLOS toward the second device 202 (i.e., via reflectors / obstacles).

[0096] In the signaling flow 200, the (multiple) second device 202 and the first device 201 execute the test program 232 to verify the reasoning of the (multiple) AI / ML model on the first device 201 side. In some example embodiments, the test program 232 may be executed in an iterative manner depending on actual requirements. In the following, the operation at the second device 202 is described, and other second devices may perform similar operations.

[0097] In some example embodiments, to enable AI / ML model testing, the first device 201 may send (205) a command to the second device 202 to enable AI / ML-based positioning or AI / ML-assisted positioning at the second device 202. Upon receiving (210) the command, the second device 202 may enable (215) AI / ML-based or AI / ML-assisted positioning for positioning measurements. The second device 202 may send (220) a confirmation of the activation of the AI / ML-based or AI / ML-assisted positioning to the first device 201. Based on receiving (225) the confirmation, the first device 201 may determine to activate the test program 232. It should be understood that in some other example embodiments, the activation of the AI / ML-based or AI / ML-assisted positioning at the second device 202 may be triggered by other network devices.

[0098] During the test procedure 232, the first device 201 configures (230) an environment for testing the AI / ML model at the second device 202, and sends (235) test configuration information to the second device 202. The test configuration information indicates a test mode of the AI / ML model relative to at least one TRP. The at least one TRP is arranged within the environment based on a test plan for at least one test channel indicator. The test channel indicator indicates a (reference) probability that a communication channel between the second device and the TRP is a LOS channel or a NLOS channel.

[0099] In some example embodiments, the test configuration information may be communicated via an air interface to the second device 202. In some example embodiments, the test configuration information may be communicated as an application layer level command.

[0100] The second device 202 receives (240) the test configuration information from the first device 201 and prepares (245) a test mode for the AI / ML model. In some example embodiments, the second device 202 may send (250) a confirmation of the test configuration information to the first device 201. Based on receiving (255) the confirmation of the test configuration information, the first device 201 may cause (260) the test plan to be executed. In some example embodiments, the confirmation of the test configuration information may be communicated to the first device 201 via an air interface. In some example embodiments, the confirmation of the test configuration information may be communicated as an application layer level command.

[0101] For any AI / ML model that derives LOS / NLOS inference, the LOS / NLOS inference needs to be tested and validated against labeled data in order to increase confidence in the training data and model output. "Label" refers to a trusted model inference and is also represented as a true value. In the case of LOS / NLOS inference, the label includes a trusted LOS / NLOS indication between the second device 202 and at least one TRP. To configure the label, an environment for testing the AI / ML model used by the second device 202 can be specifically defined.

[0102] Figure 3 A flow chart of an AI / ML model testing process 300 according to some example embodiments of the present disclosure is shown. At block 310, the first device 201 selects at least one test channel indicator for at least one TRP. As mentioned, the test channel indicator may indicate a probability that the communication channel between the second device 202 and the TRP is a LOS channel or a NLOS channel, and such a probability may also be referred to as a reference LOS / NLOS probability.

[0103] In some example embodiments, the first device 201 may select a reference LOS / NLOS probability to be tested. In some examples, the selection of the reference LOS / NLOS probability may correspond to an interval with a step size. For example, with respect to a given TRP, the first device 201 may select a reference probability from a NLOS probability range [0%-100%] with a step size of 10%. That is, the first device 201 may select one or more of the values ​​of the LOS probability {0%, 10%, . . . , 90%, 100%} that are complementary to the LOS probability (e.g., 10% of the NLOS probability corresponds to 90% of the LOS probability). It should be understood that the selection of the step or reference value may be performed according to the desired AI / ML model accuracy. In addition, in the case of multiple TRPs, a reference LOS / NLOS probability may be provided according to the TRP. In some example embodiments, in a test setting, the TRP may be a real base station or may be simulated, or it may be a combination of both.

[0104] At block 320, the first device 202 establishes a test plan based on the at least one selected test channel indicator. The test plan may include a test setup for the test channel indicator, the test setup being defined as the physical arrangement of the TRP relative to the second device 202 and one or more signal reflectors or obstacles within the environment.

[0105] For each value of the reference LOS / NLOS probability, a dedicated test setup may be defined with the desired configuration (eg, with the desired signal reflectors or obstacles) to achieve the desired LOS / NLOS probability for each test.

[0106] For example, for a LOS condition, a test setup for testing a channel indicator may be defined to indicate a probability that a communication channel between the second device 202 and the TRP is a LOS channel. The test setup may be defined as a physical arrangement of the TRP toward the second device 202. For a NLOS condition, a test setup for testing a channel indicator may be defined to indicate a probability that a communication channel between the second device 202 and the TRP is a NLOS channel. The test setup may be defined as a physical arrangement of the TRP toward at least one signal reflector so that a reference signal sent from the TRP may be reflected to the second device 202.

[0107] Figure 4A and Figure 4B An example test setup of NLOS conditions 400 and LOS conditions 401 according to some example embodiments of the present disclosure is shown. In the NLOS condition 400, one or more signal reflectors 412 may be arranged to reflect the signal transmitted from the TRP 410 to the second device 202 to recreate a multipath scenario (i.e., a NLOS use case). The TRP 410 may be arranged towards the signal reflector 412. The test setup may require placing (multiple) signal reflectors at (multiple) different locations to reflect the signal received from the TRP towards the second device 202 based on the desired selected LOS / NLOS probability of the TRP. In the LOS condition 401, the TRP 420 may be arranged towards the second device 202 to create a direct signal propagation path.

[0108] Based on the test plan, there may be a first number ("n") of TRPs arranged to point toward the second device 202 to simulate a LOS scenario, and there may be a second number ("m", where m may or may not be equal to n) of TRPs arranged to point toward the signal reflector(s) so that the reflected signals are received by the second device 202 to simulate a NLOS scenario with a desired reference LOS / NLOS probability, such as Figure 3 As shown in block 310 of . In some example embodiments, for a TRP pointing to a reflector, it should be ensured that the transmitted reference signal can reach the second device 202 based on the corresponding reference LOS / NLOS probability.

[0109] The first device 201 can configure the test environment according to the test settings in the test plan, for example, so that at least one TRP and a signal reflector (or obstacle) are arranged in the environment according to at least one test channel indicator. As indicated above, the first device 201 can also send test configuration information to the second device 202 to trigger the test mode of the AI / ML model.

[0110] The first device 201 can configure the second device 202 with the desired configuration including the AI / ML model whose LOS / NLOS reasoning is to be tested. The first device 201 can also configure the second device 202 to have a test mode to instruct the second device 202 to report the AI / ML model reasoning for all configured TRPs.

[0111] In some example embodiments, the test configuration information may include an indication of an AI / ML model to be tested and / or an indication of at least one TRP involved in a test mode. The indicated (multiple) TRPs may be TRPs configured in an environment according to a test plan. In some example embodiments, the test configuration information may include an indication of a test mode, a test command, or a test configuration for an AI / ML model. For example, the second device 202 may be pre-configured with an AI / ML model to be tested and / or (multiple) TRPs for testing the AI / ML model. By indicating a test mode, a test command, or a test configuration, the second device 202 may prepare to send (multiple) predicted channel indicators to the first device 201.

[0112] At box 330 of process 300, the first device 201 causes the test plan to be executed. In the execution of the test plan, different reference LOS / NLOS probabilities can be used iteratively to test the AI / ML model at the second device, for example, by placing (multiple) TRPs and other required reflectors / obstacles in the environment each time and causing (multiple) TRPs to send reference signals for channel measurement. At box 340, the first device 201 evaluates the test performance on different iterations of the reference LOS / NLOS probability through the selected (multiple) KPIs. KPIs for model performance evaluation can include, for example, the 90% percentile of the cumulative distribution function (CDF) value of the error between the true / reference LOS / NLOS probability and the inference result provided from the second device 202.

[0113] In some example embodiments, after configuring the test environment and after the second device 202 to be tested has confirmed receipt of the test configuration information, the first device 201 may cause at least one TRP arranged within the environment to send a reference signal. The second device 202 may then measure the reference signal and provide the measurement results as input to the AI / ML model to derive the LOS / NLOS indicator. Accordingly, the second device 202 may collect the measurement results of the reference signal from the (multiple) TRPs and provide the measurement results as input to the AI / ML model. The AI / ML model may be run to derive a predicted channel indicator indicating the predicted LOS / NLOS probability of the corresponding (multiple) TRPs.

[0114] Return to reference Figure 2, in the signaling flow 200, when executing the test plan, the second device 202 uses the AI / ML model to derive (265) at least one predicted channel indicator for at least one TRP. Specifically, when executing the test plan, at least one TRP arranged in the environment can send a reference signal. The second device 202 can detect the measurement results of the reference signal from the TRP, and input the measurement results to the AI / ML model to derive the channel indicator of the corresponding TRP. It should be understood that, depending on the AI / ML model configuration, various types of measurement results can be collected as input to the AI / ML model to determine whether the channel between the second device 202 and the corresponding TPR is a LOS channel or a NLOS channel. The scope of the present disclosure is not limited in this regard.

[0115] The second device 202 sends (270) at least one predicted channel indicator for at least one configured TRP to the first device 201. The predicted channel indicator indicates the probability that the channel between the second device 202 and the corresponding TPR is a LOS channel or a NLOS channel, as derived by the AI / ML model used.

[0116] The first device 201 receives (275) at least one predicted channel indicator from the second device 202, and determines (280) a test result of the AI / ML model based on a comparison between the at least one predicted channel indicator and the at least one test channel indicator. The test result may indicate whether the AI / ML model is verified.

[0117] In some example embodiments, the first device 201 may compare the received predicted channel indicator (e.g., LOS / NLOS probability) of each TRP with the test channel indicator (e.g., the configured LOS / NLOS probability). In some example embodiments, the first device 201 may display the comparison result in the form of a table or a graph. In some example embodiments, the comparison result may also be used as an input to an AI / ML model for calibration.

[0118] In some example embodiments, if at least one predicted channel indicator matches at least one test channel indicator, the first device 201 may determine the test result to indicate that the AI / ML model is verified. If at least one predicted channel indicator does not match at least one test channel indicator, the first device 201 may determine the test result to indicate that the AI / ML model is not verified. For example, if the inference values ​​of all configured TRPs match the configured reference values ​​(i.e., LOS or NLOS probabilities), for example, if the inference values ​​match the configured reference values ​​for a predetermined number of samples, the test may be considered to be successfully verified, and the first device 201 may declare the test of the AI / ML model as "PASSED". Otherwise, the test may be considered to have failed verification, and the first device 201 may declare the test of the AI / ML model as "FAILED".

[0119] According to the example embodiments of the present disclosure as discussed above, a test mechanism for verifying LOS / NLOS reasoning in AI / ML-based positioning or AI / ML-assisted positioning is provided. Through this test mechanism, LOS / NLOS reasoning on the terminal side can be verified to avoid introducing unqualified intermediate results in AI / ML positioning use cases.

[0120] Figure 5 A flowchart of an example method 500 implemented at a second device according to some example embodiments of the present disclosure is shown. For the purpose of discussion, Figure 2 The first device 201 (which may be Figure 1 Method 500 is described from the perspective of TE 110 in FIG.

[0121] At block 510, the first device 201 sends a Figure 2 The second device 202 or Figure 1 The test configuration information is sent to the DUT 201 in the environment. The test configuration information indicates a test mode of the AI / ML model relative to at least one TRP. The at least one TRP is arranged in the environment based on a test plan for at least one test channel indicator. The test channel indicator indicates the probability that the communication channel between the second device and the TRP is a line-of-sight channel or a non-line-of-sight channel.

[0122] At block 520, the first device 201 receives at least one predicted channel indicator for at least one TRP from the second device. The at least one predicted channel indicator is derived by the second device using an AI / ML model.

[0123] At block 530, the first device 201 determines a test result of the AI / ML model based on a comparison between the at least one predicted channel indicator and the at least one test channel indicator. The test result indicates whether the AI / ML model is verified.

[0124] In some example embodiments, the test configuration information includes at least one of: an indication of an AI / ML model to be tested, an indication of at least one TRP, or an indication of a test mode, test command, or test configuration.

[0125] In some example embodiments, the first device 201 may also select at least one test channel indicator for at least one TRP; determine a test plan based on the at least one test channel indicator, the test plan including a test setup for the test channel indicator, the test setup being defined as a physical arrangement of the TRP relative to the second device and at least one signal reflector within the environment; and cause the at least one TRP to be arranged according to the test plan.

[0126] In some example embodiments, the test plan includes: a first test setup for a test channel indicator indicating a probability that a communication channel between a second device and a first TRP is a line-of-sight channel, the first test setup being defined as a physical arrangement of the first TRP toward the second device, and a second test setup for a test channel indicator indicating a probability that a communication channel between the second device and a second TRP is a non-line-of-sight channel, the second test setup being defined as a physical arrangement of the second TRP toward at least one signal reflector.

[0127] In some example embodiments, the first device 201 may also cause at least one TRP disposed within the environment to transmit a reference signal, a measurement result of the reference signal being measured by the second device and used as an input to the AI / ML model.

[0128] In some example embodiments, the first device 201 may also receive a confirmation of the test configuration information from the second device; and cause the test plan to be executed based on receiving the confirmation of the test configuration information.

[0129] In some example embodiments, the first device 201 may also determine a test result to indicate that the AI / ML model is verified based on determining that at least one predicted channel indicator matches at least one test channel indicator; and determine a test result to indicate that the AI / ML model is not verified based on determining that at least one predicted channel indicator does not match at least one test channel indicator.

[0130] In some example embodiments, the first apparatus 201 comprises a testing device, and the second apparatus comprises a device under test.

[0131] In some example embodiments, the testing device comprises a network entity, a base station, or a terminal device; and wherein the device under test comprises a network entity, a base station, or a terminal device, wherein the terminal device comprises one or more receivers.

[0132] Figure 6 A flowchart of an example method 600 implemented at a second device according to some example embodiments of the present disclosure is shown. For the purpose of discussion, Figure 2 The second device 202 (which may be Figure 1 The method 600 is described from the perspective of the DUT 120 in FIG.

[0133] At block 610, the second device 202 receives a request from the first device (eg, Figure 1 The first device 201 or Figure 1 110 in the embodiment of the present invention receives test configuration information indicating a test mode of an AI / ML model relative to at least one TRP. At least one TRP is arranged in an environment based on a test plan for at least one test channel indicator. The test channel indicator indicates a probability that a communication channel between a second device and the TRP is a line-of-sight channel or a non-line-of-sight channel.

[0134] At block 620 , based on the test configuration information, the second device 202 derives at least one predicted channel indicator for at least one TRP using the AI / ML model.

[0135] At box 630, the second device 202 sends at least one predicted channel indicator for at least one TRP to the first device.

[0136] In some example embodiments, the test configuration information includes at least one of: an indication of an AI / ML model to be tested, an indication of at least one TRP, or an indication of a test mode, test command, or test configuration.

[0137] In some exemplary embodiments, the second device 202 may also send a confirmation of the test configuration information to the first device to trigger the test plan to be executed.

[0138] In some example embodiments, the second device 202 may also measure a reference signal sent from at least one TRP; and provide the measurement results of the reference signal as input to the AI / ML model to derive at least one predicted channel indicator.

[0139] In some example embodiments, the first apparatus comprises a testing device and the second apparatus comprises a device under test.

[0140] In some example embodiments, the testing device comprises a network entity, a base station or a terminal device, and the device under test comprises a network entity, a base station or a terminal device, the terminal device comprising one or more receivers.

[0141] In some example embodiments, any of the steps of method 500 (e.g., Figure 2The first device 201 in the method 500 may include a component for performing the corresponding operation of the method 500. The component may be implemented in any suitable form. For example, the component may be implemented in a circuit system or a software module. The first device may be implemented as or included in Figure 2 In the first device 201.

[0142] In some example embodiments, a first device includes: a component for sending test configuration information to a second device, the test configuration information indicating a test mode of an artificial intelligence / machine learning AI / ML model relative to at least one transmitting and receiving unit TRP, at least one TRP being arranged within an environment based on a test plan for at least one test channel indicator, the test channel indicator indicating a probability that a communication channel between the second device and the TRP is a line-of-sight channel or a non-line-of-sight channel; a component for receiving at least one predicted channel indicator for at least one TRP from the second device, the at least one predicted channel indicator being derived by the second device using the AI / ML model; and a component for determining a test result of the AI / ML model based on a comparison between the at least one predicted channel indicator and the at least one test channel indicator, the test result indicating whether the AI / ML model is verified.

[0143] In some example embodiments, the test configuration information includes at least one of: an indication of an AI / ML model to be tested, an indication of at least one TRP, or an indication of a test mode, test command, or test configuration.

[0144] In some example embodiments, the first device also includes: a component for selecting at least one test channel indicator for at least one TRP; a component for determining a test plan based on the at least one test channel indicator, the test plan including a test setup for the test channel indicator, the test setup being defined as a physical arrangement of the TRP relative to the second device and at least one signal reflector within the environment; and a component for causing the at least one TRP to be arranged according to the test plan.

[0145] In some example embodiments, the test plan includes: a first test setup for a test channel indicator indicating a probability that a communication channel between a second device and a first TRP is a line-of-sight channel, the first test setup being defined as a physical arrangement of the first TRP toward the second device, and a second test setup for a test channel indicator indicating a probability that a communication channel between the second device and a second TRP is a non-line-of-sight channel, the second test setup being defined as a physical arrangement of the second TRP toward at least one signal reflector.

[0146] In some example embodiments, the first device further comprises: means for causing at least one TRP arranged within the environment to transmit a reference signal, a measurement result of the reference signal being measured by the second device and used as an input to the AI / ML model.

[0147] In some example embodiments, the first apparatus further comprises: means for receiving a confirmation of the test configuration information from the second apparatus; and means for causing the test plan to be executed based on receiving the confirmation of the test configuration information.

[0148] In some example embodiments, the components for determining a test result include: components for determining the test result to indicate that the AI / ML model is verified based on determining that at least one predicted channel indicator matches at least one test channel indicator; and components for determining the test result to indicate that the AI / ML model is not verified based on determining that at least one predicted channel indicator does not match at least one test channel indicator.

[0149] In some example embodiments, the first apparatus comprises a testing device and the second apparatus comprises a device under test.

[0150] In some example embodiments, the testing device comprises a network entity, a base station or a terminal device, and the device under test comprises a network entity, a base station or a terminal device, the terminal device comprising one or more receivers.

[0151] In some example embodiments, the first device further comprises means for performing other operations in some example embodiments of the method 500 or the first device 201. In some example embodiments, the means comprises at least one processor; and at least one memory storing instructions which, when executed by the at least one processor, cause the performance of the first device.

[0152] In some example embodiments, any of the steps of method 600 (e.g., Figure 2 The second device 202 in the second device 202 may include a component for performing the corresponding operation of method 600. The component may be implemented in any suitable form. For example, the component may be implemented in a circuit system or a software module. The second device may be implemented as or included in Figure 2 In the second device 202.

[0153] In some example embodiments, the second device includes: a component for receiving test configuration information from the first device, the test configuration information indicating a test mode of an artificial intelligence / machine learning AI / ML model relative to at least one transmitting and receiving unit TRP, at least one TRP being arranged in an environment based on a test plan for at least one test channel indicator, the test channel indicator indicating the probability that a communication channel between the second device and the TRP is a line-of-sight channel or a non-line-of-sight channel; a component for deriving at least one predicted channel indicator for at least one TRP using the AI / ML model based on the test configuration information; and a component for sending at least one predicted channel indicator for at least one TRP to the first device.

[0154] In some example embodiments, the test configuration information includes at least one of: an indication of an AI / ML model to be tested, an indication of at least one TRP, or an indication of a test mode, test command, or test configuration.

[0155] In some example embodiments, the second device further comprises: means for sending a confirmation of the test configuration information to the first device to trigger execution of the test plan.

[0156] In some example embodiments, the second device further includes: a component for measuring a reference signal sent from at least one TRP; and a component for providing the measurement results of the reference signal as an input to the AI / ML model to derive at least one predicted channel indicator.

[0157] In some example embodiments, the first apparatus comprises a testing device and the second apparatus comprises a device under test.

[0158] In some example embodiments, the testing device comprises a network entity, a base station or a terminal device, and the device under test comprises a network entity, a base station or a terminal device, the terminal device comprising one or more receivers.

[0159] In some example embodiments, the second device further comprises means for performing other operations in some example embodiments of the method 600 or the second device 202. In some example embodiments, the means comprises at least one processor; and at least one memory storing instructions which, when executed by the at least one processor, cause the performance of the second device.

[0160] Figure 7 700 is a simplified block diagram of a device 700 suitable for implementing an example embodiment of the present disclosure. The device 700 may be provided to implement a communication device, such as Figure 11. As shown in the figure, the device 700 includes one or more processors 710, one or more memories 720 coupled to the processor 710, and one or more communication modules 740 coupled to the processor 710.

[0161] The communication module 740 is used for two-way communication. The communication module 740 has one or more communication interfaces to support communication with one or more other modules or devices. The communication interface may represent any interface necessary for communication with other network elements. In some example embodiments, the communication module 740 may include at least one antenna.

[0162] Processor 710 may be of any type suitable for the local technology network, and may include one or more of: a general purpose computer, a special purpose computer, a microprocessor, a digital signal processor (DSP), and a processor based on a multi-core processor architecture, as non-limiting examples. Device 700 may have multiple processors, such as application specific integrated circuit chips, which are time-slaved to the clock of a synchronized master processor.

[0163] The memory 720 may include one or more non-volatile memories and one or more volatile memories. Examples of non-volatile memories include, but are not limited to, read-only memory (ROM) 724, electrically programmable read-only memory (EPROM), flash memory, hard disk, compact disk (CD), digital video disk (DVD), optical disk, laser disk, and other magnetic storage and / or optical storage. Examples of volatile memories include, but are not limited to, random access memory (RAM) 722 and other volatile memories that do not persist over the duration of a power outage.

[0164] The computer program 730 includes computer executable instructions executed by the associated processor 710. The instructions of the program 730 may include instructions for performing the operations / actions of some example embodiments of the present disclosure. The program 730 may be stored in a memory, such as ROM 724. The processor 710 may perform any suitable actions and processes by loading the program 730 into the RAM 722.

[0165] The exemplary embodiments of the present disclosure may be implemented by means of a program 730, so that the device 700 may execute the Figures 2 to 6 Any process of the present disclosure discussed. The exemplary embodiments of the present disclosure may also be implemented by hardware or by a combination of software and hardware.

[0166] In some example embodiments, the program 730 may be tangibly embodied in a computer-readable medium that may be included in the device 700 (such as in the memory 720) or in other storage devices accessible to the device 700. The device 700 may load the program 730 from the computer-readable medium to the RAM 722 for execution. In some example embodiments, the computer-readable medium may include any type of non-transitory storage medium, such as ROM, EPROM, flash memory, hard disk, CD, DVD, etc. The term "non-transitory" as used herein is a limitation of the medium itself (i.e., tangible, not a signal), not a limitation on data storage persistence (e.g., RAM versus ROM).

[0167] Figure 8 An example of a computer readable medium 800 is shown which may be in the form of a CD, DVD or other optical storage disk. The computer readable medium 800 has a program 730 stored thereon.

[0168] In general, various embodiments of the present disclosure may be implemented in hardware or dedicated circuits, software, logic, or any combination thereof. Some aspects may be implemented in hardware, while other aspects may be implemented in firmware or software that may be executed by a controller, microprocessor, or other computing device. Although various aspects of the embodiments of the present disclosure are shown and described as block diagrams, flow charts, or using some other graphical representations, it should be understood that, as non-limiting examples, the boxes, devices, systems, techniques, or methods described herein may be implemented in hardware, software, firmware, dedicated circuits or logic, general hardware or controllers or other computing devices, or some combination thereof.

[0169] Some example embodiments of the present disclosure also provide at least one computer program product tangibly stored on a computer-readable medium (such as a non-transient computer-readable medium). The computer program product includes computer executable instructions, such as those included in a program module executed in a device on a target physical or virtual processor to perform any method as described above. Typically, a program module includes routines, programs, libraries, objects, classes, components, data structures, etc. that perform specific tasks or implement specific abstract data types. In various embodiments, the functions of the program modules can be combined or split between program modules as needed. Machine executable instructions for program modules can be executed in local or distributed devices. In distributed devices, program modules can be located in both local and remote storage media.

[0170] The program code for executing the method of the present disclosure can be written in any combination of one or more programming languages. The program code can be provided to a processor or controller of a general-purpose computer, a special-purpose computer or other programmable data processing device so that the program code, when executed by the processor or controller, enables the functions / operations specified in the flow chart and / or block diagram to be implemented. The program code can be executed entirely on the machine, partially on the machine, as an independent software package, partially on the machine and partially on a remote machine, or entirely on a remote machine or server.

[0171] In the context of the present disclosure, computer program codes or related data may be carried by any suitable carrier to enable a device, apparatus or processor to perform various processes and operations as described above. Examples of carriers include signals, computer readable media, etc.

[0172] The computer readable medium may be a computer readable signal medium or a computer readable storage medium. The computer readable medium may include, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared or semiconductor systems, devices or apparatuses, or any suitable combination of the foregoing. More specific examples of computer readable storage media would include an electrical connection with one or more wires, a portable computer floppy disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

[0173] In addition, although operations are depicted in a particular order, this should not be understood as requiring such operations to be performed in the particular order shown or in sequence, or to perform all of the operations shown, to achieve the desired result. In some cases, multitasking and parallel processing may be advantageous. Similarly, although several specific implementation details are included in the above discussion, these should not be interpreted as limitations on the scope of the present disclosure, but as descriptions of features that may be specific to a particular embodiment. Unless explicitly stated, certain features described in the context of a separate embodiment may also be implemented in combination in a single embodiment. On the contrary, unless explicitly stated, the various features described in the context of a single embodiment may also be implemented in multiple embodiments individually or in any suitable sub-combination.

[0174] Although the disclosure has been described in language specific to structural features and / or methodological acts, it should be understood that the disclosure defined in the appended claims is not necessarily limited to the specific features or acts described above. Rather, the specific features and acts described above are disclosed as example forms of implementing the claims.

[0175] Some abbreviations used in this disclosure are provided below, which are presented in unshortened form. It is worth noting that widely established and unique abbreviations may be assumed to be known. List of abbreviations AI - Artificial Intelligence ML - Machine Learning 3GPP - Third Generation Partnership Project KPI - Key Performance Indicator PRS - Positioning Reference Signal RSRP - Reference Signal Received Power RSTD - Reference Signal Time Difference LOS-Line of sight NLOS - Non Line of Sight A-GNSS - Assisted Global Navigation Satellite System TRP - Transmit and Receive Point UE - User Equipment DUT - Device Under Test TE-Test Equipment SSB - Synchronization Signal / PBCH Block PBCH - Physical Broadcast Channel

Claims

1. A first device for communication, comprising: at least one processor; as well as at least one memory storing instructions that, when executed by the at least one processor, cause the first apparatus to at least perform: sending test configuration information to a second device, the test configuration information indicating a test mode of an artificial intelligence / machine learning AI / ML model with respect to at least one transmitting and receiving unit TRP, the at least one TRP being arranged within an environment based on a test plan for at least one test channel indicator, the test channel indicator indicating a probability that a communication channel between the second device and the TRP is a line-of-sight channel or a non-line-of-sight channel; receiving, from the second device, at least one predicted channel indicator for the at least one TRP, the at least one predicted channel indicator being derived by the second device using the AI / ML model; and Based on a comparison between the at least one predicted channel indicator and the at least one test channel indicator, a test result of the AI / ML model is determined, the test result indicating whether the AI / ML model is validated.

2. The apparatus according to claim 1, wherein the test configuration information comprises at least one of the following: an indication of the AI / ML model to be tested, an indication of the at least one TRP, or An indication of the test mode, test command or test configuration.

3. The apparatus according to claim 1 or 2, wherein the first device is further caused to execute: selecting the at least one test channel indicator for the at least one TRP; determining the test plan based on the at least one test channel indicator, the test plan comprising a test setup for the test channel indicator, the test setup being defined as a physical arrangement of the TRP relative to the second device and at least one signal reflector within the environment; as well as The at least one TRP is arranged according to the test plan.

4. The apparatus of claim 3, wherein the test plan comprises: a first test setup for a test channel indicator indicating a probability that a communication channel between the second device and a first TRP is a line-of-sight channel, the first test setup being defined as a physical arrangement of the first TRP toward the second device, and A second test setup for a test channel indicator indicating a probability that a communication channel between the second device and a second TRP is a non-line-of-sight channel, wherein the second test setup is defined as a physical arrangement of the second TRP toward the at least one signal reflector.

5. The device according to claim 1 or 2, wherein the first device is further caused to execute: The at least one TRP arranged within the environment sends a reference signal, and the measurement result of the reference signal is measured by the second device and used as an input of the AI / ML model.

6. The apparatus according to claim 1 or 2, wherein the first device is further caused to execute: receiving, from the second device, a confirmation of the test configuration information; and Based on receiving a confirmation of the test configuration information, the test plan is caused to be executed.

7. The apparatus according to claim 1 or 2, wherein the first device is further caused to execute: Based on determining that the at least one predicted channel indicator matches the at least one test channel indicator, determining the test result to indicate that the AI / ML model is validated; and Based on determining that the at least one predicted channel indicator does not match the at least one test channel indicator, the test result is determined to indicate that the AI / ML model is not validated.

8. The apparatus of claim 1 or 2, wherein the first apparatus comprises a test device and the second apparatus comprises a device under test.

9. The apparatus according to claim 8, wherein the test device comprises a network entity, a base station or a terminal device; and The device under test includes a network entity, a base station or a terminal device, wherein the terminal device includes one or more receivers.

10. A second device for communication, comprising: at least one processor; as well as at least one memory storing instructions that, when executed by the at least one processor, cause the second apparatus to at least perform: receiving test configuration information from a first device, the test configuration information indicating a test mode of an artificial intelligence / machine learning AI / ML model with respect to at least one transmitting and receiving unit TRP, the at least one TRP being arranged within an environment based on a test plan for at least one test channel indicator, the test channel indicator indicating a probability that a communication channel between the second device and the TRP is a line-of-sight channel or a non-line-of-sight channel; deriving, based on the test configuration information, at least one predicted channel indicator for the at least one TRP using the AI / ML model; as well as Sending the at least one predicted channel indicator for the at least one TRP to the first device.

11. The apparatus according to claim 10, wherein the test configuration information comprises at least one of the following: an indication of the AI / ML model to be tested, an indication of the at least one TRP, or An indication of the test mode, test command or test configuration.

12. The apparatus according to claim 10 or 11, wherein the second device is further caused to execute: A confirmation of the test configuration information is sent to the first device to trigger the test plan to be executed.

13. The apparatus according to claim 10 or 11, wherein the second device is further caused to execute: measuring a reference signal sent from the at least one TRP; and The measurements of the reference signal are provided as input to the AI / ML model to derive the at least one predicted channel indicator.

14. The apparatus of claim 10 or 11, wherein the first apparatus comprises a test device and the second apparatus comprises a device under test.

15. The apparatus according to claim 14, wherein the test device comprises a network entity, a base station or a terminal device; and The device under test includes a network entity, a base station or a terminal device, wherein the terminal device includes one or more receivers.

16. A method for communication, comprising: At the first device, sending test configuration information to the second device, the test configuration information indicating a test mode of an artificial intelligence / machine learning AI / ML model with respect to at least one transmitting and receiving unit TRP, the at least one TRP being arranged within an environment based on a test plan for at least one test channel indicator, the test channel indicator indicating a probability that a communication channel between the second device and the TRP is a line-of-sight channel or a non-line-of-sight channel; receiving, from the second device, at least one predicted channel indicator for the at least one TRP, the at least one predicted channel indicator being derived by the second device using the AI / ML model; and Based on a comparison between the at least one predicted channel indicator and the at least one test channel indicator, a test result of the AI / ML model is determined, the test result indicating whether the AI / ML model is validated.

17. A method for communication, comprising: receiving, at the second device, test configuration information from the first device, the test configuration information indicating a test mode of an artificial intelligence / machine learning AI / ML model with respect to at least one transmitting and receiving unit TRP, the at least one TRP being arranged within an environment based on a test plan for at least one test channel indicator, the test channel indicator indicating a probability that a communication channel between the second device and the TRP is a line-of-sight channel or a non-line-of-sight channel; deriving, based on the test configuration information, at least one predicted channel indicator for the at least one TRP using the AI / ML model; as well as Sending the at least one predicted channel indicator for the at least one TRP to the first device.

18. A computer-readable medium comprising instructions stored thereon, the instructions being configured to cause an apparatus to at least perform the method according to claim 16 or the method according to claim 17.

Images