Test mechanism for positioning precision verification based on artificial intelligence / machine learning

Abstract

Systems, methods, apparatuses, and computer program products for verifying positioning accuracy. A method may include sending, by a test equipment, a request to measure position coordinates to a device, receiving, by the test equipment, a model reasoning including at least one position coordinate from the device, and determining, by the test equipment, whether a difference between the model reasoning and at least one ground truth value is within a threshold for a defined number of samples, wherein the at least one ground truth value comprises at least one known position coordinate of the test point.

Description

Technical Field

[0001] Some example embodiments may generally relate to mobile or wireless telecommunication systems, such as 3rd Generation Partnership Project (3GPP) Long Term Evolution (LTE), 5th Generation (5G) Radio Access Technology (RAT), New Radio (NR) access technology, 6th Generation (6G), and / or other communication systems. For example, certain example embodiments may relate to systems and / or methods for verifying positioning accuracy. Background Art

[0002] Examples of mobile or wireless telecommunication systems may include radio frequency (RF) 5G RAT, universal mobile telecommunication system (UMTS) terrestrial radio access network (UTRAN), LTE evolved UTRAN (E-UTRAN), advanced LTE (LTE-A), LTE-A Pro, NR access technology, and / or MulteFire Alliance. 5G wireless system refers to the next generation (NG) radio system and network architecture. 5G systems are typically built on 5G NR, but 5G (or NG) networks can also be built on E-UTRA radio. NR is expected to support service categories such as enhanced mobile broadband (eMBB), ultra-reliable low latency communications (URLLC), and massive machine type communications (mMTC). NR is expected to provide extremely broadband, ultra-robust, low latency connections and large-scale networks to support the Internet of Things (IoT). Next Generation Radio Access Network (NG-RAN) represents a radio access network (RAN) for 5G, which can provide radio access for NR, LTE, and LTE-A. Note that in 5G, a node that provides radio access functions to user equipment (e.g., similar to a Node B in UTRAN or an evolved Node B (eNB) in LTE) may be referred to as a next-generation Node B (gNB) when built on an NR radio, and as a next-generation eNB (NG-eNB) when built on an E-UTRA radio. Summary of the invention

[0003] According to some example embodiments, a method may include: sending, by a test rig, a request to measure a location coordinate to a device. The method may also include: receiving, by the test rig, a model inference including at least one location coordinate from the device. The method may also include: determining, by the test rig, whether a difference between the model inference and at least one ground truth value is within a threshold for a defined number of samples. The at least one ground truth value includes at least one known location coordinate of a test point.

[0004] According to certain example embodiments, an apparatus may include means for sending a request to a device for measuring a location coordinate. The apparatus may also include means for receiving a model inference including at least one location coordinate from the device. The apparatus may also include means for determining whether a difference between the model inference and at least one ground truth value is within a threshold for a defined number of samples. The at least one ground truth value includes at least one known location coordinate of a test point.

[0005] According to various example embodiments, a non-transitory computer-readable medium may include program instructions that, when executed by an apparatus, cause the apparatus to perform at least one method. The method may include sending a request to a device to measure a location coordinate. The method may also include receiving a model inference from the device including at least one location coordinate. The method may also include determining whether a difference between the model inference and at least one ground truth value is within a threshold for a defined number of samples. The at least one ground truth value includes at least one known location coordinate of a test point.

[0006] According to some example embodiments, a computer program product may perform a method. The method may include sending a request to a device to measure a location coordinate. The method may also include receiving a model inference including at least one location coordinate from the device. The method may also include determining whether a difference between the model inference and at least one ground truth value is within a threshold for a defined number of samples. The at least one ground truth value includes at least one known location coordinate of a test point.

[0007] According to certain example embodiments, an apparatus may include at least one processor and at least one memory storing instructions that, when executed by the at least one processor, cause the apparatus to at least send a request to a device for measuring a location coordinate. The at least one memory and instructions, when executed by the at least one processor, may also cause the apparatus to at least receive a model inference including at least one location coordinate from the device. The at least one memory and instructions, when executed by the at least one processor, may also cause the apparatus to at least determine whether a difference between the model inference and at least one ground truth value is within a threshold for a defined number of samples. The at least one ground truth value includes at least one known location coordinate of a test point.

[0008] According to various example embodiments, an apparatus may include a sending circuit system configured to perform sending a request to a device to measure a location coordinate. The apparatus may also include a receiving circuit system configured to perform receiving a model inference including at least one location coordinate from the device. The apparatus may also include a determining circuit system configured to perform determining whether a difference between the model inference and at least one ground truth value is within a threshold for a defined number of samples. The at least one ground truth value includes at least one known location coordinate of a test point.

[0009] According to some example embodiments, a method may include receiving, by a device, a request from a test rig to measure a location coordinate. The method may also include performing, by the device, one or more direct positioning measurements. The method may also include sending, by the device, a model inference including at least one location coordinate to the test rig.

[0010] According to certain example embodiments, an apparatus may include means for receiving a request for measuring a location coordinate from a test rig. The apparatus may also include means for performing one or more direct positioning measurements. The apparatus may also include means for sending a model inference including at least one location coordinate to the test rig.

[0011] According to various example embodiments, a non-transitory computer readable medium may include program instructions that, when executed by an apparatus, cause the apparatus to perform at least one method. The method may include receiving a request from a test rig to measure a location coordinate. The method may also include performing one or more direct positioning measurements. The method may also include sending a model inference including at least one location coordinate to the test rig.

[0012] According to some example embodiments, a computer program product may perform a method. The method may include receiving a request for measuring a location coordinate from a test rig. The method may also include performing one or more direct positioning measurements. The method may also include sending a model inference including at least one location coordinate to the test rig.

[0013] According to certain example embodiments, an apparatus may include at least one processor and at least one memory storing instructions that, when executed by the at least one processor, cause the apparatus to at least receive a request for measuring a location coordinate from a test rig. The at least one memory and instructions, when executed by the at least one processor, may also cause the apparatus to at least perform one or more direct positioning measurements. The at least one memory and instructions, when executed by the at least one processor, may also cause the apparatus to at least send a model inference including at least one location coordinate to the test rig.

[0014] According to various example embodiments, an apparatus may include receiving circuitry configured to perform receiving a request for measuring a location coordinate from a test rig. The apparatus may also include executing circuitry configured to perform performing one or more direct positioning measurements. The apparatus may also include sending circuitry configured to perform sending a model inference including at least one location coordinate to the test rig. BRIEF DESCRIPTION OF THE DRAWINGS

[0015] For a proper understanding of the example embodiments, reference should be made to the accompanying drawings, in which:

[0016] Figure 1illustrates an example of a test grid according to certain example embodiments;

[0017] Figure 2 illustrates an example of a flow chart of a method according to some example embodiments;

[0018] Figure 3 illustrates an example of a signaling diagram according to various example embodiments;

[0019] Figure 4 illustrates an example of a test scenario with a test location according to certain example embodiments;

[0020] Figure 5 illustrates an example of a mapping between test locations and radio signal parameters according to some example embodiments;

[0021] Figure 6 illustrates an example of a test setup according to various example embodiments, wherein multiple signals having TRP specific parameters are sent through a radio probe (ie, multiplexed);

[0022] Figure 7 illustrates an example of a signaling diagram according to certain example embodiments;

[0023] Figure 8 illustrates an example of a flow chart of a method according to various example embodiments;

[0024] Fig. 9 illustrates an example of a flow chart of a method according to various example embodiments;

[0025] Fig.10 illustrates examples of various network devices according to some example embodiments; and

[0026] Fig.11 An example of a 5G network and system architecture is illustrated in accordance with certain example embodiments. DETAILED DESCRIPTION

[0027] It is readily understood that the components of certain example embodiments, as generally described and illustrated in the figures herein, may be arranged and designed in a variety of different configurations. Therefore, the following detailed description of some example embodiments of systems, methods, apparatuses, and computer program products for verifying positioning accuracy is not intended to limit the scope of certain example embodiments, but is representative of selected example embodiments.

[0028] 3GPP defines test mechanisms for position estimates and latency based on Global Navigation Satellite System (GNSS) positioning. Such test mechanisms can be related to different satellites and can rely on GNSS measurements. However, there is currently no defined test mechanism to verify the positioning accuracy of the UE based on measurements made by the Positioning Reference Signal (PRS). For example, the PRS can be sent by a Transmission Reception Point (TRP) in the network. In addition, one of the challenges in testing positioning accuracy is extracting the ground truth of the actual position coordinates.

[0029] Certain example embodiments described herein may have various benefits and / or advantages to overcome the above disadvantages. For example, certain example embodiments may define a test mechanism to check the positioning accuracy of UE-based direct AI / ML positioning, which may rely on PRS transmission from test equipment (TE) or NE. Therefore, certain example embodiments discussed below are intended to improve computer-related technologies.

[0030] Some of the example embodiments discussed below relate to validating and testing a given UE-based AI / ML model. Specifically, it may be necessary to test and validate AI / ML models used for positioning (e.g., positioning estimates) against labeled data to improve the credibility of training data and model outputs. These labels can be credible and / or represented by ground truth by default. For example, for UE-based direct AI / ML positioning, positioning accuracy can be selected as a test metric / key performance indicator (KPI) for inference performance evaluation, particularly for proprietary AI / ML models deployed at the UE for positioning purposes. In addition, the network may not know the details of the UE-based AI / ML model, including its positioning accuracy and / or performance. However, the network may need to monitor the performance KPIs of the AI / ML model in order to initiate LCM-based processes (e.g., model reselection, update, switching, deactivation, etc.).

[0031] Current 5G / NR requirements assume that the UE reports measurements (e.g., reference signal received power (RSRP)) and the location management function (LMF) determines the location. Therefore, traditional requirements may be based on the accuracy and latency of measurements performed and reported by the UE, rather than the actual location. In the case of direct positioning, the UE may no longer need to report measurements (e.g., the UE only reports a position estimate). Therefore, traditional requirements may not apply, and there are no requirements or tests to verify direct UE-based positioning capabilities (e.g., positioning accuracy). Various traditional positioning methods (e.g., Assisted GNSS (A-GNSS)) may have performance requirements for position estimates and latency for GNSS-based positioning.

[0032] Various example embodiments of the testing process described herein for validating the positioning accuracy of UE-based AI / ML models may include the use of test equipment (TE) and a device under test (DUT), which may also be a positioning reference unit (PRU). Such testing may be performed in a system such as Figure 1 The test grid is shown in Figure 1. For example, the test grid represents a test area including a set of points with known position coordinates. Alternatively, the PRU can be placed at each point in the test grid to derive the position coordinates. Depending on the test requirements, the size of the test area and the spacing between points can be in the order of meters or centimeters.

[0033] The TE can configure the DUT with any desired configuration, including the AI / ML model whose performance is to be verified. The DUT can then be placed at the center or edge of the test grid; when the test process starts, the DUT can remain in the initial position for n seconds. While in this position, the DUT can calculate the position coordinates based on AI / ML model reasoning and send these coordinates to the TE. The TE (or other verification entity) can compare the position coordinates reported by the DUT with the actual position coordinates and generate / display the results (e.g., in a table and / or chart). The comparison results can be input into the AI / ML model for calibration purposes.

[0034] The DUT can also be moved within the test grid in a predefined trajectory and the above steps repeated. For example, the DUT can be held by a person, or fixed to a robot or drone, so that it can be moved within the test grid in a controlled manner during testing. If the average positioning accuracy (or based on a preconfigured threshold) is within a defined range, the test is declared as "passed"; otherwise, the test can be declared as "failed"

[0035] on the whole, Figure 2 An example of a flow chart of a method 200 according to some example embodiments described herein is illustrated, which method may be performed by a method similar to the following example. Fig.10 The method may be performed by a device of any one of the NE 1010 or UE 1020 shown. Specifically, at step 201, the method may include identifying one or more requirements of a positioning model or function based on a trained ML under consideration. Such a trained ML model may be associated with a set of requirements, assumptions, parameters, inputs (e.g., RSRP, RSTD, number of TRPs, CIR), network configuration (e.g., the presence and period of reference signals), target positioning accuracy, and optionally any delay / duration to complete the test process.

[0036] At step 202, the method may include mapping the requirements and assumptions identified at step 201 to a test plan. One or more predefined rules may be applied to derive a customized test plan. For example, a first rule may define the positioning accuracy required to select a test grid granularity. The grid granularity (e.g., in meters) should be lower than or at least equal to the target positioning accuracy of the training model. In another example, a test duration / delay requirement rule may be used to select multiple DUTs and map them to a test grid (e.g., for a mandatory low test duration, more test UEs may be required to process in parallel and achieve a complete test in a shorter time). In another example, a third rule may define the DUT trajectory under test (e.g., what is the sequence of DUT positions under test, and how to change the transmit signal if necessary).

[0037] At step 203, a test process may be executed that follows the test plan established at step 202. At step 204, the test results may be analyzed and performance may be derived according to a selected KPI (e.g., the 90% percentile of positioning accuracy achieved), and / or a determination may be made as to whether the test passed or failed.

[0038] Figure 3 An example of a signaling diagram 300 is illustrated, which depicts a method for evaluating the positioning accuracy of direct AI / ML-based positioning. According to certain example embodiments, any one of the DUT 320 and the TE 330 may be similar to Fig.10 NE 1010 or UE 1020 is shown.

[0039] At operation 301 , the TE 330 may send a command to the DUT 320 for enabling AI / ML-based direct positioning.

[0040] At operation 302 , the DUT 320 may enable AI / ML based direct positioning.

[0041] At operation 303 , the DUT 320 may send a command for confirming activation of the AI / ML-based direct positioning to the TE 330 .

[0042] At operation 304, the DUT 320 may send a command to move from a current test point to another test point to the TE 330. The command may or may not be sent over an air interface and / or may or may not be an application level command.

[0043] At operation 305 , in response to the command received at operation 304 , the DUT 320 may move to a test point indicated by the TE 330 .

[0044] At operation 306, the DUT 320 may send a confirmation command to the TE 330 confirming that the DUT 320 has moved to the indicated test point. The command may or may not be sent over the air interface and / or may or may not be an application level command. It should be noted that if the trajectory of the DUT 320 and the duration of dwelling at each test point are pre-configured and available at the DUT 320, operations 304-306 are optional.

[0045] At operation 307 , the TE 330 may send a request to the DUT 320 to measure location coordinates using AI / ML based direct positioning.

[0046] At operation 308 , the DUT 320 may perform AI / ML based direct positioning measurements.

[0047] At operation 309 , the DUT 320 may report the AI / ML model inferences (ie, location coordinates) to the TE 330 .

[0048] At operation 310 , the TE 330 may compare the inferences to known ground truth (ie, known location coordinates of the test points).

[0049] At operation 311, TE 330 may determine whether the difference between the inference and the ground truth is within an acceptable range for a defined number of samples. If so, at operation 312, TE 330 may verify the positioning accuracy. Otherwise, if it is not within an acceptable range, TE 330 may not perform verification. Operations 304-311 may be repeated.

[0050] In some example embodiments, Figure 3 The operations shown can be performed without using a test grid. Instead, GNSS-based position coordinates or any other non-AI / ML method can be used as ground truth for comparison of AI / ML model inference.

[0051] In various example embodiments, the TE 330 may provide monitoring data to the DUT 320 that may include relevant inputs (e.g., the ground truth location of the DUT 320) to perform model monitoring. The DUT 320 may use the data received from the TE 330 to validate and test its AI / ML model, and may calculate a metric "delta" indicative of its AI / ML model performance, such as based on prediction accuracy (e.g., mean square error (MSE) between predicted measurements (i.e., UE location) and ground truth (e.g., DUT 320 obtained from the TE 330)).

[0052] Certain example embodiments may include a test generalization of the model, where the input type / dimension may be similar to the training data set. For example, TE 330 may issue a command to DUT 320 to move to the next test point. Alternative methods may include using a preconfigured test plan (e.g., DUT 320 may move between different test points based on the test plan without issuing a command from TE 330 to DUT 320 each time (i.e., operations 304-306). Before starting the positioning measurement, the DUT 320 should be instructed at operation 301 or 304 to stay at each test point. The test may be performed with two UEs or a multiple SIM (MUSIM) UE with 2 SIMs, where UE1 / SIM1 may be configured so that the AI / ML model is enabled and the other UE2 / SIM2 may be configured so that the AI / ML model is disabled. The two UEs / SIMs may initially be placed in the same location, and they may move simultaneously, one operation at a time, as described above. TE 330 may compare the location coordinates of UE1 and UE2 to verify the positioning accuracy.

[0053] In certain example embodiments, the UE may remain stationary in the test setup, but the transmission parameters of the radio signal (e.g., transmission timing, Rx power, and propagation / transmission delay) may be defined before each set of tests (i.e., per expected location based on the test scenario). For example, the DUT positioning accuracy may be tested at multiple locations, such as Figure 4 The location of the TRP and the test location may be known or defined before the test is executed (eg, as shown in step 201 above).

[0054] The DUT algorithm can define the location based on measurements performed from different radio signal sources (e.g., from different TRPs / gNBs). Depending on the capabilities of the algorithm (e.g., ML function, AI / ML model, etc.) defined at step 201, different parameters such as the Rx power of the reference signal (RS) (e.g., RSRP), arrival time, arrival angle, etc. can be considered. There can be a correspondence between the radio signal parameters and the location. These parameters of the radio signal can be calculated based on the propagation model and / or the environment, or obtained from field measurements. A mapping between the test point location (i.e., coordinates) and the radio signal characteristics can be established. Figure 5 An example is depicted in which two parameters may be considered, namely, signal power (e.g., RX power of a positioning reference signal (PRS) or SSB per TRP) and propagation delay.

[0055] Simulation of the propagation environment may not require physical distribution of TRPs, since there may be a long distance between the TRPs and the UE, but the size of the test setup may also be limited. In addition, the number of TRPs / probes / radio sources may also be limited, while in practice, the UE may need to measure many sources for accurate positioning. Therefore, in the test setup, transmissions from multiple real TRPs (i.e., TRPs that the UE is able to measure) can / may be mapped to a smaller number of probes (i.e., physically implemented transmitters available in the test setup). The number of probes may be limited, and the UE may need more TRPs than the number of probes to determine its position. The signal may correspond to more TRPs (e.g., with TRP-specific parameters) sent through several probes. In this way, the UE is able to measure signals from many TRPs even if there are only a few physical transmitters. Figure 6 The scheme of shows a method to achieve this goal. In one example, signals (e.g., PRS transmissions) with TRP-specific parameters (e.g., TX / RX power / attenuation, propagation delay / transmission time, etc.) can be time-multiplexed (i.e., the signals are sent continuously). In another example, the TRP-specific signals can be combined via a radio frequency (RF) component (e.g., a radio signal combiner).

[0056] Figure 7 An example of a signaling diagram 700 depicting a direct positioning text sequence is illustrated. According to certain example embodiments, any of the DUT 720 and the TRP1-n 730 may be similar to Fig.10 NE 1010 or UE 1020 is shown.

[0057] In general, operations 701 - 703 may be similar to steps 201 - 202 described above, where a test plan is defined at step 201 and the AI / ML-based position prediction capabilities of the DUT 720 are defined / validated at step 202 (eg, model / function selection).

[0058] At operation 703, the DUT 720 may maintain a connection model with a system simulator (SS), which may include multiple cells / gNBs / TRPs / access points. A connection may be established with Cell1 / TRP1. Other TRPs may be used only to generate reference signals (RS, such as for positioning (PRS)) that may be used for positioning.

[0059] At operation 704, operations 705-713 may be repeated for each of the test points TRP1-n. For example, at operation 706, corresponding signal generation / transmission parameters may be applied.

[0060] At operation 707, the direct positioning function may be activated. The activation may also include indication of a specific AI / ML function or AI / ML model if supported by the DUT 720. Activation / indication of the model may also be performed at the application level and / or through a specific test interface.

[0061] At operation 714, the positioning function may be disabled so that measurements in one test point do not affect measurements in another test point.

[0062] At operations 715-716, TRP1-n may determine whether each test point and the overall test passed or failed, respectively. For example, success criteria may include that the inferred position accuracy of DUT 720 remains within xx meters (eg, 0.5 meters) within yy% of the report.

[0063] Figure 8 An example of a flow chart of a method 800 according to certain example embodiments is illustrated, which method may be performed by a DUT such as Fig.10 NE 1010 or UE 1020 as shown) is executed.

[0064] At step 801 , the method may include receiving a command for enabling AI / ML based direct positioning.

[0065] At step 802, the method may further include enabling AI / ML based direct positioning.

[0066] At step 803 , the method may further include sending a command for confirming activation of the AI / ML based direct positioning.

[0067] At step 804, the method may also include receiving a command to move from a current grid position to another grid position.

[0068] At step 805, the method may also include moving to another grid location.

[0069] At step 806, the method may also include sending a confirmation command confirming that the DUT has moved to the indicated grid location.

[0070] At step 807 , the method may further include receiving a request to measure positioning coordinates using AI / ML based direct positioning.

[0071] At step 808, the method may further include performing AI / ML based direct positioning measurements.

[0072] At step 809 , the method may also include reporting the AI / ML model reasoning.

[0073] Fig. 9An example of a flow chart of a method 900 according to certain example embodiments is illustrated, which method may be performed by a TE such as Fig.10 NE 1010 or UE 1020 as shown) is executed.

[0074] At step 901 , the method may include sending a command for enabling AI / ML based direct positioning.

[0075] At step 902 , the method may further include receiving a command to confirm activation of the AI / ML based direct positioning.

[0076] At step 903 , the method may further include sending a command for moving from a current test point to another test point.

[0077] At step 904, the method may also include receiving a confirmation command confirming that the DUT has moved to the indicated test point.

[0078] At step 905 , the method may further include sending a request to measure positioning coordinates using AI / ML based direct positioning.

[0079] At step 906, the method may also include receiving a report of the AI / ML model reasoning.

[0080] At step 907 , the method may further include comparing the inference to a known ground truth.

[0081] At step 908 , the method may also include determining whether the difference between the inference and the ground truth is within an acceptable range for a defined number of samples.

[0082] At step 909 , the method may further include verifying the positioning accuracy.

[0083] At step 910, the method may also include not performing verification.

[0084] In certain example embodiments, a method may include sending, by a test rig, a request to measure a location coordinate to a device. The method may also include receiving, by the test rig, a model inference from the device including at least one location coordinate. The method may also include receiving, by the test rig, a model inference from the device including at least one location coordinate.

[0085] In some example embodiments, the test equipment may include a network entity, a base station, or a user equipment. The equipment may include a network entity, a base station, or a user equipment, wherein the user equipment may include one or more receivers.

[0086] In various example embodiments, the method may include comparing, by the test equipment, the location coordinates of the first receiver of the user equipment with the location coordinates of the second receiver of the user equipment.

[0087] In certain example embodiments, the method may include adjusting, by the test equipment, at least one transmission parameter of the test equipment to simulate a change in the location of the device. The at least one transmission parameter may include at least one of transmission timing, received power, transmission power, propagation delay, or transmission delay.

[0088] In some example embodiments, the request for measuring the location coordinates may be based on at least one of: artificial intelligence-based direct positioning or machine learning-based direct positioning.

[0089] In various example embodiments, the determining may further include comparing, by the test rig, the model inferences to at least one ground truth value.

[0090] In certain example embodiments, the known location coordinates of the test point may include at least one actual location coordinate of the test point.

[0091] In some example embodiments, the method may include sending, by the test equipment, a command for the device to move from the test point to another test point, and receiving, by the test equipment, a confirmation that the device has moved to the other test point.

[0092] In various example embodiments, the test grid may include at least a test point and another test point. The granularity of the test grid may be based at least in part on the required positioning accuracy. At least one of a time period, a start time, or an end time may indicate when the device is located at the test point. The trajectory of the device may be defined or preconfigured by synchronization signaling.

[0093] In certain example embodiments, at least one of an artificial intelligence inference including at least one location coordinate or a machine learning inference including at least one location coordinate is sent by the test equipment. At least one of the artificial intelligence inference or the machine learning inference may be sent via a synchronization signal or the test equipment.

[0094] In some example embodiments, the method may further include sending, by the test equipment, a command for confirming activation of at least one of artificial intelligence-based direct positioning or machine learning-based direct positioning. The command may be sent via an air interface or an application-level command.

[0095] In certain example embodiments, the at least one ground truth value may include at least one global navigation satellite system based position coordinate.

[0096] In some example embodiments, the method may further include sending, by the test rig, monitoring data including at least one model monitoring input. The at least one model monitoring input may include at least one ground truth position.

[0097] In various example embodiments, the determining may be performed without a test grid.The determining may be performed based on position coordinates based on a global navigation satellite system.

[0098] In certain example embodiments, the method may include testing, by a test rig, generalization of the model reasoning. The testing may be performed using at least one configuration parameter that matches at least one parameter in a data set used to train the artificial intelligence or machine learning model. The at least one configuration parameter may include at least one transmitter. Generalization may refer to testing the performance of the ML model under different conditions (e.g., NLOS conditions).

[0099] In some example embodiments, the test equipment may be configured to move based on at least one preconfigured test plan. For example, at least one preconfigured test plan may define at least one movement pattern on a test grid, and a time period for the test equipment to remain at a test point. At least one preconfigured test plan may define a test plan for the device and at least one other device. The device may be configured with an artificial intelligence or machine learning model, and at least one other device does not use an artificial intelligence or machine learning model.

[0100] In certain example embodiments, the method may include comparing, by the test equipment, the location coordinates of the device with the location coordinates of at least one other device.

[0101] In certain example embodiments, a method may include receiving, by a device, a request from a test rig to measure a location coordinate. The method may also include performing, by the device, one or more direct positioning measurements. The method may also include sending, by the device, a model inference including at least one location coordinate to the test rig.

[0102] The device may include a network entity, a base station, or a user equipment. The test equipment may include a network entity, a base station, or a user equipment.

[0103] In some example embodiments, one or more direct positioning measurements may be performed based on at least one transmission parameter. For example, the at least one transmission parameter may include at least one of transmission timing, received power, propagation delay, or transmission delay.

[0104] In various example embodiments, direct positioning may include at least one of artificial intelligence-based direct positioning or machine learning-based direct positioning.

[0105] In certain example embodiments, the method may further include receiving, by the device, a command for the device to move from the test point to another test point, and sending, by the device, a confirmation confirming that the device has moved to the other test point.

[0106] In some example embodiments, the request for measuring the location coordinates may be based on at least one of direct positioning based on artificial intelligence or direct positioning based on machine learning. The test grid may include at least a test point and another test point. The granularity of the test grid may be based at least in part on the required positioning accuracy. At least one of a time period, a start time, or an end time may indicate when the device is at the test point. The trajectory of the device may be defined or preconfigured by synchronization signaling.

[0107] In various example embodiments, the method may further include receiving, by the device, a command to confirm activation of at least one of artificial intelligence-based direct positioning or machine learning-based direct positioning, and sending, by the device, a confirmation that the device has moved to another test point. The confirmation may be sent via an air interface or an application-level command. The at least one ground truth value may include at least one global navigation satellite system-based position coordinate.

[0108] Fig.10 An example of a system according to some example embodiments is illustrated. In an example embodiment, the system may include a plurality of devices, such as NE 1010 and / or UE 1020 .

[0109] NE 1010 may be one or more of a base station (e.g., a 3G UMTS NodeB, a 4G LTE evolved NodeB, or a 5G NR next-generation NodeB), a serving gateway, a server, and / or any other access node, or a combination thereof.

[0110] The NE 1010 may also include at least one gNB centralized unit (CU) that may be associated with at least one gNB distributed unit (DU). The at least one gNB-CU and the at least one gNB-DU may communicate with each other via at least one F1 interface, at least one X1 interface, and at least one X2 interface via a fifth generation core (5GC). n -C interface and / or at least one NG interface for communication.

[0111] UE 1020 may include one or more mobile devices, such as a mobile phone, a smart phone, a personal digital assistant (PDA), a tablet or portable media player, a digital camera, a pocket camera, a video game console, a navigation unit (such as a global positioning system (GPS) device), a desktop or laptop computer, a single positioning device (such as a sensor or smart meter), or any combination thereof. In addition, NE 1010 and / or UE 1020 may be one or more Citizen Broadband Radio Service Devices (CBSDs).

[0112] NE 1010 and / or UE 1020 may include at least one processor, respectively denoted as 1011 and 1021. Processors 1011 and 1021 may be embodied by any computing or data processing device, such as a central processing unit (CPU), an application specific integrated circuit (ASIC), or the like. The processor may be implemented as a single controller, or multiple controllers or processors.

[0113] As shown in 1012 and 1022, at least one memory may be provided in one or more devices. The memory may be fixed or removable. The memory may include computer code or computer program instructions contained therein. Memories 1012 and 1022 may independently be any suitable storage device, such as a non-transitory computer-readable medium. The term "non-transitory" used herein may correspond to the limitations of the medium itself (i.e., tangible, rather than a signal), rather than the limitations of data storage persistence (e.g., random access memory (RAM) versus read-only memory (ROM)). A hard disk drive (HDD), random access memory (RAM), flash memory, or other suitable memory may be used. The memory may be combined on a single integrated circuit as a processor, or may be separated from one or more processors. In addition, the computer program instructions stored in the memory and that may be processed by the processor may be computer program codes in any suitable form, for example, a compiled or interpreted computer program written in any suitable programming language.

[0114] Processors 1011 and 1021, memories 1012 and 1022, and any subset thereof may be configured to provide Figure 1-Figure 9 Components corresponding to the various boxes of . Although not shown, the device may also include positioning hardware, such as GPS or micro-electromechanical system (MEMS) hardware, which can be used to determine the location of the device. Other sensors are also allowed and can be configured to determine location, altitude, speed, orientation, etc., such as a barometer, compass, etc.

[0115] like Fig.10As shown, transceivers 1013 and 1023 may be provided, and one or more devices may also include at least one antenna, shown as 1014 and 1024, respectively. The device may have many antennas, such as an antenna array configured for multiple-input multiple-output (MIMO) communication, or multiple antennas for multiple RATs. For example, other configurations of these devices may be provided. Transceivers 1013 and 1023 may be transmitters, receivers, both transmitters and receivers, or units or devices that may be configured for both transmission and reception.

[0116] The memory and the computer program instructions may be configured to, together with the processor of a specific device, cause a hardware device (such as a UE) to perform any of the above processes (ie, Figure 1-Figure 9 ). Thus, in certain example embodiments, a non-transitory computer-readable medium may be encoded with computer instructions that, when executed in hardware, perform a process such as one of the processes described herein. Alternatively, certain example embodiments may be performed entirely in hardware.

[0117] In certain example embodiments, the apparatus may include a Figure 1-Figure 9 As used in this application, the term "circuitry" may refer to one or more or all of the following: (a) a hardware-only circuit implementation (such as an implementation in analog and / or digital circuitry only), (b) a combination of hardware circuitry and software, such as (as applicable): (i) a combination of (multiple) analog and / or digital hardware circuits and software / firmware, and (ii) any portion of (multiple) hardware processors (including (multiple) digital signal processors), software, and (multiple) memories with software that work together to enable a device (such as a mobile phone or server) to perform various functions), and (c) (multiple) hardware circuits and / or (multiple) processors, such as (multiple) microprocessors or portions of (multiple) microprocessors, which require software (e.g., firmware) to operate, but when software is not required for operation, the software may not be present. This definition of circuitry applies to all uses of the term in this application, including in any claims. As a further example, as used in this application, the term circuitry also covers an implementation of only a hardware circuit or processor (or multiple processors) or a portion of a hardware circuit or process and its accompanying software and / or firmware. For example, the term circuitry would also cover, if applicable to a particular claim element, a baseband integrated circuit or a processor integrated circuit for a mobile device, or a similar integrated circuit in a server, cellular network device, or other computing or networking equipment.

[0118] Fig.11An example of a 5G network and system architecture according to certain example embodiments is illustrated. A plurality of network functions are shown, which may be implemented as software operating as part of a network device or dedicated hardware, software operating as a network device itself or dedicated hardware, or virtual functions operating as a network device or dedicated hardware. Fig.11 The NE and UE shown may be similar to NE 1010 and UE 1020, respectively. The user plane function (UPF) may provide services such as intra-RAT and inter-RAT mobility, routing and forwarding of data packets, inspection of packets, user plane quality of service (QoS) processing, buffering of downlink packets, and / or triggering of downlink data notifications. The application function (AF) may primarily interface with the core network to facilitate application use of service routing and interact with the policy framework.

[0119] According to some example embodiments, processors 1011 and 1021 and memories 1012 and 1022 may be included in a processing circuit system or a control circuit system, or may form a part of a processing circuit system or a control circuit system. In addition, in some example embodiments, transceivers 1013 and 1023 may be included in a transceiver circuit system, or may form a part of a transceiver circuit system.

[0120] In some example embodiments, an apparatus (e.g., NE 1010 and / or UE 1020) may include a component for performing a method, process, or any variant discussed herein. Examples of the component may include one or more processors, memories, controllers, transmitters, receivers, and / or computer program codes for causing the execution of operations.

[0121] In various example embodiments, apparatus 1010 or 1020 may be controlled by memory 1012 and processor 1011 (or memory 1022 and processor 1021) to send a request to a device for measuring a location coordinate, receive a model inference including at least one location coordinate from the device, and determine whether a difference between the model inference and at least one ground truth value is within a threshold for a defined number of samples. The at least one ground truth value includes at least one known location coordinate of a test point.

[0122] Certain example embodiments may be directed to an apparatus comprising means for performing any of the methods described herein, for example, means for sending a request to a device to measure a location coordinate, means for receiving a model inference including at least one location coordinate from the device, and means for determining whether a difference between the model inference and at least one ground truth value is within a threshold for a defined number of samples. The at least one ground truth value includes at least one known location coordinate of a test point.

[0123] In various example embodiments, device 1010 or 1020 can be controlled by memory 1012 and processor 1011 (or memory 1022 and processor 1021) to receive a request for measuring a position coordinate from a test device, perform one or more direct positioning measurements, and send a model inference including at least one position coordinate to the test device.

[0124] Certain example embodiments may relate to an apparatus comprising components for performing any of the methods described herein, including, for example, components for receiving a request for measuring a location coordinate from a test rig, components for performing one or more direct positioning measurements, and components for sending a model inference including at least one location coordinate to the test rig.

[0125] The features, structures, or characteristics of the example embodiments described throughout this specification may be combined in any suitable manner in one or more example embodiments. For example, the use of the phrases "various embodiments," "certain embodiments," "some embodiments," or other similar language throughout this specification refers to the fact that a particular feature, structure, or characteristic described in conjunction with the example embodiments may be included in at least one example embodiment. Thus, the appearances of the phrases "in various embodiments," "certain embodiments," "in some embodiments," or other similar language throughout this specification do not necessarily all refer to the same set of example embodiments, and the described features, structures, or characteristics may be combined in any suitable manner in one or more example embodiments.

[0126] As used herein, “at least one of: ” and “at least one of ” and similar expressions (wherein a list of two or more elements is connected by “and” or “or”) refer to at least any one of these elements, or at least any two or more of these elements, or at least all of these elements.

[0127] In addition, if necessary, the above-mentioned different functions or processes can be performed in different orders and / or simultaneously with each other. In addition, if necessary, one or more of the above-mentioned functions or processes can be optional or can be combined. Therefore, the above description should be considered as an explanation of the principles and teachings of some example embodiments, rather than a limitation thereof.

[0128] Those skilled in the art will readily appreciate that the example embodiments discussed above may be practiced with processes in a different order and / or with hardware elements in configurations different from those disclosed. Therefore, although some embodiments have been described based on these example embodiments, it will be apparent to those skilled in the art that certain modifications, variations, and alternative constructions will be apparent while remaining within the spirit and scope of the example embodiments.

[0129] Partial Glossary

[0130] 3GPP: Third Generation Partnership Project

[0131] 5G: Fifth Generation

[0132] 5GC: Fifth Generation Core

[0133] 6G: Sixth Generation

[0134] AF: Application Function

[0135] A-GNSS: Assisted Global Navigation Satellite System

[0136] AI: Artificial Intelligence

[0137] ASIC: Application-Specific Integrated Circuit

[0138] CBSD: Citizens Broadband Radio Service Device

[0139] CIR: Channel Impulse Response

[0140] CPU: Central Processing Unit

[0141] CU: Centralized Unit

[0142] DU: Distributed Unit

[0143] DUT: Device Under Test

[0144] eMBB: enhanced mobile broadband

[0145] eNB: Evolved Node B

[0146] gNB: Next Generation Node B

[0147] GNSS: Global Navigation Satellite System

[0148] GPS: Global Positioning System

[0149] HDD: Hard Disk Drive

[0150] IoT: Internet of Things

[0151] KPI: Key Performance Indicator

[0152] LMF: Location Management Function

[0153] LoS: Line of sight

[0154] LTE: Long Term Evolution

[0155] LTE-A: Long Term Evolution Advanced

[0156] MEMS: Micro-Electro-Mechanical Systems

[0157] MIMO: Multiple Input Multiple Output

[0158] ML: Machine Learning

[0159] mMTC: Massive Machine Type Communications

[0160] MSE: Mean Squared Error

[0161] NE: Network Entity

[0162] NG: Next Generation

[0163] NG-eNB: Next Generation Evolved Node B

[0164] NG-RAN: Next Generation Radio Access Network

[0165] NLoS: Non Line of Sight

[0166] NR: New Radio

[0167] PDA: Personal Digital Assistant

[0168] PRS: Positioning Reference Signal

[0169] PRU: Positioning Reference Unit

[0170] QoS: Quality of Service

[0171] RAM: Random Access Memory

[0172] RAN: Radio Access Network

[0173] RAT: Radio Access Technology

[0174] RF: Radio Frequency

[0175] ROM: Read Only Memory

[0176] RS: Reference signal

[0177] RSRP: Reference Signal Received Power

[0178] RSTD: Reference Signal Time Difference

[0179] SS: System Simulator

[0180] SSB: Synchronization Signal Block

[0181] TE: Test Equipment

[0182] TRP: Transmission Reception Point

[0183] UE: User Equipment

[0184] UMTS: Universal Mobile Telecommunications System

[0185] UPF: User Plane Function

[0186] URLLC: Ultra-Reliable Low-Latency Communications

[0187] UTRAN: Universal Mobile Telecommunications System Terrestrial Radio Access Network

[0188] WLAN: Wireless Local Area Network

Claims

1. An apparatus for wireless telecommunications, comprising: at least one processor; as well as at least one memory storing instructions, which, when executed by the at least one processor, cause the apparatus to at least: Sending a request to the device for measuring location coordinates; receiving, from the device, model inferences including at least one location coordinate; as well as A determination is made as to whether a difference between the model inference and at least one ground truth value is within a threshold for a defined number of samples, wherein the at least one ground truth value comprises at least one known location coordinate of a test point.

2. The apparatus according to claim 1, wherein the apparatus comprises a network entity, a base station, or a user equipment.

3. The apparatus according to claim 1, wherein the device comprises a network entity, a base station, or a user equipment. The apparatus of claim 3 , wherein the user equipment comprises one or more receivers.

5. The apparatus of claim 4, wherein the at least one memory, and the instructions, when executed by the at least one processor, further cause the apparatus to at least: The location coordinates of the first receiver of the user equipment are compared with the location coordinates of the second receiver of the user equipment.

6. The apparatus of claim 1, wherein the at least one memory, and the instructions, when executed by the at least one processor, further cause the apparatus to at least: At least one transmission parameter of the apparatus is adjusted to simulate a change in the position of the device.

7. The apparatus of claim 1, wherein the request for measuring location coordinates is based on at least one of: direct positioning based on artificial intelligence, or direct positioning based on machine learning.

8. The apparatus of claim 1, wherein the at least one memory, and the instructions, when executed by the at least one processor, further cause the apparatus to at least: The model inferences are compared to the at least one ground truth.

9. The apparatus of claim 1 , wherein the known position coordinates of the test point include: At least one actual position coordinate of the test point.

10. The apparatus of claim 1, wherein the at least one memory, and the instructions, when executed by the at least one processor, further cause the apparatus to at least: A command is sent for the device to move from the test point to another test point.

11. The apparatus of claim 1, wherein at least one of a time period, a start time, or an end time indicates when the device is located at the test point.

12. The apparatus of claim 1, wherein the trajectory of the device is defined or preconfigured by synchronization signaling.

13. The apparatus of claim 1, wherein the at least one memory, and the instructions, when executed by the at least one processor, further cause the apparatus to at least: Send at least one of: an artificial intelligence inference including at least one location coordinate, or a machine learning inference including at least one location coordinate.

14. The apparatus of claim 1, wherein the at least one memory, and the instructions, when executed by the at least one processor, further cause the apparatus to at least: A command is sent to confirm activation of at least one of: artificial intelligence-based direct positioning, or machine learning-based direct positioning.

15. The apparatus of claim 1, wherein the at least one ground truth value comprises: At least one global navigation satellite system based position coordinate.

16. The apparatus of claim 1, wherein the at least one memory, and the instructions, when executed by the at least one processor, further cause the apparatus to at least: Monitoring data including at least one model monitoring input is transmitted.

17. The apparatus of claim 1, wherein the at least one memory, and the instructions, when executed by the at least one processor, further cause the apparatus to at least: Test the generalization of the model's inference.

18. The apparatus of claim 17, wherein the testing is performed using at least one configuration parameter that matches at least one parameter in a data set used to train an artificial intelligence or machine learning model.

19. The apparatus of claim 18, wherein the at least one configuration parameter comprises at least one transmitter.

20. The apparatus of any one of claims 1 to 19, wherein the apparatus is configured to move based on at least one preconfigured test plan.

21. An apparatus for wireless telecommunications, comprising: at least one processor; as well as at least one memory storing instructions, which, when executed by the at least one processor, cause the apparatus to at least: receiving a request from a test rig to measure position coordinates; performing one or more direct positioning measurements; and Model inferences including at least one position coordinate are sent to the test rig.

22. The apparatus of claim 21, wherein the apparatus comprises a network entity, a base station, or a user equipment.

23. The apparatus of claim 21, wherein the test equipment comprises a network entity, a base station, or a user equipment.

24. The apparatus of claim 21, wherein the direct positioning comprises at least one of: direct positioning based on artificial intelligence, or direct positioning based on machine learning.

25. The apparatus of claim 21, wherein the at least one memory, and the instructions, when executed by the at least one processor, further cause the apparatus to at least: A command is received for the device to move from a test point to another test point.

26. The apparatus of claim 21, wherein the request for measuring location coordinates is based on at least one of: direct positioning based on artificial intelligence, or direct positioning based on machine learning.

27. The apparatus of claim 21, wherein at least one of a time period, a start time, or an end time indicates when the device is located at the test point.

28. The apparatus of claim 21, wherein the trajectory of the apparatus is defined or preconfigured by synchronization signaling.

29. The apparatus of claim 21, wherein the at least one memory, and the instructions, when executed by the at least one processor, further cause the apparatus to at least: A command is received to confirm activation of at least one of: artificial intelligence-based direct positioning, or machine learning-based direct positioning.

30. The apparatus of any one of claims 21 to 29, wherein at least one ground truth value comprises: At least one global navigation satellite system based position coordinate.

31. A method for wireless telecommunications, comprising: Sending, by the test equipment, a request for measuring position coordinates to the device; receiving, by the test rig, from the device, model inferences including at least one location coordinate; as well as A determination is made by the test equipment whether a difference between the model inference and at least one ground truth value is within a threshold for a defined number of samples, wherein the at least one ground truth value comprises at least one known position coordinate of a test point.

32. A method for wireless telecommunications, comprising: receiving, by the device from the test equipment, a request to measure position coordinates; performing one or more direct positioning measurements by the device; as well as Model inferences including at least one position coordinate are sent by the device to the test rig.

Images