Consistency testing for AI / ML beam management: pre-alignment and generation of set A / B beams

Abstract

For the Test Equipment (TE) of the Device Under Test (DUT) used for conformance testing of ML-based Business Model (BM) use cases, it receives an instruction that the DUT supports ML-based BM conformance testing utilizing at least a combination of test probes. The TE adjusts the configuration for the DUT and the pre-test alignment information for the conformance testing. The TE sends the configuration and pre-test alignment information instructions to the DUT. Based on the configuration and pre-test alignment information, the TE performs conformance testing of the ML-based BM use cases for the DUT. The DUT sends an instruction to the TE that the DUT supports ML-based BM conformance testing and receives instructions on configuration and alignment information from the test equipment. Based on the configuration and alignment information, the DUT configures the information to be used for conformance testing and performs the conformance testing.

Description

Technical Field

[0001] The examples of embodiments described herein generally relate to wireless communication, and more specifically, to beam management in wireless communication. Background Technology

[0002] AI / ML (Artificial Intelligence / Machine Learning) models are used in wireless systems. One such use of AI / ML models is for beam management, where both the transmitting and receiving devices select the same beam for communication. For example, beam management can involve transmission beam prediction by both a UE (User Equipment) side model and a NW (Network) side model.

[0003] The spatial domain downlink (DL) transmit (Tx) beam prediction for set A beam is based on measurements of set B beam (and this is referred to as "BM-Case 1"). The temporal DL Tx beam prediction for set A beam is based on historical measurements of set B beam (and this is referred to as "BM-Case 2").

[0004] These AI / ML models used for beam management need to be tested. This testing is called conformance testing, and it means checking whether the AI / ML model implemented by the UE passes or fails the test when using both set A and set B beams. Specifically, the pass / fail decision can be made regarding whether the “best” predicted beams or the “top K” best predicted beam sets reported by the UE to the NW based on its AI / ML model are aligned with the truly best beams or the truly top K beams. The NW must know this information in the form of a baseline fact. The term “K” can be, for example, 1, 2, or 4, or is generally not a large number. This type of conformance testing can be improved. Summary of the Invention

[0005] This section is intended to include examples and not to impose limitations. For the examples in this section, an advantage includes performing alignment before conducting conformance testing, which should result in fewer issues during or in relation to the outcome of conformance testing. Additionally, multiple interference modes can be applied to the UE instead of a single interference mode. This better simulates the UE's real-world environment and helps to see how the UE handles different interference modes.

[0006] In the example, a method is disclosed, comprising: for a test device used to test a device under test (DUT) for conformance testing of a machine learning-based beam management use case, receiving an instruction that the DUT supports conformance testing of machine learning-based beam management utilizing at least a combination of test probes; adjusting the configuration for the DUT and pre-test alignment information for conformance testing of machine learning-based beam management by the test device; sending the configuration and pre-test alignment information from the test device to the DUT; and performing conformance testing of the machine learning-based beam management use case for the DUT by the test device based on the configuration and pre-test alignment information.

[0007] Additional examples include a computer program that includes instructions for performing the methods described in the preceding paragraph when the computer program is run on the device. The computer program according to this paragraph is a computer program product that includes a computer-readable medium carrying the instructions for the device embodied therein. Another example is a computer program according to this paragraph that can be directly loaded into the internal memory of the device.

[0008] An example of an apparatus includes one or more processors and one or more memories storing instructions that, when executed by the one or more processors, cause the apparatus to perform at least the following: For a test apparatus used to test a device under test (DUT) for conformance testing of a machine learning-based beam management use case, receiving an instruction that the DUT supports conformance testing of machine learning-based beam management utilizing at least a combination of test probes; adjusting the configuration for the DUT and pre-test alignment information for conformance testing of machine learning-based beam management by the test apparatus; sending an instruction from the test apparatus to the DUT regarding the configuration and pre-test alignment information; and performing conformance testing of the machine learning-based beam management use case for the DUT based on the configuration and pre-test alignment information by the test apparatus.

[0009] An example of a computer program product includes a computer-readable storage medium carrying instructions that, when executed by a device, cause the device to perform at least the following operations: receiving, for a test device used to test a device under test (DUT) for conformance testing of a machine learning-based beam management use case, an instruction that the DUT supports conformance testing of machine learning-based beam management utilizing at least a combination of test probes; adjusting the configuration for the DUT and pre-test alignment information for conformance testing of machine learning-based beam management by the test device; sending the configuration and pre-test alignment information from the test device to the DUT; and performing conformance testing of the machine learning-based beam management use case for the DUT by the test device based on the configuration and pre-test alignment information.

[0010] In another example, an apparatus includes components for performing the following operations: receiving, for a test device used to test a device under test (DUT) for conformance testing of a machine learning-based beam management use case, an instruction that the DUT supports conformance testing of machine learning-based beam management utilizing at least a combination of test probes; adjusting the configuration for the DUT and pre-test alignment information for conformance testing of machine learning-based beam management by the test device; sending the configuration and pre-test alignment information from the test device to the DUT; and performing conformance testing of the machine learning-based beam management use case for the DUT by the test device based on the configuration and pre-test alignment information.

[0011] In the example, a method is disclosed, comprising: for a device under test (DUT) used for conformance testing of a machine learning-based beam management use case, sending an indication to a test device that the DUT supports conformance testing of machine learning-based beam management utilizing at least a combination of test probes; the DUT receiving an indication from the test device of configuration and alignment information; the DUT configuring information to be used for conformance testing based on the configuration and alignment information; and the DUT performing conformance testing of the machine learning-based beam management use case based on the configuration and alignment information.

[0012] Additional examples include a computer program that includes instructions for performing the methods described in the preceding paragraph when the computer program is run on the device. The computer program according to this paragraph is a computer program product that includes a computer-readable medium carrying the instructions for the device embodied therein. Another example is a computer program according to this paragraph that can be directly loaded into the internal memory of the device.

[0013] An example of an apparatus includes one or more processors and one or more memories storing instructions that, when executed by the one or more processors, cause the apparatus to perform at least the following: for a device under test (DUT) used for conformance testing of a machine learning-based beam management use case, sending an instruction to a test device that the DUT supports conformance testing of machine learning-based beam management utilizing at least a combination of test probes; receiving an instruction from the test device for the DUT to receive configuration and alignment information; configuring information to be used for conformance testing by the DUT based on the configuration and alignment information; and performing conformance testing of the machine learning-based beam management use case by the DUT based on the configuration and alignment information.

[0014] An example of a computer program product includes a computer-readable storage medium carrying instructions that, when executed by a device, cause the device to perform at least the following operations: for a device under test (DUT) used for conformance testing of a machine learning-based beam management use case, sending an instruction to a test device indicating that the DUT supports conformance testing of machine learning-based beam management utilizing at least a combination of test probes; receiving configuration and alignment information from the test device by the DUT; configuring information to be used for conformance testing by the DUT based on the configuration and alignment information; and performing conformance testing of the machine learning-based beam management use case by the DUT based on the configuration and alignment information.

[0015] In another example, an apparatus includes components for performing the following operations: for a device under test (DUT) used for conformance testing of a machine learning-based beam management use case, sending an instruction to a test device that the DUT supports conformance testing of machine learning-based beam management utilizing at least a combination of test probes; receiving an instruction from the test device for configuration and alignment information; configuring information to be used for conformance testing based on the configuration and alignment information; and performing conformance testing of the machine learning-based beam management use case based on the configuration and alignment information. Attached Figure Description

[0016] The accompanying drawings use reference numerals, where the same reference numeral can always be used to refer to similar parts, but parts with the same reference numerals may differ in operation and composition. In the accompanying drawings:

[0017] Figure 1 This is a block diagram illustrating the AI / ML model implemented by the UE for beam management;

[0018] Figure 2 The layout of a 3D MPAC system for NR FR2 MIMO OTA testing is shown.

[0019] Figure 3 Example 1 shows a simulation of antenna mode "1" or beam "1" using 3 test probes;

[0020] Figure 4 Example 2 shows a simulation of antenna mode 2 or beam "2" using 5 test probes;

[0021] Figure 5 This is an example of the signal flow used for BM conformance testing;

[0022] Figure 5A It is used to determine in Figure 5 The logic flowchart of the multiple modes used in block 455; and

[0023] Figure 6 This is a block diagram showing the 3D MPAC system layout for NR FR2 MIMO OTA testing and the corresponding possible circuitry for the system and methods described herein. Detailed Implementation

[0024] The abbreviations that may appear in the specification and / or drawings are defined at the end of the detailed description section below.

[0025] The word "exemplary" is used herein to mean "serving as an example, instance, or illustration." Any embodiment described herein as "exemplary" is not necessarily to be construed as preferred or advantageous over other embodiments. All embodiments described in this detailed description are exemplary embodiments provided to enable those skilled in the art to make or use these examples.

[0026] When more than one reference numeral, word, or acronym is used with " / " in this specification, and as is generally used in this specification, " / " can be interpreted as "or," "and," or "both." As used herein, "at least one of the following: " and "at least one of " and similar wording, where the list of two or more elements is connected by "and" or "or," means at least any one of these elements, or at least any two or more of these elements, or at least all of the elements.

[0027] As used herein, the singular forms “a,” “an,” and “the” are also intended to include the plural forms, unless the context clearly indicates otherwise. It will be further understood that the terms “comprising,” “including,” “having,” “having,” “containing,” and / or “consisting of” as used herein specify the presence of the stated features, elements, and / or components, but do not preclude the presence or addition of one or more other features, elements, components, and / or combinations thereof.

[0028] Note that uppercase and lowercase words or phrases are considered the same in this article. For example, the words Slice, slice, and SLICE are the same, as are the phrases Network Repository Function, network repository function, and NETWORK REPOSITORY FUNCTION.

[0029] Any flowchart or signaling diagram herein is considered a logic flowchart and illustrates the operation of an exemplary method, the result of execution of computer program instructions embodied in a computer-readable storage medium, and / or the function executed by logic implemented in a circuit. For methods, flowcharts, and signaling diagrams, the order of method steps, boxes, or signaling within the flow is not critical but merely illustrative.

[0030] Now, technical context is provided for the technical fields relevant to understanding the example. This is provided as a brief overview of the potentially relevant technical fields.

[0031] A related topic is the 3GPP (3rd Generation Partnership Project) background on beam management features based on AI / ML (Artificial Intelligence / Machine Learning). A Rel-18 (Revision 18) study on Artificial Intelligence (AI) / Machine Learning (ML) for the NR (New Radio) air interface (FS_NR_AIML_Air) was completed at the RAN4#109 meeting in December 2023, and the findings regarding protocols and outstanding issues were documented in 3GPP TR 38.843 (Revision 18.0.0).

[0032] Furthermore, at the RAN#102 meeting in December 2023, a new version 19 work item (WI) on Artificial Intelligence (AI) / Machine Learning (ML) for the NR Air Interface (NR_AIML_Air) was approved to initiate standardization work on a general AI / ML framework for the air interface and enable use cases recommended in previous studies. To support AI / ML-enabled radio interfaces for subsequent cellular systems, 3GPP is currently working on the standardization phase of Rel-19 work item (WI description RP-234039), a continuation of Rel-18 research project (SI RP-213599). AI / ML-based beam management use cases target spatial and / or temporal beam prediction for overhead and reported latency reduction.

[0033] In AI / ML beam management, measurements from multiple beams (referred to as set B) are used as inputs to the UE (User Equipment)-side ML model, and the output of the same UE-side ML model is represented by one of the beams in set A. The measured beams (e.g., set B) can be configured with CSI (Channel State Information) reports. The output of the AI / ML model (e.g., a function) is represented by the beams in set A.

[0034] Go to Figure 1This is a block diagram illustrating the AI / ML model 21 implemented by UE 10 for beam management. The input to AI / ML model 21 is the measurement 14 of beams in set B, and the output of AI / ML model 21 is represented as the first 1 or first K beams 19 from set A, where K > 1. More specifically, set B is the set of beams whose measurements are taken as the input to the AI / ML model at the UE. Set A is, for example, the set of all possible DL beams used for DL ​​(downlink) beam prediction, i.e., the beam predicted by the AI / ML model on the UE side belongs to set A (the predicted beam should be one of the beams in set A). This means that the output of the AI / ML model / function / configuration is not all beams in set A, but rather the best beams (top 1 or top K) from set A (where "best" is determined by the AI ​​model). The best predicted beam can be the beam with the best predicted RSRP (Reference Signal Received Power) (if RSRP is also predicted), otherwise, according to the AI / ML model / function, the best beam represents the beam with the best baseline RSRP. (If no RSRP is predicted, and only the beam ID (identifier) ​​is predicted). RSRP is a measure of received power, and other measures can be used. Typically, set A is larger than set B. For example, set A can have 64 beams, while set B can have 8 beams. Regarding the relationship between set A and set B, note that set A and set B can be different. For example, in one case, set B beams are SSB (Synchronization Signal / Physical Broadcast Channel Block) beams, while set A beams are CSI-RS beams. Alternatively, in another case, set B can be a subset of set A (e.g., both set A beams and set B beams are CSI-RS beams).

[0035] For ease of reference and clarity, this document uses the term "AI / ML model" to refer to the element performing beam management in the example. Currently, different RAN working groups use different terminology. The term "AI / ML model" means any other term encompassing "AI / ML function" and / or "AI / ML configuration" or performing similar functions, including replacing "AI / ML" with "AI" or "ML," etc. The term "function" refers to a specific configuration of an AI / ML feature (e.g., beam management) based on UE capabilities. A function may be supported by one or more AI / ML models, depending on the implementation. In the latest 3GPP RAN1 discussions, the term "function" has been replaced by "ML configuration."

[0036] Studies supporting AI / ML-enabled radio interfaces for subsequent cellular systems should also identify areas where AI / ML can improve the performance of air interface functionality. The specification impact will be evaluated to improve the overall understanding of what is needed to enable AI / ML technologies for the air interface. Further studies of beam management use cases will also be conducted for spatial and / or temporal beam prediction. Spatial beam prediction (BM-Case 1, where BM = beam management) aims to predict the optimal Tx / Rx (transmit / receive) beam for different spatial locations. Conversely, temporal beam prediction (BM-Case 2) aims to predict the most likely beam to be used at subsequent moments, e.g., beam prediction in the spatial domain (BM-Case 1). Both BM-Case 1 and BM-Case 2 have been approved for use in the specification work of version 19.

[0037] The testability of AI / ML-based mechanisms is one of the highlights of Release 18 SI and the ongoing Release 19 WI. Here is an excerpt from 3GPP TR 38.843 (between the left and right quotes):

[0038] "General requirements and testing frameworks for AI / ML-based performance enhancements primarily focus on..."

[0039] - How to define the requirements and tests used for reasoning

[0040] - Assess the feasibility and necessity of requirements / tests for LCM.

[0041] - Requirements for data collection may need to be defined (especially for training).

[0042] Requirements / tests used for training will not be studied unless the training process is defined. Test design should ensure performance is guaranteed and avoid situations where the UE passes tests but performs poorly in the field.

[0043] The testability of AI / ML-based Business Model (BM) use cases is receiving increasing attention in 3GPP. Several companies raised this testability issue at the RAN4#110 meeting, and the discussion continued at the RAN4#110-bis meeting without reaching a conclusion. Here is the agreement reached on this topic, and it is documented in the WF agreed upon during the RAN4#110 meeting (also between the left and right quotes):

[0044] Question 2-3: Feasibility of test setup for FR2

[0045] We invite the company to provide further analysis regarding what the test setup should achieve in the test environment.

[0046] At the latest RAN4#113 meeting held in November 2024, there was still no clear agreement on the test framework for conformance testing of BM use cases enabled in version 19 with AI / ML. The Path Forward (R4-2420334) documented this outstanding issue through the following agreement, in which RAN4 invited companies to provide further analysis for test setups with single and multiple AoA (Angles of Arrival).

[0047] 2.3.2 Agreements reached at the main meetings:

[0048] We invite the company to provide further analysis for test setups with single AoA and multiple AoA.

[0049] Another related topic is the background on MIMO (Multiple-Input Multiple-Output) and OTA (Over-The-Air) test setups. The testability challenges of complex features such as multiple Rx (Multiple Receivers) and MIMO-OTA have been considered in various studies within 3GPP. These previous studies are reviewed below.

[0050] Multi-Rx test setup (3GPP TR 38.871) is another related topic. In TR 38.871, the objectives of FR2-1 OTA testing for UEs with multi-panel receivers and layers are as follows:

[0051] "For devices that can receive from multiple angles of arrival (AoA) simultaneously, define test methods for RF / RRM / demodulation requirement testing."

[0052] - Multiple AoA test setups should enable testing of up to two DL layers, each with dual polarization for each angle.

[0053] - For RRM, the goal should be to allow testing of 4 AoA, of which 2 are active simultaneously.

[0054] - Define test methods for demodulation testing of up to 4 DL MIMO layers.

[0055] - The form factor of the smartphone should be the first priority, and other UE types should be discussed as a second priority.

[0056] -Develop / adopt preliminary uncertainty assessments for the testing methodology.

[0057] The testing will take into account system reuse, system complexity, and testing time to keep the overall testing cost within a reasonable range.

[0058] MIMO OTA test setup (3GPP TR 38.827) is another topic. As discussed in 3GPP TR 38.827, the 3D MPAC (Multi-Probe Anechoic Chamber) test method is referenced for FR2 NR MIMO OTA testing. This is achieved by arranging an antenna array around the device under test (EUT) to simulate the spatial distribution of the angle of arrival in a 3D (three-dimensional) MPAC system, thus exposing the EUT to a near-field environment that appears to originate from a complex multipath far-field environment.

[0059] Signals propagate from the base station / communication tester to the EUT through a simulated multipath environment known as the spatial channel model, where appropriate channel impairments (such as Doppler and fading) are applied / imposed on each path before all directional signals are simultaneously injected into the chamber via the probe array. The resulting field distribution in the test area is then integrated by the EUT antenna and processed by the receiver, just as a receiver would in any non-simulated multipath environment. For NR FR2 MIMO OTA testing, the following 3D MPAC system is permitted, where six dual-polarized probes are placed on a sector with a minimum radius of 0.75m from the center of the test area.

[0060] Figure 2 A 3D MPAC system layout for NR FR2 MIMO OTA testing is shown. The SS (System Simulator) 100 includes a base station simulator 110, a channel simulator 115, and a radio head unit 120. The base station simulator 110 sends information to the channel simulator 115, which outputs to radio heads 120-1 through 120-6, which are coupled 125 to probes 210-1 through 210-6 respectively. The channel simulator 115 modifies the power level of the radio heads 120 and can also modify Doppler and fading. A test area 130 with a diameter of 20 cm is provided, comprising sectors 135, each sector having six test probes 210 at a fixed minimum radius of 0.75 cm from the center (center) of the test area. Other SS implementations are not excluded. The DUT (Device Under Test) 145 (such as UE 10) will be placed in the test area 130 (e.g., its center). SS can also be referred to as TE (Test Equipment). Note that the terms EUT and DUT are considered to be the same, and UE is considered to be either the EUT or DUT in the examples presented herein.

[0061] One of the active Release 19 topics currently being discussed at the RAN4 working group meeting is a test framework for AI / ML-enabled beam management use cases. The purpose of AI / ML-enabled beam management conformance testing is to verify whether the UE correctly predicts the top 1 or top K beams from set A (e.g., ranked as "best") based on measurements of beams from set B (which will be the input to the UE-side AI / ML model, which, as previously stated, may include functionality and / or configuration). As previously mentioned, the best predicted beam can be the beam with the best predicted RSRP, if the RSRP is also predicted. If no RSRP is predicted, and only the beam ID is predicted, then according to the AI / ML model / functionality, the best beam represents the beam with the best baseline fact RSRP. To test this, the availability of baseline facts at the test equipment is required, which the test equipment will use to determine whether the UE under test passes or fails the test. Since the baseline facts depend on the UE architecture and implementation, each UE under test will need to report the baseline facts and prediction results back to the test equipment.

[0062] Furthermore, in normal testing, the number of test probes used depends on the characteristics. For example, in multi-Rx testing, only two of all available probes are used simultaneously, and the total number of probes is fixed. In another example of MIMO, for CDL testing, all six probes are used. Because the number of probes used is fixed, only one interference mode is received by the UE.

[0063] The size of the test chamber (e.g., an anechoic chamber) depends on the number of beams to be simulated, the maximum size of the device under test (DUT), the placement of test probes, etc. For CSI-RS (Channel State Information-Reference Signal) beams, the number of set A beams equals the total number of CSI-RS that can be configured for the UE. Since a large number of set A beams (e.g., 32 CSI-RS beams) need to be simulated during AI / ML-enabled BM conformance testing to obtain baseline facts for the set A beams, generating such a large number of beams in a fixed-size (e.g., finite-size) test chamber would be challenging and time-consuming, provided UE rotation is allowed / implemented. This would not be an economical solution, as the cost of such a test system would also increase, and its availability would be limited because such an enhanced test system is only needed in limited conformance tests (such as AI / ML-enabled BM use cases).

[0064] This article addresses the following issues:

[0065] 1) How to generate set A and set B beams during conformance testing of AI / ML-enabled BM features without: (1) using UE rotation, (2) using a larger size enhanced test system with many test probes.

[0066] 2) How to resolve the pre-alignment between the TE (Test Equipment) and the DUT / UE before the start of conformance testing.

[0067] To address these and other issues, several techniques are proposed that enable the generation of set A and / or set B beams during conformance testing of AI / ML-enabled BM features without: (1) using UE rotation, and (2) using a larger-sized enhanced test system with numerous test probes. For example, a larger chamber implies a larger chamber volume and a larger quiet zone radius. For instance, the chamber size could be tens of meters. An enhanced test system refers to a test system with a significantly larger number of test probes (e.g., more than 10) and greater complexity compared to existing test systems that typically have a maximum of six probes. The proposed techniques will ensure cost-effective and efficient conformance testing of AI / ML-enabled BM features. An example in this paper also proposes a signaling scheme to perform pre-alignment between the TE and DUT prior to the commencement of conformance testing.

[0068] The main examples in this article include one or more of the following:

[0069] 1) Different sets of A and / or B beams can be simulated using the number of dynamically available test probes in the test chamber. For example, multiple beams can be generated by using, for example, 4, 5, or 6 probes.

[0070] 2) The transmit power of each test probe can be selected / adjusted by combining specific test probe combinations to simulate specific set A beam / set B beam / antenna modes.

[0071] 3) New UE capabilities can be introduced, which are intended for use by the UE to notify the TE that the UE supports conformance testing based on the examples provided in this article.

[0072] 4) New signaling can be introduced, which is intended for use by the UE and TE to align with each other (e.g., pre-alignment) for conformance testing of BM features enabled by AI / ML.

[0073] Because the number of test probes and the selection of test probes from the available test probes (which represent the beam configuration used for testing) can be dynamically chosen, multiple interference modes can be applied to the UE instead of a single interference mode. This better simulates the UE's real-world environment and helps to see how the UE handles different interference modes.

[0074] Having provided a main example as an overview, more details will now be provided. These details involve at least the simulation of multiple set A / set B beams and the proposed signaling.

[0075] The simulation of multiple beam sets A / B is now described.

[0076] Each Tx beam from set A and / or set B can be considered as a specific Tx antenna pattern that produces a unique constructive / destructive interference pattern at the UE receiving antenna. Consider a test chamber (e.g., an anechoic chamber) equipped with “M” test probes (each test probe can be single-polarized or dual-polarized), where each test probe contributes to forming a definite Tx antenna pattern (beam). The formation of the Tx antenna pattern is also affected by the amount of Tx power of each test probe.

[0077] Since the test chamber has "M" test probes, a specific Tx antenna mode or beam can be formed by selecting / activating any "N" test probes from the available "M" probes. This means that if "N" test probes are activated, "P" can be formed. N "A Tx antenna mode, in which P N = M C N (M Choose N), that is,

[0078] If only one probe is selected / activated, it can form M C1 Tx antenna modes.

[0079] If two probes are selected / activated, a formation can be achieved. M C2 Tx antenna modes.

[0080] If you select / activate 3 probes, you can form M C3 Tx antenna modes.

[0081] …

[0082] If "M-1" probes are selected / activated, then a formation can be achieved. M C M-1 One Tx antenna mode.

[0083] If you select / activate "M" probes, you can form M C M One Tx antenna mode.

[0084] Therefore, a test system with "M" test probes in the test chamber can be simulated. M C1 + M C2 + M C3 +…+ M C M-1 + M C M Each Tx antenna radiation mode (or beam).

[0085] For example,

[0086] 1) The actual test system equipped with 6 test probes can simulate up to 6 C1 + 6 C2 + 6 C3 + 6 C4 + 6 C5+ 6 C6 = 6 + 15 + 20 + 15 + 6 + 1 = 63 Tx antenna radiation modes (or beams).

[0087] 2) The actual test system equipped with 8 test probes can simulate up to 8 C1 + 8 C2 + 8 C3 + 8 C4 + 8 C5+ 8 C6 + 8 C7 + 8 C8 = 8 + 28 + 56 + 70 + 56 + 28 + 8 + 1 = 255 Tx antenna radiation modes (or beams).

[0088] This also means (as an example):

[0089] 1) For a test system with 6 test probes, selecting at least 3 test probes will result in 20 + 15 + 6 + 1 = 42 Tx antenna patterns. Note that 20 is achieved by using 3 probes; 15 by using 4 test probes; 6 by using 5 test probes; and 1 by using all 6 test probes.

[0090] 2) For a test system with 8 test probes, selecting at least 4 test probes will result in 70 + 56 + 28 + 8 + 1 = 163 Tx antenna patterns.

[0091] 3) For a test system with 8 test probes, selecting at least 5 test probes will result in 56 + 28 + 8 + 1 = 93 Tx antenna patterns.

[0092] Therefore, using this concept, a test system with at least six test probes can simulate 32 or more Tx beams (or Tx antenna patterns) using at least three simultaneously active probes. Alternatively, a test system with at least eight test probes can simulate 64 or more Tx beams (or Tx antenna patterns) using at least five simultaneously active test probes.

[0093] refer to Figure 3 The accompanying figure is Example 1 illustrating a simulation of antenna pattern "1" or beam "1" using three test probes. The DUT (UE 10) is placed in test chamber 250, and TE 70 and channel simulator 115 are coupled to probes 210-1 to 210-6. In this example, it is assumed that TE 70 includes base station simulator 110 and radio head end 120. Note that the terms TE and SS are generally used interchangeably in this field. For ease of reference and clarity, the term TE is used primarily herein, but this implies the inclusion of SS or any other apparatus used for testing. In Example 1, a unique antenna pattern is simulated using three test probes (probe 1 210-1, probe 2210-2, and probe 6 210-6) by utilizing constructive / destructive interference generated by the multi-antenna / probe signals simultaneously transmitted by probes 1 210-1, 2210-2, and 6 210-6. The following probes are disabled (i.e., not used): Probe 3 210-3, Probe 4 210-4, and Probe 210-5. The resulting beam / antenna pattern (antenna pattern 1 or beam 1 230-1) also depends on P_11 220-1, P_21 220-2, and P_61 220-6, which are the transmit powers of Probe 1 210-1, Probe 2 210-2, and Probe 6 210-6, respectively.

[0094] Regarding the terms "antenna pattern" and "beam," consider the following. Current specifications use the term "antenna pattern," while work projects for AI / ML focus on "beam." An "antenna pattern" describes how an antenna radiates radio waves in all directions and displays signal strength at different angles, while a "beam" specifically refers to the concentrated and focused direction of maximum radiation within that pattern, often considered the antenna's main lobe where most signal energy is directed. In the context of the examples in this paper, "antenna pattern" refers to the combined radiation pattern of all antennas (test probes), which may contain one or more beams. For ease of reference, this paper primarily uses the term "antenna pattern," but there are exceptions, such as set A beams and set B beams, which are the commonly used terms.

[0095] Go to Figure 4The attached figure is Example 2 illustrating a simulation of antenna mode 2 or beam "2" using five test probes. In Example 2, a single antenna mode (antenna mode 2 or beam 2 230-2) is simulated using five test probes (probe 1 210-1, probe 2 220-2, probe 3 210-3, probe 4 210-4, and probe 5 210-5) by utilizing constructive / destructive interference generated by multiple antenna signals simultaneously transmitted by probes 1 210-1, 2 210-2, 3 210-3, 4 210-4, and 5 210-5. The resulting beam / antenna pattern also depends on P_12 220-1, P_22 220-2, P_32 220-3, P_42 220-4, and P_52 220-5, which are the transmit powers of probe 1 420-1, probe 2 420-2, probe 3 420-3, probe 4 420-4, and probe 5 420-5, respectively.

[0096] Now describe the suggested signaling. Use Figure 5 To describe the suggested signaling, Figure 5 This is an example of the signaling flow for BM conformance testing using method 400. Method 400 is used to test the device under test (e.g., UE) using a test equipment (TE) for conformance testing of a machine learning-based beam management use case. For a practical implementation example, some signaling is used, which... Figure 5 The message flow diagram is shown. This signaling occurs between DUT 10 (e.g., UE) and TE 70. DUT 10 will also be referred to as UE 10.

[0097] The main steps (e.g., key steps) shown in the signaling flowchart are described below, while the remaining steps are self-explanatory. The first part 420 of method 400 is used for pre-alignment prior to conformance testing of the AI / ML-enabled beam management use case. This first part pertains to steps 1-6.

[0098] Steps 1 and 2: UE 10 notifies TE 70 that the UE supports AI / ML-enabled BM conformance testing with dynamic selection and combination of test probes (e.g., through capabilities supported by the UE). Upon receiving this information, TE 70 adjusts the transmit (Tx) parameters and / or conformance test parameters, for example, as pre-test alignment information, and determines the configuration of DUT 10 for conformance testing.

[0099] Step 3: TE 70 sends configuration and pre-test alignment information (e.g., parameters related to the codebook of the dynamic test probe beam / antenna mode, parameters for adjusting the UE-side model (e.g., functions) for conformance testing, etc.) to DUT 10, and DUT uses the pre-test alignment information provided by TE 70 to configure itself (e.g., align its model, such as functions, parameters, etc.).

[0100] Step 4: The UE uses the configuration and pre-test alignment information to configure itself. This may include using the configuration and pre-test alignment information to align its model (e.g., and the corresponding functions and / or parameters).

[0101] Steps 5 and 6: UE 10 sends a pre-test alignment report and / or other relevant alignment information to TE 70. TE 70 can use this information to ensure that the training parameters of the UE model and the test environment / parameters are aligned with each other. TE 70 performs this alignment check confirmation.

[0102] Regarding steps 5 and 6, a possible example is that the TE 70 sends parameters related to the beam codebook used for the conformance test to the UE (e.g., step 3). The UE then checks whether these parameters (provided by the TE) are aligned with its own parameters (used to train its model) (step 4), and / or the UE attempts to align its parameters with pre-alignment information provided by the TE (step 4). If the TE's parameters and the UE's training parameters are aligned, the UE can send the same positive confirmation to the TE (step 5). If they are not aligned, the UE can send a negative confirmation and / or some information about the UE-side parameters (e.g., its preferred parameters) to the TE (also step 5). Afterward, the TE performs a check / alignment / confirmation using the confirmation and / or other information provided by the UE (in step 6). The conformance test begins after confirming that the test parameters between the UE sides (e.g., parameters for the AI / ML model / function and related parameters) are aligned with those provided by the TE.

[0103] Regarding the relevant parameters, in one example, this refers to the parameters required by the UE to configure its AI / ML model / function / configuration for conformance testing. In another example, if the UE has implemented a fully explicit model / function for conformance testing based on a test dataset (collected by the UE, provided by the TE vendor, or specified in the 3GPP specification), then "relevant parameters" could refer to the activation, deactivation, configuration, etc. of the UE model used for conformance testing.

[0104] Part 2, 430, is the conformance test section 1, involving the simulation and baseline fact reporting of set A beams. This involves steps 7 through 10 in loop 431. Loop 431 can be considered as a beam scan of (all) set A beams. Part 2, 430, is used for set A beams (all set A beams) and is used for baseline fact extraction by the DUT and baseline fact reporting from the DUT to the TE. This is necessary because the UE-side AI / ML model predicts the best "K" beams in set A, and baseline fact values ​​corresponding to each set A beam are needed to verify the quality or correctness of these predictions.

[0105] Step 7: Once pre-alignment is performed (e.g., first part 420), TE 70 generates a unique antenna pattern or beam (e.g., from set A codebook) by selecting a unique combination of power levels (i.e., P_1j, P_2j, P_3j, ...) for each test probe as suggested, where “j” indicates the beam index.

[0106] Steps 8-10: The UE measures the RSRP (Reference Signal Received Power) of the set A beam transmitted in step 8 as a possible received power metric and reports the measured RSRP (Baseline Fact) to the UE in step 9. This reporting can be performed after each set A beam measurement, or alternatively after measurements of several or all set A beams. The TE 70 stores the reported RSRP value (in step 10) along with the corresponding beam ID (identifier) ​​and power level value for future set B inference. Loop 431 ends when the set A beam scan is complete.

[0107] Part 3, 440, relates to the conformity test section 2, which involves the simulation and interferometry report of the set B beams. Part 3, 440 relates to steps 11 and 12 in loop 441. Loop 441 can be considered as a beam scan of (all) set B beams. Loop 441 ends when all set B beams have been scanned.

[0108] Step 11: For the simulation and interferometric report of set B beams, the TE 70 generates an antenna pattern or beam (from the set B codebook) by selecting a unique power level combination (i.e., P_1l, P_2l, P_3l, ...) for each test probe from the beams generated in conformity test section 1 (section 430), where "l" indicates the beam index. Note: Set A and set B can be generated in reverse order; that is, set B beams can be generated first, and then set A beams can be generated.

[0109] Steps 12-16: The UE measures the RSRP of the beams in set B transmitted in step 12. Once the UE has completed measuring the RSRP of each beam in set B, the UE provides the measurement as input to its AI / ML model for prediction from the best or top K beams in set A. See step 13. The UE predicts (step 14) the best or top K beams and indicates to the TE the characteristics of the predicted top one or more beams (e.g., RSRP and / or beam ID, etc.). The TE verifies (step 15) whether the top one or more beams predicted by the UE are the same as or sufficiently close to the actual top one or more beams (e.g., satisfying one or more thresholds, such as one or more criteria that may be one of them by satisfying one or more thresholds), and declares a test iteration decision (pass / fail). In other words, the top 1 or top K beams may satisfy criteria or a set of criteria, for example, the top 1 beam should be beam X; or the top 1 beam should be beam X or beam Y; or the top 1 beam should be the beam from beam X1 to X2. Similarly, but possibly more complex, the criteria could be a set used for the first K beams. See step 16. Note that the reported RSRP values ​​stored in step 10 will be used for verification in step 15.

[0110] In operation 450, the inference phase is completed, and the test results / decision are declared.

[0111] Steps 7 and 11 can be performed at least in part by using operation 455. In operation 455, TE 70 selects one of several antenna modes (e.g., beam) determined using the probe set and corresponding power levels to form the desired number of beams. This is done by referring to... Figure 5A To provide the best explanation, Figure 5A It is used to determine in Figure 5 The flowchart illustrates the logic of the multiple modes used in block 455. Flowchart 460 is used to select the multiple modes used in block 455. Set A antenna modes are used in part for illustration. Figure 5A For beam set A, there are 32 antenna modes. As previously mentioned, individual antenna modes form one or more beams, and for ease of reference, the term "antenna mode" is used. Antenna modes for beam set B can be selected similarly.

[0112] In operation 465, the quantity is set for the set of antenna patterns to be used. This example uses 32 antenna patterns for set A beams. In operation 470, a first quantity (e.g., 3) of all (e.g., 6) probes 210 is selected, and it is determined how many antenna patterns can be generated (e.g., 20). Reference 490 shows various probe combinations 491-1 and corresponding antenna patterns / beams 492-1. In this case, using 3 of the 6 probes results in 20 possible probe combinations 491 and corresponding beams 492. In operation 475, some or all of these antenna patterns are selected. In this example, all 20 antenna patterns are selected, meaning 12 patterns remain.

[0113] In operation 480, a second number (e.g., 4) of all (e.g., 6) test probes 210 are selected, and additional antenna patterns are selected to complete all (e.g., set A of 32) antenna patterns. Reference 495 shows 15 probe combinations 491-2 that can be used and corresponding antenna patterns / beams 492-2. Of the 15 probe combinations 491-2, 12 will be selected to add 12 beams 492-2 to complete the required 32 antenna beams.

[0114] Note that this only uses two probes, 3 or 4. However, many other options exist. If you want to generate (e.g., simulate) 32 ensemble A-beams during conformance testing, there are several ways to choose:

[0115] Option 1: Use all 20 antenna modes with 3 probes and use 12 antenna modes with 4 probes.

[0116] Option 2: 17 antenna modes via 3 probes, and all 15 antenna modes via 4 probes.

[0117] Option 3: Use 1 antenna mode with 6 probes, 6 antenna modes with 5 probes, 15 antenna modes with 4 probes, and 10 antenna modes with 3 probes.

[0118] It can have many other options.

[0119] Operation 485 instructs process 490 to be executed dynamically (meaning that when executing...). Figure 5 (in signaling), or statically (e.g., using in execution) Figure 5 The signaling diagram is previously fixed in a table. The first option (dynamic selection) can be a high-level option, but to achieve this, the UE vendor should train its model for all such possible combinations. For the latter option (using a table to generate, etc.), the complexity can be reduced.

[0120] Multiple mode sets can also be used to test the DUT 10, such as using different probe combination sets 491 to test multiple sets. As a simple example, consider using 32 beams (20 beams from all 20 probe combinations 491-1 and the remaining 12 beams from 21-32 from probe combination 491-2) for a single pass. Figure 5 Method 400; and then 32 beams (17 beams from 17 probe combinations 491-1 and all 15 probe combinations 491-2) were used for another pass. Figure 5 Method 400.

[0121] Figure 6 This is a block diagram illustrating the 3D MPAC system layout for NR FR2 (as a device under test, DUT) MIMO OTA testing and the corresponding possible circuitry for the system and methods described herein. Figure 6 In this example, UE 10 is in OTA test chamber 250 and uses FR2 for communication. Test probe 210 generates probe signal 610 used to form antenna pattern or beam 230. TE 70 outputs to channel emulator 115, the output of which is coupled to test probe 210. In this example, TE 70 may include... Figure 2 The base station simulator 110 and the radio head terminal 120.

[0122] UE 10 includes one or more processors 13, one or more memories 15, and other circuitry 16. The other circuitry 16 includes one or more (wired and / or wireless) receivers (Rx) 17 and one or more (wired and / or wireless) transmitters (Tx) 18. Instructions 12 (e.g., as a program) are used to cause UE 10 to perform the operations described herein. For UE 10, the other circuitry 16 may include circuitry such as for user interface elements (not shown) (e.g., a display). Instructions 12 may be implemented via a program stored in memory 15 and executed by processor 13 after being retrieved from memory 15; or implemented by circuitry as part of a processor or other circuitry element; or both.

[0123] TE 70 includes one or more processors 73, one or more memories 75, and other circuitry 76. The other circuitry 76 includes one or more receivers (Rx) 77 and one or more transmitters (Tx) 78. Instructions 72 are used to cause TE 70 to perform the operations described herein. Instructions 72 may be implemented via a program stored in memory 75 and executed by processor 73 after being retrieved from memory 75; or implemented by circuitry as part of a processor or other circuit element; or both.

[0124] Instructions 12 and 72 can be implemented by (different) programs stored in one or more corresponding memories 15 or 75. When these instructions are retrieved and executed by one or more corresponding processors 13 or 73, they cause the corresponding device 10 or 70 to perform the operations described herein. Computer program products 94-1 and 94-2 are shown. A corresponding computer-readable medium 94 contains instructions that, when downloaded and installed into instructions 12 or 72 and / or memories 15 or 75 of the corresponding UE 10 or TE 70 and executed by processor 13 or 73, cause the corresponding device to perform the corresponding actions described herein. The computer-readable medium 94 can be implemented for downloading using an optical disc or memory stick or any other physical storage device, such as memory in a data network.

[0125] Computer-readable storage devices 15 and 75 are circuits and can be of any type suitable for the local technical environment (e.g., storage devices 15 and 75 can be completely different), and can be implemented using any suitable data storage technology, such as semiconductor-based memory devices, flash memory, firmware, magnetic storage devices and systems, optical storage devices and systems, fixed memory, and removable memory. Processors 13 and 73 are circuits and can be of any type suitable for the local technical environment (e.g., processors 13 and 73 can be completely different). For example, as a non-limiting example, these processors may include one or more of a general-purpose computer, a special-purpose computer, a microprocessor, a digital signal processor (DSP), a processor based on a multi-core processor architecture, and may also include special-purpose circuits such as field-programmable gate arrays (FPGAs), application-specific integrated circuits (ASICs), signal processing devices, and other devices or combinations thereof. Processors 13 and 73 are circuits that can be programmed to perform functions via software, firmware, etc. (including microcode), but not only software.

[0126] Receivers 17 and 77, and transmitters 18 and 78, can implement a wireless interface. These receivers and transmitters can be combined together as a transceiver.

[0127] Typically, various embodiments of user equipment 10 may include, but are not limited to, devices implementing cellular technologies (such as smartphones, mobile phones, cellular phones, Voice over Internet Protocol (VoIP) phones, and / or wireless local loop phones), tablet computers, portable computers, vehicles or in-vehicle devices for wireless V2X (vehicle-to-everything) communication, image capture devices (such as digital cameras, gaming devices, music storage and playback devices), internet devices (including Internet of Things (IoT) devices), IoT devices with sensors and / or actuators for applications such as automation, and portable units or terminals combining these functions, laptop embedded devices (LEE), laptop mounted devices (LME), Universal Serial Bus (USB) dongles, smart devices, wireless client devices (CPE), Internet of Things (IoT) devices, watches or other wearable devices, head-mounted displays (HMDs), vehicles, drones, medical devices and applications (e.g., remote surgery), industrial devices and applications (e.g., robots and / or other wireless devices operating in the context of industrial and / or automated processing chains), consumer electronic devices, devices operating on commercial and / or industrial wireless networks, etc. In other words, UE 10 can be any terminal device capable of wireless communication. By way of example and not limitation, UE may also be referred to as a communication device, terminal device (MT), subscriber station (SS), portable subscriber station, mobile station (MS), or access terminal (AT).

[0128] Without limiting the scope, interpretation, or application of the claims that follow in any way, the technical effects and / or advantages of one or more example embodiments disclosed herein lie in providing a selection of the number of dynamic test probes to simulate set A or set B beams, which provides multiple interference modes applied to the UE rather than a single interference mode. This better simulates the UE's real-world environment and helps to see how the UE handles different interference modes. Another technical effect and / or advantage of one or more example embodiments disclosed herein is providing the UE with the capability intended for use by the UE to notify the TE that the UE supports conformance testing based on the examples provided herein. Another technical effect and / or advantage of one or more example embodiments disclosed herein is that signaling can be introduced, which is intended for use by the UE and TE to align (e.g., pre-align) parameters of each other for conformance testing of BM features enabling AI / ML.

[0129] The following are additional examples.

[0130] Example 1. A method comprising: for a test device used to test a device under test (DUT) for a conformance test of a machine learning-based beam management use case, receiving an indication that the DUT supports a conformance test of machine learning-based beam management utilizing at least a combination of test probes; adjusting a configuration for the DUT and pre-test alignment information for the conformance test of the machine learning-based beam management by the test device; sending the configuration and the pre-test alignment information from the test device to the DUT; and performing the conformance test of the machine learning-based beam management use case for the DUT by the test device based on the configuration and the pre-test alignment information.

[0131] Example 2. The method according to Example 1 further includes: prior to the execution: receiving, by the test device, one or both of a report or alignment information from the device under test; and aligning, based on, the parameters of the model or function or both of the model and function used by the device under test for the conformance test with the parameters provided by the test device.

[0132] Example 3. The method according to any one of Examples 1 or 2, wherein the pre-test alignment information includes one or more of the following: parameters related to the codebook of the dynamic test probe beam or antenna pattern used for the conformance test; or parameters for adjusting the model or function or both the model and function used by the device under test for the conformance test.

[0133] Example 4. The method according to any one of Examples 1 to 3, wherein the conformance test is performed by the test equipment after the test equipment confirms that one or more test parameters used by the device under test are aligned with one or more corresponding test parameters provided by the test equipment.

[0134] Example 5. The method according to Example 4, wherein the one or more test parameters include parameters for a machine learning model or function, or both a model and a function.

[0135] Example 6. The method according to any one of Examples 1 to 5, wherein performing the conformance test comprises: generating a unique transmit antenna pattern or beam to be sent to the device under test by: selecting a unique power level combination for one or more test probes for transmitting the unique transmit antenna pattern or beam to the device under test; receiving from the device under test an indication of a metric of received power corresponding to the unique transmit antenna pattern or beam by the test device; and storing by the test device information including the indication of the metric and an indication of the beam identifier corresponding to the unique transmit antenna pattern or beam and the unique power level combination.

[0136] Example 7. The method according to Example 6, wherein the generation, receiving, and storing are performed for multiple unique transmit antenna modes or beams.

[0137] Example 8. The method according to Example 6 or 7, wherein: the stored unique transmit antenna pattern or beam is for a first beam set; and the method further includes: the test device using the stored information to generate a transmit antenna pattern or beam from a second beam set by: selecting a unique power level combination for a plurality of test probes used to form the transmit antenna pattern to transmit at least one beam to the device under test.

[0138] Example 9. The method according to Example 8 further includes: receiving from the device under test an indication of one or more beams from the second beam set that satisfy criteria at the device under test; verifying, when comparing the one or more beams from the second beam set with one or more beams from the first beam set, whether the indicated one or more beams from the second beam set that satisfy criteria at the device under test satisfy one or more thresholds; determining, based on the verification, whether the result of the conformance test is pass or fail; and outputting the pass or fail indication.

[0139] Example 10. The method according to any one of Examples 6 to 9, wherein the indication of the received power metric includes an indication of the received power of a reference signal.

[0140] Example 11. The method according to any one of Examples 1 to 10, wherein the device under test is a user equipment.

[0141] Example 12. A method comprising: for a device under test (DUT) used for conformance testing of a machine learning-based beam management use case, sending an indication to a test device that the DUT supports conformance testing of machine learning-based beam management utilizing at least a combination of test probes; the DUT receiving an indication from the test device of configuration and alignment information; the DUT configuring information to be used for the conformance testing based on the configuration and alignment information; and the DUT performing the conformance testing of the machine learning-based beam management use case based on the configuration and alignment information.

[0142] Example 13. The method according to Example 12 further includes: prior to the execution: sending one or both of a report or alignment information from the device under test to the test device for alignment by the test device.

[0143] Example 14. The method according to Example 13, wherein the alignment information from the device under test includes one or more parameters of a model or function or both a model and a function used by the device under test.

[0144] Example 15. The method according to any one of Examples 12 to 14, wherein the alignment information from the test device includes one or more of the following: parameters related to the codebook of the dynamic test probe beam or antenna pattern used for the conformance test; or parameters for adjusting the model used by the device under test for the conformance test.

[0145] Example 16. The method according to any one of Examples 12 to 15, wherein performing the conformance test comprises: receiving a unique transmit antenna pattern or beam transmitted from the test equipment, the unique transmit antenna pattern or beam having a unique power level combination for one or more test probes used to transmit the unique transmit antenna pattern or beam; and the device under test transmitting to the test equipment an indication of a metric of the received power corresponding to the unique transmit antenna pattern or beam.

[0146] Example 17. The method according to Example 16, wherein the receiving and transmitting are performed for a plurality of unique transmit antenna modes or beams.

[0147] Example 18. The method according to Example 16 or 17, wherein the method further comprises: the device under test receiving from the test equipment a transmit antenna pattern having at least one beam from a second beam set, the transmit antenna pattern having a unique combination of power levels for a plurality of test probes used to transmit the transmit antenna pattern or beam.

[0148] Example 19. The method according to Example 18 further includes: the device under test sending to the test device an indication of one or more beams from the second beam set that satisfy the criteria at the device under test.

[0149] Example 20. The method according to any one of Examples 16 to 19, wherein the indication of the received power metric includes an indication of the received power of a reference signal.

[0150] Example 21. An apparatus comprising components for performing the following operations: for a test device used to test a device under test (DUT) for a conformance test of a machine learning-based beam management use case, receiving an indication that the DUT supports a conformance test of machine learning-based beam management utilizing at least a combination of test probes; adjusting a configuration for the DUT and pre-test alignment information for the conformance test of the machine learning-based beam management by the test device; sending an indication from the test device to the DUT of the configuration and the pre-test alignment information; and performing the conformance test of the machine learning-based beam management use case for the DUT by the test device based on the configuration and the pre-test alignment information.

[0151] Example 22. The apparatus according to Example 21, wherein the component is further configured to: receive, prior to the execution: one or both of a report or alignment information from the device under test by the test device; and align, based on the one or both of the report or the alignment information, the parameters of a model or function or both of a model and function used by the device under test for the conformance test with the parameters provided by the test device.

[0152] Example 23. An apparatus according to any one of Examples 21 or 22, wherein the pre-test alignment information includes one or more of the following: parameters related to the codebook of the dynamic test probe beam or antenna pattern used for the conformance test; or parameters for adjusting the model or function, or both the model and function, used by the device under test for the conformance test.

[0153] Example 24. The apparatus according to any one of Examples 21 to 23, wherein the conformance test is performed by the test equipment after the test equipment confirms that one or more test parameters used by the device under test are aligned with one or more corresponding test parameters provided by the test equipment.

[0154] Example 25. The apparatus according to Example 24, wherein the one or more test parameters include parameters for a machine learning model or function, or both a model and a function.

[0155] Example 26. An apparatus according to any one of Examples 21 to 25, wherein performing the conformance test comprises: generating a unique transmit antenna pattern or beam to be sent to the device under test by: selecting a unique power level combination for one or more test probes for transmitting the unique transmit antenna pattern or beam to the device under test; receiving from the device under test an indication of a metric of received power corresponding to the unique transmit antenna pattern or beam by the test device; and storing by the test device information including the indication of the metric and an indication of the beam identifier corresponding to the unique transmit antenna pattern or beam and the unique power level combination.

[0156] Example 27. The apparatus according to Example 26, wherein the generation, receiving, and storing are performed for a plurality of unique transmit antenna patterns or beams.

[0157] Example 28. The apparatus according to Example 26 or 27, wherein: the stored unique transmit antenna pattern or beam is for a first beam set; and the component is further configured to: generate a transmit antenna pattern or beam from a second beam set by the test device using the stored information by: selecting a unique power level combination for a plurality of test probes used to form the transmit antenna pattern to transmit at least one beam to the device under test.

[0158] Example 29. The apparatus according to Example 28, wherein the component is further configured to: receive from the device under test an indication of one or more beams from the second beam set that satisfy criteria at the device under test; verify, when comparing the one or more beams from the second beam set with one or more beams from the first beam set, whether the indicated one or more beams from the second beam set that satisfy criteria at the device under test satisfy one or more thresholds; based on the verification, determine whether the result of the conformance test is pass or fail; and output the pass or fail indication.

[0159] Example 30. The apparatus according to any one of Examples 26 to 29, wherein the indication of the measurement of the received power includes an indication of the received power of a reference signal.

[0160] Example 31. The apparatus according to any one of Examples 21 to 30, wherein the device under test is a user equipment.

[0161] Example 32. An apparatus comprising components for performing the following operations: for a device under test (DUT) for a conformance test of a machine learning-based beam management use case, sending an indication to a test device that the DUT supports a conformance test of machine learning-based beam management utilizing at least a combination of test probes; receiving an indication from the test device by the DUT of configuration and alignment information; configuring information to be used in the conformance test by the DUT based on the configuration and alignment information; and performing the conformance test of the machine learning-based beam management use case by the DUT based on the configuration and alignment information.

[0162] Example 33. The apparatus according to Example 32, wherein the component is further configured to: prior to the execution: send one or both of a report or alignment information from the device under test to the test device for alignment by the test device.

[0163] Example 34. The apparatus according to Example 33, wherein the alignment information from the device under test includes one or more parameters of a model or function, or both a model and a function, used by the device under test.

[0164] Example 35. An apparatus according to any one of Examples 32 to 34, wherein the alignment information from the test equipment includes one or more of the following: parameters related to the codebook of the dynamic test probe beam or antenna pattern used for the conformance test; or parameters for adjusting the model used by the device under test for the conformance test.

[0165] Example 36. An apparatus according to any one of Examples 32 to 35, wherein performing the conformance test comprises: receiving a unique transmit antenna pattern or beam transmitted from the test equipment, the unique transmit antenna pattern or beam having a unique power level combination for one or more test probes used to transmit the unique transmit antenna pattern or beam; and the device under test transmitting to the test equipment an indication of a metric of the received power corresponding to the unique transmit antenna pattern or beam.

[0166] Example 37. The apparatus according to Example 36, wherein the receiving and transmitting are performed for a plurality of unique transmit antenna patterns or beams.

[0167] Example 38. An apparatus according to Example 36 or 37, wherein the component is further configured to: receive, by the device under test, a transmit antenna pattern having at least one beam from a second beam set from the test equipment, the transmit antenna pattern having a unique combination of power levels for a plurality of test probes used to transmit the transmit antenna pattern or beam.

[0168] Example 39. The apparatus according to Example 38, wherein the component is further configured to: send from the device under test to the test device an indication of one or more beams from the second beam set that satisfy the criteria at the device under test.

[0169] Example 40. The apparatus according to any one of Examples 36 to 39, wherein the indication of the received power measurement includes an indication of the received power of a reference signal.

[0170] Example 41. An apparatus comprising: one or more processors; and one or more memories storing instructions, which, when executed by the one or more processors, cause the apparatus to perform at least: receiving, for a test device used to test a device under test for a conformance test of a machine learning-based beam management use case, an indication that the device under test supports a conformance test of machine learning-based beam management utilizing at least a combination of test probes; adjusting by the test device a configuration for the device under test and pre-test alignment information for the conformance test of the machine learning-based beam management; sending from the test device to the device under test an indication of the configuration and the pre-test alignment information; and performing by the test device, based on the configuration and the pre-test alignment information, the conformance test of the machine learning-based beam management use case for the device under test.

[0171] Example 42. The apparatus according to Example 41, wherein the one or more memories further store instructions that, when executed by the one or more processors, cause the apparatus to perform at least the following actions prior to the execution: receiving, by the test device, one or both of a report or alignment information from the device under test; and aligning, based on, the parameters of a model or function, or both of a model and function, used by the device under test for the conformance test with parameters provided by the test device.

[0172] Example 43. An apparatus according to any one of Examples 41 or 42, wherein the pre-test alignment information includes one or more of the following: parameters related to the codebook of the dynamic test probe beam or antenna pattern used for the conformance test; or parameters for adjusting the model or function, or both the model and function, used by the device under test for the conformance test.

[0173] Example 44. The method according to any one of Examples 41 to 43, wherein the conformance test is performed by the test equipment after the test equipment confirms that one or more test parameters used by the device under test are aligned with one or more corresponding test parameters provided by the test equipment.

[0174] Example 45. The apparatus according to Example 44, wherein the one or more test parameters include parameters for a machine learning model or function, or both a model and a function.

[0175] Example 46. An apparatus according to any one of Examples 41 to 45, wherein performing the conformance test comprises: generating a unique transmit antenna pattern or beam to be sent to the device under test by: selecting a unique power level combination for one or more test probes for transmitting the unique transmit antenna pattern or beam to the device under test; receiving from the device under test an indication of a metric of received power corresponding to the unique transmit antenna pattern or beam by the test device; and storing by the test device information including the indication of the metric and an indication of the beam identifier corresponding to the unique transmit antenna pattern or beam and the unique power level combination.

[0176] Example 47. The apparatus according to Example 46, wherein the generation, receiving, and storing are performed for a plurality of unique transmit antenna patterns or beams.

[0177] Example 48. An apparatus according to Example 46 or 47, wherein: the stored unique transmit antenna pattern or beam is for a first beam set; and the one or more memories further store instructions that, when executed by the one or more processors, cause the apparatus to at least perform: using the stored information, the test device generates a transmit antenna pattern or beam from a second beam set by selecting a unique power level combination for a plurality of test probes used to form the transmit antenna pattern to transmit at least one beam to the device under test.

[0178] Example 49. The apparatus according to Example 48, wherein the one or more memories further store instructions that, when executed by the one or more processors, cause the apparatus to perform at least: receiving from the device under test an indication of one or more beams from the second beam set that satisfy criteria at the device under test; verifying, when comparing the one or more beams from the second beam set with one or more beams from the first beam set, whether the indicated one or more beams from the second beam set that satisfy criteria at the device under test satisfy one or more thresholds; determining, based on the verification, whether the result of the conformance test is pass or fail; and outputting the pass or fail indication.

[0179] Example 50. The apparatus according to any one of Examples 46 to 49, wherein the indication of the received power measurement includes an indication of the received power of a reference signal.

[0180] Example 51. The apparatus according to any one of Examples 41 to 50, wherein the device under test is a user equipment.

[0181] Example 52. An apparatus comprising: one or more processors; and one or more memories storing instructions, which, when executed by the one or more processors, cause the apparatus to perform at least: for a device under test (DUT) for conformance testing of a machine learning-based beam management use case, sending to a test device an indication that the DUT supports conformance testing of machine learning-based beam management utilizing at least a combination of test probes; receiving from the test device by the DUT configuration and alignment information; configuring by the DUT information to be used for the conformance test based on the configuration and alignment information; and performing the conformance test of the machine learning-based beam management use case by the DUT based on the configuration and alignment information.

[0182] Example 53. The apparatus according to Example 52, wherein the one or more memories further store instructions that, when executed by the one or more processors, cause the apparatus to perform at least the following prior to execution: sending one or both of a report or alignment information from the device under test to the test device for alignment by the test device.

[0183] Example 54. The apparatus according to Example 53, wherein the alignment information from the device under test includes one or more parameters of a model or function, or both a model and a function, used by the device under test.

[0184] Example 55. An apparatus according to any one of Examples 52 to 54, wherein the alignment information from the test equipment includes one or more of the following: parameters related to the codebook of the dynamic test probe beam or antenna pattern used for the conformance test; or parameters for adjusting the model used by the device under test for the conformance test.

[0185] Example 56. An apparatus according to any one of Examples 52 to 55, wherein performing the conformance test comprises: receiving a unique transmit antenna pattern or beam transmitted from the test equipment, the unique transmit antenna pattern or beam having a unique power level combination for one or more test probes used to transmit the unique transmit antenna pattern or beam; and the device under test transmitting to the test equipment an indication of a metric of the received power corresponding to the unique transmit antenna pattern or beam.

[0186] Example 57. The apparatus according to Example 56, wherein the receiving and transmitting are performed for a plurality of unique transmit antenna modes or beams.

[0187] Example 58. An apparatus according to Example 56 or 57, wherein the one or more memories further store instructions that, when executed by the one or more processors, cause the apparatus to perform at least: receiving, by the device under test, a transmit antenna pattern having at least one beam from a second beam set from the test device, the transmit antenna pattern having a unique combination of power levels for a plurality of test probes used to transmit the transmit antenna pattern or beam.

[0188] Example 59. The apparatus according to Example 58, wherein the one or more memories further store instructions that, when executed by the one or more processors, cause the apparatus to perform at least: sending an indication from the device under test to the test device of one or more beams from the second beam set that satisfy the criteria at the device under test.

[0189] Example 60. The apparatus according to any one of Examples 56 to 59, wherein the indication of the measurement of the received power includes an indication of the received power of a reference signal.

[0190] Example 61. A computer program comprising instructions that, when executed by a device, cause the device to perform the method according to any one of Examples 1 to 20.

[0191] Example 62. A computer program according to Example 61, wherein the computer program is a computer program product, the computer program product including a computer-readable medium carrying instructions for the device embodied therein.

[0192] Example 63. The computer program according to Example 61, wherein the computer program can be directly loaded into the internal memory of the device.

[0193] As used in this application, the term "circuit" may refer to one or more of the following:

[0194] (a) Hardware circuit implementation only (such as implementations in analog, digital, and / or quantum circuits); and

[0195] (b) Combinations of hardware circuitry and software, such as (if applicable): (i) combinations of analog, digital, and / or quantum hardware circuitry with software / firmware, and (ii) any or all portions of a hardware processor (including digital and / or quantum processors) and memory having software, which work together to enable a device (such as a mobile device, computing device, or server) to perform various functions; and

[0196] (c) Any or all parts of a hardware circuit, such as one or more microprocessors, one or more processors and / or quantum processors, require software (e.g., firmware) to operate, but the software may be absent when it is not required to operate.

[0197] This definition of "circuit" applies to all uses of the term in this application (including any claims). As a further example, as used in this application, the term "circuit" also covers only hardware circuitry or a processor (or processors) or a portion thereof and its accompanying software and / or firmware implementation. The term "circuit" also covers, for example and if applicable to a particular claim element, baseband integrated circuits or processor integrated circuits for mobile devices or similar integrated circuits in servers, cellular network devices or other computing or network devices.

[0198] In example embodiments, as used herein, software (e.g., application logic, instruction set) is maintained on any of a variety of conventional computer-readable media. In the context of this document, "computer-readable media" can be any medium or component that can contain, store, communicate, propagate, or transmit instructions for use by or in connection with an instruction execution system, apparatus, or device, such as a computer, where one example of a computer is, for example, a... Figure 6 The computer-readable medium may include computer-readable storage media (e.g., memories 15 and 75 or other devices), which may be any medium or component that can contain, store, and / or transmit instructions for use by or in connection with an instruction execution system, apparatus, or device (such as a computer). Computer-readable storage media do not include propagated signals and can therefore be considered non-transitory. As used herein, the term "non-transitory" is a limitation on the medium itself (i.e., tangible, not signaling), not a limitation on the persistence of data storage (e.g., RAM, random access memory versus ROM, read-only memory).

[0199] If necessary, the different functions discussed in this article may be executed in different orders and / or simultaneously with each other. Furthermore, if necessary, one or more of the above functions may be optional or may be combined.

[0200] Although various aspects of the invention are set forth in the independent claims, other aspects of the invention include other combinations of features from the described embodiments and / or dependent claims with features of the independent claims, and not only the combinations expressly set forth in the claims.

[0201] It should also be noted in this document that although exemplary embodiments of the invention have been described above, these descriptions should not be construed as limiting. Rather, several changes and modifications may be made without departing from the scope of the invention as defined in the appended claims.

[0202] The following abbreviations, which may appear in the specification and / or drawings, are defined as follows:

[0203] 3D

[0204] 3GPP Third Generation Partnership Project

[0205] 5G fifth generation

[0206] AI (Artificial Intelligence)

[0207] AI / ML (Artificial Intelligence / Machine Learning)

[0208] AMF access and mobility management functions

[0209] AoA Arrival Angle

[0210] BM Beam Management

[0211] CSI Channel Status Information

[0212] CSI-RS Channel State Information - Reference Signal

[0213] CU Central Unit

[0214] DL downlink (from network to UE)

[0215] DU Distributed Unit

[0216] DUT (Device Under Test)

[0217] E-SMLC Evolution Service Mobile Location Center

[0218] eNB (or eNodeB) evolved Node B (e.g., LTE base station)

[0219] EUT (Equipment Under Test)

[0220] GMLC Gateway Mobile Positioning Center

[0221] gNB (or gNodeB) is used for 5G / NR base stations.

[0222] ID identifier

[0223] I / F interface

[0224] LMF location management function

[0225] LTE Long Term Evolution

[0226] MIMO (Multiple Input Multiple Output)

[0227] ML machine learning

[0228] MME Mobility Management Entity

[0229] MPAC Multi-Probe Anechoic Chamber

[0230] NF Network Functions

[0231] ng or NG next generation

[0232] NR New Radio

[0233] NRF network storage function

[0234] N / W or NW network

[0235] OTA (Over-the-Air)

[0236] RAN Radio Access Network

[0237] Rel version

[0238] RSRP reference signal received power

[0239] RU radio unit

[0240] Rx receiver

[0241] SGW Service Gateway

[0242] SSB Synchronization Signal / Physical Broadcast Channel Block

[0243] SI Research Project

[0244] SMF Session Management Function

[0245] SS System Simulator

[0246] TE testing equipment

[0247] TRP Send-Receive Point

[0248] Tx transmitter

[0249] UDM Unified Data Management

[0250] UDR Unified Data Storage

[0251] UE (User Equipment) (e.g., wireless equipment, typically mobile equipment)

[0252] UPF User Face Functions

[0253] WI work projects

Claims

1. A method comprising: For a test device used to test a device under test for conformance testing of a machine learning-based beam management use case, the test device receives an indication that the device under test supports conformance testing of machine learning-based beam management using at least a combination of test probes. The test equipment adjusts the configuration for the device under test and the pre-test alignment information for the conformance test based on machine learning beam management; Instructions to send the configuration and pre-test alignment information from the test device to the device under test; as well as The test equipment performs the conformance test for the machine learning-based beam management use case of the device under test based on the configuration and the pre-test alignment information.

2. A method comprising: For a device under test (DUT) used for conformance testing of machine learning-based beam management use cases, an instruction is sent to the test device that the DUT supports conformance testing of machine learning-based beam management using at least a combination of test probes. The device under test receives instructions from the test equipment regarding configuration and alignment information. The device under test configures the information to be used in the conformance test based on the configuration and the alignment information. as well as The device under test performs the conformance test of the machine learning-based beam management use case based on the configuration and the alignment information.

3. An apparatus comprising components for performing the following operations: For a test device used to test a device under test for conformance testing of a machine learning-based beam management use case, the test device receives an indication that the device under test supports conformance testing of machine learning-based beam management using at least a combination of test probes. The test equipment adjusts the configuration for the device under test and the pre-test alignment information for the conformance test based on machine learning beam management; Instructions to send the configuration and pre-test alignment information from the test device to the device under test; as well as The test equipment performs the conformance test for the machine learning-based beam management use case of the device under test based on the configuration and the pre-test alignment information.

4. The apparatus according to claim 3, wherein, The component is further configured to: prior to the execution: The test equipment receives one or both of a report or alignment information from the device under test; and Based on one or both of the report or the alignment information, the parameters of the model or function, or both of the model and function, used by the device under test for the conformance test are aligned with the parameters provided by the test device.

5. The apparatus according to any one of claims 3 or 4, wherein, The pre-test alignment information includes one or more of the following: parameters related to the codebook of the dynamic test probe beam or antenna pattern used for the conformance test; or parameters for adjusting the model or function, or both, used by the device under test for the conformance test.

6. The apparatus according to any one of claims 3 to 5, wherein, The conformance test is performed by the test equipment after the test equipment confirms that one or more test parameters used by the device under test are aligned with one or more corresponding test parameters provided by the test equipment.

7. The apparatus according to claim 6, wherein, The one or more test parameters include parameters used for the machine learning model or function, or both the model and function.

8. The apparatus according to any one of claims 3 to 7, wherein, Performing the consistency test includes: To generate a unique transmit antenna pattern or beam to be sent to the device under test, a unique power level combination is selected for one or more test probes used to send the unique transmit antenna pattern or beam to the device under test. The test equipment receives an indication of the received power metric corresponding to the unique transmit antenna mode or beam from the device under test; and The test equipment stores information including the indication of the metric, as well as an indication of the combination of the beam identifier corresponding to the unique transmit antenna pattern or beam and the unique power level.

9. The apparatus according to claim 8, wherein, The generation, reception, and storage are performed for multiple unique transmit antenna modes or beams.

10. The apparatus according to claim 8 or 9, wherein: The stored unique transmit antenna pattern or beam is for the first beam set; and The component is further configured to: The test equipment uses the stored information to generate a transmit antenna pattern or beam from the second beam set by selecting a unique combination of power levels for multiple test probes used to form the transmit antenna pattern to send at least one beam to the device under test.

11. The apparatus according to claim 10, wherein, The component is further configured to: The test equipment receives from the device under test an indication of one or more beams from the second beam set that satisfy the criteria at the device under test; When comparing one or more beams from the second beam set with one or more beams from the first beam set, verify whether one or more beams from the second beam set that meet the criteria at the device under test meet one or more thresholds; Based on the verification, determine whether the consistency test passes or fails; as well as Output the indicated pass or fail.

12. The apparatus according to any one of claims 3 to 11, wherein, The device under test is a user equipment.

13. An apparatus comprising components for performing the following operations: For a device under test (DUT) used for conformance testing of machine learning-based beam management use cases, an instruction is sent to the test device that the DUT supports conformance testing of machine learning-based beam management using at least a combination of test probes. The device under test receives instructions from the test equipment regarding configuration and alignment information. The device under test configures the information to be used in the conformance test based on the configuration and the alignment information. as well as The device under test performs the conformance test of the machine learning-based beam management use case based on the configuration and the alignment information.

14. The apparatus according to claim 13, wherein, The component is further configured to: prior to the execution: The device under test (DUT) sends one or both of its reports or alignment information to the test device for alignment purposes.

15. The apparatus according to claim 14, wherein, The alignment information from the device under test includes one or more parameters of the model or function, or both, used by the device under test.

16. The apparatus according to any one of claims 13 to 15, wherein, The alignment information from the test equipment includes one or more of the following: parameters related to the codebook of the dynamic test probe beam or antenna pattern used for the conformance test; Alternatively, it can be used to adjust the parameters of the model used by the device under test for the conformance test.

17. The apparatus according to any one of claims 13 to 16, wherein, Performing the consistency test includes: Receive a unique transmit antenna pattern or beam transmitted from the test equipment, the unique transmit antenna pattern or beam having a unique combination of power levels for one or more test probes used to transmit the unique transmit antenna pattern or beam; and The device under test sends an indication to the test equipment of the received power measurement corresponding to the unique transmit antenna mode or beam.

18. The apparatus according to claim 17, wherein, The receiving and transmitting are performed for multiple unique transmit antenna modes or beams.

19. The apparatus according to claim 17 or 18, wherein, The component is further configured to: The device under test receives from the test equipment a transmit antenna pattern having at least one beam from a second beam set, the transmit antenna pattern having a unique combination of power levels for a plurality of test probes used to transmit the transmit antenna pattern or beam.

20. The apparatus according to claim 19, wherein, The component is further configured to: The device under test (DUT) sends an indication from the second beam set of one or more beams that satisfy the criteria at the DUT to the test equipment.

Images