Method and setup for testing artificial intellegence machine learning based beam prediction

Abstract

In accordance with example embodiments of the invention there is at least a method and apparatus to perform receiving from a second apparatus of a communication network information comprising a test command to enable or activate at least one of a training or inference test mode for conformance testing of a machine learning model or functionality based beam management use case during the at least one of a training or inference phase; based on the information, communicating an acknowledgement message with the second apparatus confirming an activation of the at least one of a training or inference test mode for the at least one of a training or inference phase; wherein the activation is using a configuration to perform the at least one of a training or inference validation of the machine learning model or functionality of the first apparatus using a testing dataset during the conformance testing. Further, to perform sending towards a second apparatus of a communication network information comprising a test command to enable or activate at least one of a training or inference test mode at the second apparatus for conformance testing of a machine learning model or functionality based beam management use case during the at least one of a training or inference phase; and receiving from the second apparatus of the communication network information comprising an acknowledgement message confirming an activation of the at least one of a training or inference test mode for the at least one of a training or inference phase, wherein the activation is using a configuration to perform the at least one of a training or inference validation of the machine learning model or functionality of the first apparatus using a testing dataset during the conformance testing

Description

METHOD AND SETUP FOR TESTING ARTIFICIAL INTELLEGENCE MACHINE LEARNING BASED BEAM PREDICTIONTECHNICAL FIELD:

[0001] The teachings in accordance with the exemplary embodiments of this invention relate generally to a new machine learning-dedicated bearer for machine learning or artificial intelligence and, more specifically, relate to a new machine learning-dedicated bearer for machine learning or artificial intelligence related to a new machine learning-dedicated bearer for conformance testing of an improved AI / ML based beam management feature.BACKGROUND:

[0002] This section is intended to provide a background or context to the invention that is recited in the claims. The description herein may include concepts that could be pursued but are not necessarily ones that have been previously conceived or pursued. Therefore, unless otherwise indicated herein, what is described in this section is not prior art to the description and claims in this application and is not admitted to be prior art by inclusion in this section.

[0003] Certain abbreviations that may be found in the description and / or in the Figures are herewith defined as follows:3GPP 3rdGeneration Partnership ProjectAl Artificial IntelligenceBM Beam ManagementCSI Channel State InformationCSLRS Channel State Information Reference SignalDL DownlinkDUT Device Under TestEUT Equipment Under TestKPI Key Performance IndicatorLCM Life Cycle ManagementMIMO Multiple Input Multiple OutputML Machine LearningMP AC Multi-Probe Anechoic ChamberNR New RadioNW NetworkOTA Over The AirRAN Radio Access NetworkRANI Radio Access Network Working Group 1RAN4 Radio Access Network Working Group 4Rel. ReleaseRF Radio FrequencyRRM Radio Resource ManagementRSRP Reference Signal Received PowerSI Study ItemSSB Synchronization Signal BlockTE Test EquipmentTR Technical ReportTS Technical SpecificationUE User EquipmentWF Way ForwardWI Work Item

[0004] Some standards at the time of this application are based on studies on Artificial Intelligence (AI) / Machine Learning (ML) for NR Air Interface. One goal is to support a new AI / ML-enabled radio interface for the next cellular systems, at the time of this application 3GPP is working on normative phase of a work item based on beam management use-case targets spatial and / or time-domain beam prediction for overhead and reporting latency reduction.

[0005] Example embodiments of this invention propose improved operations for at least this beam management.SUMMARY:

[0006] This section contains examples of possible implementations and is not meant to be limiting.

[0007] In another example aspect of the invention, there is an apparatus, such as a user equipment side apparatus, comprising: at least one processor; and at least one non-transitory memory storing instructions, that when executed by the at least one processor, cause the apparatus at least to: receive from a second apparatus of a communication network information comprising a test command to enable or activate at least one of a training or inference test mode for conformance testing of a machine learning model or functionality based beam management use case during the at least one of a training or inference phase; based on the information, communicate an acknowledgement message with the second apparatus confirming an activation of the at least one of a training or inference test mode for the at least one of a training or inference phase; wherein the activation is using a configuration to perform the at least one of a training or inference validation of the machine learning model or functionality of the first apparatus using a testing dataset during the conformance testing.

[0008] In still another example aspect of the invention, there is a method, comprising: receiving from a second apparatus of a communication network information comprising a test command to enable or activate at least one of a training or inference test mode for conformance testing of a machine learning model or functionality based beam management use case during the at least one of a training or inference phase; based on the information, communicating an acknowledgement message with the second apparatus confirming an activation of the at least one of a training or inference test mode for the at least one of a training or inference phase; wherein the activation is using a configuration to perform the at least one of a training or inference validation of the machine learning model or functionality of the first apparatus using a testing dataset during the conformance testing.

[0009] A further example embodiment is an apparatus and a method comprising the apparatus and the method of the previous paragraphs, wherein the informationcomprises a test command to enable or activate the at least one of a training or inference test mode during or before conformance testing of a machine learning based beam management use case, wherein there is, based on training data using at least one subset of at least one dataset, train the machine learning model or functionality of the apparatus for beam management use case to predict at least a reference signal received power of a best or all beams in a Set A beams, based on the measured reference signal received power of the beams in Set B beams, wherein the at least one dataset is for first apparatus side artificial intelligence or machine learning functionality or model training for conformance test purposes, wherein there is accumulating several measurements, wherein the measurements use M*P, and wherein P is a number of probes, before performing at least one of inference or prediction, wherein the measurements are distributed in time, wherein there is activating the test mode for the inference phase; sending towards the second apparatus an acknowledgement confirming that the test mode for the inference phase is activated; receiving from the second apparatus a test mode command providing a test configuration to perform beam sweeping in one or more iterations and measure a reference signal received power of any corresponding beams; based on the test configuration, performing beam sweeping in one or more iterations; and transmitting an acknowledgement message to the second apparatus confirming readiness to perform beam sweeping in one or more iterations.

[0010] Wherein there is receiving from the second apparatus a set of beams (Set B), wherein each beam from the set of beams is transmitted in a sequential manner, and wherein each beam of the set of beams is transmitted from a unique combination of test probes and first apparatus orientation; and measuring a reference signal received power of each beam of the set of beams (Set B), wherein there is measuring a reference signal received power of each beam of the set of beams (Set B), wherein the set of beams (Set B) is a subset of set A beams; providing as input to the machine learning model or functionality model for at least a best beam prediction from the set A beams; and reporting at least a best Set A beam, wherein the reporting comprises an indication of characteristics of the at least best beam prediction, further comprising at least a beam identification, and / or a reference signal received power of the best predicted beam, wherein the indication provides a reference signal received power and a beamidentification of the best predicted beam to the second apparatus for validation, and wherein there is providing as input to the machine learning model or functionality for a best Set A beam prediction, wherein a predicted best beam and characteristics of the predicted best beam are indicated to the second apparatus.

[0011] A non-transitory computer-readable medium storing program code, the program code executed by at least one processor to perform at least the method as described in the paragraphs above.

[0012] In yet another example aspect of the invention, there is an apparatus comprising: means for receiving from a second apparatus of a communication network information comprising a test command to enable or activate at least one of a training or inference test mode for conformance testing of a machine learning model or functionality based beam management use case during the at least one of a training or inference phase; means, based on the information, for communicating an acknowledgement message with the second apparatus confirming an activation of the at least one of a training or inference test mode for the at least one of a training or inference phase; wherein the activation is using a configuration to perform the at least one of a training or inference validation of the machine learning model or functionality of the first apparatus using a testing dataset during the conformance testing.

[0013] In accordance with the example embodiments as described in the paragraph above, at least the means for receiving and communicating comprises a network interface, and computer program code stored on a computer-readable medium and executed by at least one processor.

[0014] In another example aspect of the invention, there is an apparatus, such as a network side apparatus, comprising: at least one processor; and at least one non- transitory memory storing instructions, that when executed by the at least one processor, cause the apparatus at least to: send towards a second apparatus of a communication network information comprising a test command to enable or activate at least one of a training or inference test mode at the second apparatus for conformance testing of a machine learning model or functionality based beam management use case during the at least one of a training or inference phase; receive from the secondapparatus of the communication network information comprising an acknowledgement message confirming an activation of the at least one of a training or inference test mode for the at least one of a training or inference phase, wherein the activation is using a configuration to perform the at least one of a training or inference validation of the machine learning model or functionality of the first apparatus using a testing dataset during the conformance testing.

[0015] In still another example aspect of the invention, there is a method, comprising: sending towards a second apparatus of a communication network information comprising a test command to enable or activate at least one of a training or inference test mode at the second apparatus for conformance testing of a machine learning model or functionality based beam management use case during the at least one of a training or inference phase; receiving from the second apparatus of the communication network information comprising an acknowledgement message confirming an activation of the at least one of a training or inference test mode for the at least one of a training or inference phase, wherein the activation is using a configuration to perform the at least one of a training or inference validation of the machine learning model or functionality of the first apparatus using a testing dataset during the conformance testing.

[0016] A further example embodiment is an apparatus and a method comprising the apparatus and the method of the previous paragraphs, wherein the information comprises a test command to enable or activate the at least one of a training or inference test mode during or before conformance testing of a machine learning based beam management use case, wherein there is communicating with the second apparatus training data, using the at least one subset of at least one dataset, for the training of second apparatus side machine learning model or functionality to perform inference during the conformance testing, wherein the training data is for use to train the machine learning model or functionality of the second apparatus for beam management use case to predict at least a reference signal received power of a best or all beams in a Set A beams, based on the measured reference signal received power of the beams in Set B beams wherein the at least one dataset is for second apparatus side artificial intelligence or machine learning functionality or model training for conformance test purposes, wherein there is performing operations comprising: selecting a power level for each ofat least one combination of test probes in a test chamber; determining an orientation or a test-point over a 3D test sphere and fix the second apparatus in a corresponding orientation; and utilizing a plurality of beams sequentially, wherein each beam is utilized from a different combination of test-probe and second apparatus orientation, one at a time, and wherein the second apparatus measures a reference signal received power of the plurality of beams sequentially, and wherein there is configuring the different combination of test-probe and second apparatus orientation to generate set B beams for the second apparatus to predict at least the RSRP of the best or all of the Set A beam(s) using the reference signal received power and beam identification of Set B beams generated using different combinations of different power levels of the different combination of test-probe and second apparatus orientations^wherein the best or all of the SET A beams are predicted but less than all of the SET A beams are transmitted to the second apparatus

[0017] Wherein the operations are performed M times, wherein M is a ceiling function or smallest integer function of a ratio of a number of beams in the set of beams (Set B) to a total number of a combination of test probes available in the test chamber, and wherein M is an positive integer, wherein there is accumulating several measurements, wherein the measurements use M*P, and wherein P is a number of probes, before performing at least one of inference or prediction, wherein the measurements are distributed in time., wherein the apparatus is embodied in a test equipment, wherein there is receiving from the second apparatus an acknowledgement confirming that the test mode for the inference phase is activated; sending towards the second apparatus a test mode command providing a test configuration to perform beam sweeping in one or more iterations and measure a reference signal received power of any corresponding beams, wherein based on the test configuration there is beam sweeping in one or more iterations; and receiving from the second apparatus an acknowledgement message confirming readiness to perform beam sweeping in one or more iterations at the second apparatus, wherein there is sending towards the second apparatus a set of beams, wherein the set of beams is sequential, wherein each beam of the set of beams is from a different combination of test probes and second apparatus orientations, and wherein based on the set of beams, a reference signal received powerof each beam of the set of beams is measured by the second apparatus to perform machine learning based beam prediction

[0018] Wherein there is selecting a power level of each test probe and select an orientation of the second apparatus for each of the different combination of test probes and second apparatus orientations over the 3D test sphere, wherein selecting an orientation of the second apparatus comprising toning pair of theta and phi and fixing the second apparatus in a corresponding orientation, wherein there is randomly selecting Set B beams from a generated dataset and generate only a set of beams (Set B) in a testing chamber; transmitting a subset of beams sequentially, wherein each of the beam in the subset is transmitted from a different test-probe and sent one at a time, wherein there is receiving from the second apparatus a report indicating at least a best Set A beam, wherein the reporting comprises an indication of characteristics of the at least best predicted beam, comprising at least a beam identification, and / or a reference signal received power of the at least best beam, wherein the report is based on a predicted output the second apparatus based on the reference signal received power of each beam of the set of beams (Set B), wherein the set of beams (Set B) is a subset of set A beams, wherein the indication provides at least one of the reference signal received power or a beam identification to the second apparatus for inference validation, wherein there is determining information comprising an input to the machine learning model or functionality of the second apparatus for at least a best Set A beam prediction, wherein at least a best beam is predicted and characteristics of the predicted best beam are indicated to the apparatus.

[0019] Wherein there is, based on the information, verifying whether at least the best set A beam prediction is same or close enough to a true best beam, wherein there is, based on the best set A beam prediction being same as the true best beam, determining that true best beam validation is successful and test iteration has passed, and wherein there is based on the best set A beam prediction not being same as the true best beam, determine that true best beam validation and test iteration was not successful and has failed.

[0020] In yet another example aspect of the invention, there is an apparatus comprising: means for sending towards a second apparatus of a communication network information comprising a test command to enable or activate at least one of a training or inference test mode at the second apparatus for conformance testing of a machine learning model or functionality based beam management use case during the at least one of a training or inference phase; and means for receiving from the second apparatus of the communication network information comprising an acknowledgement message confirming an activation of the at least one of a training or inference test mode for the at least one of a training or inference phase, wherein the activation is using a configuration to perform the at least one of a training or inference validation of the machine learning model or functionality of the first apparatus using a testing dataset during the conformance testing.

[0021] In accordance with the example embodiments as described in the paragraph above, at least the means for sending and receiving comprises a network interface, and computer program code stored on a computer-readable medium and executed by at least one processor.

[0022] A communication system comprising the network side apparatus and the user equipment side apparatus performing operations as described above.BRIEF DESCRIPTION OF THE DRAWINGS:

[0023] The above and other aspects, features, and benefits of various embodiments of the present disclosure will become more fully apparent from the following detailed description with reference to the accompanying drawings, in which like reference signs are used to designate like or equivalent elements. The drawings are illustrated for facilitating better understanding of the embodiments of the disclosure and are not necessarily drawn to scale, in which:

[0024] FIG. 1 shows a 3D MP AC system layout for NR FR2 MIMO OTA testing;

[0025] FIG. 2 shows an illustration of difference in between the legacy and AI / ML based testing scenario / setup;

[0026] FIG. 3 shows an OTA test setup with four test probes;

[0027] FIG. 4 shows a first part of a message flow sequence in accordance with example embodiments of the invention;

[0028] FIG. 5 shows a next part of a message flow sequence in accordance with example embodiments of the invention;

[0029] FIG. 6 shows an example dataset;

[0030] FIG. 7 shows a high-level block diagram of various devices used in carrying out various aspects of the invention; and

[0031] FIG. 8A and FIG. 8B each show a method in accordance with example embodiments of the invention which may be performed by an apparatus.DETAILED DESCRIPTION:

[0032] In example embodiments of this invention there is proposed at least a method and apparatus for a new machine learning-dedicated bearer for machine learning or artificial intelligence related to a new machine learning-dedicated bearer for conformance testing of an improved AI / ML based beam management feature.

[0033] As similarly stated above, to support a new AI / ML-enabled radio interface for the next cellular systems, 3GPP is currently working on normative phase of a Rel-19 Work Item (WI description RP-234039), which is a continuation of Rel-18 SI (RP-213599). AI / ML-based beam management use-case targets spatial and / or time beam prediction for overhead and reporting latency reduction.

[0034] In AI / ML beam management, the measurements of multiple beam (referred to as Set B) are used as input to the ML model and the output of AI / ML model or functionality is represented by Set A beams.

[0035] The measured beams, e.g., Set B can be provided with the CSI reporting configurations. Set A contains “true (KPIs and ID) of all SSB / CSI-RS beams” whereas the output of AI / ML model or functionality is the:1) Prediction of KPI (e.g., RSRP) for each of the beam in Set A, and / or2) Prediction of beam ID and / or RSRP etc. of the BEST beam in Set A.

[0036] The study should also identify areas where AI / ML could improve the performance of air-interface functions. Specification impact will be assessed to improve the overall understanding of what would be required to enable AI / ML techniques for the air interface. The beam management use case is further studied for spatial and / or time beam prediction. The scope of spatial-domain beam prediction (BM-Casel) is to predict the best Tx / Rx beams in different spatial locations. Conversely, time-domain beam predictions (BM-Case2) aim to predict the most likely beam to use for next time instants, e.g., beam prediction in the spatial domain (BM- Casel). Both BM-Case 1 and BM-Case2 have been approved for the normative work in Rel. 19.

[0037] Testability of AI / ML based mechanisms is one of the highlights of rel. 18 SI as well as the ongoing Rel. 19 WI. Here is an extract from the TR 38.843:

[0038] The general requirements and testing frameworks for AI / ML based performance enhancements mainly focus on:- how to define requirements and tests for inference;- evaluate feasibility and necessity of requirements / tests for LCM; requirements for data collection (in particular for training) could / need be defined.

[0039] Requirements / tests for training will not be studied unless training procedures are defined. The design of test should ensure performance is guaranteed and avoid that a UE can pass the test but perform poorly in the field.

[0040] Testability of AI / ML based BM use cases is getting more and more attention in 3GPP. This testability issue was brought by multiple companies at RAN4#110 meeting and the discussion continued without any conclusion at RAN4#110-bis. Here is the agreement achieved on this topic and it is documented in the agreed WF during RAN4#110 meeting:

[0041] Companies are invited to provide further analysis on what the test setup should enable in terms of test environment

[0042] Background on MIMO and OTA testing setups

[0043] The challenge of testability of complex features such as multiRx and MIMO OTA has already been considered in different studies in 3GPP. In the following, previous studies are addressed:

[0044] multiRx testing setup TR 38.871

[0045] The objectives of FR2- 1 OTA testing for UEs with multi-panel reception and 4DE-layer are captured as follows in TR 38.871:

[0046] Define a test methodology for RF / RRM / Demodulation requirements testing for devices that can receive simultaneously from multiple Angle of Arrival (AoA).

[0047] The multiple AoA test setup should enable testing of up to 2 DE Layers with dual polarization for each angle.

[0048] For RRM, the target should be to allow testing of 4 AoAs with 2 simultaneously active AoAs.

[0049] There can be defining a test methodology for up to 4 DL MIMO layer demodulation testing.

[0050] Smartphone form factor should be the first priority, other UE types should also be discussed as 2nd priority.

[0051] Further, there can be development of related preliminary uncertainty assessments for the test methodologies.

[0052] The tests shall take the test system reuse, test system complexity and test time into account to keep the whole test costs within a reasonable level.

[0053] MIMO OTA testing setup TR 38.827

[0054] As discussed in TR 38.827, for FR2 NR MIMO OTA testing, the 3D MP AC (Multi-Probe Anechoic Chamber) test method is the reference. This is achieved by arranging an array of antennas around the Equipment Under Test (EUT), a spatial distribution of angles of arrival in the 3D MP AC system may be simulated to expose the EUT to a near field environment that appears to have originated from a complex multipath far field environment.

[0055] Signals propagate from the base station / communication tester to the EUT through a simulated multipath environment known as a spatial channel model, where appropriate channel impairments such as Doppler and fading are applied to each path prior to injecting all of the directional signals into the chamber simultaneously through the probe array. The resulting field distribution in the test zone is then integrated by the EUT antenna(s) and processed by the receiver(s) just as it would do so in any non-simulated multipath environment. The 3D MP AC system with 6 dualpolarized probes placed on a sector with minimum radius of 0.75m from the centre or center of the test zone is permitted for NR FR2 MIMO OTA testing.

[0056] FIG. 1 shows a 3D MP AC system layout for NR FR2 MIMO OTA testing. As shown in FIG. 1 there is a base station simulator communicating with a channel model emulator communicating with a radio head to probes, which will transmit and / or receive the signal(s) from the DUT inside a test zone.

[0057] During RAN4#1 lObis meeting there was proposed one solution based on CDL channel model and using all the probes to create a single beam. This test setup is described in their tdoc R4-2404215 and the setup is quite similar to what is used in TR 38.827. Here are some drawbacks found in this method:• CDL is emulated inside the chamber using 3 probes, which is not sufficient to completely replicate CDL model that contains much higher path components. Therefore, it won’t be feasible to train any UE side models based on only CDL channel models characteristics;• The testing setup proposed in R4-2404215 is more complex (due to the use of more complex CDL channel model) and would require relatively much larger time as compared to the AWGN based setup, as proposed in this IR;• The necessity of using CDL channel is also challenged by multiple companies during the discussions.

[0058] At the time of this invention there are not known any existing low complexity testability solutions (e.g. based on AWGN channel) for testing AI / ML based Beam Management in a multi beam environment.

[0059] Traditional / legacy beam management is based on the UE evaluation of the quality of reference symbols transmitted by gNB (e.g., SSB or CSI-RS) and on reporting of these measured quality metrics (e.g., RSRP, RSRQ, SINR) back to the NW. In other words, there is a direct relation in between transmitted RS and reported measurement. Existing requirements (measurement accuracy defined in TS 38.133, Clause 10), test cases, and testing setups are also based on this assumption. Thedifference between legacy testing approach / scenario and the one needed for AI / ML- based beam management is illustrated in Error! Reference source not found. 2.

[0060] FIG. 2 shows an illustration of a difference between the legacy and Ai / ML based testing scenario / setup. As shown in FIG. 2 there is legacy beam measurements testing scenario with a UE served by a gNB / SS communicating measurement reporting with a gNB / SS that is communicating measurement and predictions reporting with another UE served by another gNB / SS. Dashed beams on the right are beams that are not transmitted but predicted.

[0061] Based on Rel-18 SI in Rel-19 WI, a new AI / ML based beam management mechanism is being specified. One of the main differences of this new approach is that it is data driven, i.e., prediction of the beam(s) measurement (either in spatial domain or on certain time instance in the future) is based on the measurement of a large set of beams usually referred as Set B.

[0062] Firstly, (1) the receive power of Set B beams is not random / uncorrelated or completely independent on each other. If it would be the case, the model would not be able to learn any patterns in the transmitted beams and to build any predictions.

[0063] Secondly, (2) the problem is that the current RRM-based conformance testing setups are designed to support very limited number of probes. Usually only two radio probes are used and based on the latest TR on the OTA testing methods defined in TR 38.871, it is still challenging to support even 4 probes. However, there are testing setups (TS) that might support up to 6 probes, as defined in TR 38.827.

[0064] (3) Even 6 probes are not sufficient to emulate the realistic scenario where the UE needs to measure 16 narrow / fine beams (set B) to predict the other 16 beams (i.e., Set A overlaps with Set B and Set A consists of 32 beams). Whereas such assumptions are considered in RANI evaluations based on TR 38.843.

[0065] This testability issue was brought by multiple companies at RAN4#110 meeting and the discussion continued without any conclusion at RAN4#110-bis.

[0066] In short, one main challenge is how to use existing conformance testing setup with minimal enhancements / changes to test AI / ML based beam prediction features?

[0067] Other additional challenges that need to be treated is that:1) For SSB beams all the beams transmitted by the cell shall be contained in the SSB burst, i.e., it is not possible to distribute SSB transmission in time so that in one burst only a sub-set of beams is transmitted, then in the next burst another sub-set and so on;2) Beam transmission and measurement periodicities periodicity is limited to relatively short time intervals, i.e., it practically impossible to move UE or probe with periodicity of the order of a few 100 ms.

[0068] In accordance with example embodiments of the invention there is proposed a framework for the conformance testing of AI / ML based beam management feature and enables:(1) the beam sweeping of Set A and / or Set B beams in an iterative manner, by using a combination of the available test probes and different UE orientations / rotations, during the RAN4 conformance testing using the OTA test chamber where single or multiple test probe(s) are available;(2) a method for the conformance testing of the AI / ML based beam management feature;(3) a new UE test mode is added to emulate full beam sweeping cycle by generating subsets of different beam transmissions sequentially. For example, UE accumulates several measurements (N*P in where P is the number of probes)before it performs inference / prediction, i.e., measurements are distributed in time.

[0069] Before describing the example embodiments as disclosed herein in detail, reference is made to FIG. 7 for illustrating a simplified block diagram of various electronic devices that are suitable for use in practicing the example embodiments of this invention.

[0070] FIG. 7 shows a block diagram of one possible and non-limiting exemplary system in which the example embodiments may be practiced. In FIG. 7, a user equipment (UE) 10 is in wireless communication with a wireless network 1 or network, 1 as in FIG. 7. The wireless network 1 or network 1 as in FIG. 7 can comprise a communication network such as a mobile network as disclosed herein. Any reference herein to a wireless network 1 as in FIG. 7 can be seen as a reference to any wireless network as disclosed herein. Further, the wireless network 1 as in FIG. 7 can also comprise hardwired features as may be required by a communication network. A UE is a wireless, typically mobile device that can access a wireless network. The UE, for example, may be a mobile phone (or called a "cellular" phone) and / or a computer with a mobile terminal function. For example, the UE or mobile terminal may also be a portable, pocket, handheld, computer-embedded or vehicle-mounted mobile device and performs a language signaling and / or data exchange with the RAN.

[0071] The UE 10 includes one or more processors DP 10A, one or more memories MEM 10B, and one or more transceivers TRANS 10D interconnected through one or more buses. Each of the one or more transceivers TRANS 10D includes a receiver and a transmitter. The one or more buses may be address, data, or control buses, and may include any interconnection mechanism, such as a series of lines on a motherboard or integrated circuit, fiber optics or other optical communication equipment, and the like. The one or more transceivers TRANS 10D which can be optionally connected to one or more antennas for communication to Network node 12 and Network node 13, respectively. The one or more memories MEM 10B include computer program code PROG 10C. The UE 10 communicates with Network node 12 and / or Network node 13 via a wireless link 11 or 16. The UE 10 may communicatewith a gNB or eNB, or a device such as the Network node 12 and / or the Network node 13 such as via link 6.

[0072] The Network node 12 (NR / 5G / 6G Node B, an evolved NB, or LTE device) is a network node such as a master or secondary node base station (e.g., for NR or LTE long term evolution) that communicates with devices such as Network node 13 and UE 10 of FIG. 7. The Network node 12 provides access to wireless devices such as the UE 10 to the wireless network 1. The Network node 12 includes one or more processors DP 12 A, one or more memories MEM 12B, and one or more transceivers TRANS 12D interconnected through one or more buses. In accordance with the example embodiments these TRANS 12D can include X2 and / or Xn interfaces for use to perform the example embodiments. Each of the one or more transceivers TRANS 12D includes a receiver and a transmitter. The one or more transceivers TRANS 12D can be optionally connected to one or more antennas for communication over at least link 11 with the UE 10. The one or more memories MEM 12B and the computer program code PROG 12C are configured to cause, with the one or more processors DP 12 A, the Network node 12 to perform one or more of the operations as described herein. The Network node 12 may communicate with another gNB or eNB, or a device such as the Network node 13 such as via link 16 or link 18. Further, the link 11, link 16 and / or any other link may be wired or wireless or both and may implement, e.g., an X2 or Xn interface. Further the link 11 and / or link 16 and / or link 18 may be through other network devices such as, but not limited to an NCE / MME / SGW / UDM / PCF / AMF / SMF / LMF 14 device as in FIG. 7. The Network node 12 may perform functionalities of an MME (Mobility Management Entity) or SGW (Serving Gateway), such as a User Plane Functionality, and / or an Access Management functionality for LTE and similar functionality for 5G or 6G.

[0073] The Network node 13 can be for WiFi or Bluetooth or other wireless device associated with a mobility function device such as an AMF or SMF, further the Network node 13 may comprise a NR / 5G / 6G Node B or possibly an evolved NB a base station such as a master or secondary node base station (e.g., for NR or LTE long term evolution) that communicates with devices such as the Network node 12 and / or UE 10 and / or the wireless network 1. The Network node 13 includes one or more processors DP 13A, one or more memories MEM 13B, one or more network interfaces, and oneor more transceivers TRANS 13D interconnected through one or more buses. In accordance with the example embodiments these network interfaces of Network node 13 can include X2 and / or Xn interfaces for use to perform the example embodiments. Each of the one or more transceivers TRANS 13D includes a receiver and a transmitter that can optionally be connected to one or more antennas. The one or more memories MEM 13B include computer program code PROG 13C. For instance, the one or more memories MEM 13B and the computer program code PROG 13C are configured to cause, with the one or more processors DP 13 A, the Network node 13 to perform one or more of the operations as described herein. The Network node 13 may communicate with another mobility function device and / or eNB such as the Network node 12 and the UE 10 or any other device using, e.g., link 11 or link 16 or link 18 or another link. The link 16 or link 18 as shown in FIG. 7 can be used for communication with the NN 12 or a network node. These links maybe wired or wireless or both and may implement, e.g., an X2 or Xn interface. Further, as stated above the link 11 and / or link 16 and / or link 18 may be through other network devices such as, but not limited to an NCE / MME / SGW device such as the NCE / MME / SGW / UDM / PCF / AMF / SMF / LMF 14 of FIG. 7.

[0074] The one or more buses of the device of FIG. 7 may be address, data, or control buses, and may include any interconnection mechanism, such as a series of lines on a motherboard or integrated circuit, fiber optics or other optical communication equipment, wireless channels, and the like. For example, the one or more transceivers TRANS 12D, TRANS 13D and / or TRANS 10D may be implemented as a remote radio head (RRH), with the other elements of the Network node 12 being physically in a different location from the RRH, and these devices can include one or more buses that could be implemented in part as fiber optic cable to connect the other elements of the Network node 12 to an RRH.

[0075] It is noted that although FIG. 7 shows a network nodes such as Network node 12 and Network node 13, any of these nodes may can incorporate or be incorporated into an eNodeB or eNB or gNB such as for ETE and NR and would still be configurable to perform example embodiments.

[0076] Also, it is noted that description herein indicates that “cells” perform functions, but it should be clear that the gNB that forms the cell and / or a user equipmentand / or mobility management function device that will perform the functions. In addition, the cell makes up part of a gNB, and there can be multiple cells per gNB.

[0077] The wireless network 1 or any network it can represent may or may not include a NCE / MME / SGW / UDM / PCF / AMF / SMF / LMF 14 that may include (NCE) network control element functionality, MME (Mobility Management Entity) / SGW (Serving Gateway) functionality, and / or serving gateway (SGW), and / or MME (Mobility Management Entity) and / or SGW (Serving Gateway) functionality, and / or user data management functionality (UDM), and / or PCF (Policy Control) functionality, and / or Access and Mobility Management Function (AMF) functionality, and / or Session Management (SMF) functionality, and / or Location Management Function (LMF), and / or Authentication Server (AUSF) functionality and which provides connectivity with a further network, such as a telephone network and / or a data communications network (e.g., the Internet), and which is configured to perform any 5G, 6G, and / or NR operations in addition to or instead of other standard operations at the time of this application. The NCE / MME / SGW / UDM / PCF / AMF / SMF / LMF 14 is configurable to perform operations in accordance with example embodiments in any of an LTE, NR, 5G, 6G, and / or any standards-based communication technologies being performed or discussed at the time of this application. In addition, it is noted that the operations in accordance with example embodiments, as performed by the Network node 12 and / or Network node 13, may also be performed at the NCE / MME / SGW / UDM / PCF / AMF / SMF / LMF 14.

[0078] The NCE / MME / SGW / UDM / PCF / AMF / SMF / LMF 14 includes one or more processors DP 14A, one or more memories MEM 14B, and one or more network interfaces (N / W I / F(s)), interconnected through one or more buses coupled with the link 130 and / or link 6 and / or link 16 and / or link 18. In accordance with the example embodiments these network interfaces can include X2 and / or Xn interfaces for use to perform the example embodiments. The one or more memories MEM 14B include computer program code PROG 14C. The one or more memories MEM14B and the computer program code PROG 14C are configured to, with the one or more processors DP 14A, cause the NCE / MME / SGW / UDM / PCF / AMF / SMF / LMF 14 to perform one or more operations which may be needed to support the operations in accordance with the example embodiments.

[0079] It is noted that that the Network node 12 and / or Network node 13 and / or UE 10 can be configured (e.g. based on standards implementations etc.) to perform functionality of a Location Management Function (LMF). The LMF functionality may be embodied in any of these network devices or other devices associated with these devices. In addition, an LMF such as the LMF of the MME / SGW / UDM / PCF / AMF / SMF / LMF 14 of FIG. 7, as at least described below, can be co-located with UE 10 such as to be separate from the Network node 12 and / or Network node 13 of FIG. 7 for performing operations in accordance with example embodiments as disclosed herein.

[0080] The wireless Network 1 may implement network virtualization, which is the process of combining hardware and software network resources and network functionality into a single, software-based administrative entity, a virtual network. Network virtualization involves platform virtualization, often combined with resource virtualization. Network virtualization is categorized as either external, combining many networks, or parts of networks, into a virtual unit, or internal, providing network-like functionality to software containers on a single system. Note that the virtualized entities that result from the network virtualization are still implemented, at some level, using hardware such as processors DP10, DP12A, DP13A, and / or DP14A and memories MEM 10B, MEM 12B, MEM 13B, and / or MEM 14B, and also such virtualized entities create technical effects.

[0081] The computer readable memories MEM 12B, MEM 13B, and MEM 14B may be of any type suitable to the local technical environment and may be implemented using any suitable data storage technology, such as semiconductor-based memory devices, flash memory, magnetic memory devices and systems, optical memory devices and systems, fixed memory and removable memory. The computer readable memories MEM 12B, MEM 13B, and MEM 14B may be means for performing storage functions. The processors DP10, DP12A, DP13A, and DP14A may be of any type suitable to the local technical environment, and may include one or more of general-purpose computers, special purpose computers, microprocessors, digital signal processors (DSPs) and processors based on a multi-core processor architecture, as non-limiting examples. The processors DP10, DP12A, DP13A, and DP14A may bemeans for performing functions, such as controlling the UE 10, Network node 12, Network node 13, and other functions as described herein.

[0082] In general, various embodiments of any of these devices can include, but are not limited to, cellular telephones such as smart phones, tablets, personal digital assistants (PDAs) having wireless communication capabilities, portable computers having wireless communication capabilities, image capture devices such as digital cameras having wireless communication capabilities, gaming devices having wireless communication capabilities, music storage and playback appliances having wireless communication capabilities, Internet appliances permitting wireless Internet access and browsing, tablets with wireless communication capabilities, as well as portable units or terminals that incorporate combinations of such functions.

[0083] Further, the various embodiments of any of these devices can be used with a UE vehicle, a High-Altitude Platform Station, or any other such type node associated with a terrestrial network or any drone type radio or a radio in aircraft or other airborne vehicle or a vessel that travels on water such as a boat.

[0084] As similarly stated above, in accordance with example embodiments of the invention there is proposed a framework for the conformance testing of AI / ML based beam management feature and enables:(1) the beam sweeping of Set A and / or Set B beams in an iterative manner, by using a combination of the available test probes and different UE orientations / rotations, during the RAN4 conformance testing using the OTA test chamber where single or multiple test probe(s) are available;(2) a method for the conformance testing of the AI / ML based beam management feature;(3) a new UE test mode is added to emulate full beam sweeping cycle by generating subsets of different beam transmissions sequentially. For example, UE accumulates several measurements (N*P in where P is the number of probes)before it performs inference / prediction, i.e., measurements are distributed in time.

[0085] The proposed test framework can be divided into at least three phases as discussed herein.

[0086] Data collection / ground truth extraction phase:

[0087] To collect the test data at the test equipment for generating the testing dataset and validating the UE predicted beam against the true value. Ensure the consistency (power levels, Angles of Arrival, propagation delays, etc.) between different beams transmitted to a device under test, or a UE:• Consistency may be achieved via using system-level simulator as a source of generated data,• It can also be achieved by creating a large dataset that would be used for both training and inference stages.

[0088] Training Phase (Optional):

[0089] To ensure that the device under test (DUT) is trained for the test data and can make correct or meaningful predictions for the Set A beams emulated by the test system by using the measurements on Set B beams emulated by the same test system. AI / ML model is trained (re-trained or fine-tuned) based on the data provided by the TE, if a generic beam prediction model deployed in the UE cannot be used directly in the test.

[0090] Inference phase:

[0091] To verify whether the best beam predicted by the DUT is same or close enough to the true best beam. Based on the signals transmitted by the probes, the UE measures RSRPs of the signals and then makes the inference for Set A beams. The predictions are either compared to the expected best beams / measurements or comparedto the measurements periodically reported by the UE for the whole Set A (e.g. using the monitoring mechanism).

[0092] Note that even though the example embodiments of the invention are demonstrated on the example of AI / ML-bascd beam prediction use-case (i.e., when the beams belong to the same cell and LI measurements are performed), the same approach can be used also when the beams belong to different cells and different type of measurements (e.g., L3 measurements) are assumed.

[0093] In this section, there is provided a detailed description of the proposed framework in accordance with example embodiments of the invention for the conformance testing of AI / ML based beam management feature. The proposed framework enables:(1) the beam sweeping of Set A and / or Set B beams in an iterative manner, by using a combination of the available test probes and different UE orientations / rotations, during the RAN4 conformance testing using the OTA test chamber where single or multiple test probe(s) are available;(2) a method for the conformance testing of the AI / ML based beam management feature;(3) The proposed test framework for the AI / ML based beam management use-case can be divided into the following three phases:(1) Data collection / ground truth extraction phase,(2) Training phase, and(3) Inference / testing phase.

[0094] (1) Data collection / ground truth extraction phase:

[0095] The purpose of data collection / ground truth extraction phase is to collect the test data (e.g., UE RSRP measurements inside the test chamber) at the test equipment which will then be used for:I. Generating test dataset for: a. UE-side AI / ML functionality / model training for conformance test purposes, and / or b. Acquiring ground truth of Set A beams, e.g., RSRP of each beam in Set A for a given / fixed test configuration (fixed test probe power, fixed UE orientation, etc.), and / or c. Selecting Set B beams during the AI / ML beam management conformance testing.II. Validating the prediction output of the UE-side AI / ML functionality / model against the true value during the conformance testing.III. Ensure the consistency (power levels, Angles of Arrival, propagation delays, etc.) between different beams transmitted to a device under test, or a UE: a. Consistency may be achieved via using system-level simulator as a source of generated data, b. It can also be achieved by creating a large dataset that would be used for both training and inference stages.

[0096] (2) Training Phase (Optional):

[0097] The purpose of training phase is to ensure that the device under test (DUT) is trained for the test data and can make correct or meaningful predictions for the Set A beams emulated by the test system (Test Equipment + Channel Emulator + Test Chamber) by using the measurements on Set B beams emulated by the same test system. The training for the conformance test can be done in multiple ways, e.g.:I. Only by the UE vendors (without intervention from TE vendors), orII. Before starting testing phase by using UE and TE collaboration (as mentioned in the call flow), orIII. By a standardized dataset agreed in 3GPP (for example, RAN4 agreed testing dataset).

[0098] (3) Inference phase:

[0099] The purpose of inference phase is to verify whether the best beam predicted by the UE side AI / ML based beam management functionality / model is same (or close enough) as the true best beam or not. If the best beam predicted by the DUT is same or close enough to the true best beam for that test configuration, the corresponding test iteration is passed otherwise is declared failed.

[0100] Test System Assumptions:1. The channel model to be used during the test is additive white gaussian noise (AWGN) model, i.e., no fading is assumed;2. The OTA test chamber may be equipped with either a single test probe or multiple test probes (e.g., 2 or 4 or 6 test probes). The example of an OTA test chamber set up with 4 test probes is shown in FIG. 3;3. During the test, one beam is generated by using only one of the available test probe(s).

[0101] Depending upon the test probe characteristics (whether each test probe is single or dual polarized), each test probe can generate either one beam (if the probe is single polarized) or 2 beams (if the probe is dual polarized).

[0102] However, without loss of generality, it can be assumed that in accordance with example embodiments of the invention each test probe can generate only one beam at a time.

[0103] FIG. 3 shows an OTA test setup with four test probes. As shown in FIG. 3 there is a system simulator / test equipment (SS / TE) communicating with a channel emulator (by varying power levels) that is communicating with an OTA test chamber for a UE that is using a test probe to each of multiple beams. Also as shown in FIG. 3 the UE is performing user plane rotations.

[0104] Signaling Call Flow:

[0105] Example Signaling Call Flow of the proposed solution is provided in FIG. 4 and FIG. 5.

[0106] The details of the steps 1-36 of FIG. 5 and FIG. 6 are provided below.

[0107] Note that the Steps from 1 to 20 can be placed in a different (e.g., preparational phase) of testing in this phase may be even optional. In the latter case, the test itself will start from the Inference phase from Step 21.

[0108] Steps 1-12 of FIG.4: AI / ML based BM use case conformance testing - Data Collection PhaseIt is noted that novel steps in accordance with example embodiments of the invention are marked with an asterisk in FIG. 4. These novel steps include steps 1-6, 12, and 13- 20 of FIG. 4.Step 1. TE transmits the test command to the DUT to enable / activate the Test Mode for “Data Collection” phase of AI / ML based BM use case;Step 2. DUT enables / activates the Test Mode for Data Collection phase;Step 3. DUT transmits an acknowledgement message to the TE confirming “Data Collection Test Mode Activated” at the DUT;Step 4. TE transmits a test mode command to the PUT providing Test Configuration to perform beam sweeping in multiple iterations and measure the RSRP of the corresponding beams;Step 5. DUT configures itself to perform the beam sweeping in multiple iterations;Step 6. DUT transmits an acknowledgement message to the TE confirming that the DUT is ready to perform beam sweeping in multiple iterations:TE starts configuring the test setup to generate dataset (e,g„ beam ID. Ll-RSRP) of Set A beams by using different combinations of test probes power levels and UE orientations;Repeat Steps 7-12 ‘N’ times where N is the ceiling function or smallest integer function of the ratio of the number of beams in Set A to the total number of test probes available in the test chamber;Step 7. TE selects the power level of each of the Test Probe(s), i.e., Pl, P2, P3, P4 (if there are 4 test probes in the test chamber);Step 8. TE decides the UE Orientation or a test-point over the 3D test sphere, i.e., one pair of theta and phi (not previously used) and fix the DUT in the corresponding orientation;Step 9. TE transmit four beams sequentially (each from a different test-probe but one at a time);Step 10. UE measures the RSRP of four beams sequentially;Step 11. DUT reports the RSRP of the measured four beams back to the TE / NW ;Step 12. TE stores the RSRP values of four beams reported by the DUT along with their corresponding beam IDs, Pl, P2, P3, P4, Theta and Phi values for dataset generation.

[0109] Data Collection Phase is complete, and Dataset is ready for Training / Inference.

[0110] Note that:1) The explanation of TE configuring the test setup to generate Set A beams by using different combinations of test probes power levels and UE orientations is as follows: Let us assume that there are 16 beams in Set A and 4 test probes in the test chamber, then 16 beams can be emulated in the test set up by using all 4 test probes for each fixed UE orientation to emulate 4 different Tx beams and then UE orientation is changed 16 / 4 = 4 times to emulate a total of 4*4 = 16 Set A beams,2) In different iterations, probes will represent beams with different TCI states. For example, if there are 4 probes, then probe 1 may represent TCI state 1 in first iteration while it will represent TCI state 5 in second iteration,3) It is worth mentioning that changing the UE orientation actually changes the UE’s Rx beam while the effect of Tx beam is kept constant. However, it is mathematically correct to say that the effect of change in UE orientation on the RSRP will be same as corresponding change in the Tx beam while assuming the Rx beam to be the same,4) In step 7 and 8 above, in one example the power levels, Theta and Phi values follow the output of a system level simulation scenario.

[0111] In the example dataset as shown in FIG. 6, the power levels of the testprobes are kept constant for each row and each beam-sweeping iteration has a different theta and phi values.

[0112] Steps 13-20 of FIG. 4: AI / ML based BM use case conformance testing - Training Phase (Optional)Step 13. TE transmits the test command to the DUT to enable / activate the Test Mode for the “Training Phase” of the AI / ML based BM use case;Step 14. DUT enables / activates the Test Mode for Training phase;Step 15. DUT transmits an acknowledgement message to the TE confirming “Training Test Mode Activated” at the DUT;Step 16. TE transmits a test mode command to the DUT providing Test Configuration to perform AI / ML model training during / before conformance testing;Step 17. DUT configures itself to perform its AI / ML model / functionality training using testing dataset during / before the conformance testing;Step 18. DUT transmits an acknowledgement message to the TE confirming that the DUT is ready to train its AI / ML model / functionality;Step 19. TE transmits the training data;Step 20. UE trains its model using the training data provided by the TE.Training Phase is complete, and DUT is trained over Testing Dataset to perform Inference during the conformance testing.

[0113] It is Noted that: As explained in the start of Section 7, the training for the conformance test can be done in multiple ways, e.g.:I. Only by the UE vendors (without intervention from TE vendors), orII. Before starting testing phase by using UE and TE collaboration (as mentioned in the call flow), orIII. By a standardized dataset agreed in 3GPP (for example, RAN4 agreed testing dataset).

[0114] Steps 21-36 of FIG. 5; AI / ML based BM use case conformance testing - Inference PhaseIt is noted that some novel steps in accordance with example embodiments of the Invention are marked with asterisks in FIG. 5. These novel steps include steps 21-26, 32, and 36 of FIG. 5.Step 21. TE transmits the test command to the DUT to enable / activate Test Mode for Inference phase of AI / ML based BM use case;Step 22. DUT enables / activates the Test Mode for Inference phase;Step 23. DUT transmits an acknowledgement message to the TE confirming “Inference Test Mode Activated” at the DUT;Step 24. TE transmits a test mode command to the DUT providing Test Configuration to perform beam sweeping in multiple iterations and measure the RSRP of the corresponding beams;Step 25. DUT configures itself to perform beam sweeping in multiple iterations;Step 26. DUT transmits an acknowledgement message to the TE confirming that the DUT is ready to perform beam sweeping in multiple iterations.TE starts configuring the test setup to generate Set B beams so that the DUT can predict best Set A beam (RSRP, beam ID, etc.) using RSRP measurements of SetB beams generated using different combinations of test probes power levels and UE orientations;Repeat Steps 27-32 ‘M’ times where M is the ceiling function or smallest integer function of the ratio of the number of beams in Set B to the total number of test probes available in the test chamber;Step 27. TE selects the power level of each of the Test Probe(s), i.e., Pl, P2, P3, P4 (if there are 4 test probes in the test chamber);Step 28. TE decides the UE Orientation or a test-point over the 3D test sphere, i.e., one pair of theta and phi (not previously used) and fix the DUT in the corresponding orientation;TE will randomly select Set B beams from the generated dataset and will generate only Set B in the testing chamber;Step 29. TE transmit four beams sequentially (each from a different testprobe but one at a time);Step 30. UE measures the RSRP of four beams sequentially;Step 31. (Optional) DUT reports the RSRP of the measured four beams back to the TE / NW;Step 32. (Optional) TE validates the RSRP values of the four beams reported by the DUT and their corresponding beam IDs, Pl, P2, P3, P4, Theta and Phi values using the dataset generated during the “Data Collection” phase;Step 33. DUT has measured RSRP of each beam in Set B and provides it as input to its AI / ML model for best Set A beam prediction;Step 34. DUT predicts the best beam and indicates the characteristics of the predicted best beam e.g., RSRP and / or beam ID, etc. to the TE / NW for validation;Step 35. TE / NW verifies whether the best UE predicted beam is same or close enough to the true best beam or not;Step 36. If the best beam predicted by the DUT’s AI / ML model / functionality is close enough to the true best beam in the dataset, the validation is successful, and the corresponding test iteration is marked as pass otherwise it is marked as fail.

[0115] Inference Phase is complete, and Test Result / Verdict is declared.

[0116] It is Noted that:1. Similar to the note in “Data Collection” phase, let us assume that there are 8 beams in Set B and 4 test probes in the test chamber, then 8 Set B beams can be emulated in the test set up by using all 4 test probes for each fixed UE orientation to emulate 4 different Tx beams and then UE orientation is changed 8 / 4 = 2 times to emulate a total of 4*2 = 8 Set B beams;2. In the context of example embodiments of the invention, if the predicted best beam’s RSRP is within X dB from the RSRP of the true best beam in the dataset, it can be called “close enough”. The value of X can be defined for example by RAN4, for example, by using simulation results.

[0117] FIG. 8A and FIG. 8B each show a method in accordance with example embodiments of the invention which may be performed by an apparatus.

[0118] FIG. 8A illustrates operations which may be performed by a device such as, but not limited to, a device under test or a second apparatus (e.g., the UE 10 as in FIG. 7). As shown in block 810 of FIG. 8A there is receiving from a second apparatusof a communication network information comprising a test command to enable or activate at least one of a training or inference test mode for conformance testing of a machine learning model or functionality based beam management use case during the at least one of a training or inference phase. As shown in block 820 of FIG. 8 A there is, based on the information, communicating an acknowledgement message with the second apparatus confirming an activation of the at least one of a training or inference test mode for the at least one of a training or inference phase. Then as shown in block 830 of FIG. 8A wherein the activation is using a configuration to perform the at least one of a training or inference validation of the machine learning model or functionality of the first apparatus using a testing dataset during the conformance testing.

[0119] In accordance with the example embodiments as described in the paragraph above, wherein the information comprises a test command to enable or activate the at least one of a training or inference test mode during or before conformance testing of a machine learning based beam management use case.

[0120] In accordance with the example embodiments as described in the paragraphs above, wherein there is, based on training data using at least one subset of at least one dataset, train the machine learning model or functionality of the apparatus for beam management use case to predict at least a reference signal received power of a best or all beams in a Set A beams, based on the measured reference signal received power of the beams in Set B beams, wherein the at least one dataset is for first apparatus side artificial intelligence or machine learning functionality or model training for conformance test purposes.

[0121] In accordance with the example embodiments as described in the paragraphs above, wherein there is accumulating several measurements, wherein the measurements use M*P, and wherein P is a number of probes, before performing at least one of inference or prediction.

[0122] In accordance with the example embodiments as described in the paragraphs above, wherein the measurements are distributed in time.

[0123] In accordance with the example embodiments as described in the paragraphs above, wherein there is activating the test mode for the inference phase;sending towards the second apparatus an acknowledgement confirming that the test mode for the inference phase is activated; receiving from the second apparatus a test mode command providing a test configuration to perform beam sweeping in one or more iterations and measure a reference signal received power of any corresponding beams; based on the test configuration, performing beam sweeping in one or more iterations; and transmitting an acknowledgement message to the second apparatus confirming readiness to perform beam sweeping in one or more iterations.

[0124] In accordance with the example embodiments as described in the paragraphs above, wherein there is receiving from the second apparatus a set of beams (Set B), wherein each beam from the set of beams is transmitted in a sequential manner, and wherein each beam of the set of beams is transmitted from a unique combination of test probes and first apparatus orientation; and measuring a reference signal received power of each beam of the set of beams (Set B).

[0125] In accordance with the example embodiments as described in the paragraphs above, wherein there is measuring a reference signal received power of each beam of the set of beams (Set B), wherein the set of beams (Set B) is a subset of set A beams; providing as input to the machine learning model or functionality model for at least a best beam prediction from the set A beams; and reporting at least a best Set A beam, wherein the reporting comprises an indication of characteristics of the at least best beam prediction, further comprising at least a beam identification, and / or a reference signal received power of the best predicted beam.

[0126] In accordance with the example embodiments as described in the paragraphs above, wherein the indication provides a reference signal received power and a beam identification of the best predicted beam to the second apparatus for validation.

[0127] In accordance with the example embodiments as described in the paragraphs above, wherein there is providing as input to the machine learning model or functionality for a best Set A beam prediction, wherein a predicted best beam and characteristics of the predicted best beam are indicated to the second apparatus.

[0128] In accordance with the example embodiments as described in the paragraphs above, wherein the apparatus is embodied in a device under test.

[0129] A non-transitory computer-readable medium (MEM 10B as in FIG. 6) storing program code (PROG 10C as in FIG. 6), the program code executed by at least one processor (DP 10A as in FIG. 6) to perform the operations as at least described in the paragraphs above.

[0130] In accordance with an example embodiment of the invention as described above there is an apparatus comprising: means for receiving (one or more transceivers 10D; MEM 10B; PROG 10C; and DP 10A as in FIG. 7) from a second apparatus (NN 12 and / or NN13 as in FIG. 7) of a communication network (Network 1 as in FIG. 7) information comprising a test command to enable or activate at least one of a training or inference test mode for conformance testing of a machine learning model or functionality based beam management use case during the at least one of a training or inference phase; means, based on the information, for communicating (one or more transceivers 10D; MEM 10B; PROG 10C; and DP 10A as in FIG. 7) an acknowledgement message with the second apparatus confirming an activation of the at least one of a training or inference test mode for the at least one of a training or inference phase; wherein the activation is using a configuration to perform the at least one of a training or inference validation of the machine learning model or functionality of the first apparatus using a testing dataset during the conformance testing.

[0131] In the example aspects according to the paragraph above, wherein at least the means for receiving and communicating comprises a non-transitory computer readable medium [MEM 10B] encoded with a computer program [PROG 10C] executable by at least one processor [DP 10A].

[0132] FIG. 8B illustrates operations which may be performed by a device such as, but not limited to, a test equipment or a second apparatus (e.g., the NN12 and / or NN13 as in FIG. 7). As shown in block 850 of FIG. 8B there is send towards a second apparatus of a communication network information comprising a test command to enable or activate at least one of a training or inference test mode at the secondapparatus for conformance testing of a machine learning model or functionality based beam management use case during the at least one of a training or inference phase. As shown in block 860 of FIG. 8B wherein the test command for data collection is to enable or activate a test mode at the second apparatus for a data collection phase during the conformance testing. Then as shown in block 870 of FIG. 8B wherein the activation is using a configuration to perform the at least one of a training or inference validation of the machine learning model or functionality of the first apparatus using a testing dataset during the conformance testing.

[0133] In accordance with the example embodiments as described in the paragraph above, wherein the information comprises a test command to enable or activate the at least one of a training or inference test mode during or before conformance testing of a machine learning based beam management use case.

[0134] In accordance with the example embodiments as described in the paragraphs above, wherein there is communicating with the second apparatus training data, using the at least one subset of at least one dataset, for the training of second apparatus side machine learning model or functionality to perform inference during the conformance testing.

[0135] In accordance with the example embodiments as described in the paragraphs above, wherein the training data is for use to train the machine learning model or functionality of the second apparatus for beam management use case to predict at least a reference signal received power of a best or all beams in a Set A beams, based on the measured reference signal received power of the beams in Set B beams wherein the at least one dataset is for second apparatus side artificial intelligence or machine learning functionality or model training for conformance test purposes.

[0136] In accordance with the example embodiments as described in the paragraphs above, wherein there is performing operations comprising: selecting a power level for each of at least one combination of test probes in a test chamber; determining an orientation or a test-point over a 3D test sphere and fix the second apparatus in a corresponding orientation; and utilizing a plurality of beams sequentially,wherein each beam is utilized from a different combination of test-probe and second apparatus orientation, one at a time, and wherein the second apparatus measures a reference signal received power of the plurality of beams sequentially.

[0137] In accordance with the example embodiments as described in the paragraphs above, wherein there is configuring the different combination of test-probe and second apparatus orientation to generate set B beams for the second apparatus to predict at least the RSRP of the best or all of the Set A beam(s) using the reference signal received power and beam identification of Set B beams generated using different combinations of different power levels of the different combination of test -probe and second apparatus orientations^wherein the best or all of the SET A beams are predicted but less than all of the SET A beams are transmitted to the second apparatus.

[0138] In accordance with the example embodiments as described in the paragraphs above, wherein the operations are performed M times, wherein M is a ceiling function or smallest integer function of a ratio of a number of beams in the set of beams (Set B) to a total number of a combination of test probes available in the test chamber, and wherein M is an positive integer.

[0139] In accordance with the example embodiments as described in the paragraphs above, wherein there is accumulating several measurements, wherein the measurements use M*P, and wherein P is a number of probes, before performing at least one of inference or prediction.

[0140] In accordance with the example embodiments as described in the paragraphs above, wherein the measurements are distributed in time.

[0141] In accordance with the example embodiments as described in the paragraphs above, wherein the apparatus is embodied in a test equipment.

[0142] In accordance with the example embodiments as described in the paragraphs above, wherein there is receiving from the second apparatus an acknowledgement confirming that the test mode for the inference phase is activated;sending towards the second apparatus a test mode command providing a test configuration to perform beam sweeping in one or more iterations and measure a reference signal received power of any corresponding beams, wherein based on the test configuration there is beam sweeping in one or more iterations; and receiving from the second apparatus an acknowledgement message confirming readiness to perform beam sweeping in one or more iterations at the second apparatus.

[0143] In accordance with the example embodiments as described in the paragraphs above, wherein there is sending towards the second apparatus a set of beams, wherein the set of beams is sequential, wherein each beam of the set of beams is from a different combination of test probes and second apparatus orientations, and wherein based on the set of beams, a reference signal received power of each beam of the set of beams is measured by the second apparatus to perform machine learning based beam prediction.

[0144] In accordance with the example embodiments as described in the paragraphs above, wherein there is selecting a power level of each test probe and select an orientation of the second apparatus for each of the different combination of test probes and second apparatus orientations over the 3D test sphere, wherein selecting an orientation of the second apparatus comprising toning pair of theta and phi and fixing the second apparatus in a corresponding orientation.

[0145] In accordance with the example embodiments as described in the paragraphs above, wherein there is randomly selecting Set B beams from a generated dataset and generate only a set of beams (Set B) in a testing chamber; transmitting a subset of beams sequentially, wherein each of the beam in the subset is transmitted from a different test-probe and sent one at a time.

[0146] In accordance with the example embodiments as described in the paragraphs above, wherein there is receiving from the second apparatus a report indicating at least a best Set A beam, wherein the reporting comprises an indication of characteristics of the at least best predicted beam, comprising at least a beam identification, and / or a reference signal received power of the at least best beam, Inaccordance with the example embodiments as described in the paragraphs above, wherein the report is based on a predicted output the second apparatus based on the reference signal received power of each beam of the set of beams (Set B), wherein the set of beams (Set B) is a subset of set A beams.

[0147] In accordance with the example embodiments as described in the paragraphs above, wherein the indication provides at least one of the reference signal received power or a beam identification to the second apparatus for inference validation.

[0148] In accordance with the example embodiments as described in the paragraphs above, wherein there is determining information comprising an input to the machine learning model or functionality of the second apparatus for at least a best Set A beam prediction, wherein at least a best beam is predicted and characteristics of the predicted best beam are indicated to the apparatus.

[0149] In accordance with the example embodiments as described in the paragraphs above, wherein there is, based on the information, verifying whether at least the best set A beam prediction is same or close enough to a true best beam.

[0150] In accordance with the example embodiments as described in the paragraphs above, wherein there is, based on the best set A beam prediction being same as the true best beam, determining that true best beam validation is successful and test iteration has passed.

[0151] In accordance with the example embodiments as described in the paragraphs above, wherein there is based on the best set A beam prediction not being same as the true best beam, determine that true best beam validation and test iteration was not successful and has failed.

[0152] A non-transitory computer-readable medium (MEM 12B and / or MEM 13B as in FIG. 7) storing program code (PROG 12C and / or PROG 13C as in FIG. 7),the program code executed by at least one processor (DP 12A and / or DP 13A as in FIG. 7) to perform the operations as at least described in the paragraphs above.

[0153] In accordance with an example embodiment as described above there is an apparatus comprising: means for sending (TRANS 12D and / or TRANS 13D; MEM 12B and / or MEM 13B, PROG 12C and / or PROG 13C, and DP 12A and / or DP 13A as in FIG. 7) towards a second apparatus (UE 10 as in FIG. 7) of a communication network (Network 1 as in FIG. 7) information comprising a test command to enable or activate at least one of a training or inference test mode at the second apparatus for conformance testing of a machine learning model or functionality based beam management use case during the at least one of a training or inference phase; and means for receiving (TRANS 12D and / or TRANS 13D; MEM 12B and / or MEM 13B, PROG 12C and / or PROG 13C, and DP 12A and / or DP 13A as in FIG. 7) from the second apparatus of the communication network information comprising an acknowledgement message confirming an activation of the at least one of a training or inference test mode for the at least one of a training or inference phase, wherein the activation is using a configuration to perform the at least one of a training or inference validation of the machine learning model or functionality of the first apparatus using a testing dataset during the conformance testing.

[0154] In the example aspects according to the paragraph above, wherein at least the means for sending, and receiving comprises a non-transitory computer readable medium [MEM 12B and / or MEM 13B] encoded with a computer program [PROG 12C and / or PROG 13C] executable by at least one processor [DP 12A and / or DP 13A],

[0155] It is noted that computer-implemented inventions (CII) may be claimed as apparatus claims, method claims, and software claims. In some jurisdictions, such as in Europe, signal claims can also be made. In the U.S., a software claim must be claimed as a non-transitory computer program product or a non-transitory computer readable medium.

[0156] The term “non-transitory,” as may be used herein, is a limitation of the medium itself (i.e., tangible, not a signal) as opposed to a limitation on data storage persistency (e.g., RAM vs. ROM).

[0157] In other jurisdictions, a software claim can be claimed as a computer program, a data structure, and / or a computer readable medium.

[0158] Further, in accordance with example embodiments of the invention there is circuitry for performing operations in accordance with example embodiments of the invention as disclosed herein. This circuitry can include any type of circuitry including content coding circuitry, content decoding circuitry, processing circuitry, image generation circuitry, data analysis circuitry, etc.). Further, this circuitry can include discrete circuitry, application-specific integrated circuitry (ASIC), and / or field- programmable gate array circuitry (FPGA), etc. as well as a processor specifically configured by software to perform the respective function, or dual-core processors with software and corresponding digital signal processors, etc.). Additionally, there are provided necessary inputs to and outputs from the circuitry, the function performed by the circuitry and the interconnection (perhaps via the inputs and outputs) of the circuitry with other components that may include other circuitry in order to perform example embodiments of the invention as described herein.

[0159] In accordance with example embodiments of the invention as disclosed in this application this application, the “circuitry” provided can include at least one or more or all of the following:(a) hardware-only circuit implementations (such as implementations in only analog and / or digital circuitry);(b) combinations of hardware circuits and software, such as (as applicable):(i) a combination of analog and / or digital hardware circuit(s) with software / firmware; and(ii) any portions of hardware processor(s) with software (including digital signal processor(s)), software, and memory(ies) that work together to cause an apparatus, such as a mobile phone or server, to perform various functions, such as functions or operations in accordance with example embodiments of the invention as disclosed herein); and(c) hardware circuit(s) and or processor(s), such as a microprocessor(s) or a portion of a microprocessor(s), that requires software (e.g., firmware) for operation, but the software may not be present when it is not needed for operation.”

[0160] In accordance with example embodiments of the invention, there is adequate circuitry for performing at least novel operations in accordance with example embodiments of the invention as disclosed in this application, this 'circuitry' as may be used herein refers to at least the following:(a) hardware-only circuit implementations (such as implementations in only analog and / or digital circuitry); and(b) to combinations of circuits and software (and / or firmware), such as (as applicable): (i) to a combination of processor(s) or (ii) to portions of processor(s) / software (including digital signal processor(s)), software, and memory(ies) that work together to cause an apparatus, such as a mobile phone or server, to perform various functions); and(c) to circuits, such as a microprocessor(s) or a portion of a microprocessor(s), that require software or firmware for operation, even if the software or firmware is not physically present.

[0161] This definition of 'circuitry' applies to all uses of this term in this application, including in any claims. As a further example, as used in this application, the term "circuitry" would also cover an implementation of merely a processor (or multiple processors) or portion of a processor and its (or their) accompanying software and / or firmware. The term "circuitry" would also cover, for example and if applicable to the particular claim element, a baseband integrated circuit or applications processorintegrated circuit for a mobile phone or a similar integrated circuit in a server, a cellular network device, or other network device.

[0162] In general, the various embodiments may be implemented in hardware or special purpose circuits, software, logic or any combination thereof. For example, some aspects may be implemented in hardware, while other aspects may be implemented in firmware or software which may be executed by a controller, microprocessor or other computing device, although the invention is not limited thereto. While various aspects of the invention may be illustrated and described as block diagrams, flow charts, or using some other pictorial representation, it is well understood that these blocks, apparatus, systems, techniques or methods described herein may be implemented in, as non-limiting examples, hardware, software, firmware, special purpose circuits or logic, general purpose hardware or controller or other computing devices, or some combination thereof.

[0163] Embodiments of the inventions may be practiced in various components such as integrated circuit modules. The design of integrated circuits is by and large a highly automated process. Complex and powerful software tools are available for converting a logic level design into a semiconductor circuit design ready to be etched and formed on a semiconductor substrate.

[0164] The word "exemplary" is used herein to mean "serving as an example, instance, or illustration." Any embodiment described herein as "exemplary" is not necessarily to be construed as preferred or advantageous over other embodiments. All of the embodiments described in this Detailed Description are exemplary embodiments provided to enable persons skilled in the art to make or use the invention and not to limit the scope of the invention which is defined by the claims.

[0165] The foregoing description has provided by way of exemplary and nonlimiting examples a full and informative description of the best method and apparatus presently contemplated by the inventors for carrying out the invention. However, various modifications and adaptations may become apparent to those skilled in the relevant arts in view of the foregoing description, when read in conjunction with the accompanying drawings and the appended claims. However, all such and similarmodifications of the teachings of example embodiments of this invention will still fall within the scope of this invention.

[0166] It should be noted that the terms "connected," "coupled," or any variant thereof, mean any connection or coupling, either direct or indirect, between two or more elements, and may encompass the presence of one or more intermediate elements between two elements that are "connected" or "coupled" together. The coupling or connection between the elements can be physical, logical, or a combination thereof. As employed herein two elements may be considered to be "connected" or "coupled" together by the use of one or more wires, cables and / or printed electrical connections, as well as by the use of electromagnetic energy, such as electromagnetic energy having wavelengths in the radio frequency region, the microwave region and the optical (both visible and invisible) region, as several non-limiting and non-exhaustive examples.

[0167] Furthermore, some of the features of the preferred embodiments of this invention could be used to advantage without the corresponding use of other features. As such, the foregoing description should be considered as merely illustrative of the principles of the invention, and not in limitation thereof.

Claims

CLAIMSWhat is claimed is:

1. An apparatus, comprising: at least one processor; and at least one memory storing instructions, that when executed by the at least one processor, cause the apparatus at least to: receive from a second apparatus of a communication network information comprising a test command to enable or activate at least one of a training or inference test mode for conformance testing of a machine learning model or functionality based beam management use case during the at least one of a training or inference phase; and based on the information, communicate an acknowledgement message with the second apparatus confirming an activation of the at least one of a training or inference test mode for the at least one of a training or inference phase; wherein the activation is using a configuration to perform the at least one of a training or inference validation of the machine learning model or functionality of the first apparatus using a testing dataset during the conformance testing.

2. The apparatus of claim 1, wherein the information comprises a test command to enable or activate the at least one of a training or inference test mode during or before conformance testing of a machine learning based beam management use case.

3. The apparatus of claim 1, wherein the at least one memory is storing instructions, that when executed by the at least one processor, cause the apparatus at least to: based on training data using at least one subset of at least one dataset, train the machine learning model or functionality of the apparatus for beam management use case to predict at least a reference signal received power of a best or all beams in a Set A beams, based on the measured reference signal received power of the beams in Set B beams,wherein the at least one dataset is for first apparatus side artificial intelligence or machine learning functionality or model training for conformance test purposes.

4. The apparatus of claim 1 , wherein the at least one memory is storing instructions, that when executed by the at least one processor, cause the apparatus at least to: accumulate several measurements, wherein the measurements use M*P, and wherein P is a number of probes, before performing at least one of inference or prediction.

5. The apparatus of claim 4, wherein the measurements are distributed in time.

6. The apparatus of claim 1, wherein the at least one processor; and the at least one memory is storing instructions, that when executed by the at least one processor, cause the apparatus at least to: activate the test mode for the inference phase; send towards the second apparatus an acknowledgement confirming that the test mode for the inference phase is activated; receive from the second apparatus a test mode command providing a test configuration to perform beam sweeping in one or more iterations and measure a reference signal received power of any corresponding beams; based on the test configuration, perform beam sweeping in one or more iterations; and transmits an acknowledgement message to the second apparatus confirming readiness to perform beam sweeping in one or more iterations.

7. The apparatus of claim 6, wherein the at least one processor; and the at least one memory is storing instructions, that when executed by the at least one processor, cause the apparatus at least to: receive from the second apparatus a set of beams (Set B), wherein each beam from the set of beams is transmitted in a sequential manner, andwherein each beam of the set of beams is transmitted from a unique combination of test probes and first apparatus orientation; and measure a reference signal received power of each beam of the set of beams (Set B).

8. The apparatus of claim 6, wherein the at least one processor; and the at least one memory is storing instructions, that when executed by the at least one processor, cause the apparatus at least to: measure a reference signal received power of each beam of the set of beams (Set B), wherein the set of beams (Set B) is a subset of set A beams; provide as input to the machine learning model or functionality model for at least a best beam prediction from the set A beams; and report at least a best Set A beam, wherein the reporting comprises an indication of characteristics of the at least best beam prediction, further comprising at least one of a beam identification, or a reference signal received power of the best predicted beam.

9. The apparatus of claim 8, wherein the indication provides a reference signal received power and a beam identification of the best predicted beam to the second apparatus for validation.

10. The apparatus of claim 6, wherein the at least one processor; and the at least one memory is storing instructions, that when executed by the at least one processor, cause the apparatus at least to: provide as input to the machine learning model or functionality for a best Set A beam prediction, wherein a predicted best beam and characteristics of the predicted best beam are indicated to the second apparatus.

11. The apparatus of claim 1 , embodied in a device under test.

12. A method, comprising:receiving from a second apparatus of a communication network information comprising a test command to enable or activate at least one of a training or inference test mode for conformance testing of a machine learning model or functionality based beam management use case during the at least one of a training or inference phase; and based on the information, communicating an acknowledgement message with the second apparatus confirming an activation of the at least one of a training or inference test mode for the at least one of a training or inference phase; wherein the activation is using a configuration to perform the at least one of a training or inference validation of the machine learning model or functionality of the first apparatus using a testing dataset during the conformance testing.

13. An apparatus, comprising: at least one processor; and at least one memory storing instructions, that when executed by the at least one processor, cause the apparatus at least to: send towards a second apparatus of a communication network information comprising a test command to enable or activate at least one of a training or inference test mode at the second apparatus for conformance testing of a machine learning model or functionality based beam management use case during the at least one of a training or inference phase; and receive from the second apparatus of the communication network information comprising an acknowledgement message confirming an activation of the at least one of a training or inference test mode for the at least one of a training or inference phase, wherein the activation is using a configuration to perform the at least one of a training or inference validation of the machine learning model or functionality of the first apparatus using a testing dataset during the conformance testing.

14. The apparatus of claim 13, wherein the information comprises a test command to enable or activate the at least one of a training or inference test mode during or before conformance testing of a machine learning based beam management use case.

15. The apparatus of claim 13, wherein the at least one memory is storing instructions, that when executed by the at least one processor, cause the apparatus at least to: communicate with the second apparatus training data, using the at least one subset of at least one dataset, for the training of second apparatus side machine learning model or functionality to perform inference during the conformance testing.

16. The apparatus of claim 13, wherein the training data is for use to train the machine learning model or functionality of the second apparatus for beam management use case to predict at least a reference signal received power of a best or all beams in a Set A beams, based on the measured reference signal received power of the beams in Set B beams wherein the at least one dataset is for second apparatus side artificial intelligence or machine learning functionality or model training for conformance test purposes.

17. The apparatus of claim 13, wherein the at least one memory is storing instructions, that when executed by the at least one processor, cause the apparatus at least to: perform operations comprising: selecting a power level for each of at least one combination of test probes in a test chamber; determining an orientation or a test-point over a 3D test sphere and fix the second apparatus in a corresponding orientation; and utilizing a plurality of beams sequentially, wherein each beam is utilized from a different combination of test-probe and second apparatus orientation, one at a time, and wherein the second apparatus measures a reference signal received power of the plurality of beams sequentially.

18. The apparatus of claim 13, wherein the at least one memory is storing instructions, that when executed by the at least one processor, cause the apparatus at least to: configure the different combination of test-probe and second apparatus orientation to generate set B beams for the second apparatus to predict at least theRSRP of the best or all of the Set A beam(s) using the reference signal received power and beam identification of Set B beams generated using different combinations of different power levels of the different combination of test-probe and second apparatus orientations^ wherein the best or all of the SET A beams are predicted but less than all of the SET A beams are transmitted to the second apparatus.

19. The apparatus of claim 13, wherein the operations are performed M times, wherein M is a ceiling function or smallest integer function of a ratio of a number of beams in the set of beams (Set B) to a total number of a combination of test probes available in the test chamber, and wherein M is an positive integer.

20. The apparatus of claim 13, wherein the at least one memory is storing instructions, that when executed by the at least one processor, cause the apparatus at least to: accumulate several measurements, wherein the measurements use M*P, and wherein P is a number of probes, before performing at least one of inference or prediction.

21. The apparatus of claim 13, wherein the measurements are distributed in time.

22. The apparatus of claim 13, embodied in a test equipment.

23. The apparatus of claim 13, wherein the at least one processor; and the at least one memory is storing instructions, that when executed by the at least one processor, cause the apparatus at least to: receive from the second apparatus an acknowledgement confirming that the test mode for the inference phase is activated; send towards the second apparatus a test mode command providing a test configuration to perform beam sweeping in one or more iterations and measure a reference signal received power of any corresponding beams,wherein based on the test configuration there is beam sweeping in one or more iterations; and receive from the second apparatus an acknowledgement message confirming readiness to perform beam sweeping in one or more iterations at the second apparatus.

24. The apparatus of claim 13, wherein the at least one processor; and the at least one memory is storing instructions, that when executed by the at least one processor, cause the apparatus at least to: send towards the second apparatus a set of beams, wherein the set of beams is sequential, wherein each beam of the set of beams is from a different combination of test probes and second apparatus orientations, and wherein based on the set of beams, a reference signal received power of each beam of the set of beams is measured by the second apparatus to perform machine learning based beam prediction.

25. The apparatus of claim 24, wherein the at least one processor; and the at least one memory is storing instructions, that when executed by the at least one processor, cause the apparatus at least to: select a power level of each test probe and select an orientation of the second apparatus for each of the different combination of test probes and second apparatus orientations over the 3D test sphere, wherein selecting an orientation of the second apparatus comprising toning pair of theta and phi and fixing the second apparatus in a corresponding orientation.

26. The apparatus of claim 25, wherein the at least one processor; and the at least one memory is storing instructions, that when executed by the at least one processor, cause the apparatus at least to: randomly select Set B beams from a generated dataset and generate only a set of beams (Set B) in a testing chamber;transmit a subset of beams sequentially, wherein each of the beam in the subset is transmitted from a different test -probe and sent one at a time.

27. The apparatus of claim 13, wherein the at least one processor; and the at least one memory is storing instructions, that when executed by the at least one processor, cause the apparatus at least to: receive from the second apparatus a report indicating at least a best Set A beam, wherein the reporting comprises an indication of characteristics of the at least best predicted beam, comprising at least a beam identification, and / or a reference signal received power of the at least best beam, wherein the report is based on a predicted output the second apparatus based on the reference signal received power of each beam of the set of beams (Set B), wherein the set of beams (Set B) is a subset of set A beams.

28. The apparatus of claim 27, wherein the indication provides at least one of the reference signal received power or a beam identification to the second apparatus for inference validation.

29. The apparatus of claim 27, wherein the at least one processor; and the at least one memory is storing instructions, that when executed by the at least one processor, cause the apparatus at least to: determine information comprising an input to the machine learning model or functionality of the second apparatus for at least a best Set A beam prediction, wherein at least a best beam is predicted and characteristics of the predicted best beam are indicated to the apparatus.

30. The apparatus of claim 27, wherein the at least one processor; and the at least one memory is storing instructions, that when executed by the at least one processor, cause the apparatus at least to: based on the information verify whether at least the best set A beam prediction is same or close enough to a true best beam.

31. The apparatus of claim 27, wherein the at least one processor; and the at least one memory is storing instructions, that when executed by the at least one processor, cause the apparatus at least to: based on the best set A beam prediction being same as the true best beam, determine that true best beam validation is successful and test iteration has passed.

32. The apparatus of claim 27, wherein the at least one processor; and the at least one memory is storing instructions, that when executed by the at least one processor, cause the apparatus at least to: based on the best set A beam prediction not being same as the true best beam, determine that true best beam validation and test iteration was not successful and has failed.

33. A method, comprising: sending towards a second apparatus of a communication network information comprising a test command to enable or activate at least one of a training or inference test mode at the second apparatus for conformance testing of a machine learning model or functionality based beam management use case during the at least one of a training or inference phase; and receiving from the second apparatus of the communication network information comprising an acknowledgement message confirming an activation of the at least one of a training or inference test mode for the at least one of a training or inference phase, wherein the activation is using a configuration to perform the at least one of a training or inference validation of the machine learning model or functionality of the first apparatus using a testing dataset during the conformance testing.

34. A computer program product comprising at least one non-transitory computer- readable storage medium having computer-executable program code instructions stored therein, the computer-executable program code instructions comprising program code instructions configured to: receive from a second apparatus of a communication network informationcomprising a test command to enable or activate at least one of a training or inference test mode for conformance testing of a machine learning model or functionality based beam management use case during the at least one of a training or inference phase; and communicate, based on the information, an acknowledgement message with the second apparatus confirming an activation of the at least one of a training or inference test mode for the at least one of a training or inference phase; wherein the activation is using a configuration to perform the at least one of a training or inference validation of the machine learning model or functionality of the first apparatus using a testing dataset during the conformance testing.

35. An apparatus, comprising: means for receiving from a second apparatus of a communication network information comprising a test command to enable or activate at least one of a training or inference test mode for conformance testing of a machine learning model or functionality based beam management use case during the at least one of a training or inference phase; and means for communicating, based on the information, an acknowledgement message with the second apparatus confirming an activation of the at least one of a training or inference test mode for the at least one of a training or inference phase; wherein the activation is using a configuration to perform the at least one of a training or inference validation of the machine learning model or functionality of the first apparatus using a testing dataset during the conformance testing.

36. A computer program product comprising at least one non-transitory computer- readable storage medium having computer-executable program code instructions stored therein, the computer-executable program code instructions comprising program code instructions configured to: send towards a second apparatus of a communication network information comprising a test command to enable or activate at least one of a training or inference test mode at the second apparatus for conformance testing of a machine learning model or functionality based beam management use case during the at least one of a training or inference phase; andreceive from the second apparatus of the communication network information comprising an acknowledgement message confirming an activation of the at least one of a training or inference test mode for the at least one of a training or inference phase, wherein the activation is using a configuration to perform the at least one of a training or inference validation of the machine learning model or functionality of the first apparatus using a testing dataset during the conformance testing.

37. An apparatus, comprising: means for sending towards a second apparatus of a communication network information comprising a test command to enable or activate at least one of a training or inference test mode at the second apparatus for conformance testing of a machine learning model or functionality based beam management use case during the at least one of a training or inference phase; and means for receiving from the second apparatus of the communication network information comprising an acknowledgement message confirming an activation of the at least one of a training or inference test mode for the at least one of a training or inference phase, wherein the activation is using a configuration to perform the at least one of a training or inference validation of the machine learning model or functionality of the first apparatus using a testing dataset during the conformance testing.

Images