UE special conformance testing function for ai / ML use cases and associated features

Abstract

A DUT, as part of conformance testing, receives indication to activate an AI / ML test function. The AI / ML test function includes indication of a use case, as one of multiple testing features associated with the AI / ML test function, that corresponds to one or both of an AI / ML model or functionality for which the conformance testing is performed. The DUT activates the AI / ML test function and associated one or more of the multiple testing features corresponding to the use case. The DUT performs, based on the activated AI / ML test function, the one or both of the AI / ML model or functionality for the indicated use case and associated testing features. A SS, as part of the conformance testing of the DUT, sends indication to the DUT to activate the AI / ML test function, receives indication that at least the AI / ML test function has been activated, and participates in the conformance testing of the DUT.

Description

UE SPECIAL CONFORMANCE TESTING FUNCTION FOR AI / ML USE CASES ANDASSOCIATED FEATURESTECHNICAL FIELD

[0001] Examples of embodiments herein relate generally to telecommunication systems, and, more specifically, relate to conformance testing for use in the telecommunication systems.BACKGROUND

[0002] In wireless systems such as cellular systems, devices that will be used in the system will undergo conformance testing to ensure that they meet conformance requirements defined by 3GPP (third generation partnership project). For instance, in a cellular system such as a 5G system (a fifth-generation system), a UE (user equipment, a wireless, typically mobile device) has to conform to and meet certain criteria. To support the conformance testing of 5GS, UE special conformance test functions are used. They form a part of the core requirements for a cellular system and thus have a direct impact on the design of the UE. The UE special conformance test functions vary depending on the conformance testing functionality they are designed to support, but UEs have to conform to criteria that have been set. While the UE special conformance test functions are important, these and other test functions can be improved.BRIEF SUMMARY

[0003] This section is intended to include examples and is not intended to be limiting.

[0004] In an exemplary embodiment, a method is disclosed that includes, by a device under test (DUT) as part of conformance testing of the DUT, receiving indication to activate an artificial intelligence / machine learning (AI / ML) test function for the conformance testing, the AI / ML test function comprising indication of a use case, as one of multiple testing features associated with the AI / ML test function, that corresponds to one or both of an AI / ML model or functionality for which the conformance testing is to be performed; activating, by the DUT, the AI / ML test function and associated one or more of the multiple testing features corresponding to the indicated use case; and performing, by the DUT based on the activated AI / ML test function,the one or both of the AI / ML model or functionality for the indicated use case and associated testing features as part of the conformance testing.

[0005] An additional exemplary embodiment includes a computer program, comprising instructions for performing the method of the previous paragraph, when the computer program is run on an apparatus. The computer program according to this paragraph, wherein the computer program is a computer program product comprising a computer-readable medium bearing the instructions embodied therein for use with the apparatus. Another example is the computer program according to this paragraph, wherein the program is directly loadable into an internal memory of the apparatus.

[0006] An exemplary apparatus includes one or more processors and one or more memories storing instructions that, when executed by the one or more processors, cause the apparatus at least to perform: by a device under test (DUT) as part of conformance testing of the DUT, receiving indication to activate an artificial intelligence / machine learning (AI / ML) test function for the conformance testing, the Al / ML test function comprising indication of a use case, as one of multiple testing features associated with the Al / ML test function, that corresponds to one or both of an Al / ML model or functionality for which the conformance testing is to be performed; activating, by the DUT, the AI / ML test function and associated one or more of the multiple testing features corresponding to the indicated use case; and performing, by the DUT based on the activated AI / ML test function, the one or both of the AI / ML model or functionality for the indicated use case and associated testing features as part of the conformance testing.

[0007] An exemplary computer program product includes a computer-readable storage medium bearing instructions that, when executed by an apparatus, cause the apparatus to perform at least the following: by a device under test (DUT) as part of conformance testing of the DUT, receiving indication to activate an artificial intelligence / machine learning (AI / ML) test function for the conformance testing, the AI / ML test function comprising indication of a use case, as one of multiple testing features associated with the AI / ML test function, that corresponds to one or both of an AI / ML model or functionality for which the conformance testing is to be performed; activating, by the DUT, the AI / ML test function and associated one or more of the multiple testing features corresponding to the indicated use case; and performing, by the DUT based on the activated AI / ML test function, the one or both of the AI / ML model or functionality for the indicated use case and associated testing features as part of the conformance testing.

[0008] In another exemplary embodiment, an apparatus comprises means for: by a device under test (DUT) as part of conformance testing of the DUT, receiving indication to activate an artificial intelligence / machine learning (AI / ML) test function for the conformance testing, the AI / ML test function comprising indication of a use case, as one of multiple testing features associated with the AI / ML test function, that corresponds to one or both of an AI / ML model or functionality for which the conformance testing is to be performed; activating, by the DUT, the AI / ML test function and associated one or more of the multiple testing features corresponding to the indicated use case; and performing, by the DUT based on the activated AI / ML test function, the one or both of the AI / ML model or functionality for the indicated use case and associated testing features as part of the conformance testing.

[0009] In an exemplary embodiment, a method is disclosed that includes, by a system simulator (SS) as part of conformance testing of a device under test (DUT), sending indication to the DUT to activate an artificial intelligence / machine learning (AI / ML) test function used for the conformance testing, the AI / ML test function comprising indication of a use case, as one of multiple testing features, that corresponds to one or both of an AI / ML model or functionality for which the conformance testing is to be performed; receiving, by the SS from the DUT, indication that at least the AI / ML test function has been activated; and participating, by the SS based on the activated AI / ML test function and in conjunction with the DUT, in the conformance testing of the DUT.

[0010] An additional exemplary embodiment includes a computer program, comprising instructions for performing the method of the previous paragraph, when the computer program is run on an apparatus. The computer program according to this paragraph, wherein the computer program is a computer program product comprising a computer-readable medium bearing the instructions embodied therein for use with the apparatus. Another example is the computer program according to this paragraph, wherein the program is directly loadable into an internal memory of the apparatus.

[0011] An exemplary apparatus includes one or more processors and one or more memories storing instructions that, when executed by the one or more processors, cause the apparatus at least to perform: by a system simulator (SS) as part of conformance testing of a device under test (DUT), sending indication to the DUT to activate an artificial intelligence / machine learning (AI / ML) test function used for the conformance testing, the AI / MLtest function comprising indication of a use case, as one of multiple testing features, that corresponds to one or both of an AI / ML model or functionality for which the conformance testing is to be performed; receiving, by the SS from the DUT, indication that at least the AI / ML test function has been activated; and participating, by the SS based on the activated Al / ML test function and in conjunction with the DUT, in the conformance testing of the DUT.

[0012] An exemplary computer program product includes a computer-readable storage medium bearing instructions that, when executed by an apparatus, cause the apparatus to perform at least the following: by a system simulator (SS) as part of conformance testing of a device under test (DUT), sending indication to the DUT to activate an artificial intelligence / machine learning (Al / ML) test function used for the conformance testing, the AI / ML test function comprising indication of a use case, as one of multiple testing features, that corresponds to one or both of an AI / ML model or functionality for which the conformance testing is to be performed; receiving, by the SS from the DUT, indication that at least the AI / ML test function has been activated; and participating, by the SS based on the activated AI / ML test function and in conjunction with the DUT, in the conformance testing of the DUT.

[0013] In another exemplary embodiment, an apparatus comprises means for: by a system simulator (SS) as part of conformance testing of a device under test (DUT), sending indication to the DUT to activate an artificial intelligence / machine learning (AI / ML) test function used for the conformance testing, the AI / ML test function comprising indication of a use case, as one of multiple testing features, that corresponds to one or both of an AI / ML model or functionality for which the conformance testing is to be performed; receiving, by the SS from the DUT, indication that at least the AI / ML test function has been activated; and participating, by the SS based on the activated AI / ML test function and in conjunction with the DUT, in the conformance testing of the DUT.BRIEF DESCRIPTION OF THE DRAWINGS

[0014] The accompanying drawings use reference numerals, where the same reference numerals may be used to refer to like parts throughout, but parts having the same reference numeral can differ in operation and components. In the attached drawings:

[0015] FIG. 1 is a signaling diagram illustrating OTA test commands for test loops;

[0016] FIG. 2 is an example illustrating an AI / ML test function and corresponding features;

[0017] FIG. 3 is a signaling diagram illustrating a UE AI / ML test mode activation procedure;

[0018] FIG. 4 is a signaling diagram illustrating a UE AI / ML test mode deactivation procedure;

[0019] FIG. 5 is a signaling diagram illustrating an OTA UE conformance test function -based example for the conformance testing of AI / ML use-cases / other testing features;

[0020] FIG. 6 is a signaling diagram illustrating an AT Command based example for the conformance testing of AI / ML use-cases / other testing features;

[0021] FIGS. 7A and 7B are block diagrams of possible and non-limiting exemplary systems in which the exemplary embodiments may be practiced, where FIG. 7 A is for a conducted mode conformance testing and FIG. 7B is for a radiated (OTA) mode conformance testing; and

[0022] FIG. 8 is a block diagram illustrating possible circuitry for the systems of FIGS. 7 A and 7B.DETAILED DESCRIPTION OF THE DRAWINGS

[0023] Abbreviations that may be found in the specification and / or the drawing figures are defined below, at the end of the detailed description section.

[0024] The word “exemplary” is used herein to mean “serving as an example, instance, or illustration.” Any embodiment described herein as “exemplary” is not necessarily to be construed as preferred or advantageous over other embodiments. All of the embodiments described in this Detailed Description are exemplary embodiments provided to enable persons skilled in the art to make or use the examples.

[0025] When more than one drawing reference numeral, word, or acronym is used within this description with “ / ”, and in general as used within this description, the “ / ” may be interpreted as “or”, “and”, or “both”. As used herein, “at least one of the following: ” and “at least one of ” and similar wording, where the list of two or more elements are joined by “and” or “or,” mean at least any one of the elements, or at least any two or more of the elements, or at least all the elements.

[0026] As used herein, the singular forms “a”, “an” and “the” are intended to include the plural forms as well, unless the context clearly indicates otherwise. It will be further understood that the terms “comprises”, “comprising”, “has”, “having”, “includes” and / or “including”, when used herein, specify the presence of stated features, elements, and / or components etc., but do not preclude the presence or addition of one or more other features, elements, components and / or combinations thereof.

[0027] It is noted that capital and lowercase words or phrases are considered to be the same herein. For instance, the words Slice, slice, and SLICE are the same, as are the phrases Network Repository Function, network repository function, and NETWORK REPOSITORY FUNCTION.

[0028] Any signaling diagram (e.g., FIGS. 1, 3, 5 and 6) herein is considered to be a logic flow diagram, and illustrates the operation of an exemplary method, results of execution of computer program instructions embodied on a computer readable memory, and / or functions performed by logic implemented in circuitry. For methods, flow diagrams, and signaling diagrams, the orders of method steps, blocks in the flow, or signaling are not critical and instead are examples.

[0029] Technical context is now provided for technical areas related to the understanding of the examples. The following uses a nested topical structure for ease of reference. This nested topical structure has the following key: I, II, III... are main categories; 1, 2, and 3... are subcategories; a, b, c... are further subcategories; and i, ii, iii... are even further subcategories. The wording used in the categories and subcategories is used to provide guidance as to the contents, but is not intended to be limiting.

[0030] I. UE (user equipment) Special Conformance Test Functions

[0031] To support the conformance testing of 5G systems (5GS), UE special conformance test functions are required. They form a part of the core requirements and thus have a direct impact on the design of the UE. The UE special conformance test functions vary depending on the conformance testing functionality they are designed to support and are broadly classified into the following two groups:

[0032] 1.1 Test Loop Functions. Functions that require a loop to be established between the UE and the System Simulator (SS) to allow, e.g., Downlink (DL) data packets sent by the SS to be looped back in uplink (UL) by the UE.

[0033] 1.2 General Test Functions. Commands sent by the SS, e.g., to trigger a certainUE behavior which may be a behavior determined by 3 GPP (third generation partnership project) core specification requirements or needed to perform conformance testing and not being part of any 3GPP core specification requirements, or, to provide to the UE information needed for the conformance testing.

[0034] The utilization of any UE special conformance test functions is considered as putting the UE in a test mode. The duration of the test mode depends on the UE special conformance test function and, in most of the cases, will be delimited by an activation and a deactivation command. The UE special conformance test functions, including any relevant procedure and the Test Mode Control (TMC) message contents used for information exchange, are defined in 3GPP TS 36.509 (for LTE, long term evolution) and 38.509 (extensions for NR, new radio).

[0035] The UE special conformance test functions provide access to isolated functions of the UE via the radio interface without introducing new physical interfaces just for the reason of conformance testing. However, in certain cases, the usage of AT Commands may be required, which will require an external interface, e.g., EMMI (Electrical Man Machine Interface). Furthermore, conformance testing may be performed through conducted (a physical “wire” or set of wires) tests or radiated (through the air) tests, though possibly these could be combined. Wireless / OTA (Over the Air) conformance test commands fall under a 3GPP RAN air interface, although there are at least two possibilities: for FR2 (frequency range 20, the conformance testing using OTA; for FR1 (frequency range 1), the conformance testing may use OTA or AT commands.

[0036] Depending on the conformance testing functionality supported, the UE special conformance test functions may comprise:

[0037] I.2.a A single DL message (e.g., a test function intended to provide UE information needed for the conformance testing).

[0038] I.2.b A Request / Acknowledgement type of two-message exchange, a DL message followed by a UL message (e.g., a test function intended to request the UE to execute an action which requires acknowledgment that the request was received and acted upon).

[0039] I.2.c Two, or more, UE special conformance test functions may be needed to be executed in a particular sequence before a specific target UE behavior can be assumed. Anexample for this is the Activate UE test mode and Close UE test loop functions. The former needs to be executed first, at a particular moment of time, in order for a specific type of test bearer terminated in a particular UE protocol layer to be established. Followed by the latter, executed at different point of time, which will instruct the UE to start looping back the received packets.

[0040] II. Test Methodology

[0041] While testing devices, one method is to test the device as a black box and the most common use case is having the device connected to a network simulator via the air interface only. However, it is not possible in all use cases. For example, when testing data transfer, the network simulator needs to have control of what the device is sending. For example: in the case of UL data, what source is there within the device to send a specific test data set?

[0042] For such a case, specific Over the Air (OTA) test commands are used. The test commands have their own protocol discriminator / protocol end point, a protocol running in parallel to other protocols at layer 3 level, like RRC (radio resource control), MM, and the like. The test commands are specified in 3GPP TS 38.509 / TS 36.509.

[0043] A general principle is shown in FIG. 1 , which is a signaling diagram illustrating OTA test commands for test loops. A UE 10 communicates with an SS (system simulator) 70. The SS may be referred to by other terms, such as network simulator and test equipment. In the example of FIG. 1, the UE 10 is in RRC_Connected mode in block 110, the SS 70 sends a Close UE test loop message in signaling 120, and the UE 10 responds with Close UE test loop complete message in signaling 130. One set of commands is for loop back of data, meaning the network simulator sends (e.g., signaling 120) data towards the device (UE 10 in this example) and the UE 10 is configured to send (e.g., signaling 130) the same frames back in the uplink. The mapping between downlink and uplink and at which protocol layers can be configured.

[0044] Another way to control the device is via MT / TE communication as suggested in 3GPP TS 27.007, which specifies a profile of AT (attention) commands and recommends that this profile be used for controlling Mobile Termination (MT) functions and network services. This requires in some instances a physical interface separate to the 3 GPP air interface which allows the exchange of AT commands. For instance, this is typically a wired serial interface between the UE and a terminal. It is also possible this can be wireless via BT (Bluetooth, a short-range wireless technology standard that is used for exchanging data between fixed and mobile devices over short distances and building personal area networks) or WiFi (a family of wireless network protocols based on the IEEE 802.11 family of standards). There is a dedicated connection besides the RAN air interface. It just requires that we are able to setup a terminal connection. In this way, it is possible to both command and control test procedures in the device. An example is for V2X (vehicle to everything) testing, which is given in Section 15 of 3GPP TS 27.007.

[0045] III. Description of Possible Issues with Current Technology

[0046] In 3 GPP Release 18 study item titled “Study on Artificial Intelligence (AI) / Machine Learning (ML) for NR (New Radio) Air Interface” (FS_NR_AIML_Air), the application of AI / ML (artificial intelligence / machine learning) techniques to NR air interface has been studied and the outcome of the study has been documented in 3GPP TS 38.843. It is noted that both the terms AI / ML and AIML are treated as being the same herein, and also the term AI / ML could represent Al, ML, or both Al and ML, to the extent there are differences between Al and ML.

[0047] In 3GPP Release 19, there is an ongoing work item titled “New WID (Working item description) on Artificial Intelligence (AI) / Machine Learning (ML) for NR Air Interface” (NR_AIML_Air) which aims to provide the normative support for the general framework for AI / ML for air interface and to enable the recommended use cases in the preceding study.

[0048] The normative work for AI / ML enabled use cases (e.g., Beam Management, Positioning) and corresponding features has already been started in 3GPP RAN1-RAN4 where the work of RAN4 is to develop the conformance requirements for the AI / ML enabled use cases and the corresponding performance test cases. An AI / ML use case (e.g., an application where UE can predict future CSI) can be enabled by deploying an AI / ML model or functionality (e.g., which may be software based) to a UE and such a UE is equipped with the required / dedicated circuitry (e.g., and instructions) to run the corresponding AI / ML model or functionality (and the associated AI / ML algorithms). These performance tests will further be developed by the RAN5 for which special UE conformance test functions for AI / ML conformance testing (e.g., conformance testing of AI / ML enabled 3GPP use-case(s) and / or features) will be needed to conduct these tests with controlled UE behavior during the conformance testing.

[0049] The AI / ML test function differs from the traditional non-AI / ML test functions in the following ways:

[0050] III.l Additional support should be provided for training data exchange between the SS 70 and the DUT (device under test, e.g., the UE 10), if the feature demands.

[0051] III.2 Additional messages to ensure that the training is complete and the DUT is ready to perform the inference.

[0052] III.3 Additional use-case specific data, required for the conformance testing of AI / ML features, should be provided to the DUT.

[0053] While the terms UE and DUT are considered to be interchangeable herein, it is noted that a UE becomes a DUT when the UE is being tested, e.g., by activating a test mode (e.g., for at least the AI / ML test function 230).

[0054] It is believed there is no UE special conformance test function defined currently in 3GPP TS 38.509 or other applicable specifications to be used for the testing of UE’s AI / ML features, e.g., as the scope of the UE related AI / ML normative work in 3GPP is quite new.

[0055] One problem that examples herein address is the following. How to design a UE special conformance test function, which can not only be used for the conformance testing of AI / ML use cases currently being discussed in 3 GPP but is designed to also support at least the basic conformance testing framework for AI / ML use cases to be agreed / decided in the future (e.g., 6G)?

[0056] IV. Overview of Examples

[0057] Examples herein propose a UE special conformance test function, referred to as “UE Artificial Intelligence (AI) / Machine Learning (ML) Test Function” or “UE AI / ML Test Function” (or other suitable phrases), designed to enable the conformance testing of UE features supporting AI-ML use cases (e.g., Al-enabled Beam Management, Al-enabled CSI (channel state information) compression, Al-enabled CSI prediction, Al-enabled positioning) in a controlled environment / manner. The proposed test function may also support the basic conformance testing framework for new AI / ML use cases to be agreed / decided in the future (e.g., 6G). For CSI prediction, AI / ML functionality / model(s) predict future channel states based on past and present CSI information, improving resource allocation and reducing latency. For CSI compression, the UE uses AI / ML encoder(s) to compress the estimated downlink channeland reports the quantized approximation of it to the gNB, where the AI / ML decoder at the gNB dequantizes and recovers the compressed CSI, received from the UE, back to the original information. In beam management, AI / ML model(s) assist in selecting the optimal beams and mitigating interference with the limited number of beam measurements. For positioning accuracy, AI / ML model(s) can be used either directly providing user positioning or assisting in positioning to refine user location for improved accuracy.

[0058] Refer to FIG. 2, which provides an example illustrating an AI / ML test function 230 and its associated testing features 240. The testing features 240 include any elements that can be used to help perform the AI / ML test function 230. Examples of features of the proposed UE Artificial Intelligence (AI) / Machine Learning (ML) Test Function are one or more of the following.

[0059] IV.1 May provide an indication about the AI / ML use case for which the conformance testing is required. See testing features 240- 1. This indication will be used by the UE / DUT to read the use-case specific fields (optional) containing the information required for the conformance testing of a particular use case. The AI / ML use case is one of the following examples: AI / ML based positioning (of the UE within the network); AI / ML based beam management; AI / ML based CSI compression; and AI / ML based CSI predication. These indications are for use cases (e.g., and corresponding model / functionality) so that UE is aware which use case specific fields will be present in the later part of the message. These use cases are examples and other use cases can be handled (e.g., such as use cases that are added in the future).

[0060] IV.2 May provide an indication about the AI / ML mode required for the conformance testing. See testing features 240-2. The basic modes may be one of the following examples: Data Collection Mode; Training Mode; or Inference Mode. The mode is performed for a selected AI / ML use case.

[0061] IV.3 May provide a field to be used for the Data (e.g., Training Data) Collection during the conformance testing (e.g., targeting a 6G use-case). See testing features 240-3. This field will contain at least an indication about the size of the training data to be provided to the UE / DUT. The training data will then be provided to the UE either using the same / similar field inside the UE AI / ML Test Function or through the PDSCH (physical downlink shared channel) resources or through the AT command(s) or possibly through other resources. In more detail, consider the following options: Option 1, where training data isappended inside the AI / ML test function; Option 2, where, after the AI / ML test function informs the UE that training data is coming, the UE monitors PDSCH and considers the data the UE receives to be the training data; and / or Option 3, where training data is pushed to the UE through AT command(s).

[0062] IV.4 May provide use-case specific field(s) to carry the use-case specific information to the UE required for the conformance testing. See testing features 240-4.

[0063] The proposed AI / ML Test Function can be implemented by 3GPP using one of the following alternatives.

[0064] Alternative 1 : Explicitly using the UE special conformance testing function(s) defined for the 3GPP OTA interface (such as using the specification 3GPP TS 38.509 or its 6G equivalent). See Section V.l below for more details.

[0065] Alternative 2: Explicitly using the standardized AT commands defined by the 3GPP which requires the external interface between the SS and the DUT (such as using the target specification: TS 27.007 or its 6G equivalent). See Section V.2 below for more details.

[0066] Alternative 3: Using a combination of the OTA Test Function and AT commands. (Not explicitly explained herein but can be a straight-forward combination of the details provided in Sections V.1 and V.2).

[0067] V. Additional details

[0068] Now that an overview has been provided, additional details are provided. In this document, a UE special conformance test function is proposed named “UE Artificial Intelligence (AI) / Machine Learning (ML) Test Function” or “UE AI / ML Test Function” (or other suitable phrase) which is designed to enable the conformance testing of the UE testing features supporting AI / ML use cases (e.g., AI / ML-enabled Beam Management, AI / ML-enabled CSI compression, AI / ML-enabled CSI prediction, AI / ML-enabled positioning) in a controlled environment / manner. It is noted that, based on the RAN5 future agreement(s), the exact name of the proposed UE special conformance test function can be different from the name(s) proposed above. The details of the proposed UE AI / ML Test Function and its testing features are provided below.

[0069] V.1 Alternative 1 : UE Artificial Intelligence ( AI) / Machine Learning (ML) Test Function (UAMF)

[0070] V.1.1 General

[0071] The UE AI / ML test function is intended for making the UE activate and / or use the testing features of this test function to perform at least one of the following procedures either before or during the UE conformance testing.

[0072] ( 1 ) Data collection to be used for UE side AI / ML model / functionality-based training,

[0073] (2) UE side AI / ML model / functionality-based training, and / or

[0074] (3) UE side AI / ML model / functionality-based inference.

[0075] In terms of the AI / ML model / functionality, there is discussion as to whether the models should be individually known (e.g., and chosen for use) or whether the functionality should be known, and the functionality could be performed by any or certain model or models. The terms “AI / ML model / functionality” is mainly used herein, and this term is intended to mean “one or both of AI / ML model or functionality” and to cover at least the following: (i) functionality that is performed by a single model or multiple models that implement the functionality; (ii) an AI / ML-enabled feature or feature group that perform the functionality; and / or model IDs (identifications), where the Network / UE may activate / deactivate / select / switch individual AI / ML models via model ID. For instance, for the use case of positioning, the AI / ML functionality may be UE position within a network, which may be performed by a single model. There may be multiple models that meet the requirements for the AI / ML functionality, and any of these models could be used. Alternatively, there could also be multiple models that together determine the UE position within a network. This may be extended to use of model IDs, e.g., where a unique ID is used, e.g., at least for models for individual use cases or possibly for all models. In other words, if a single model with a unique ID is used, the single model would perform the AI / ML functionality. In terms of use cases, it is mainly assumed herein that one use case corresponds to one functionality, but it might be possible for one use case to correspond to multiple functionalities.

[0076] Some of these procedures may be needed in preparation for subsequent test procedures. The UE AI / ML test function is useful for applicable UEs supporting AI / ML usecases requiring one-sided or two-sided model(s).

[0077] The SS 70 may use the UE AI / ML test mode activation procedure to command the UE to activate the UE special conformance testing features indicated via the UE AI / ML test function. See FIG. 3, which is a signaling diagram illustrating a UE AI / ML test modeactivation procedure. The SS 70 sends an Activate AI / ML test function message in signaling 210. It is noted that the UE is “preprogrammed” with the AI / ML test function 230, such that the AI / ML test function already is resident in (e.g., long term) memory prior to receiving the signaling 210. The UE 10 responds with an Activate AI / ML test function complete message in signaling 220.

[0078] After and based on the Activate AI / ML test function message, the UE 10 activates the AI / ML test function and the corresponding AI / ML functionality indicated by the AI / ML test function in block 260. The UE 10 performs, in block 270, the (activated) AI / ML functionality indicated by the AI / ML test function to perform (part of) the process for the conformance test.

[0079] As described above, there are three alternatives for implementing the AI / ML test function 230 (and its corresponding AI / ML functionality): Alt. (alterative) 1: implement using at least a newly defined UE special conformance testing function(s), see block 280-1; Alt. 2: implement using the newly defined AT command(s), see block 280-2; or Alt. 3: implement using a combination of the OTA Test Eunction and AT commands.

[0080] Each one of the AI / ML use cases from testing features 240- 1 has a corresponding AI / ML functionality that is performed in block 270 for different use cases: positioning has functionality (e.g., model(s)) 250-1 for positioning; beam management has functionality (e.g., model(s)) 250-2 for beam management; CSI compression has functionality (e.g., model(s)) 250-3 for CSI compression; and CSI prediction has functionality (e.g., model(s)) 250-4 for CSI prediction.

[0081] The SS uses the UE AI / ML test mode deactivation procedure to command the UE to deactivate the UE special conformance testing features indicated via the UE AI / ML test function. See EIG. 4, which is a signaling diagram illustrating a UE AI / ML test mode deactivation procedure. The SS 70 sends a Deactivate AI / ML test function message in signaling 310, and the UE 10 responds with a Deactivate AI / ML test function complete message in signaling 320. Based on the Deactivate AI / ML test function message, the UE 10 deactivates the AI / ML test function and the corresponding AI / ML functionality indicated by the AI / ML test function. See block 460.

[0082] V.1.2 Activate UE AI / ML test mode procedure

[0083] This section illustrates an example.

[0084] V.1.2.1 Initiation

[0085] The SS 70 requests the UE to activate AI / ML test mode by transmitting an ACTIVATE AI / ML TEST FUNCTION message.

[0086] V.1.2.2 Reception of ACTIVATE AI / ML TEST FUNCTION message by UE

[0087] When the UE 10 receives ACTIVATE AI / ML TEST FUNCTION message then the UE performs the following. Note that in the following (1) indicates the same level of indention, as does (2) and (3). In other words, the lines marked with (1) are all at the same level of indentation; the lines marked with (2) are all at the same level of indentation, but indented from (1); and the lines marked with (3) are all at the same level of indentation, but indented from (2).

[0088] ( 1 ) If the UE is in RRC_CONNECTED state:

[0089] (2) If the AI / ML Use Case field in the UE AI / ML TEST FUNCTION = 0:

[0090] (3) Consider the Use Case to be AI / ML based positioning and configure theUE accordingly;

[0091] (2) else if the AI / ML Use Case field in the UE AI / ML TEST FUNCTION = 1 :

[0092] (3) Consider the Use Case to be AI / ML based beam management and configure the UE accordingly;

[0093] (2) else if the AI / ML Use Case field in the UE AI / ML TEST FUNCTION = 2:

[0094] (3) Consider the Use Case to be AI / ML based CSI compression and configure the UE accordingly;

[0095] (2) else if the AI / ML Use Case field in the UE AI / ML TEST FUNCTION = 3:

[0096] (3) Consider the Use Case to be AI / ML based CSI prediction and configure the UE accordingly;

[0097] (2) if the AI / ML Mode field in the UE AI / ML TEST FUNCTION = 0:

[0098] (3) Configure and / or Activate the Data Collection mode at the UE;

[0099] (2) else if the AI / ML Mode field in the UE AI / ML TEST FUNCTION = 1 :

[0100] (3) Configure and / or Activate the Training mode at the UE;

[0101] (2) else if the AI / ML Mode field in the UE AI / ML TEST FUNCTION = 2

[0102] (3) Configure and / or Activate the Inference mode at the UE;

[0103] (2) Transmit ACTIVATE AI / ML TEST FUNCTION COMPLETE message;

[0104] (l) else:

[0105] (2) the UE behavior is unspecified.

[0106] V.1.3 Deactivate UE AI / ML test mode procedure

[0107] V.1.3.1 Initiation

[0108] The SS 70 requests the UE 10 to deactivate AI / ML test mode by transmitting, e.g., a DEACTIVATE AI / ML TEST FUNCTION message. The SS should perform this when the UE is in the RRC_CONNECTED state.

[0109] V.1.3.2 Reception of DEACTIVATE AI / ML TEST FUNCTION message by UE

[0110] When the UE receives DEACTIVATE AI / ML TEST FUNCTION message, then the UE performs the following in this example. As previously described, in the following, (1) indicates the same level of indention, as does (2) and (3).

[0111] (1) if the UE is in RRC_CONNECTED state AND the UE AI / ML TESTFUNCTION is active:

[0112] (2) Deactivate the testing features 240 of the UE AI / ML TEST FUNCTION and transmit DEACTIVATE AI / ML TEST FUNCTION COMPLETE message;

[0113] (l) else:

[0114] (2) the UE behavior is unspecified.

[0115] V.1.3.3 Release of AI / ML TEST FUNCTION by the UE

[0116] When the UE leaves the RRC_CONNECTED state, the UE performs the following:

[0117] ( 1 ) if the UE AI / ML TEST FUNCTION is active:

[0118] (2) deactivate the testing features 240 of the UE AI / ML TEST FUNCTION.

[0119] V.2 Alternative 1 : UE AI / ML TEST FUNCTION messages

[0120] V.2.1 ACTIVATE AI / ML TEST FUNCTION

[0121] This message (an activating AI / ML test function message) is only sent in the direction SS to UE. This example includes the following:

[0122] The presence, format, and length entries have parameters that are known, e.g., see 3GPP TS 24.007. It is noted that TBD is “to be determined”. CV denotes conditionally present fields. Other definitions: V = Value Only; TLV = Type / Tag, Length, Value. It is noted that the tables herein and their lEs / parameters / values are presented as examples and other tables / IEs / parameters / values may be used.

[0123] The message type is the following in this example:

[0124] It is noted that the message type field is used to identify ACTIVATE AI / ML TEST FUNCTION command here. The value of the “message type” field in the ACTIVATE AI / ML TEST FUNCTION may be uniquely decided by the RAN5 based on the currently available (e.g., not used) 8-bit sequences, e.g., a set of bits. The UE AI / ML Test Function (one such example) is the following:AI / ML Use Case to the UE in this example. The AI / ML use case field will serve as an indication to the UE that use-case specific information which is required for the conformance testing will be provided in the conditional fields (e.g., octets, which are different sets of bits). The value ofthe AI / ML use case field is selected from the pre-defined list of examples of use-cases as shown in Table X.l below.

[0126] Bit numbers 5 and 6 of octet 1 (i.e., X6X5), which are in the same set of bits, are used to indicate the AI / ML Mode (i.e., Data Collection mode or Training mode or Inference mode) to the UE in this example. The value of the AI / ML Mode field is selected from the predefined list examples of agreed modes as shown in Table X.2 below.

[0127] The “Data Collection” mode is used in this example to indicate to the DUT (e.g., UE 10) that the upcoming data (downlink) is intended for the training of the UE side AI / ML model / functionality. If the bit combination in AI / ML Mode field is set to “Data Collection”, the “Training Data PDU” field will be present which will at least contain either an indication about the size of the upcoming training data or the actual data for UE side AI / ML model / functionality training.

[0128] As an alternative, the training data can be sent to the DUT during the Data Collection Mode either using proprietary AT command(s) or an AT command explicitly defined for “Data Collection” that may be added to the standardized AT commands list, e.g., in 3GPP TS 27.007.

[0129] The “Training” mode is to indicate to the DUT that the data for UE side AI / ML model / functionality training is completely received, and the DUT can now start training its AI / ML model / functionality.

[0130] As an alternative, the Training Mode indication can be sent to the DUT either using proprietary AT command(s) or a AT command explicitly defined for “Training Mode” to be added to the standardized AT commands list, e.g., in 3GPP TS 27.007.

[0131] It is noted that, in an alternative design, the “Training Mode” field may be absent, and the completion of Data Collection Mode will implicitly indicate to the DUT to perform the training of its AI / ML model / functionality.

[0132] The “Inference” mode may be used to indicate to the DUT to start performing the use case inference(s) using the trained AI / ML model / functionality at the DUT side.

[0133] As an alternative, the Inference Mode indication can be sent to the DUT using proprietary AT command(s), or via an AT command explicitly defined for “Inference Mode” that may be added to the standardized AT commands list in 3GPP TS 27.007.

[0134] The “Reserved” mode is reserved for future use.

[0135] It is noted that, after completion of each of the Data Collection, Training and Inference modes, the DUT may transmit an acknowledgement / completion message to the SS confirming the completion of the associated tasks of a particular mode. This acknowledgement can be sent either through the:

[0136] ( 1 ) Proposed ACTIVATE AI / ML TEST FUNCTION COMPLETE message(see Section V .2.2 herein), or

[0137] (2) A new acknowledgement message for each of the Data Collection,Training and Inference modes.

[0138] Bit numbers 7 and 8 of octet 1 (i.e., X8 and X7) are also reserved for future use.

[0139] The following are also noted.

[0140] In one alternative, different bit length and / or combinations of octet 1 can be used to indicate the AI / ML use case field and AI / ML mode field. Examples:

[0141] Example 1: Bit numbers 1-5 denote AI / ML use case, bit numbers 6-8 denote AI / ML mode; or

[0142] Example 2: Bit number 1-3 denote AI / ML use case, bit numbers 4-6 denote AI / ML mode, bit numbers 7-8 are reserved for future use.

[0143] In other alternative, different octets are used to indicate the AI / ML use case field and AI / ML mode field.

[0144] Consider this example. Some or all 8 bits of Octet 1 (e.g., one set of bits) are used to denote the AI / ML use case field whereas bits of octet 2 (some or all 8 bits) (e.g., another,different set of bits) are used to indicate the AI / ML mode. The unused bits of octet 1 and octet 2 are reserved for future use.

[0145] The remaining IES (information elements) are optional and are conditionally present according to the conditions defined below. It is worth mentioning that some of the below defined fields may not be a part of the proposed AI / ML Test Function, depending on RAN5 agreements. Table X.3 below illustrates an example of Conditions and Explanations:

[0146] The IE Training Data PDU is to be used in this example to transmit the training data to the DUT / UE to train its AI / ML model / functionality.

[0147] The following are further noted.

[0148] As an alternative, instead of sending the training data to the DUT / UE using the proposed IE “Training Data PDU” in the “UE AI / ML Test Function” message / command, the training data can be transmitted to the DUT / UE through the PDSCH resources. In another alternative, the training data can be transmitted to the DUT / UE through AT command(s).

[0149] The IE “Positioning Use Case” is proposed to carry / transmit the test related information to the DUT / UE required for the conformance testing of Positioning use case.

[0150] The IE “Beam Management Use Case” is proposed to carry / transmit the test related information to the DUT / UE required for the conformance testing of Beam Management use case.

[0151] The IE “CSI Compression Use Case” is proposed to carry / transmit the test related information to the DUT / UE required for the conformance testing of CSI Compression use case.

[0152] The IE “CSI Prediction Use Case” is proposed to carry / transmit the test related information to the DUT / UE required for the conformance testing of CSI Prediction use case.

[0153] The IE “Reserved for future use cases” is proposed to carry / transmit the test related information to the UE required for the conformance testing of future use cases.

[0154] V.2.2 ACTIVATE AI / ML TEST FUNCTION COMPLETE

[0155] This message is only sent in the direction UE to SS.

[0156] Message type is the following in this example:

[0157] It is noted that the message type field is used to identify ACTIVATE AI / ML TEST FUNCTION COMPLETE command here. The value of the “message type” field in ACTIVATE AI / ML TEST FUNCTION COMPLETE will be uniquely decided by the RAN5 based on the currently available (not used) 8-bit sequences.

[0158] V.2.3 DEACTIVATE AI / ML TEST FUNCTION

[0159] This message is only sent in the direction SS to UE.

[0160] It is noted that the message type field is used to identify DEACTIVATE AI / ML TEST FUNCTION command here. The value of the “message type” field in DEACTIVATE AI / ML TEST FUNCTION will be uniquely decided by the RAN5 based on the currently available (not used) 8-bit sequences.

[0161] V.2.4 DEACTIVATE AI / ML TEST FUNCTION COMPLETE

[0162] This message is only sent in the direction UE to SS.

[0163] The message type is the following in this example:

[0164] It is noted that the message type field may be used to identify DEACTIVATE AI / ML TEST FUNCTION COMPLETE command here. The value of the “message type” field in DEACTIVATE AI / ML TEST FUNCTION COMPLETE may be uniquely decided by the RAN5 based on the currently available (not used) 8-bit sequences.

[0165] FIG. 5 shows an example message flow sequence in a signaling diagram for the conformance testing of AI / ML use-cases / other testing features based on the OTA UE conformance test function. This illustrates a process 400 that is a process for the conformance testing of AI / ML enabled testing features / use-case(s), which is divided into this example by three phases, illustrated by blocks 420, 430, and 440. It is noted that, in block 420, the UE 10 becomes a DUT, and is no longer under test after the test mode is deactivated by the UE. Block 420 indicates that test mode is going to be entered in this phase. This occurs using signaling 1 and 2. The SS 70 in signaling 1 activates the test mode. The UE 10 responds in signaling 2 with an Activate test mode complete message, where OK is a known response.

[0166] Block 430 indicates the next phase of the process, which involves an OTA Comm (communication) command to Activate / Deactivate AI / ML Test Function and the associated testing features. In signaling 3, the SS 70 sends an ACTIVATE AI / ML TEST FUNCTION (Contains Conditionally Present IES and indication for AI / ML use case, AI / ML mode). As previously described, the UE 10 activates the AI / ML test function 230 and activates and performs the corresponding AI / ML functionality 250. This is shown in block 470 as performing (e.g., at least) block 260 of EIG. 2. The UE 10 responds with an Activate AI / ML test function complete message. This occurs in signaling 4.

[0167] Also, it is noted that the “Activate AI / ML test function” message (e.g., in signaling 3) can be sent multiple times to the DUT before “Deactivate AI / ML test function” message (e.g., in signaling 4) will be sent. Consider the following example. Eirstly, the “Activate AI / ML test function” message is sent to indicate to the DUT to activate the “Data Collection” mode and “receive the training data” from the SS. Thereafter, a second “Activate AI / ML test function” message is sent to indicate the DUT to “train its model / functionality”. Then, a third time, the “AI / ML test function” message is sent to indicate the DUT to “perform inference” by activating “inference mode”. In this example, the “Deactivate AI / ML test function” message has not yet been sent to the DUT. The “Deactivate AI / ML test function” message will only be needed once the conformance test is complete, and this special test function is not needed anymore. In another alternative, each “Activate AI / ML test function” message is ended by a “Deactivate AI / ML test function” message before another functionality or mode can be activated.

[0168] In signaling 5, the UE and SS run (e.g., participate in) a Conformance Test and Generate Inference(s). It is noted that the signaling 5 is performed by both the UE 10 (as the DUT) and the SS 70, in conjunction with each other. Lor instance, the SS 70 will perform the conformance test and will observe the UE behavior during the test, based on which the SS will generate the verdict of the test, i.e., whether the UE had passed the test (e.g., is in compliance with one or more conformance requirements) or failed the test (e.g., is in not compliance with one or more conformance requirements). The UE 10, for example, could be performing positioning, and need to report within a certain time period. The SS 70 can determine whether the time period in which the UE reports falls within the permissible requirement limit(s) or not,thereby passing (if within the limit(s)) or failing (if not within the limits) this (e.g., part of) the conformance test.

[0169] The SS 70 in signaling 6 sends a DEACTIVATE AI / ML TEST FUNCTION message to the UE, and the UE 10 responds in signaling 7 with a DEACTIVATE AI / ML TEST FUNCTION COMPLETE message. Block 480 indicates that block 460 is performed based on the DEACTIVATE AI / ML TEST FUNCTION message, and the UE 10 deactivates the AI / ML test function and the corresponding AI / ML functionality indicated by the AI / ML test function.

[0170] Block 440 illustrates a third phase, which ends test mode. In signaling 8, the SS 7 sends a Deactivate test mode message to the UE and the UE responds in signaling 9 with a Deactivate test mode complete message.

[0171] V.3 Alternative 2: AT Command based solution for the conformance testing of AI / ML use cases

[0172] This section is described in part by reference to FIG. 6, which is a signaling diagram illustrating an AT Command based example for the conformance testing of AI / ML use- cases / other testing features. This illustrates a process 500, which is a process for AI / ML conformance testing using AT commands. Three phases are illustrated by blocks 420, 530, and 440. Block 110 was described in FIG. 5. Block 420 indicates that test mode is going to be entered in this phase. This occurs using signaling 1 and 2. The SS 70 in signaling 1 activates the test mode (+CATM). The +CATM command activates and deactivates the UE test mode procedure and sets the UE into test mode. For this and other extended commands (the “extended” indicated by the “+”), see, e.g., at least section 4.1 of 3GPP TS 27.007 (e.g., 3GPP TS 27.007 VI 8.7.0 (2024-06)), which describes a basic structure of a command line and possible extended commands. The UE 10 responds in signaling 2 with an Activate test mode complete message, where OK is a known response. In block 530, this illustrates a phase for an AT Command to Activate / Deactivate AI / ML Test Function and the Associated Features. In signaling 3, the SS 70 sends to the UE 10 an ACTIVATE AI / ML TEST FUNCTION (+CAAIMLTF) message with parameters of +CAAIMLTF=<status=l>[,<use-case>[,<mode>]], where use-case and / or mode may be optional. The parameter of +CAAIMLTF=<status=l> indicates that this message is for activation of the AI / ML test function and associated testing features. As indicated previously, the UE 10 activates the AI / ML test function 230 and activates and performs the correspondingAI / ML functionality 250. This occurs in block 535, where (e.g., at least) block 260 of FIG. 2 is performed. The UE 10 responds in signaling 4 with an OK message.

[0173] In signaling 5, the SS sends a TRANSMIT DATA COLLECTION PDU (+CDCP) message with parameters of +CDCP=<action>[,<data_size>], meaning that data_size is optional. The UE 10 receives (block 540) the data collection PDU, and the UE 10 responds in signaling 6 with an OK message.

[0174] In signaling 7, the SS 70 sends a TRANSMIT USE CASE SPECIFIC PDU (+CUCSP) message with the parameters of +CUCSP=<action>[,<data_size>], where data_size is optional. The UE receives (block 550) the use case specific PDU and, in signaling 8, the UE 10 responds with an OK message. In signaling 9, the UE and SS communicate to Run a Conformance Test and Generate an Inference.

[0175] In signaling 10, there is a DEACTIVATE AI / ML TEST FUNCTION message with the parameter of +CAAIMLTF=<status=0>. The parameter of +CAAIMLTF=<status=0> indicates that this message is for deactivation of the AI / ML test function and associated testing features. Based on the signaling 10, the UE performs (see block 555) block 460 (as indicated by block 550), so the UE deactivates the AI / ML test function and the corresponding AI / ML functionality indicated by the AI / ML test function. The UE 10 responds in signaling 11 with an OK message. It is noted that the status = 1 indicates activation of the AI / ML test function and the status = 0 indicates deactivation of the AI / ML test function, but these values of the status are merely examples and others may be used.

[0176] Block 440 is the same as in FIG. 5, and signaling 12 and 13 from FIG. 6 are the same as signaling 8 and 9, respectively, in FIG. 5.

[0177] Here, the following (e.g., extended) AT commands are defined. The AT command named “+CAAIMLTF” (signaling 3 of FIG. 6), “+CDCP” (signaling 5 of FIG. 6) and “+CUCSP” (signaling 7 of FIG. 6) to indicate the activation / deactivation of AI / ML TEST FUNCTION, TRAINING DATA PDUs, and USE CASE SPECIFIC PDUs, respectively. The TRAINING DATA PDU and USE CASE SPECIFIC PDUs commands are optional. The USE CASE SPECIFIC PDU command is used to transmit the use case specific information to the DUT for the conformance testing.

[0178] Table X.4 illustrates possible +CAAIMLTF parameter command syntax:

[0179] Table X.5 illustrates possible +CDCP parameter command syntax:

[0180] Description of (+CAAIMLTF) is as follows.

[0181] The set command is used to activate or deactivate the UE AI / ML Test Function (proposed in this invention). The AT command is only applicable when test mode is activated. When <status>=l, the set command activates the UE AI / ML Test Function. When <status>=0, the set command deactivates the UE AI / ML Test Function. If <status>=l, the parameters <use-case> and <mode> are required to configure the testing features of UE AI / ML Test Function.

[0182] The <use-case> indicates the AI / ML use-case for which the conformance test is to be performed. The parameter <mode> indicates the AI / ML mode to be run / used either for preparation or during the conformance test. All parameters are discarded when <status>=0 or when the UE test mode is deactivated.

[0183] The read command returns the current settings. The test command returns values supported as compound values.

[0184] Defined values (+CAAIMLTF) are as follows.

[0185] <status>: integer type. Indicates the state of UE AI / ML Test Function.

[0186] 0 deactivated; and

[0187] 1 activated.

[0188] <use-case>: integer type. Indicates the AI / ML use case under the test. See the following table with examples:

[0189] <mode>: integer type. Indicates the AI / ML mode to be used. See the following table with examples:

[0190] Description (+CDCP) is presented now.

[0191] The set command is used to start transmitting the data to the UE for the UE side AI / ML model / functionality training. The data can be transmitted either using AT interface or PDSCH resources. The AT command is only applicable when test mode is activated. The <action> indicates whether the UE shall start or stop receiving data. If <action>= 1 , the parameter <data_size> must be provided. All parameter(s) are discarded when <action>=0 or when the UE test mode is deactivated.

[0192] The read command returns the current settings. The test command returns values supported as compound values.

[0193] Defined values (+CDCP) are as follows.

[0194] <action>: integer type. Indicates whether the UE shall start or stop receiving the training data.

[0195] 0 stop receiving data; and

[0196] 1 start receiving data.

[0197] <data_size>: integer type. Indicates how many bytes of data the UE shall receive for the AI / ML model / functionality training.

[0198] It is noted that the AT commands proposed here are intended to explain the examples, but these can be changed and other or similar AT commands used.

[0199] Referring to FIGS. 7A and 7B, these are block diagrams of possible and nonlimiting exemplary systems in which the exemplary embodiments may be practiced, where thesystem model in FIG. 7A is for a conducted mode conformance testing and the system model in FIG. 7B is for a radiated (OTA) mode conformance testing.

[0200] In FIG. 7A, for a conducted testing system model, there is an SS (system simulator) 70-1, which is also referred to as a test equipment (TE), which produces an output vector x 710 made of four complex baseband symbols / components {x1, x2, x3, x4}. There is a channel emulator that emulates the desired channel HLduring the conformance testing and applies that to the output vector x to produce an output HLx 730 that is carried over cables (e.g., coaxial cables) to the test location 750, in which the UE 10 is located. There is an effective channel Hcabie740, and the cables are routed to physical inputs (H, V) 760-1 and 760-2 which represents the of the antenna port inputs of the UE 10. The main example has the SS 70-1 being separate from the channel emulator 720, but these can be combined to form an SS 70-2 that would produce the output HLx 730.

[0201] In FIG. 7B, for an OTA model, this is similar, but the UE 10 is placed into an OTA testing chamber (e.g., anechoic chamber) 770, and the output HLx 730 is routed to two sets 790-1 and 790-2 of dual polarized test probes, where set 790-1 emits a dual-polarized beaml 780-1 that is received by a polarized antenna panel 775-1 on the UE 10, and the set 790-2 emits a dual-polarized beam2 780-2 that is received by a polarized antenna panel 775-2 on the UE 10. There is a dual polarized antenna panels 775-1 and 775-2 in this example. As an alternative, the two sets 790-1 and 790-2 of dual polarized test probes can be replaced with 4 single polarized test probes.

[0202] FIG. 8 is a block diagram illustrating possible circuitry for the systems of FIGS. 7A and 7B. In FIG. 8, a user equipment (UE) 10 may be in conducted (e.g., wired) communication via cabling 79 with the SS 70, which corresponds to the system of FIG. 7A; or be in wireless communication via radio link 11 with the SS 70, which corresponds to the system of FIG. 7B.

[0203] The UE 10 includes one or more processors 13, one or more memories 15, and other circuitry 16. The other circuitry 16 includes one or more (wired and / or wireless) receivers (Rx(s)) 17 and one or more (wired and / or wireless) transmitters (Tx(s)) 18. A program 12 is used to cause the UE 10 to perform the operations described herein. For a UE 10, the other circuitry 16 could include circuitry such as for user interface elements (not shown) like a display. Theprogram 12 may be implemented via instructions stored in memory / memories 15 and executed by processor(s) 13, or by circuitry such being implemented as part of the processor(s) or other circuitry elements, or both.

[0204] The SS 70 includes one or more processors 73, one or more memories 75, and other circuitry 76. The other circuitry 76 includes one or more receivers (Rx(s)) 77 and one or more transmitters (Tx(s)) 78. A program 72 is used to cause the base station 70 to perform the operations described herein. The program 72 may be implemented via instructions stored in memory / memories 75 and executed by processor(s) 73, or by circuitry such being implemented as part of the processor(s) or other circuitry elements, or both.

[0205] The programs 12 and 72 contain instructions stored by corresponding one or more memories 15 or 75. These instructions, when executed by the corresponding one or more processors 13 or 73, cause the corresponding apparatus 10 or 70, to perform the operations described herein. Computer program products 94- 1 and 94-2 are illustrated. The respective computer-readable medium 94 contains instructions that, when downloaded and installed into the program 12 or 72 and / or the memories 15 or 75 of the corresponding UE 10 or SS 70, and executed by processor(s) 13 or 73, cause the respective device to perform corresponding actions described herein. The computer-readable medium 94 may be implemented for downloading using a compact disc or memory stick or any other memory such as memory in a data network.

[0206] The computer readable memories 15, 75, are circuitry and may be of any type suitable to the local technical environment (e.g., memories 15, 75, could all be different) and may be implemented using any suitable data storage technology, such as semiconductor-based memory devices, flash memory, firmware, magnetic memory devices and systems, optical memory devices and systems, fixed memory and removable memory. The processors 13, 73 are circuitry and may be of any type suitable to the local technical environment (e.g., processors 13, 73 could all be different). For example, these processors may include one or more of general- purpose computers, special purpose computers, microprocessors, digital signal processors (DSPs), processors based on a multi-core processor architecture, and may also include specialized circuits such as field-programmable gate arrays (FPGAs), application specific circuits (ASICs), signal processing devices and other devices, or combinations of these devices, as non-limiting examples. The processors 13 and 73 are circuitry that can be programmed toperform functions via software, firmware or the like (including microcode), but are not solely software.

[0207] The receivers 17 and 77, and the transmitters 18 and 78 may implement wired or wireless interfaces. The receivers and transmitters may be grouped together as transceivers.

[0208] In general, the various embodiments of the user equipment 10 can include, but are not limited to, devices implementing cellular technologies (such as smart phones, mobile phones, cellular phones, voice over Internet Protocol (IP) (VoIP) phones, and / or wireless local loop phones), tablets, portable computers, vehicles or vehicle-mounted devices for, e.g., wireless V2X (vehicle-to-everything) communication, image capture devices such as digital cameras, gaming devices, music storage and playback appliances, Internet appliances (including Internet of Things, loT, devices), loT devices with sensors and / or actuators for, e.g., automation applications, as well as portable units or terminals that incorporate combinations of such functions, laptop-embedded equipment (LEE), laptop-mounted equipment (LME), Universal Serial Bus (USB) dongles, smart devices, wireless customer-premises equipment (CPE), an Internet of Things (loT) device, a watch or other wearable, a head-mounted display (HMD), a vehicle, a drone, a medical device and applications (e.g., remote surgery), an industrial device and applications (e.g., a robot and / or other wireless devices operating in an industrial and / or an automated processing chain contexts), a consumer electronics device, a device operating on commercial and / or industrial wireless networks, and the like. That is, the UE 10 could be any end device that may be capable of wireless communication. By way of example rather than limitation, the UE may also be referred to as a communication device, terminal device (MT), a Subscriber Station (SS), a Portable Subscriber Station, a Mobile Station (MS), or an Access Terminal (AT).

[0209] Without in any way limiting the scope, interpretation, or application of the claims appearing below, a technical effect and / or advantage of one or more of the example embodiments disclosed herein is that no “UE special conformance test function” currently exists to assist in the conformance testing of AI / ML enabled features / use-cases / functionalities. Without the proposed UE AI / ML test function, such conformance tests may not be performed.

[0210] The following are additional examples.

[0211] Example 1. A method, comprising: by a device under test (DUT) as part of conformance testing of the DUT, receiving indication to activate an artificialintelligence / machine learning (Al / ML) test function for the conformance testing, the Al / ML test function comprising indication of a use case, as one of multiple testing features associated with the Al / ML test function, that corresponds to one or both of an Al / ML model or functionality for which the conformance testing is to be performed; activating, by the DUT, the Al / ML test function and associated one or more of the multiple testing features corresponding to the indicated use case; and performing, by the DUT based on the activated Al / ML test function, the one or both of the Al / ML model or functionality for the indicated use case and associated testing features as part of the conformance testing.

[0212] Example 2. The method according to example 1, further comprising: receiving, by the DUT, an indicator indicating the Al / ML test function should be deactivated; and deactivating, by the DUT, the Al / ML test function and the testing features corresponding to the one or both of the Al / ML model or functionality.

[0213] Example 3. The method according to any of examples 1 to 2, wherein the use case comprises one of the following: Al / ML enabled positioning of the DUT; Al / ML enabled beam management; Al / ML enabled channel state information (CSI) compression; or Al / ML enabled CSI prediction, and the use case is used to select a corresponding Al / ML based functionality.

[0214] Example 4. The method according to example 3, wherein the corresponding Al / ML based functionality comprises one of the following that corresponds to the use case: Al / ML based positioning; Al / ML based channel state information (CSI) compression; Al / ML based beam management; or Al / ML based CSI prediction.

[0215] Example 5. The method according to any of examples 1 to 5, wherein: the Al / ML test function further indicates a mode to be performed by the DUT, the mode being one of the associated testing features, wherein the mode comprises one of the following: data collection mode; training mode; or inference mode; and performing, by the DUT, the indicated mode, as part of the conformance testing, for the one or both of an Al / ML model or functionality.

[0216] Example 6. The method according to example 5, wherein: the mode is the data collection mode; the Al / ML test function further contains one or more fields to be used for providing training data to the DUT, to train the one or more of the Al / ML model or functionality at least on a DUT side, during the conformance testing; and the method further comprises usingcontent of the one or more fields during the data collection mode as part of the conformance testing.

[0217] Example 7. The method according to example 6 wherein the one or more fields contain at least an indication about a size of the training data to be provided to the DUT during the conformance testing.

[0218] Example 8. The method according to example 7, further comprising: receiving, by the DUT, the training data using the one or more fields inside the AI / ML test function, or through the PDSCH (physical downlink shared channel) resources, or through AT command(s), or through other resources.

[0219] Example 9. The method according to any of examples 6 to 8, wherein the method further comprises: activating, by the DUT, the training mode based on another AI / ML test function indicating the mode to be performed by the DUT is the training mode; and performing, by the DUT, training of the one or both of the Al / ML model or functionality using the training data from the data collection mode.

[0220] Example 10. The method according to any of examples 1 to 9, wherein the AI / ML test function and its associated testing features, including a mode which applies to the use case, corresponding to the one or both of the AI / ML model or functionality are provided to the DUT using at least one defined UE special conformance testing function for a third generation partnership project (3GPP) over-the-air (OTA) interface.

[0221] Example 11. The method according to any of examples 1 to 9, wherein the AI / ML test function and its associated testing features, including a mode which applies to the use case, corresponding to the one or both of the AI / ML model or functionality are provided to the DUT using standardized attention (AT) commands which requires an external interface between a system simulator (SS) and the DUT.

[0222] Example 12. The method according to any of examples 1 to 9, wherein the AI / ML test function and its associated testing features, including a mode which applies to the use case, corresponding to the one or both of the AI / ML model or functionality is provided to the DUT using a combination of the following: at least one defined UE special conformance testing function for a third generation partnership project (3GPP) over-the-air (OTA) interface; and standardized attention (AT) commands which requires an external interface between a system simulator (SS) and the DUT.

[0223] Example 13. The method according to any of examples 1 to 12, wherein: the associated one or more of the multiple testing features comprise one or more specific fields that carry information specific to the use case; and the method further comprises using by the DUT the information specific to the use case in the one or more specific fields as part of the conformance testing.

[0224] Example 14. The method according to any of examples 1 to 13, wherein: the AI / ML test function further indicates the following using a single set of bits: a use case is indicated using a first portion of the single set of bits; and a mode is indicated using a second portion of the single set of bits.

[0225] Example 15. The method according to any of examples 1 to 13, wherein: the AI / ML test function further indicates the following: a use case is indicated using a first set of bits; and a mode is indicated using a second set of bits, wherein the first and second sets of bits are different sets of bits not within a single set of bits.

[0226] Example 16. A method, comprising: by a system simulator (SS) as part of conformance testing of a device under test (DUT), sending indication to the DUT to activate an artificial intelligence / machine learning (AI / ML) test function used for the conformance testing, the AI / ML test function comprising indication of a use case, as one of multiple testing features, that corresponds to one or both of an AI / ML model or functionality for which the conformance testing is to be performed; receiving, by the SS from the DUT, indication that at least the AI / ML test function has been activated; and participating, by the SS based on the activated AI / ML test function and in conjunction with the DUT, in the conformance testing of the DUT.

[0227] Example 17. The method according to example 16, further comprising: sending, from the SS to the DUT, an indicator indicating the AI / ML test function should be deactivated; and receiving, by the SS from the DUT, indication that at least the AI / ML test function has been deactivated.

[0228] Example 18. The method according to any of examples 16 or 17, wherein participating in the conformance testing of the DUT comprises running a conformance test and generating an inference of whether or not the DUT is in compliance with one or more conformance requirements.

[0229] Example 19. An apparatus, comprising means for: by a device under test(DUT) as part of conformance testing of the DUT, receiving indication to activate an artificialintelligence / machine learning (Al / ML) test function for the conformance testing, the Al / ML test function comprising indication of a use case, as one of multiple testing features associated with the Al / ML test function, that corresponds to one or both of an Al / ML model or functionality for which the conformance testing is to be performed; activating, by the DUT, the Al / ML test function and associated one or more of the multiple testing features corresponding to the indicated use case; and performing, by the DUT based on the activated Al / ML test function, the one or both of the Al / ML model or functionality for the indicated use case and associated testing features as part of the conformance testing.

[0230] Example 20. The apparatus according to example 19, wherein the means are further configured for: receiving, by the DUT, an indicator indicating the Al / ML test function should be deactivated; and deactivating, by the DUT, the Al / ML test function and the testing features corresponding to the one or both of the Al / ML model or functionality.

[0231] Example 21. The apparatus according to any of examples 19 to 20, wherein the use case comprises one of the following: Al / ML enabled positioning of the DUT; Al / ML enabled beam management; Al / ML enabled channel state information (CSI) compression; or Al / ML enabled CSI prediction, and the use case is used to select a corresponding Al / ML based functionality.

[0232] Example 22. The apparatus according to example 21, wherein the corresponding Al / ML based functionality comprises one of the following that corresponds to the use case: Al / ML based positioning; Al / ML based channel state information (CSI) compression; Al / ML based beam management; or Al / ML based CSI prediction.

[0233] Example 23. The apparatus according to any of examples 19 to 23, wherein: the Al / ML test function further indicates a mode to be performed by the DUT, the mode being one of the associated testing features, wherein the mode comprises one of the following: data collection mode; training mode; or inference mode; and performing, by the DUT, the indicated mode, as part of the conformance testing, for the one or both of an Al / ML model or functionality.

[0234] Example 24. The apparatus according to example 23, wherein: the mode is the data collection mode; the Al / ML test function further contains one or more fields to be used for providing training data to the DUT, to train the one or more of the Al / ML model or functionality at least on a DUT side, during the conformance testing; and the means are furtherconfigured for using content of the one or more fields during the data collection mode as part of the conformance testing.

[0235] Example 25. The apparatus according to example 24 wherein the one or more fields contain at least an indication about a size of the training data to be provided to the DUT during the conformance testing.

[0236] Example 26. The apparatus according to example 25, wherein the means are further configured for: receiving, by the DUT, the training data using the one or more fields inside the AI / ML test function, or through the PDSCH (physical downlink shared channel) resources, or through AT command(s), or through other resources.

[0237] Example 27. The apparatus according to any of examples 24 to 26, wherein the means are further configured for : activating, by the DUT, the training mode based on another AI / ML test function indicating the mode to be performed by the DUT is the training mode; and performing, by the DUT, training of the one or both of the AI / ML model or functionality using the training data from the data collection mode.

[0238] Example 28. The apparatus according to any of examples 19 to 27, wherein the AI / ML test function and its associated testing features, including a mode which applies to the use case, corresponding to the one or both of the AI / ML model or functionality are provided to the DUT using at least one defined UE special conformance testing function for a third generation partnership project (3GPP) over-the-air (OTA) interface.

[0239] Example 29. The apparatus according to any of examples 19 to 27, wherein the AI / ML test function and its associated testing features, including a mode which applies to the use case, corresponding to the one or both of the AI / ML model or functionality are provided to the DUT using standardized attention (AT) commands which requires an external interface between a system simulator (SS) and the DUT.

[0240] Example 30. The apparatus according to any of examples 19 to 27, wherein the AI / ML test function and its associated testing features, including a mode which applies to the use case, corresponding to the one or both of the AI / ML model or functionality is provided to the DUT using a combination of the following: at least one defined UE special conformance testing function for a third generation partnership project (3GPP) over-the-air (OTA) interface; and standardized attention (AT) commands which requires an external interface between a system simulator (SS) and the DUT.

[0241] Example 31. The apparatus according to any of examples 19 to 30, wherein: the associated one or more of the multiple testing features comprise one or more specific fields that carry information specific to the use case; and the means are further configured for using by the DUT the information specific to the use case in the one or more specific fields as part of the conformance testing.

[0242] Example 32. The apparatus according to any of examples 19 to 31, wherein: the AI / ML test function further indicates the following using a single set of bits: a use case is indicated using a first portion of the single set of bits; and a mode is indicated using a second portion of the single set of bits.

[0243] Example 33. The apparatus according to any of examples 19 to 31, wherein: the AI / ML test function further indicates the following: a use case is indicated using a first set of bits; and a mode is indicated using a second set of bits, wherein the first and second sets of bits are different sets of bits not within a single set of bits.

[0244] Example 34. An apparatus, comprising means for: by a system simulator (SS) as part of conformance testing of a device under test (DUT), sending indication to the DUT to activate an artificial intelligence / machine learning (AI / ML) test function used for the conformance testing, the AI / ML test function comprising indication of a use case, as one of multiple testing features, that corresponds to one or both of an AI / ML model or functionality for which the conformance testing is to be performed; receiving, by the SS from the DUT, indication that at least the AI / ML test function has been activated; and participating, by the SS based on the activated AI / ML test function and in conjunction with the DUT, in the conformance testing of the DUT.

[0245] Example 35. The apparatus according to example 34, wherein the means are further configured for: sending, from the SS to the DUT, an indicator indicating the AI / ML test function should be deactivated; and receiving, by the SS from the DUT, indication that at least the AI / ML test function has been deactivated.

[0246] Example 36. The apparatus according to any of examples 34 or 35, wherein participating in the conformance testing of the DUT comprises running a conformance test and generating an inference of whether or not the DUT is in compliance with one or more conformance requirements.

[0247] Example 37. An apparatus, comprising: one or more processors; and one or more memories storing instructions that, when executed by the one or more processors, cause the apparatus at least to perform: by a device under test (DUT) as part of conformance testing of the DUT, receiving indication to activate an artificial intelligence / machine learning (AI / ML) test function for the conformance testing, the AI / ML test function comprising indication of a use case, as one of multiple testing features associated with the Al / ML test function, that corresponds to one or both of an Al / ML model or functionality for which the conformance testing is to be performed; activating, by the DUT, the AI / ML test function and associated one or more of the multiple testing features corresponding to the indicated use case; and performing, by the DUT based on the activated AI / ML test function, the one or both of the AI / ML model or functionality for the indicated use case and associated testing features as part of the conformance testing.

[0248] Example 38. The apparatus according to example 37, wherein the one or more memories further store instructions that, when executed by the one or more processors, cause the apparatus at least to perform: receiving, by the DUT, an indicator indicating the AI / ML test function should be deactivated; and deactivating, by the DUT, the AI / ML test function and the testing features corresponding to the one or both of the AI / ML model or functionality.

[0249] Example 39. The apparatus according to any of examples 37 to 38, wherein the use case comprises one of the following: AI / ML enabled positioning of the DUT; AI / ML enabled beam management; AI / ML enabled channel state information (CSI) compression; or AI / ML enabled CSI prediction, and the use case is used to select a corresponding AI / ML based functionality.

[0250] Example 40. The apparatus according to example 39, wherein the corresponding AI / ML based functionality comprises one of the following that corresponds to the use case: AI / ML based positioning; AI / ML based channel state information (CSI) compression; AI / ML based beam management; or AI / ML based CSI prediction.

[0251] Example 41. The apparatus according to any of examples 37 to 41, wherein: the AI / ML test function further indicates a mode to be performed by the DUT, the mode being one of the associated testing features, wherein the mode comprises one of the following: data collection mode; training mode; or inference mode; and performing, by the DUT, the indicated mode, as part of the conformance testing, for the one or both of an AI / ML model or functionality.

[0252] Example 42. The apparatus according to example 41, wherein: the mode is the data collection mode; the AI / ML test function further contains one or more fields to be used for providing training data to the DUT, to train the one or more of the AI / ML model or functionality at least on a DUT side, during the conformance testing; and the one or more memories further store instructions that, when executed by the one or more processors, cause the apparatus at least to perform using content of the one or more fields during the data collection mode as part of the conformance testing.

[0253] Example 43. The apparatus according to example 42 wherein the one or more fields contain at least an indication about a size of the training data to be provided to the DUT during the conformance testing.

[0254] Example 44. The apparatus according to example 43, wherein the one or more memories further store instructions that, when executed by the one or more processors, cause the apparatus at least to perform: receiving, by the DUT, the training data using the one or more fields inside the Al / ML test function, or through the PDSCH (physical downlink shared channel) resources, or through AT command(s), or through other resources.

[0255] Example 45. The apparatus according to any of examples 42 to 44, wherein the one or more memories further store instructions that, when executed by the one or more processors, cause the apparatus at least to perform: activating, by the DUT, the training mode based on another AI / ML test function indicating the mode to be performed by the DUT is the training mode; and performing, by the DUT, training of the one or both of the AI / ML model or functionality using the training data from the data collection mode.

[0256] Example 46. The apparatus according to any of examples 37 to 45, wherein the AI / ML test function and its associated testing features, including a mode which applies to the use case, corresponding to the one or both of the AI / ML model or functionality are provided to the DUT using at least one defined UE special conformance testing function for a third generation partnership project (3GPP) over-the-air (OTA) interface.

[0257] Example 47. The apparatus according to any of examples 37 to 45, wherein the AI / ML test function and its associated testing features, including a mode which applies to the use case, corresponding to the one or both of the AI / ML model or functionality are provided to the DUT using standardized attention (AT) commands which requires an external interface between a system simulator (SS) and the DUT.

[0258] Example 48. The apparatus according to any of examples 37 to 45, wherein the AI / ML test function and its associated testing features, including a mode which applies to the use case, corresponding to the one or both of the AI / ML model or functionality is provided to the DUT using a combination of the following: at least one defined UE special conformance testing function for a third generation partnership project (3GPP) over-the-air (OTA) interface; and standardized attention (AT) commands which requires an external interface between a system simulator (SS) and the DUT.

[0259] Example 49. The apparatus according to any of examples 37 to 48, wherein: the associated one or more of the multiple testing features comprise one or more specific fields that carry information specific to the use case; and the one or more memories further store instructions that, when executed by the one or more processors, cause the apparatus at least to perform using by the DUT the information specific to the use case in the one or more specific fields as part of the conformance testing.

[0260] Example 50. The apparatus according to any of examples 37 to 49, wherein: the Al / ML test function further indicates the following using a single set of bits: a use case is indicated using a first portion of the single set of bits; and a mode is indicated using a second portion of the single set of bits.

[0261] Example 51. The apparatus according to any of examples 37 to 49, wherein: the Al / ML test function further indicates the following: a use case is indicated using a first set of bits; and a mode is indicated using a second set of bits, wherein the first and second sets of bits are different sets of bits not within a single set of bits.

[0262] Example 52. An apparatus, comprising: one or more processors; and one or more memories storing instructions that, when executed by the one or more processors, cause the apparatus at least to perform: by a system simulator (SS) as part of conformance testing of a device under test (DUT), sending indication to the DUT to activate an artificial intelligence / machine learning (AI / ML) test function used for the conformance testing, the AI / ML test function comprising indication of a use case, as one of multiple testing features, that corresponds to one or both of an AI / ML model or functionality for which the conformance testing is to be performed; receiving, by the SS from the DUT, indication that at least the AI / ML test function has been activated; and participating, by the SS based on the activated AI / ML test function and in conjunction with the DUT, in the conformance testing of the DUT.

[0263] Example 53. The apparatus according to example 52, wherein the one or more memories further store instructions that, when executed by the one or more processors, cause the apparatus at least to perform: sending, from the SS to the DUT, an indicator indicating the Al / ML test function should be deactivated; and receiving, by the SS from the DUT, indication that at least the AI / ML test function has been deactivated.

[0264] Example 54. The apparatus according to any of examples 52 or 53, wherein participating in the conformance testing of the DUT comprises running a conformance test and generating an inference of whether or not the DUT is in compliance with one or more conformance requirements.

[0265] Example 55. A computer program, comprising instructions which, when the program is executed by an apparatus, cause the apparatus to carry out the methods of any of examples 1 to 18.

[0266] Example 56. The computer program according to example 55, wherein the computer program is a computer program product comprising a computer-readable medium bearing the instructions embodied therein for use with the apparatus.

[0267] Example 57. The computer program according to example 55, wherein the computer program is directly loadable into an internal memory of the apparatus.

[0268] As used in this application, the term “circuitry” may refer to one or more or all of the following:

[0269] (a) hardware-only circuit implementations (such as implementations in only analog and / or digital circuitry) and

[0270] (b) combinations of hardware circuits and software, such as (as applicable): (i) a combination of analog and / or digital hardware circuit(s) with software / firmware and (ii) any portions of hardware processor(s) (including digital signal processor(s)) with software, and memory(ies) that work together to cause an apparatus, such as a mobile phone or server, to perform various functions) and

[0271] (c) hardware circuit(s) and or processor(s), such as a microprocessor(s) or a portion of a microprocessor(s), that requires software (e.g., firmware) for operation, but the software may not be present when it is not needed for operation.

[0272] This definition of circuitry applies to all uses of this term in this application, including in any claims. As a further example, as used in this application, the term circuitry alsocovers an implementation of merely a hardware circuit or processor (or multiple processors) or portion of a hardware circuit or processor and its (or their) accompanying software and / or firmware. The term circuitry also covers, for example and if applicable to the particular claim element, a baseband integrated circuit or processor integrated circuit for a mobile device or a similar integrated circuit in server, a cellular network device, or other computing or network device.

[0273] Embodiments herein may be implemented in software (executed by one or more processors), hardware (e.g., an application specific integrated circuit), or a combination of software and hardware. In an example embodiment, the software (e.g., application logic, an instruction set) is maintained on any one of various conventional computer-readable media. In the context of this document, a “computer-readable medium” may be any media or means that can contain, store, communicate, propagate or transport the instructions for use by or in connection with an instruction execution system, apparatus, or device, such as a computer, with one example of a computer described and depicted, e.g., in FIG. 8. A computer-readable medium may comprise a computer-readable storage medium (e.g., memories 15 and 75 or other device) that may be any media or means that can contain, store, and / or transport the instructions for use by or in connection with an instruction execution system, apparatus, or device, such as a computer. A computer-readable storage medium does not comprise propagating signals, and therefore may be considered to be non-transitory. The term “non-transitory”, as used herein, is a limitation of the medium itself (i.e., tangible, not a signal) as opposed to a limitation on data storage persistency (e.g., RAM, random access memory, versus ROM, read-only memory).

[0274] If desired, the different functions discussed herein may be performed in a different order and / or concurrently with each other. Furthermore, if desired, one or more of the above-described functions may be optional or may be combined.

[0275] Although various aspects of the invention are set out in the independent claims, other aspects of the invention comprise other combinations of features from the described embodiments and / or the dependent claims with the features of the independent claims, and not solely the combinations explicitly set out in the claims.

[0276] It is also noted herein that while the above describes example embodiments of the invention, these descriptions should not be viewed in a limiting sense. Rather, there areseveral variations and modifications which may be made without departing from the scope of the present invention as defined in the appended claims.

[0277] The following abbreviations that may be found in the specification and / or the drawing figures are defined as follows:

[0278] 3GPP third generation partnership project

[0279] 5G fifth generation

[0280] 6G sixth generation

[0281] 5GS fifth generation system

[0282] A I / ML or AIML artificial intelligence / machine learning

[0283] AMF access and mobility management function

[0284] AT Command attention command

[0285] CSI channel state information

[0286] DL downlink (from network to UE)

[0287] DUT device under test

[0288] E-SMLC evolved serving mobile location center

[0289] EMMI Electrical Man Machine Interface

[0290] eNB (or eNodeB) evolved Node B (e.g., an LTE base station)

[0291] GMLC Gateway Mobile Location Center

[0292] gNB (or gNodeB) base station for 5G / NR

[0293] IE information element

[0294] I / F interface

[0295] LMF Location Management Function

[0296] LTE long term evolution

[0297] MM Mobility management

[0298] MME mobility management entity

[0299] MT Mobile Termination

[0300] NF network function

[0301] ng or NG next generation

[0302] NR new radio

[0303] NRF Network Repository Function

[0304] N / W or NW network

[0305] OTA over the air

[0306] PDSCH physical downlink shared channel

[0307] RAN radio access network

[0308] RRC radio resource control

[0309] Rx receiver

[0310] SGW serving gateway

[0311] SMF session management function

[0312] SS system simulator

[0313] TA terminal adapter

[0314] TE Terminal Equipment

[0315] TMC Test Mode Control

[0316] TRP transmission-reception point

[0317] Tx transmitter

[0318] UDM unified data management

[0319] UDR unified data repository

[0320] UE user equipment (e.g., a wireless, typically mobile device)

[0321] UL uplink (from UE to network)

[0322] UPF user plane function

[0323] V2X vehicle to everything

[0324] WID Working item description

Claims

What is claimed is:

1. A method, comprising: by a device under test (DUT) as part of conformance testing of the DUT, receiving indication to activate an artificial intelligence / machine learning (AI / ML) test function for the conformance testing, the AI / ML test function comprising indication of a use case, as one of multiple testing features associated with the Al / ML test function, that corresponds to one or both of an Al / ML model or functionality for which the conformance testing is to be performed; activating, by the DUT, the Al / ML test function and associated one or more of the multiple testing features corresponding to the indicated use case; and performing, by the DUT based on the activated AI / ML test function, the one or both of the AI / ML model or functionality for the indicated use case and associated testing features as part of the conformance testing.

2. The method according to claim 1, further comprising: receiving, by the DUT, an indicator indicating the AI / ML test function should be deactivated; and deactivating, by the DUT, the AI / ML test function and the testing features corresponding to the one or both of the AI / ML model or functionality.

3. The method according to any of claims 1 to 2, wherein the use case comprises one of the following: AI / ML enabled positioning of the DUT; AI / ML enabled beam management; AI / ML enabled channel state information (CSI) compression; or AI / ML enabled CSI prediction, and the use case is used to select a corresponding AI / ML based functionality.

4. The method according to claim 3, wherein the corresponding AI / ML based functionality comprises one of the following that corresponds to the use case: AI / ML basedpositioning; AI / ML based channel state information (CSI) compression; AI / ML based beam management; or AI / ML based CSI prediction.

5. The method according to any of claims 1 to 5, wherein: the AI / ML test function further indicates a mode to be performed by the DUT, the mode being one of the associated testing features, wherein the mode comprises one of the following: data collection mode; training mode; or inference mode; and performing, by the DUT, the indicated mode, as part of the conformance testing, for the one or both of an AI / ML model or functionality.

6. The method according to claim 5, wherein: the mode is the data collection mode; the AI / ML test function further contains one or more fields to be used for providing training data to the DUT, to train the one or more of the AI / ML model or functionality at least on a DUT side, during the conformance testing; and the method further comprises using content of the one or more fields during the data collection mode as part of the conformance testing.

7. The method according to claim 6 wherein the one or more fields contain at least an indication about a size of the training data to be provided to the DUT during the conformance testing.

8. The method according to claim 7, further comprising: receiving, by the DUT, the training data using the one or more fields inside the AI / ML test function, or through the PDSCH (physical downlink shared channel) resources, or through AT command(s), or through other resources.

9. The method according to any of claims 6 to 8, wherein the method further comprises: activating, by the DUT, the training mode based on another AI / ML test function indicating the mode to be performed by the DUT is the training mode; andperforming, by the DUT, training of the one or both of the Al / ML model or functionality using the training data from the data collection mode.

10. The method according to any of claims 1 to 9, wherein the Al / ML test function and its associated testing features, including a mode which applies to the use case, corresponding to the one or both of the Al / ML model or functionality are provided to the DUT using at least one defined UE special conformance testing function for a third generation partnership project (3GPP) over-the-air (OTA) interface.

11. The method according to any of claims 1 to 9, wherein the Al / ML test function and its associated testing features, including a mode which applies to the use case, corresponding to the one or both of the Al / ML model or functionality are provided to the DUT using standardized attention (AT) commands which requires an external interface between a system simulator (SS) and the DUT.

12. The method according to any of claims 1 to 9, wherein the Al / ML test function and its associated testing features, including a mode which applies to the use case, corresponding to the one or both of the Al / ML model or functionality is provided to the DUT using a combination of the following: at least one defined UE special conformance testing function for a third generation partnership project (3GPP) over-the-air (OTA) interface; and standardized attention (AT) commands which requires an external interface between a system simulator (SS) and the DUT.

13. The method according to any of claims 1 to 12, wherein: the associated one or more of the multiple testing features comprise one or more specific fields that carry information specific to the use case; and the method further comprises using by the DUT the information specific to the use case in the one or more specific fields as part of the conformance testing.

14. The method according to any of claims 1 to 13, wherein: the AI / ML test function further indicates the following using a single set of bits: a use case is indicated using a first portion of the single set of bits; and a mode is indicated using a second portion of the single set of bits.

15. The method according to any of claims 1 to 13, wherein: the AI / ML test function further indicates the following: a use case is indicated using a first set of bits; and a mode is indicated using a second set of bits, wherein the first and second sets of bits are different sets of bits not within a single set of bits.

16. A method, comprising: by a system simulator (SS) as part of conformance testing of a device under test (DUT), sending indication to the DUT to activate an artificial intelligence / machine learning (AI / ML) test function used for the conformance testing, the AI / ML test function comprising indication of a use case, as one of multiple testing features, that corresponds to one or both of an AI / ML model or functionality for which the conformance testing is to be performed; receiving, by the SS from the DUT, indication that at least the AI / ML test function has been activated; and participating, by the SS based on the activated AI / ML test function and in conjunction with the DUT, in the conformance testing of the DUT.

17. The method according to claim 16, further comprising: sending, from the SS to the DUT, an indicator indicating the AI / ML test function should be deactivated; and receiving, by the SS from the DUT, indication that at least the AI / ML test function has been deactivated.

18. The method according to any of claims 16 or 17, wherein participating in the conformance testing of the DUT comprises running a conformance test and generating aninference of whether or not the DUT is in compliance with one or more conformance requirements.

19. An apparatus, comprising means for: by a device under test (DUT) as part of conformance testing of the DUT, receiving indication to activate an artificial intelligence / machine learning (AI / ML) test function for the conformance testing, the AI / ML test function comprising indication of a use case, as one of multiple testing features associated with the AI / ML test function, that corresponds to one or both of an AI / ML model or functionality for which the conformance testing is to be performed; activating, by the DUT, the AI / ML test function and associated one or more of the multiple testing features corresponding to the indicated use case; and performing, by the DUT based on the activated AI / ML test function, the one or both of the AI / ML model or functionality for the indicated use case and associated testing features as part of the conformance testing.

20. The apparatus according to claim 19, wherein the means are further configured for: receiving, by the DUT, an indicator indicating the AI / ML test function should be deactivated; and deactivating, by the DUT, the AI / ML test function and the testing features corresponding to the one or both of the AI / ML model or functionality.

21. The apparatus according to any of claims 19 to 20, wherein the use case comprises one of the following: AI / ML enabled positioning of the DUT; AI / ML enabled beam management; AI / ML enabled channel state information (CSI) compression; or AI / ML enabled CSI prediction, and the use case is used to select a corresponding AI / ML based functionality.

22. The apparatus according to claim 21, wherein the corresponding AI / ML based functionality comprises one of the following that corresponds to the use case: AI / MLbased positioning; AI / ML based channel state information (CSI) compression; AI / ML based beam management; or Al / ML based CSI prediction.

23. The apparatus according to any of claims 19 to 23, wherein: the AI / ML test function further indicates a mode to be performed by the DUT, the mode being one of the associated testing features, wherein the mode comprises one of the following: data collection mode; training mode; or inference mode; and performing, by the DUT, the indicated mode, as part of the conformance testing, for the one or both of an AI / ML model or functionality.

24. The apparatus according to claim 23, wherein: the mode is the data collection mode; the AI / ML test function further contains one or more fields to be used for providing training data to the DUT, to train the one or more of the AI / ML model or functionality at least on a DUT side, during the conformance testing; and the means are further configured for using content of the one or more fields during the data collection mode as part of the conformance testing.

25. The apparatus according to claim 24 wherein the one or more fields contain at least an indication about a size of the training data to be provided to the DUT during the conformance testing.

26. The apparatus according to claim 25, wherein the means are further configured for: receiving, by the DUT, the training data using the one or more fields inside the AI / ML test function, or through the PDSCH (physical downlink shared channel) resources, or through AT command(s), or through other resources.

27. The apparatus according to any of claims 24 to 26, wherein the means are further configured for : activating, by the DUT, the training mode based on another AI / ML test function indicating the mode to be performed by the DUT is the training mode; andperforming, by the DUT, training of the one or both of the Al / ML model or functionality using the training data from the data collection mode.

28. The apparatus according to any of claims 19 to 27, wherein the Al / ML test function and its associated testing features, including a mode which applies to the use case, corresponding to the one or both of the Al / ML model or functionality are provided to the DUT using at least one defined UE special conformance testing function for a third generation partnership project (3GPP) over-the-air (OTA) interface.

29. The apparatus according to any of claims 19 to 27, wherein the Al / ML test function and its associated testing features, including a mode which applies to the use case, corresponding to the one or both of the Al / ML model or functionality are provided to the DUT using standardized attention (AT) commands which requires an external interface between a system simulator (SS) and the DUT.

30. The apparatus according to any of claims 19 to 27, wherein the Al / ML test function and its associated testing features, including a mode which applies to the use case, corresponding to the one or both of the Al / ML model or functionality is provided to the DUT using a combination of the following: at least one defined UE special conformance testing function for a third generation partnership project (3GPP) over-the-air (OTA) interface; and standardized attention (AT) commands which requires an external interface between a system simulator (SS) and the DUT.

31. The apparatus according to any of claims 19 to 30, wherein: the associated one or more of the multiple testing features comprise one or more specific fields that carry information specific to the use case; and the means are further configured for using by the DUT the information specific to the use case in the one or more specific fields as part of the conformance testing.

32. The apparatus according to any of claims 19 to 31, wherein: the AI / ML test function further indicates the following using a single set of bits: a use case is indicated using a first portion of the single set of bits; and a mode is indicated using a second portion of the single set of bits.

33. The apparatus according to any of claims 19 to 31, wherein: the AI / ML test function further indicates the following: a use case is indicated using a first set of bits; and a mode is indicated using a second set of bits, wherein the first and second sets of bits are different sets of bits not within a single set of bits.

34. An apparatus, comprising means for: by a system simulator (SS) as part of conformance testing of a device under test (DUT), sending indication to the DUT to activate an artificial intelligence / machine learning (AI / ML) test function used for the conformance testing, the AI / ML test function comprising indication of a use case, as one of multiple testing features, that corresponds to one or both of an AI / ML model or functionality for which the conformance testing is to be performed; receiving, by the SS from the DUT, indication that at least the AI / ML test function has been activated; and participating, by the SS based on the activated AI / ML test function and in conjunction with the DUT, in the conformance testing of the DUT.

35. The apparatus according to claim 34, wherein the means are further configured for: sending, from the SS to the DUT, an indicator indicating the AI / ML test function should be deactivated; and receiving, by the SS from the DUT, indication that at least the AI / ML test function has been deactivated.

36. The apparatus according to any of claims 34 or 35, wherein participating in the conformance testing of the DUT comprises running a conformance test and generating aninference of whether or not the DUT is in compliance with one or more conformance requirements.

37. An apparatus, comprising: one or more processors; and one or more memories storing instructions that, when executed by the one or more processors, cause the apparatus at least to perform: by a device under test (DUT) as part of conformance testing of the DUT, receiving indication to activate an artificial intelligence / machine learning (AI / ML) test function for the conformance testing, the AI / ML test function comprising indication of a use case, as one of multiple testing features associated with the AI / ML test function, that corresponds to one or both of an AI / ML model or functionality for which the conformance testing is to be performed; activating, by the DUT, the AI / ML test function and associated one or more of the multiple testing features corresponding to the indicated use case; and performing, by the DUT based on the activated AI / ML test function, the one or both of the AI / ML model or functionality for the indicated use case and associated testing features as part of the conformance testing.

38. The apparatus according to claim 37, wherein the one or more memories further store instructions that, when executed by the one or more processors, cause the apparatus at least to perform: receiving, by the DUT, an indicator indicating the AI / ML test function should be deactivated; and deactivating, by the DUT, the AI / ML test function and the testing features corresponding to the one or both of the AI / ML model or functionality.

39. The apparatus according to any of claims 37 to 38, wherein the use case comprises one of the following: AI / ML enabled positioning of the DUT; AI / ML enabled beam management; AI / ML enabled channel state information (CSI) compression; or AI / MLenabled CSI prediction, and the use case is used to select a corresponding Al / ML based functionality.

40. The apparatus according to claim 39, wherein the corresponding Al / ML based functionality comprises one of the following that corresponds to the use case: Al / ML based positioning; Al / ML based channel state information (CSI) compression; Al / ML based beam management; or Al / ML based CSI prediction.

41. The apparatus according to any of claims 37 to 41, wherein: the Al / ML test function further indicates a mode to be performed by the DUT, the mode being one of the associated testing features, wherein the mode comprises one of the following: data collection mode; training mode; or inference mode; and performing, by the DUT, the indicated mode, as part of the conformance testing, for the one or both of an Al / ML model or functionality.

42. The apparatus according to claim 41, wherein: the mode is the data collection mode; the Al / ML test function further contains one or more fields to be used for providing training data to the DUT, to train the one or more of the Al / ML model or functionality at least on a DUT side, during the conformance testing; and the one or more memories further store instructions that, when executed by the one or more processors, cause the apparatus at least to perform using content of the one or more fields during the data collection mode as part of the conformance testing.

43. The apparatus according to claim 42 wherein the one or more fields contain at least an indication about a size of the training data to be provided to the DUT during the conformance testing.

44. The apparatus according to claim 43, wherein the one or more memories further store instructions that, when executed by the one or more processors, cause the apparatus at least to perform: receiving, by the DUT, the training data using the one or more fields inside the Al / ML test function, or through the PDSCH (physical downlink shared channel) resources, or through AT command(s), or through other resources.

45. The apparatus according to any of claims 42 to 44, wherein the one or more memories further store instructions that, when executed by the one or more processors, cause the apparatus at least to perform: activating, by the DUT, the training mode based on another Al / ML test function indicating the mode to be performed by the DUT is the training mode; and performing, by the DUT, training of the one or both of the Al / ML model or functionality using the training data from the data collection mode.

46. The apparatus according to any of claims 37 to 45, wherein the Al / ML test function and its associated testing features, including a mode which applies to the use case, corresponding to the one or both of the Al / ML model or functionality are provided to the DUT using at least one defined UE special conformance testing function for a third generation partnership project (3GPP) over-the-air (OTA) interface.

47. The apparatus according to any of claims 37 to 45, wherein the Al / ML test function and its associated testing features, including a mode which applies to the use case, corresponding to the one or both of the Al / ML model or functionality are provided to the DUT using standardized attention (AT) commands which requires an external interface between a system simulator (SS) and the DUT.

48. The apparatus according to any of claims 37 to 45, wherein the Al / ML test function and its associated testing features, including a mode which applies to the use case,corresponding to the one or both of the AI / ML model or functionality is provided to the DUT using a combination of the following: at least one defined UE special conformance testing function for a third generation partnership project (3GPP) over-the-air (OTA) interface; and standardized attention (AT) commands which requires an external interface between a system simulator (SS) and the DUT.

49. The apparatus according to any of claims 37 to 48, wherein: the associated one or more of the multiple testing features comprise one or more specific fields that carry information specific to the use case; and the one or more memories further store instructions that, when executed by the one or more processors, cause the apparatus at least to perform using by the DUT the information specific to the use case in the one or more specific fields as part of the conformance testing.

50. The apparatus according to any of claims 37 to 49, wherein: the AI / ML test function further indicates the following using a single set of bits: a use case is indicated using a first portion of the single set of bits; and a mode is indicated using a second portion of the single set of bits.

51. The apparatus according to any of claims 37 to 49, wherein: the AI / ML test function further indicates the following: a use case is indicated using a first set of bits; and a mode is indicated using a second set of bits, wherein the first and second sets of bits are different sets of bits not within a single set of bits.

52. An apparatus, comprising: one or more processors; and one or more memories storing instructions that, when executed by the one or more processors, cause the apparatus at least to perform:by a system simulator (SS) as part of conformance testing of a device under test (DUT), sending indication to the DUT to activate an artificial intelligence / machine learning (AI / ML) test function used for the conformance testing, the AI / ML test function comprising indication of a use case, as one of multiple testing features, that corresponds to one or both of an AI / ML model or functionality for which the conformance testing is to be performed; receiving, by the SS from the DUT, indication that at least the AI / ML test function has been activated; and participating, by the SS based on the activated AI / ML test function and in conjunction with the DUT, in the conformance testing of the DUT.

53. The apparatus according to claim 52, wherein the one or more memories further store instructions that, when executed by the one or more processors, cause the apparatus at least to perform: sending, from the SS to the DUT, an indicator indicating the AI / ML test function should be deactivated; and receiving, by the SS from the DUT, indication that at least the AI / ML test function has been deactivated.

54. The apparatus according to any of claims 52 or 53, wherein participating in the conformance testing of the DUT comprises running a conformance test and generating an inference of whether or not the DUT is in compliance with one or more conformance requirements.

55. A computer program, comprising instructions which, when the program is executed by an apparatus, cause the apparatus to carry out the methods of any of claims 1 to 18.

56. The computer program according to claim 55, wherein the computer program is a computer program product comprising a computer-readable medium bearing the instructions embodied therein for use with the apparatus.

57. The computer program according to claim 55, wherein the computer program is directly loadable into an internal memory of the apparatus.

Images