Application and verification of AI / ML model capability

Abstract

A terminal device 120 transmits 405, to test equipment 110, capability information indicating generalization capability of an artificial intelligence / machine learning (AI / ML) model or functionality in the terminal device 120, the generalization capability at least comprising pair information, the pair information indicating a plurality of pairs of scenarios and communication configurations in enabling the AI / ML model or functionality. The terminal device 120 receives 430, from the test equipment 110, an indication of a generalization score of the AI / ML model or functionality. The terminal device 120 may also receive 420, from the test equipment 110, an instruction of running the AI / ML model or functionality according to the plurality of pairs of scenarios and communication configurations and, in response to the instruction, cause the AI / ML model or functionality to be run according to the plurality of pairs of scenarios and communication configurations. The AI / ML model or functionality may be configured for channel state information (CSI) compression and is paired with a further AI / ML model or functionality in the test equipment 110 that is configured for CSI decompression.

Description

[0002] In the telecommunication industry, artificial intelligence / machine learning (AI / ML) have been employed in telecommunication systems to improve the performance. The 3rd Generation Partnership Project (3GPP) Release-18 started the study on AI / ML for New Radio (NR) air interface. The goal is to explore the benefits of augmenting the air interface with features enabling improved support of AI / ML-based algorithms for enhanced performance and / or reduced complexity / overhead. Several use cases are considered to enable the identification of a common AI / ML framework, including functional requirements of AI / ML architecture, which could be used in subsequent projects. It also aims to test and verify various aspects of capability of the AI / ML models or functionalities in the communication systems. SUMMARY

[0003] In a first aspect of the present disclosure, there is provided a first apparatus. The first apparatus comprises at least one processor; and at least one memory storing instructions that, when executed by the at least one processor, cause the first apparatus at least to: transmit, to a second apparatus, capability information indicating generalization capability of an artificial intelligence / machine learning (AI / ML) model or functionality in the first apparatus, the generalization capability at least comprising pair information, the pair information indicating a plurality of pairs of scenarios and communication configurations in enabling the AI / ML model or functionality; and receive, from the second apparatus, an indication of a generalization score of the AI / ML model or functionality.

[0004] In a second aspect of the present disclosure, there is provided a second apparatus. The second apparatus comprises at least one processor; and at least one memory storing instructions that, when executed by the at least one processor, cause the second apparatus at least to: receive, from a first apparatus capability information indicating generalization capability of an artificial intelligence (AI) / machine learning (ML) model or functionality in the first apparatus, the generalization capability at least comprising pair information, the pair information indicating a plurality of pairs of scenarios and communication configurations in enabling the AI / ML model or functionality; perform generalization test on the AI / ML model in the first apparatus based on the generalization capability of the AI / ML model; and transmit, to the first apparatus and based on a result of the generalization test, an indication of a generalization score of the AI / ML model or functionality.

[0005] In a third aspect of the present disclosure, there is provided a method. The method comprises: transmitting, to a second apparatus, capability information indicating generalization capability of an artificial intelligence / machine learning (AI / ML) model or functionality in the first apparatus, the generalization capability at least comprising pair information, the pair information indicating a plurality of pairs of scenarios and communication configurations in enabling the AI / ML model or functionality; and receiving, from the second apparatus, an indication of a generalization score of the AI / ML model or functionality.

[0006] In a fourth aspect of the present disclosure, there is provided a method. The method comprises: receiving, from a first apparatus capability information indicating generalization capability of an artificial intelligence (AI) / machine learning (ML) model or functionality in the first apparatus, the generalization capability at least comprising pair information, the pair information indicating a plurality of pairs of scenarios and communication configurations in enabling the AI / ML model or functionality; performing generalization test on the AI / ML model in the first apparatus based on the generalization capability of the AI / ML model; and transmitting, to the first apparatus and based on a result of the generalization test, an indication of a generalization score of the AI / ML model or functionality.

[0007] In a fifth aspect of the present disclosure, there is provided a first apparatus. The first apparatus comprises means for transmitting, to a second apparatus, capability information indicating generalization capability of an artificial intelligence / machine learning (AI / ML) model or functionality in the first apparatus, the generalization capability at least comprising pair information, the pair information indicating a plurality of pairs of scenarios and communication configurations in enabling the AI / ML model or functionality; and means for receiving, from the second apparatus, an indication of a generalization score of the AI / ML model or functionality.

[0008] In a sixth aspect of the present disclosure, there is provided a second apparatus. The second apparatus comprises means for receiving, from a first apparatus capability information indicating generalization capability of an artificial intelligence (AI) / machine learning (ML) model or functionality in the first apparatus, the generalization capability at least comprising pair information, the pair information indicating a plurality of pairs of scenarios and communication configurations in enabling the AI / ML model or functionality; means for performing generalization test on the AI / ML model in the first apparatus based on the generalization capability of the AI / ML model; and means for transmitting, to the first apparatus and based on a result of the generalization test, an indication of a generalization score of the AI / ML model or functionality.

[0009] In a seventh aspect of the present disclosure, there is provided a computer readable medium. The computer readable medium comprises instructions stored thereon for causing an apparatus to perform at least the method according to the third aspect.

[0010] In an eighth aspect of the present disclosure, there is provided a computer readable medium. The computer readable medium comprises instructions stored thereon for causing an apparatus to perform at least the method according to the fourth aspect.

[0011] It is to be understood that the Summary section is not intended to identify key or essential features of embodiments of the present disclosure, nor is it intended to be used to limit the scope of the present disclosure. Other features of the present disclosure will become easily comprehensible through the following description. BRIEF DESCRIPTION OF THE DRAWINGS

[0012] Some example embodiments will now be described with reference to the accompanying drawings, where:

[0013] FIG. 1 illustrates an example communication environment in which example embodiments of the present disclosure can be implemented;

[0014] FIG. 2 illustrates an example architecture of an artificial intelligence / machine learning (AI / ML) model or functionality;

[0015] FIG. 3A illustrates generalization capabilities of AI / ML models or functionalities;

[0016] FIG. 3B illustrates performance of generalized models or functionalities with respect to time;

[0017] FIG. 4 illustrates a signaling flow for application and verification of AI / ML model or functionality capability in a test stage in accordance with some example embodiments of the present disclosure;

[0018] FIG. 5 illustrates a signaling flow for application and verification of AI / ML model or functionality capability in an in-field stage in accordance with some example embodiments of the present disclosure;

[0019] FIG. 6A illustrates a block diagram of the generalization test framework for AI / ML model or functionality in accordance with some example embodiments of the present disclosure;

[0020] FIG. 6B illustrates a detailed signaling flow for application and verification of AI / ML model or functionality capability in a test stage in accordance with some further example embodiments of the present disclosure;

[0021] FIG. 7 illustrates a detailed signaling flow for application and verification of AI / ML model or functionality capability in an in-field stage in accordance with some further example embodiments of the present disclosure;

[0022] FIG. 8 illustrates a flowchart of a method implemented at a first apparatus in an in-field stage in accordance with some example embodiments of the present disclosure;

[0023] FIG. 9 illustrates a flowchart of a method implemented at a second apparatus in an in-field stage in accordance with some example embodiments of the present disclosure;

[0024] FIG. 10 illustrates a flowchart of a method implemented at a first apparatus in a test stage in accordance with some example embodiments of the present disclosure;

[0025] FIG. 11 illustrates a flowchart of a method implemented at a second apparatus in a test stage in accordance with some example embodiments of the present disclosure;

[0026] FIG. 12 illustrates a simplified block diagram of a device that is suitable for implementing example embodiments of the present disclosure; and

[0027] FIG. 13 illustrates a block diagram of an example computer readable medium in accordance with some example embodiments of the present disclosure.

[0028] Throughout the drawings, the same or similar reference numerals represent the same or similar element. DETAILED DESCRIPTION

[0029] Principle of the present disclosure will now be described with reference to some example embodiments. It is to be understood that these embodiments are described only for the purpose of illustration and help those skilled in the art to understand and implement the present disclosure, without suggesting any limitation as to the scope of the disclosure. Embodiments described herein can be implemented in various manners other than the ones described below.

[0030] In the following description and claims, unless defined otherwise, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skills in the art to which this disclosure belongs.

[0031] References in the present disclosure to “one embodiment,” “an embodiment,” “an example embodiment,” and the like indicate that the embodiment described may include a particular feature, structure, or characteristic, but it is not necessary that every embodiment includes the particular feature, structure, or characteristic. Moreover, such phrases are not necessarily referring to the same embodiment. Further, when a particular feature, structure, or characteristic is described in connection with an embodiment, it is submitted that it is within the knowledge of one skilled in the art to affect such feature, structure, or characteristic in connection with other embodiments whether or not explicitly described.

[0032] It shall be understood that although the terms “first,” “second,”..., etc. in front of noun(s) and the like may be used herein to describe various elements, these elements should not be limited by these terms. These terms are only used to distinguish one element from another and they do not limit the order of the noun(s). For example, a first element could be termed a second element, and similarly, a second element could be termed a first element, without departing from the scope of example embodiments. As used herein, the term “and / or” includes any and all combinations of one or more of the listed terms.

[0033] As used herein, “at least one of the following: ” and “at least one of ” and similar wording, where the list of two or more elements are joined by “and” or “or”, mean at least any one of the elements, or at least any two or more of the elements, or at least all the elements. 5

[0034] As used herein, unless stated explicitly, performing a step “in response to A” does not indicate that the step is performed immediately after “A” occurs and one or more intervening steps may be included.

[0035] The terminology used herein is for the purpose of describing particular embodiments only and is not intended to be limiting of example embodiments. As used 10 herein, the singular forms “a”, “an” and “the” are intended to include the plural forms as well, unless the context clearly indicates otherwise. It will be further understood that the terms “comprises”, “comprising”, “has”, “having”, “includes” and / or “including”, when used herein, specify the presence of stated features, elements, and / or components etc., but do not preclude the presence or addition of one or more other features, elements, 15 components and / or combinations thereof.

[0036] As used in this application, the term “circuitry” may refer to one or more or all of the following: (a) hardware-only circuit implementations (such as implementations in only analog and / or digital circuitry) and (b) combinations of hardware circuits and software, such as (as applicable): (i) a combination of analog and / or digital hardware circuit(s) with software / firmware and (ii) any portions of hardware processor(s) with software (including digital signal processor(s)), software, and memory(ies) that work together to cause an apparatus, such as a mobile phone or server, to perform various functions) and (c) hardware circuit(s) and or processor(s), such as a microprocessor(s) or a portion of a microprocessor(s), that requires software (e.g., firmware) for operation, but the software may not be present when it is not needed for operation.

[0037] This definition of circuitry applies to all uses of this term in this application, including in any claims. As a further example, as used in this application, the term circuitry also covers an implementation of merely a hardware circuit or processor (or multiple processors) or portion of a hardware circuit or processor and its (or their) accompanying software and / or firmware. The term circuitry also covers, for example and if applicable to the particular claim element, a baseband integrated circuit or processor integrated circuit for a mobile device or a similar integrated circuit in server, a cellular network device, or other computing or network device.

[0038] As used herein, the term “communication network” refers to a network following any suitable communication standards, such as New Radio (NR), Long Term Evolution (LTE), LTE-Advanced (LTE-A), Wideband Code Division Multiple Access (WCDMA), High-Speed Packet Access (HSPA), Narrow Band Internet of Things (NB-IoT) and so on. Furthermore, the communications between a terminal device and a network device in the communication network may be performed according to any suitable generation communication protocols, including, but not limited to, the first generation (1G), the second generation (2G), 2.5G, 2.75G, the third generation (3G), the fourth generation (4G), 4.5G, the fifth generation (5G), the sixth generation (6G) communication protocols, and / or any other protocols either currently known or to be developed in the future. Embodiments of the present disclosure may be applied in various communication systems. Given the rapid development in communications, there will of course also be future type communication technologies and systems with which the present disclosure may be embodied. It should not be seen as limiting the scope of the present disclosure to only the aforementioned system. Moreover, the communication may utilize any proper wireless communication technology, comprising but not limited to: Code Division Multiple Access (CDMA), Frequency Division Multiple Access (FDMA), Time Division Multiple Access (TDMA), Frequency Division Duplex (FDD), Time Division Duplex (TDD), Multiple-Input Multiple-Output (MIMO), Orthogonal Frequency Division Multiple (OFDM), Discrete Fourier Transform spread OFDM (DFT-s-OFDM) and / or any other technologies currently known or to be developed in the future.

[0039] As used herein, the term “network device” refers to a node in a communication network via which a terminal device accesses the network and receives services therefrom. The network device may refer to a base station (BS) or an access point (AP), for example, a node B (NodeB or NB), an evolved NodeB (eNodeB or eNB), an NR NB (also referred to as a gNB), a Remote Radio Unit (RRU), a radio header (RH), a remote radio head (RRH), a relay, an Integrated Access and Backhaul (IAB) node, a low power node such as a femto, a pico, a non-terrestrial network (NTN) or non-ground network device such as a satellite network device, a low earth orbit (LEO) satellite and a geosynchronous earth orbit (GEO) satellite, an aircraft network device, and so forth, depending on the applied terminology and technology. In some example embodiments, radio access network (RAN) split architecture comprises a Centralized Unit (CU) and a Distributed Unit (DU) at an IAB donor node. An IAB node comprises a Mobile Terminal (IAB-MT) part that behaves like a UE toward the parent node, and a DU part of an IAB node behaves like a base station toward the next-hop IAB node.

[0040] The term “terminal device” refers to any end device that may be capable of wireless communication. By way of example rather than limitation, a terminal device may also be referred to as a communication device, user equipment (UE), a Subscriber Station (SS), a Portable Subscriber Station, a Mobile Station (MS), or an Access Terminal (AT). The terminal device may include, but not limited to, a mobile phone, a cellular phone, a smart phone, voice over IP (VoIP) phones, wireless local loop phones, a tablet, a wearable terminal device, a personal digital assistant (PDA), portable computers, desktop computer, image capture terminal devices such as digital cameras, gaming terminal devices, music storage and playback appliances, vehicle-mounted wireless terminal devices, wireless endpoints, mobile stations, laptop-embedded equipment (LEE), laptop-mounted equipment (LME), USB dongles, smart devices, wireless customer-premises equipment (CPE), an Internet of Things (loT) device, a watch or other wearable, a head-mounted display (HMD), a vehicle, a drone, a medical device and applications (e.g., remote surgery), an industrial device and applications (e.g., a robot and / or other wireless devices operating in an industrial and / or an automated processing chain contexts), a consumer electronics device, a device operating on commercial and / or industrial wireless networks, and the like. The terminal device may also correspond to a Mobile Termination (MT) part of an IAB node (e.g., a relay node). In the following description, the terms “terminal device”, “communication device”, “terminal”, “user equipment” and “UE” may be used interchangeably.

[0041] As used herein, the term “resource,” “transmission resource,” “resource block,” “physical resource block” (PRB), “uplink resource,” or “downlink resource” may refer to any resource for performing a communication, for example, a communication between a terminal device and a network device, such as a resource in time domain, a resource in frequency domain, a resource in space domain, a resource in code domain, or any other combination of the time, frequency, space and / or code domain resource enabling a communication, and the like. In the following, unless explicitly stated, a resource in both frequency domain and time domain will be used as an example of a transmission resource for describing some example embodiments of the present disclosure. It is noted that example embodiments of the present disclosure are equally applicable to other resources in other domains.

[0042] As used herein, the term “model” is referred to as an association between an input and an output learned from training data, and thus a corresponding output may be generated for a given input after the training. The generation of the model may be based on machine learning (ML) techniques. The machine learning techniques may also be referred to as artificial intelligence (AI) techniques. In general, a machine learning model can be built, which receives input information and makes predictions based on the input information. For example, a classification model may predict a class of the input information among a predetermined set of classes. As used herein, “model” may also be referred to as “machine learning model”, “AI / ML model”, “learning model”, “machine learning network”, or “learning network,” which are used interchangeably herein. An “AI / ML functionality” may include at least one AI / ML model.

[0043] Generally, model lifecycle management may usually include three stages, i.e., a training stage, a validation stage, and an application stage (also referred to as an inference stage). At the training stage, a given AI / ML model may be trained (or optimized) iteratively using a great amount of training data until the model can make inference close to desired outputs in the training or labelled dataset. During the training, a set of parameter values of the model is iteratively updated until a training objective is reached. Through the training process, the AI / ML model may be regarded as being capable of learning the association between the input and the output (also referred to an input-output mapping) from the training data. At the validation stage, a validation input is applied to the trained AI / ML model to test whether the model can provide a correct output, so as to determine the performance of the model. Generally, the validation stage may be considered as a step in a training process, or may be omitted in some cases. At the inference stage, the resulting AI / ML model may be used to process a real-world model input based on the trained model obtained from the training process and to determine the corresponding model output. In some cases, a retraining or updating stage may be included in the model lifecycle management, to enable the model evolved to have better performance.

[0044] To facilitate understanding of the terminologies, some definitions of the list of terminologies used for AI / ML are provided below.

[0045] AI / ML Model: A data driven algorithm that applies AI / ML techniques to generate a set of outputs based on a set of inputs.

[0046] AI / ML model delivery: A generic term referring to delivery of an AI / ML model from one entity to another entity in any manner. Note: An entity could mean a network node / function (e.g., gNB, location management function (LMF), etc.), UE, proprietary server, etc.

[0047] AI / ML model Inference: A process of using a trained AI / ML model to produce a set of outputs based on a set of inputs.

[0048] AI / ML model testing: A subprocess of training, to evaluate the performance of a final AI / ML model using a dataset different from one used for model training and validation. Differently from AI / ML model validation, testing does not assume subsequent tuning of the model.

[0049] AI / ML model training: A process to train an AI / ML Model [by learning the input / output relationship] in a data driven manner and obtain the trained AI / ML Model for inference.

[0050] AI / ML model transfer: Delivery of an AI / ML model over the air interface in a manner that is not transparent to 3GPP signalling, either parameters of a model structure known at the receiving end or a new model with parameters. Delivery may contain a full model or a partial model.

[0051] AI / ML model validation: A subprocess of training, to evaluate the quality of an AI / ML model using a dataset different from one used for model training, that helps selecting model parameters that generalize beyond the dataset used for model training.

[0052] Data collection: A process of collecting data by the network nodes, management entity, or UE for the purpose of AI / ML model training, data analytics and inference.

[0053] Federated learning / federated training: A machine learning technique that trains an AI / ML model across multiple decentralized edge nodes (e.g., UEs, gNBs) each performing local model training using local data samples. The technique requires multiple interactions of the model, but no exchange of local data samples.

[0054] Functionality identification: A process / method of identifying an AI / ML functionality for the common understanding between the network and the UE. Note: Information regarding the AI / ML functionality may be shared during functionality identification. Where AI / ML functionality resides depends on the specific use cases and sub use cases.

[0055] Model activation: enable an AI / ML model for a specific function.

[0056] Model deactivation: disable an AI / ML model for a specific function.

[0057] Model download: Model transfer from the network to UE.

[0058] Model identification: A process / method of identifying an AI / ML model for the common understanding between the network (NW) and the UE. Note: The process / method of model identification may or may not be applicable. Note: Information regarding the AI / ML model may be shared during model identification.

[0059] Model monitoring: A procedure that monitors the inference performance of the AI / ML model.

[0060] Model parameter update: Process of updating the model parameters of a model.

[0061] Model selection: The process of selecting an AI / ML model for activation among multiple models for the same AI / ML enabled feature. Note: Model selection may or may not be carried out simultaneously with model activation.

[0062] Model switching: Deactivating a currently active AI / ML model and activating a different AI / ML model for a specific function.

[0063] Model update: Process of updating the model parameters and / or model structure of a model.

[0064] Model upload: Model transfer from UE to the network.

[0065] Network-side (AI / ML) model: An AI / ML Model whose inference is performed entirely at the network.

[0066] Offline field data: The data collected from field and used for offline training of the AI / ML model.

[0067] Offline training: An AI / ML training process where the model is trained based on collected dataset, and where the trained model is later used or delivered for inference. Note: This definition only serves as a guidance. There may be cases that may not exactly conform to this definition but could still be categorized as offline training by commonly accepted conventions.

[0068] Online field data: The data collected from field and used for online training of the AI / ML model.

[0069] Online training: An AI / ML training process where the model being used for inference) is (typically continuously) trained in (near) real-time with the arrival of new training samples. Note: the notion of (near) real-time vs. non real-time is context-dependent and is relative to the inference time-scale. Note: This definition only serves as a guidance. There may be cases that may not exactly conform to this definition but could still be categorized as online training by commonly accepted conventions. Note: Fine-tuning / re-training may be done via online or offline training. (This note could be removed when we define the term fine-tuning.)

[0070] Reinforcement Learning (RL): A process of training an AI / ML model from input (a.k.a. state) and a feedback signal (a.k.a. reward) resulting from the model’s output (a.k.a. action) in an environment the model is interacting with.

[0071] Semi-supervised learning: A process of training a model with a mix of labelled data and unlabelled data.

[0072] Two-sided (AI / ML) model: A paired AI / ML Model(s) over which joint inference is performed, where joint inference comprises AI / ML Inference whose inference is performed jointly across the UE and the network, i.e, the first part of inference is firstly performed by UE and then the remaining part is performed by the gNB, or vice versa.

[0073] UE-side (AI / ML) model: An AI / ML Model whose inference is performed entirely at the UE.

[0074] Unsupervised learning: A process of training a model without labelled data.

[0075] Proprietary-form at models: ML models of vendor- / device-specific proprietary format, from 3GPP perspective. They are not mutually recognizable across vendors and hide model design information from other vendors when shared. Note: An example is a device-specific binary executable format.

[0076] Open-format models: ML models of specified format that are mutually recognizable across vendors and allow interoperability, from the 3GPP perspective. They are mutually recognizable between vendors and do not hide model design information from other vendors when shared.

[0077] Model identification: A process / method of identifying an AI / ML model for the common understanding between the network device and the UE. Note: The process / method of model identification may or may not be applicable. Note: Information regarding the AI / ML model may be shared during model identification.

[0078] Functionality identification: A process / method of identifying an AI / ML functionality for the common understanding between the network device and the UE. Note: information regarding the AI / ML functionality may be shared during functionality identification.

[0079] FIG. 1 illustrates an example communication environment 100 in which example embodiments of the present disclosure can be implemented. It is to be understood that the elements shown in the communication environment 100 are intended to represent main functions provided within the system. As such, the blocks shown in FIG. 1 refer to specific elements in communication networks that provide these main functions. However, other network elements may be used to implement some or all of the main functions represented. Also, it is to be understood that not all functions of a communication network are depicted in FIG. 1. Rather, functions that facilitate an explanation of illustrative embodiments are represented. Further, the number of the elements shown in FIG. 1 is also for the purpose of illustration only and there may be any number of elements.

[0080] As shown, the communication environment 100 comprises a plurality of communication devices, including test equipment (TE) 110, one or more terminal devices 120-1, 120-2, ..., 120-N (collectively or individually referred to as terminal devices 120) and / or a network device 130. A serving area of the network device 130 may be called a cell. The terminal devices 120 may perform signal transmission and reception with the network device 130.

[0081] In some example embodiments, one or more AI / ML models or functionalities 125-1, 125-2, ..., 125-N (collectively or individually referred to as AI / ML models or functionalities 125) may be used by the one or more terminal devices 120. In some example embodiments, one or more AI / ML models or functionalities 135-1, 135-2 (collectively or individually referred to as AI / ML models or functionalities 135) may be used by the network device 130.

[0082] An AI / ML model or functionality may sometimes be referred to as an AI model or functionality or an ML model or functionality for short. Different AI / ML models or functionalities may be configured to implement the same different algorithms in the communication environment 100. The AI / ML model or functionality 125 used by a terminal device 120 may sometimes include to either a UE-side model or functionality. The AI / ML model or functionality 135 used by a network device 130 may sometimes include to either a network (NW)-side model or functionality.

[0083] In some example embodiments, the test equipment 110 is configured to test the AI / ML model(s) or functionality(ies) 125 used by the terminal devices 120. In some example embodiments, test equipment 110 may be a network entity in the core network, a base station (e.g., gNB, eNB) in RAN, or may be a terminal device (e.g., UE) or any other device that is configured for AI / ML testing. In the test stage, a terminal device 120 is considered as a device under test (DUT).

[0084] Generally, during the test stage, the terminal devices(s) 120 may only interact with the test equipment 110 which may behave like the network device 130. During the in-field stage, the terminal device(s) 120 and the network device 130 may communicate with each other, without the test equipment 110 involved.

[0085] In some example embodiments, two sided models may be applied to support some features in the communication environment. An AI / ML model or functionality 125 at the terminal device 120 is paired with an AI / ML model or functionality 135 at the network device 130, each to provide features to be implemented by the terminal device 120 or the network device 130. In such cases, in the test stage, to test the AI / ML model or functionality 125 at the terminal device 120, the test equipment 110 may also select an AI / ML model or functionality 115 which is similar to the paired AI / ML model or functionality 135 at the network device 130.

[0086] As an example, in the current standards as part of the channel state information (CSI) feedback framework, RI (rand indicator), PMI (Pre-coding Matrix Indication) and CQI (Channel Quality Indicator) could be jointly reported by UE to gNB according to the given configuration(s) by the gNB, where CQI may need more resources for feedback in the case of sub-band reporting. For codebook-based solutions, UE determines the CQI for reporting based on the precoding matrix indicated by the PMI and its associated receiver.

[0087] With the introduction of MU-MIMO (Multi-User Multiple-Input Multiple-Output) system, the overhead required to transmit high-resolution CSI feedback at high ranks in the uplink has increased by many folds. AI / ML-based solutions can help in reducing the overhead by compressing the CSI report. Various approaches are provided to achieve this using AI / ML based solutions using two sided models (Autoencoders).

[0088] Autoencoders are, by definition, matching the problem of CSI feedback compression. Indeed, autoencoders are an unsupervised learning technique where a bottleneck is imposed in the network to force a compressed knowledge representation of the original input. The main challenge remains the reconstruction of the original input.

[0089] Before training the autoencoder, four hyperparameters, among others, need to be set: 1) the size of the codeword / bottleneck, 2) the number of layers, 3) the number of nodes per layer, and 4) the loss function to be used, e.g., mean squared error (MSE), cosine similarity. The number of nodes per layer typically decreases with each subsequent layer of the encoder and increases back in the decoder. The decoder is symmetric to the encoder in terms of the layer structure.

[0090] An autoencoder consists of three parts, including the encoder, the bottleneck (codeword here), and the decoder. The encoder aims at compressing the input data, e.g., the channel matrix H or the eigenvectors, into a codeword that is of dimension smaller than the original information. The bottleneck, e.g., the codeword, is the compressed representation of the original information. The bottleneck is followed by the decoder, a module that decompress the codeword and reconstruct the data: the recovered information H. H is then compared to H. It is a lossy process, and the recovered matrix H will not be the same as H. An example architecture of autoencoders is illustrated in FIG. 2. The AI / ML model or functionality 125 at the terminal device 120 may be configured as the encoder of the autoencoder, to encode or compress CSI information to be reported by the terminal device 120. The AI / ML model or functionality 135 at the network device 130 may be configured as the decoder of the autoencoder, to decode or decompress the compressed information received by the network device 130, to reconstruct the CSI information.

[0091] The 3GPP Release 18 provides study on artificial intelligence (AI) / machine learning (ML) for NR air interface (FS NR AIML Air).

[0092] The 3GPP Release 9 starts the normative work for the general AI / ML framework for air interface and to enable the recommended use cases in the preceding study.

[0093] Since there are number of open issues left after the end of Release-18 study item, Release-19 WI will start with the continuation of the study. The following RAN4 5 objectives can be found in the new AI / ML WID. Table 1 Testability and interoperability [RAN4]: o Finalize the testing framework and procedure for one-sided models and further analyze the various testing options for two-sided models, in collaboration with RANI, and including at least: ■ Relation to legacy requirements ■ Performance monitoring and LCM aspects considering use-case specifics ■ Generalization aspects ■ Static / non-static scenarios / conditions and propagation conditions for testing (e.g., CDL, field data, etc.) ■ UE processing capability and limitations ■ Post-deployment validation due to model change / drift RAN5 aspects related to testability and interoperability to be addressed on a request basis

[0094] Generalization refers to the model's ability to adapt properly to new, previously unseen data, drawn from the same distribution as the one used to create the model. In other words, generalization examines how well a model can digest new data (mostly 10 corresponding to new environment / scenario) and make correct predictions (for unseen / new environment or scenario) after getting trained on a training set.

[0095] If a model is trained too well on training data, it will be incapable of generalizing. In such cases, it will end up making erroneous predictions when it’s given new data. This would make the model ineffective even though it’s capable of making correct predictions 15 for the training data set. This is known as overfitting.

[0096] There are several techniques widely used to avoid the problem of overfitting and to keep good generalization capabilities of the model, such as dropOut, batch normalization, early stopping, etc.

[0097] The following cases were considered for verifying the generalization performance of an AI / ML model over various scenarios / configurations. Table 2 • Case 1: The AI / ML model is trained based on training dataset from one Scenario# A / Configuration#A, and then the AI / ML model performs inference / test on a dataset from the same Scenario# A / Configuration#A • Case 2: The AI / ML model is trained based on training dataset from one Sce-nario#A / Configuration#A, and then the AI / ML model performs inference / test on a different dataset than Scenario#A / Configuration#A, e.g., Scenario#B / Configu-ration#B, Scenario#A / Configuration#B • Case 3: The AI / ML model is trained based on training dataset constructed by mixing datasets from multiple scenarios / configurations including Sce-nario#A / Configuration#A and a different dataset than Scenario#A / Configura-tion#A, e.g., Scenario#B / Configuration#B, Scenario#A / Configuration#B, and then the AI / ML model performs inference / test on a dataset from a single Sce-nario / Configuration from the multiple scenarios / configurations, e.g., Scenario# A / Configuration#A, Scenario#B / Configuration#B, Scenario# A / Configu-ration#B. o Note: Companies to report the ratio for dataset mixing o Note: number of the multiple scenarios / configurations can be larger than two

[0098] In brief, Case 1 focuses on training dataset from Scenario A / Configuration A 5 and performing inference on the same Scenario / Configuration. Case 2 focuses on training dataset from Scenario A / Configuration A and performing inference on a different Scenario / Configuration. Case 2A investigates training on a selected Scenario A / Configuration A and generalizing to Scenario B / Configuration B via fine-tuning (e.g., transfer learning) as an evolved version of Case 2. Case 3 investigates using mixed 10 training dataset on both Scenario A / Configuration A and Scenario B / Configuration B and test to any combination of of Scenario A / Configuration A and Scenario B / Configuration B.

[0099] Furthermore, 3GPP TR 38.843 also covers following aspects for verification of generalization performance over various scenarios and configurations for CSI compression use case. Table 3 To verify the generalization performance of an AI / ML model over various scenarios, the set of scenarios are considered focusing on one or more of the following aspects: - Various deployment scenarios (e.g., UMa, UMi, InH) - Various outdoor / indoor UE distributions for UMa / UMi (e.g., 10:0, 8:2, 5:5, 2:8, 0:10) - Various carrier frequencies (e.g., 2GHz, 3.5GHz) - Other aspects of scenarios are not precluded, e.g., various antenna spacing, various antenna virtualization (TxRU mapping), various ISDs, various UE speeds, etc. - Companies to report the selected scenarios for generalization verification To verify the generalization / scalability performance of an AI / ML model over various configurations (e.g., which may potentially lead to different dimensions of model in-put / output), the set of configurations are considered focusing on one or more of the following aspects: - Various bandwidths (e.g., 10MHz, 20MHz) and / or frequency granularities, (e.g., size of subband) - Various sizes of CSI feedback payloads - Various antenna port layouts, e.g., (N1 / N2 / P) and / or antenna port numbers (e.g., 32 ports, 16 ports) - Various UE speeds (e.g., lOkm / h, 30km / h, 60km / h, 120km / h, etc.) for CSI prediction sub use case - Other aspects of configurations are not precluded, e.g., various numerologies, various rank numbers / layers, etc. - The selected configurations for generalization verification - The method to achieve generalization over various configurations to achieve scalability of the AI / ML input / output, including pre-processing, post-processing, etc For evaluating the generalization / scalability over various configurations for CSI compression, to achieve the scalability over different input / output dimensions, companies to report which case(s) are evaluated from the following list: - Case 0 (benchmark for comparison): One CSI generation part with fixed input and output dimensions to 1 CSI reconstruction part with fixed input and output dimensions for each of the different input and / or output dimensions. - Case 1: One CSI generation part with scalable input and / or output dimensions to N>1 separate CSI reconstruction parts each with fixed and different output and / or input dimensions - Case 2: M>1 separate CSI generation parts each with fixed and different input and / or output dimensions to one CSI reconstruction part with scalable output and / or input dimensions - Case 3: A pair of CSI generation part with scalable input / output dimensions and CSI reconstruction part with scalable output and / or input dimensions

[0100] There are some meeting discussions to address generalization of AI / ML models / functionality as one of the main challenges for the AI / ML enabled use cases. In practice, if a model has been trained for a given radio condition or a certain parameter setting then encountering a different radio condition or parameter settings can severely 5 impact the performance of the functionality.

[0101] Furthermore, verification of model generalization is an important topic for AI / ML enabled use cases. Currently, there is no such test mechanism that can validate generalization aspects of AI / ML enabled functionalities for CSI compression use-case.

[0102] The 3GPP TR 38.843 covers following aspects for verification of 10 generalizability. Table 4 The necessity and feasibility of defining requirements or test to verify the generalization of AI / ML is studied. The goals of generalization test are to verify whether the minimum level of performance of AI / ML functionality / model can be achieved / maintain under the identified scenarios and / or configurations, while the performance won’t be significantly degraded in other scenarios and / or configurations. The following aspects should be considered for generalization / scalability related testing: - details about the scenarios and / or configurations for test and the corresponding AI / ML models / functionality - what the minimum level performance for each identified scenario and / or configuration is - what the significant degradation for other scenarios and / or configurations is It should also be considered that generalization and / or scalability related requirements for different scenarios / configurations can be implicitly handled in the test case definition. As for the handling of generalization tests, the following option is considered as baseline: Signalling based LCM procedures and performance monitoring are considered in dedicated test cases and are excluded in tests verifying generalization. RAN4 may define multiple tests with different conditions. In each of the test, TE configures the same specified UE configuration, and therefore the same specified UE configuration is tested under different conditions to verify its generalizability, (environment differs in each test but not changing dynamically during the test) - Specified UE configuration includes functionality and / or model ID if defined

[0103] There are some agreements on test encoder / decoder for 2-sided models, which are reproduced below. Table 5 7.3.2.3 Test encoder / decoder for 2-sided model (NOTE: At the current stage the framework in this session applies to CSI compression case.) In order to determine the test encoder / decoder, the following issues are considered: Common assumptions for proposals of the test decoder / encoder (and the paired encoder / decoder) for tester The need for and potential definition and derivation procedure of intermediate KPI for decoder evaluation and selection Data collection / generation for decoder evaluation, and the common assumptions / environment needed for data collection / generation How to minimize the impact of possible variations / differences in the test decoder / test encoder design / implementation on UE / gNB performance verification The impact of test decoder / encoder for testing complexity to UE / gNB performance verification, and the advantage / disadvantage analysis of high / low complexity decoders. The test decoder / encoder design should take into account complexity limitations based on e.g., feasibility of TE implementation and complexity levels considered feasible by network vendors / UE vendors for decoder / encoder deployment. The choice of test decoder / encoder should aim as much as possible to avoid limiting the implementation choices, including e.g. complexity, back-bone model etc, of UE / gNB encoders / decoders operating in the field (this principle may not be fully achievable in practice). Specification on the test may include some high-level parameters for the test decoder / encoder (e.g. parameters related to processing complexity, model structure, etc). Following the above principles, the considered options of test decoder are listed below Option 1: DUT provides the decoder Option 2: Infra vendor provides the decoder Option 3: Full decoder specification in standard Option 4: TE vendor provides the decoder

[0104] AI / ML models and functionalities have the limits on the conditions where they can be used. These limitations can be imposed by the training procedure or dataset, based on the definition of the UE capabilities, and importantly based on the generalization capabilities. One way to capture these conditions is to introduce additional conditions. 5 However, when specific configurations / conditions associated with UE capability are determined / identified between UE-side and NW-side, they do not describe / formalize the generalization capabilities of the models, i.e., how well is the model / functionality performance outside these conditions or how fast this performance may degrade when the UE leaves these conditions. 10

[0105] Two schematic examples are provided in Error! Reference source not found., where one model / functionality has good generalization capabilities, the other one does not generalize that well. Also, FIG. 3B illustrates how the performance of the models or functionalities is dependent on their generalizing capabilities especially in the edge areas of the model capability. This could result in the NW frequently changing the 15 models / functionalities at the UE as the NW is not aware of the generalizing capability of the model / functionality at the UE. This kind of ping-pong between the models selected at the UE could impact the performance of the functionality at the UE. This could also result in other side effects at the UE like power consumption.

[0106] For AI / ML enabled CSI compression use case, two sided models are required, where one AI / ML model runs at the UE / DUT side (e.g., encoder for CSI compression usecase in the uplink) and the other runs at the gNB / TE side (e.g., decoder for CSI compression use-case in the uplink). This is completely different from one-sided model use-cases like beam management where the AI / ML model runs only at the UE side and the interoperability of the model (in terms of joint working of UE and network / TE side models) is not required.

[0107] The joint working of UE side model (encoder) and network / TE side model (decoder) in CSI compression use-case is of utmost importance because without this, the decoder will not fairly reconstruct the CSI compressed by the encoder resulting in performance degradation.

[0108] Currently, there is no test mechanism for validating generalization aspects of the AI / ML model or functionality in the communication system, especially for the two-sided models (such as those used in the AI / ML based CSI compression use case).

[0109] Example embodiments of the present disclosure provide solutions for application and verification of generalization aspects of AI / ML models or functionalities. A terminal device can report generalization capability to a network device in an in-field stage or to a test equipment in the test stage.

[0110] Some example embodiments define and report the generalization capabilities of AI / ML models or functionalities at the terminal devices. Some example embodiments also provide a solution to handle different model implementations with different generalization capabilities in field. Some example embodiments allow the network device to confirm or verify the generalization capability reported by the terminal device.

[0111] Some example embodiments further define how different generalization capabilities will impact LCM procedures of the AI / ML models or functionalities.

[0112] Some example embodiments provide a mechanism to handle model / functionality management to avoid frequent switching or ping-pongs between the AI / ML models / functionalities; particularly in the edge area between the models / functionalities.

[0113] Some example embodiments can address the testability of the generalization aspects of AI / ML enabled CSI compression functionality with single / static test decoder which involves two-sided models for the case when the test decoder is either fully specified in the standard or being implemented by the TE vendor.

[0114] Example embodiments of the present disclosure will be described in detail below with reference to the accompanying drawings.

[0115] FIG. 4 illustrates a flowchart of a signaling flow 400 for application and verification of AI / ML model capability in a stage of a terminal device in accordance with some example embodiments of the present disclosure. For purpose of illustration, the signaling flow 400 is described with reference to FIG. 1. As shown in FIG.4, the signaling flow 400 involves the test equipment 110 and the terminal device 120. In this stage, the terminal device 120 is sometimes called a device under test (DUT).

[0116] In the signaling flow 400, it is assumed that the terminal device 120 has an AI / ML model or functionality 125 running. In some embodiments, the AI / ML model or functionality 125 may be configured for CSI compression, and is paired with an AI / ML model or functionality 135 in the network side which is configured for CSI compression. In some other embodiments, the AI / ML model or functionality 125 in the terminal device 120 may be configured for other features in the communication environment.

[0117] In the signaling flow 400, the terminal device 120 transmits (405), to a test equipment 110, capability information indicating generalization capability of an artificial intelligence / machine learning (AI / ML) model or functionality in the terminal device 120, the generalization capability at least includes pair information which indicates a plurality of pairs of scenarios and communication configurations in enabling the AI / ML model or functionality 125.

[0118] The generalization of an AI / ML model or functionality may be related to the following main aspects. A first aspect is the changing of the scenarios (e.g., channel or radio conditions. If the configured AI / ML functionality / model has been trained with a dataset representing mainly certain radio condition environment (e.g., Urban Macro (UMA) propagation conditions with low mobility UEs) then this AI / ML functionality / model may experience degraded performance if different channel conditions are met in the field. To avoid performance degradation, it should be guaranteed that the AI / ML configured functionality / model has been trained with a diverse dataset and can perform well with acceptable tolerance margin / threshold in varying channel conditions. This should also be verified testing to completely validate the configured AI / ML functionality / model. In the present disclosure, the term “scenario” and “channel condition” or “radio condition” are used interchangeably.

[0119] A second aspect is the changing of communication configurations or parameters settings. The impact of generalization on the performance of various AI / ML use cases (each use case configured with different AI / ML functionality / model) depends heavily on the configuration and parameter settings used for dataset generation for the training. For example, for AI / ML CSI compression use-case, configurations should cover different bandwidths, different size of CSI feedback payload, different antenna port layouts (such as Nl) or antenna port number (such as 32 ports), etc. E.g., UE side model / functionality for CSI compression in UMa scenario with low mobility, B' MHz channel bandwidth, ‘A’ antenna ports and ‘C’ bytes as the CSI payload.

[0120] Considering the above aspects, the capability information from the terminal device 120 with the AI / ML model or functionality 125 to be tested may indicate a plurality of pairs of scenarios and communication configurations in enabling the AI / ML model or functionality 125.

[0121] In some example embodiments, the generalization capability of the AI / ML model or functionality 125 at the terminal device 120 may further comprises a generalization score of the AI / ML model or functionality, or characteristics of the AI / ML model or functionality 125. In some example embodiments, the capability information may further indicate the AI / ML features and configurations supported by the terminal device 120.

[0122] The terminal device 120, as a part of its capability exchange with the test equipment 110 or (with the network device 130 as will be discussed below), can provide a generalization capability of its AI / ML model or functionality 125. The generalization capability is sometime referred to as a generalization matrix (GM).

[0123] This generalization capability will provide information on the generalization aspects of the model(s) across different pre-agreed / standardized pairs of scenarios and communication configurations. This generalization capability will also be exchanged by the terminal device 120 with the network device 130 when there is a change / update in the AI / ML model or functionality.

[0124] In principle these pairs of scenarios and communication configurations can be those that are commonly encountered by the terminal device 120 in the field.

[0125] The generalization capability can be used in different ways as below, based on where it is used. A first way is for conformance testing. In case of conformance testing the test equipment 110 can select only those pairs that the terminal device 120 has mentioned in its capability information and do the tests to ascertain the generalization capability of the AI / ML model or functionality 125. A second way is for in-field UE monitoring. During in-field monitoring, the test equipment 110 at the NW side can use the generalization capability to map the current channel conditions of the terminal device 120 with the one in the generalization capability and use that to make decisions to take actions in the functionality / model LCM, e.g., model update, switching.

[0126] One of the sample methods to represent the generalization capability can be to use a bit stream to represent different pairs and every bit that is enabled will denote that the model is generalizable for the corresponding pair implicitly denoted by that bit. Few possible examples for this new capability are shown below. Table 6 j GeneralizationMatrixCSICompression j UE N / ! No i No j Pools of pre-defined generalization pairs that the UE supports on ! A ! j the NR AI / ML enabled CSI compression functionality j 1

[0127] The information element (IE) GeneralizationMatrixCSICompression indicates the generalization capability of an AI / ML model or functionally for CSI compression.

[0128] In some examples, when the generalization capability contains a global generalization score, it may be defined as in Table 7. Table 7 GeneralizationMatrixCSICompression information element GeneralizationMatrixCSICompression ::= SEQUENCE { generalizationscore INTEGER, } TAG - GEN E AL I EAT I ON E' LI R E C 01C OMERE SSI ON - E IO E

[0129] In some examples, when the generalization capability contains the pair information for which the AI / ML model or functionality is generalized along with the global generalization score. The generalization pairs that are supported are represented as a bit string as in Table 8, where each bit represents a pair of standardized scenario and configuration set.

[0130] The test equipment 110 receives (410), from the terminal device 120, the capability information indicating the generalization capability of the AI / ML model or functionality 125 at the terminal device 120. The test equipment 110 performs (415) generalization test on the AI / ML model or functionality in the terminal device 120 based on the generalization capability of the AI / ML model or functionality 125.

[0131] During the testing, the test equipment 110 may transmit, to the terminal device 120, an instruction of running the AI / ML model or functionality according to the plurality of pairs of scenarios and communication configurations. In response to receiving (420) the instruction from the test equipment 110, the terminal device 120 may cause the AI / ML model or functionality 125 to be run according to the plurality of pairs of scenarios and communication configurations.

[0132] Some more aspects of selection / identification of different scenarios and communication configurations for generalization are discussed in detail in the following.

[0133] In some example embodiments, in the use case of two side models, in order to test the AI / ML model or functionality 125 in the terminal device 120, the test equipment 110 may determine, based on the characteristics of the AI / ML model or functionality 125 in the terminal device 120, a further AI / ML model or functionality 115 to be applied in the test equipment 110. Then the test equipment 110 may perform the generalization test on the AI / ML model or functionality 125 by also running the AI / ML model or functionality 115 at the test equipment 110.

[0134] In some example embodiments, by testing the plurality of pairs of scenarios and communication configurations, the test equipment 110 may calculate a test generalization score for the AI / ML model or functionality 125. This generalization score may be an overall confidence on the generalization of the AI / ML model or functionality 125. The test generalization score calculated by the test equipment 110 may be included in the generalization capability of the AI / ML model or functionality 125 or may be used to replace the original generalization score included therein.

[0135] To test the model / functionality generalization, the test equipment 110 may determine, from the generalization capability, a baseline pair of scenario and communication configuration and at least one non-baseline pair of scenario and communication configuration. Then the test equipment 110 may determine a baseline performance indicator (e.g., key performance indicator, KPI) for the AI / ML model or functionality 125 by causing the AI / ML model or functionality 125 at the terminal device 120 to run according to the baseline pair of scenario and communication configuration. The test equipment 110 may also determine at least one non-baseline performance indicator for the AI / ML model or functionality 125 by causing the AI / ML model or functionality 125 at the terminal device 120 to run according to the at least one nonbaseline pair of scenario and communication configuration. The test equipment 110 may transmit an instruction of running the AI / ML model or functionality 125 according to the baseline and non-baseline pair of scenarios and communication configurations.

[0136] Then the test equipment 110 may determine if there is any performance degradation of the at least one non-baseline performance indicator from the baseline performance indicator. A generalization score for the AI / ML model or functionality 125 is calculated based on a performance degradation of the at least one non-baseline performance indicator from the baseline performance indicator.

[0137] After the testing, a validation result about the generalization of the AI / ML model or functionality 125 is generated. The test equipment 110 may determine a generalization score of the AI / ML model or functionality. In some example embodiments, the test equipment 110 may further determine whether the AI / ML model or functionality 125 is valid or to be enabled for application at least in the aspect of generalization. The test equipment 110 transmits (425), to the terminal device 120 and based on a result of the generalization test, an indication of a generalization score of the AI / ML model or functionality. In some example embodiments, after the generalization test, the test equipment 110 may also transmit an indication of enabling or disabling the AI / ML model or functionality 125. The terminal device 120 receives (430), from the test equipment 110, the indication of the generalization score of the AI / ML model or functionality. In some example embodiments, the terminal device 120 may receive an indication of enabling or disabling the AI / ML model or functionality 125. In this way, it is able to verify the generalization capability of the AI / ML model or functionality 125 at the terminal device 120.

[0138] Some details of the test stage will be further discussed in the following with reference to FIG. 6A and FIG. 6B.

[0139] FIG. 5 illustrates a signaling flow 500 for application and verification of AI / ML model capability in an in-field stage in accordance with some example embodiments of the present disclosure. For purpose of illustration, the signaling flow 500 is described with reference to FIG. 1. As shown in FIG.5, the signaling flow 500 involves the terminal device 120 and the network device 130.

[0140] In the signaling flow 500, it is assumed that the terminal device 120 has an AI / ML model or functionality 125 running. In some embodiments, the AI / ML model or functionality 125 may be configured for CSI compression, and is paired with an AI / ML model or functionality 135 in the network device 130 which is configured for CSI compression. In some other embodiments, the AI / ML model or functionality 125 in the terminal device 120 and the AI / ML model or functionality 135 in the network device 130 may be configured for other features in the communication environment.

[0141] The terminal device 120 obtains (505) capability information indicating generalization capability of the AI / ML model or functionality 125 in the terminal device 120, the generalization capability at least comprising a generalization score of the AI / ML model or functionality 125. In some example embodiments, the generalization score of the AI / ML model or functionality 125 may be the generalization score output in the test stage. This score will be used to represent the generalizability of the AI / ML model or functionality 125, e.g., the generalization of the CSI compression functionality. In some examples, the generalization score may be calculated to represent performance of the AI / ML model or functionality 125. In such cases, a higher generalization score represents a higher performance level in the aspect of generalization. In some examples, the generalization score may be calculated to represent performance degradation of the AI / ML model or functionality 125. In such cases, a higher generalization score represents a lower performance level in the aspect of generalization. The representation of the generalization score can be defined as required.

[0142] In some example embodiments, the generalization capability of the AI / ML model or functionality 125 may further include pair information, the pair information indicating a plurality of pairs of scenarios and communication configurations supported in enabling the AI / ML model or functionality 125, and / or characteristics of the AI / ML model or functionality 125. In some example embodiments, the capability information may further indicate the AI / ML features and configurations supported by the terminal device 120.

[0143] The terminal device 120 transmits (510) the capability information to the network device 130. The network device 130 receives (515), from the terminal device 120, capability information indicating generalization capability of a first artificial intelligence (AI) / machine learning (ML) model or functionality in the terminal device, the generalization capability at least comprising a first generalization score of the AI / ML model or functionality 125. Then the network device 130 determines (520) enablement or activation of the AI / ML model or functionality 125 at the terminal device 120 based on a determination of whether the generalization score of the AI / ML model or functionality 125 satisfied with a generalization requirement.

[0144] The generalization requirement is defined based on the representation of the generalization score. In the case where the generalization score represents performance of the AI / ML model or functionality, the generalization requirement may be defined as the generalization score exceeding or equal to a generalization threshold. In this case, the generalization threshold may represent minimum acceptable performance. In the case where the generalization score represents performance degradation of the AI / ML model or functionality, the generalization requirement may be defined as the generalization score being below another generalization threshold. In this case, the generalization threshold may represent the maximum acceptable / allowed performance degradation. In the below embodiments, unless explicitly stated, most of the examples are described with respect to the case where the generalization score represents performance of the AI / ML model or functionality.

[0145] In some example embodiments, if the generalization score of the current AI / ML model or functional 125 fails to satisfy the generalization requirement (e.g., the generalization score is below a corresponding generalization threshold), the network device 130 may transmit (525), to the terminal device 120, a request for selecting an AI / ML model or functionality with better generalization than the current AI / ML model or functionality 125 (e.g., with a higher generalization score than the generalization score of the current AI / ML model or functionality). In some example embodiments, if the generalization score of the current AI / ML model or functional 125 satisfies the generalization requirement (e.g., the generalization score exceeds or equal to the corresponding generalization threshold), the network device 130 may decide to enable or activate the current AI / ML model or functional 125 at the terminal device 120.

[0146] The terminal device 120 may have one or more different AI / ML models or functionalities 125 available and is aware of at least their corresponding generalization scores. If the terminal device 120 receives (530) the request for selecting an AI / ML model or functionality with a higher generalization score than the generalization score of the AI / ML model or functionality 125, it may determine if there is such an AI / ML model or functionality with a higher generalization score available. Then the terminal device 12TE0 may transmit (540), to the network device 130, a response to the request indicating whether an AI / ML model or functionality 125 with a higher generalization score is available at the terminal device 120.

[0147] The network device 130 receives (540) the response indicating whether an AI / ML model or functionality 125 with a higher generalization score is available at the terminal device. If the response indicates that an AI / ML model or functionality with a higher generalization score is available at the terminal device 120, the network device 130 may decide to enable or active the AI / ML model or functionality with better generalization score.

[0148] In some example embodiments, when transmitting the response indicating whether an AI / ML model or functionality 125 with a higher generalization score is available, the terminal device 120 may also transmit generalization capability of that AI / ML model or functionality 125 with a higher generalization score. In some example embodiments, the generalization capability of that AI / ML model or functionality 125 with a higher generalization score is transmitted in response to a request from the network device 130.

[0149] In some example embodiments, if the response from the terminal device 120 indicates that an AI / ML model or functionality with a higher generalization score is unavailable at the terminal device 120, the network device 130 may determine whether to enable the AI / ML model or functionality 125 with the generalization score lower than the threshold or switch to a non-AI / ML functionality at the terminal device 120. For the latter case, the network device 130 may instruct the terminal device 120 to apply a non-AI / ML feature, e.g., to report the CSI information in a legacy way.

[0150] In some example embodiments, if any AI / ML model or functionality 125 is determined to be enabled or activated, the network device 130 may transmit an indication of enablement or activation of the AI / ML model or functionality 125 to the terminal device 120. By receiving such an indication, the terminal device 120 may determine which AI / ML model or functionality 125 is to be enabled or activated.

[0151] In some example embodiments, if an AI / ML model or functionality 125 in the terminal device 120 is to be enabled or activated, the network device 130 may determine, based on characteristics of the AI / ML model or functionality 125 to be enabled or activated, an AI / ML model or functionality 135 at the network device 130.

[0152] In some examples, the network device 130 may select appropriate input and / or output dimensions of the NW sided AI / ML model or functionality 135 which is suitable for use with the corresponding AI / ML model or functionality 125 at the terminal device 120. The input and / or output dimensions of the NW sided AI / ML model or functionality 135 may be determined based on the characteristics of the AI / ML model or functionality 125 to be enabled or activated.

[0153] In some examples, the network device 130 may use the generalization capability reported in capability information of the terminal device to adapt parameters related to performance monitoring mechanism. One example of these parameters may be the time required before taking an LCM action (such as switching to another model / functionality or fallback to legacy) once certain threshold of performance degradation is detected. In some example embodiments, the network device 130 may determine, based at least one the generalization capability of the AI / ML model or functionality 125, a reporting frequency for performance monitoring on at least one AI / ML model or functionality 125 in the terminal device 120.

[0154] The configuration of the reporting frequency would be helpful in avoiding transmission of unnecessary LCM commands to the terminal device 120, since the confidence level of the network on a more generalized UE-side AI / ML model or functionality (based on Generalization Matrix reported by the terminal device 120) would be higher. It would be more likely that the temporary performance degradation may be linked to some other on-field incidents (such as temporary blockage around the UE) and may not be linked to the changing environment, since the terminal device 120 has good generalization capabilities. The network device 130 may transmit to the terminal device 120 a configuration indicating the reporting frequency. The terminal device 120 may perform the LCM management on its AI / ML model or functionality 125 based on the configured reporting frequency.

[0155] In addition to the reporting frequency for performance monitoring, the network device 130 may configure, based on the generalization capability reported by the terminal device 120, other parameters related to the AI / ML models or functionalities at the terminal device or the network device. The scope of the present disclosure is not limited in this regard.

[0156] In the following, detailed discussion will be made to the test stage and then to the in-field stage.

[0157] FIG. 6A illustrates a block diagram of the generalization test framework 600A for AI / ML model or functionality in accordance with some example embodiments of the present disclosure. This framework may be applied for the test setup use case or the use case with NO / minimum modification in the real field deployments for monitoring the generalization performance during LCM.

[0158] The framework 600A mainly consists of a terminal device 120 whose AI / ML model or functionality (e.g., AI / ML enabled CSI compression functionality (model)) is being tested for the generalization, and a test equipment (TE) 110 which will help to test the generalization of the AI / ML model or functionality.

[0159] To realize the generalization tests at the test equipment 110, the following architectural blocks at the test equipment 110 are provided in some example embodiments of the present disclosure.

[0160] The test equipment 110 comprises a test management block 670 which is responsible for creation / identification of the scenarios, configurations, and conditions that are needed to test the generalizability of the functionality. The baseline and nonbaseline pairs for generalization testing are identified here.

[0161] The generalization threshold (for each selected ‘non-baseline pair’) is also configured in this test management block 670. These are the permissible margins / thresholds within which the results of the generalization capability should vary with respect to the baseline pair to ensure the generalization over other non-baseline pair(s).

[0162] This test management block 670 is also responsible for the selection of a real or test AI / ML model or functionality at the test equipment (and if required, the test AI / ML model or functionality at the terminal device 120). In some cases, the real or test AI / ML model or functionality at the test equipment 110 may be a decoder that helps to reconstruct the CSI feedback that is compressed by the encoder residing in the terminal device 120.

[0163] The test equipment 110 further comprises a test execution block 660. Once the baseline and non-baseline pairs, decoder, and the respective generalization thresholds are identified, this test execution block 660 will help to execute the selected pair(s) and obtain the test results. The results of non-baseline pairs are compared against the KPIs of baseline pair to ascertain the generalizability of the AI / ML model or functionality in the terminal device 120.

[0164] The main steps of the proposed framework can be described as follows:

[0165] (1) Define a test AI / ML model or functionality (single / static), a reference / baseline pair of scenarios and configurations (referred to as just “baseline pair” in subsequent sections). o Examples of test AI / ML model or functionality include either a test decoder fully specified in the standard, or a test AI / ML model or functionality specified by the TE vendor or a test decoder specified by the collaboration between TE vendor(s) and other vendor(s). o Examples of reference / baseline pair can be combination of specific scenario, and a particular communication configuration along with its required parameters to perform corresponding features of the AI / ML model or functionality (e.g., the AI / ML enabled CSI compression). As an example, the communication configuration may include UE side AI / ML model / functionality for CSI compression in UMa scenario with low mobility, ‘B’ MHz channel bandwidth, ‘A’ antenna ports and ‘C’ bytes as the CSI payload, and the like.

[0166] (2) Keeping the same test AI / ML model or functionality (single / static), define non-baseline / different pair(s) of scenarios and communication configurations (referred to as just “non-baseline pair” in subsequent sections). o Same test AI / ML model or functionality referred in this example as a test decoder will be used for generalization testing where different pairs will be tested for generalization. o Examples of non-baseline pair can be combination of scenario, and a particular UE configuration along with its required parameters to perform AIML enabled CSI compression which is different from the baseline pair. E.g., UE side AI / ML model / functionality for CSI compression in UMa scenario with low mobility, ‘Z’ MHz channel bandwidth, ‘D’ antenna ports and ‘W’ bytes as the CSI payload.

[0167] (3) Measure / calculate the UE performance KPIs at the test equipment 110 for the baseline pair, and each of the identified / supported non-baseline pair(s) as reported by the terminal device 120.

[0168] (4) Calculate the relative performance degradation in the KPI for each nonbaseline pair compared to the baseline pair.

[0169] (5) Define a generalization threshold for all non-baseline pairs. The generalization threshold may be a 3GPP defined target.

[0170] (6) Perform test on all selected / identified non-baseline pairs, calculate the generalization verdict for each Pair, calculate the overall generalization score and report results to the generalization performance validation entity / TE.

[0171] (7) The test equipment validates the tests based on the defined generalization thresholds and overall generalization score for different combinations.

[0172] In the present disclosure, reference test / pair is also used interchangeably by Baseline Test / pair. The reference test / pair is used to get reference values of the KPI(s). In the present disclosure, other scenario and configuration test / pair is also used interchangeably by non-baseline test / pair. Other scenario and configuration test / pair is used to test different pairs whose results are compared against reference test.

[0173] FIG. 6B illustrates a detailed signaling flow 600B for application and verification of AI / ML model capability in a test stage in accordance with some further example embodiments of the present disclosure. The signaling flow 600B involves the test equipment 110 and the terminal device 120 (e.g., DUT). The signaling flow 600B is considered as an example implementation of the signaling flow 400.

[0174] In the signaling flow 600B, the test equipment 110 is testing AI / ML enabled feature of the terminal device 120. At 608, the terminal device 120 transmits capability information to the test equipment 110. The capability information including generalization capability of an AI / ML model or functionality that supports the testing AI / ML enabled feature. The generalization capability at least includes pair information which indicates a plurality of pairs of scenarios and communication configurations in enabling the AI / ML model or functionality 125.

[0175] In some example embodiments, the generalization capability of the AI / ML model or functionality 125 at the terminal device 120 may further comprises a generalization score of the AI / ML model or functionality, or characteristics of the AI / ML model or functionality 125. In some example embodiments, the capability information may further indicate the AI / ML features and configurations supported by the terminal device 120.

[0176] At 610, the test equipment 110 triggers the terminal device 120 to move to an AI / ML test mode. In other examples, this step is optional. At 612, the test equipment 110 transmits an indication to enable the AI / ML enabled feature of the terminal device 120 under test.

[0177] At 614, the test equipment 110 performs test pair selection.

[0178] Specifically, at 616, the test equipment 110 selects a test AI / ML model or functionality by selecting its input and / or output dimensions based on the characteristics of the AI / ML model or functionality at the terminal device 120.

[0179] To select the appropriate input and / or output dimensions of the test AI / ML model or functionality at the test equipment to test the AI / ML model or functionality 125 at the terminal device 120, the test equipment 110 needs the characteristics (input / output dimensions) of the AI / ML model or functionality 125 at the terminal device 120.

[0180] In some examples related to the CSI compression and reconstruction, there may be M>1 separate CSI generation parts each with fixed and different input and / or output dimensions to one CSI reconstruction part with scalable output and / or input dimensions. In this case, there are multiple CSI generation parts (i.e., encoder) with fixed and different input and / or output dimensions whereas there is only 1 CSI reconstruction part (i.e., decoder) with scalable output and / or input dimensions.

[0181] As mentioned in previously, Option 3 and 4 are the preferred candidates for test decoder construction / selection for the conformance testing. Considering Option 3 and 4 together with Case 2 (as shown above), there are two possibilities.

[0182] In a first possibility, the AI / ML models or functionalities provided by different UE vendors may have different input and / or output dimensions. In this case, the test AI / ML model or functionality needs to be scalable in the sense that it should be able to scale its input and / or output dimensions to match with the corresponding dimensions of a particular AI / ML model or functionality at the terminal device 120. For this, a signaling framework / solution is required through which the terminal device 120 will share / inform to the test equipment 110 about the input and output dimensions of its AI / ML model or functionality, e.g., via the generalization capability.

[0183] In a second possibility, the AI / ML models or functionalities provided by different UE vendors may have the same input and / or output dimensions. In this case, the input and output dimensions for all AI / ML models or functionalities at the terminal devices and the test AI / ML models or functionalities needs to be fixed and defined in the standard. For this, there is no need to share the input and output dimensions of the AI / ML models or functionalities at the terminal device 120 to the test equipment 110.

[0184] As an alternative, the test equipment 110 can also learn about the AI / ML model or functionality at the terminal device 120 based on the model ID information shared by the terminal device 120.

[0185] In this case, the input and output dimensions for all terminal device 120 encoders and the test decoder needs to be fixed and defined in the standard. For this, there is no need to share the terminal device 120’s encoder’s input and output dimensions to the test equipment 110 / network.

[0186] The test equipment 110 may select the appropriate KPI to evaluate the UE / DUT performance in different scenario(s) and configuration(s) pair(s). For example, KPI can be throughput, NMSE, SGCS, etc. At 618, based on the generalization capability reported by the terminal device 120, the test equipment 110 selects appropriate KPIs to test, a corresponding baseline pair of scenario and communication configuration, and one or more non-baseline pairs of scenarios and communication configurations (which may be based on applicability conditions or specifications).

[0187] The performance KPI(s) may include, for example, through-put, NMSE (normalized mean squared error), SGCS, etc. The test equipment 110 also defines or identifies the baseline set up (scenario and / or configuration and / or condition) for which the functionality (model) was developed. In some examples, examples of reference / baseline pair can be combination of specific scenario, and a particular UE communication configuration along with its required parameters to perform AI / ML enabled feature (e.g., the AI / ML enabled CSI compression). The configuration may be, for example, UE side model / functionality for CSI compression in UMa scenario with low mobility, ‘B’ MHz channel bandwidth, ‘A’ antenna ports and ‘C’ bytes as the CSI payload.

[0188] The principle of selection of reference decoder, scenario / condition and configuration is to select test decoder, scenario / condition and configuration that will be used as a baseline for AI / ML CSI compression use-case conformance testing. Baseline scenario / conditions are either the scenario / conditions generally being tested in RAN4 verification (such as AWGN conditions) or can be the scenario / conditions which are mainly encountered by the UE in the field after deployment (such as a slow-moving UE in macro urban conditions) for which the corresponding AI / ML functionality / model was developed.

[0189] The performance KPI(s), such as through-put, NMSE, SGCS, etc., for the reference scenario / conditions / configuration will be part of the performance requirement testing of the AI / ML use case. For the validation of generalization aspects of AI / ML model or functionality 125, it is not sufficient to only validate performance KPIs for a given (reference) scenario / condition / configuration and therefore, it is necessary to test and validate the AI / ML model or functionality 125 for a variety of other scenario, conditions, and configurations.

[0190] Secondly, the AI / ML model or functionality 125 may be further tested for various other non-baseline scenarios / conditions / configurations in order to validate generalization aspects of the AI / ML model or functionality 125.

[0191] This will be performed by using a predefined list of scenarios with a variety of conditions and configurations. These scenarios may be a combination of generally tested conditions in RAN4 such as TDL-A (Tapped Delay Line A), TDL-C channel conditions with changing parameters to calibrate normal and extreme radio conditions. The list of scenarios can also be influenced by the most present scenario in the field such as slow moving UEs / high speed UEs in micro / macro urban / rural conditions for different channel bandwidths, different numerology and different antenna port number and layouts.

[0192] After the test pair selection, at 620, the test equipment 110 performs baseline pair test. Specifically, at 622, the test equipment 110 performs a baseline test according to the baseline pair of scenario and communication configuration. At 624, the test equipment 110 instructs the terminal device 120 to run the AI / ML model or functionality for a reference scenario (e.g., the baseline scenario). At 626, the test equipment 110 validates accuracy target and stores the baseline KPIs (represented as KPIbase).

[0193] At 630, the test equipment 110 performs non-baseline pair test. Specifically, for each identified non-baseline pair of scenario and communication configuration, at 632, the test equipment 110 performs a test according to an identified non-baseline pair of scenario and communication configuration. At 634, the test equipment 110 instructs the terminal device 120 to run the AI / ML model or functionality for the identified pair. At 636, the test equipment 110 measures and stores the corresponding KPIs for each pair of scenario and configuration. The KPI for each of these non-baseline pair are represented as KPIi, KPL, .., KPIn, for each of the ‘N’ non-baseline pairs.

[0194] For example, if it is identified that there are 3 scenarios and 3 configurations of interest, then there will be 9 different pairs (Scenario 1, Configuration I), (Scenario 1, Configuration 2), (Scenario 1, Configuration 3), (Scenario 2, Configuration 1), ..., (Scenario 3, Configuration 3). Any one pair out of these 9 pairs can be the baseline setup (depending on functionality / model implementation) and the remaining 8 pairs will then be used as other set ups to test the generalization aspects.

[0195] At 640, the test equipment 110 performs generalization performance evaluation. At 642, the test equipment 110 calculates the relative performance degradation of each of the non-baseline pair, and at 644, the test equipment 110 compares the KPI results. The test results are compared with the reference or baseline KPI. For example, the test equipment 110 may compare the KPIs of non-baseline pairs with KPIbase to evaluate the relative performance difference / degradation (e.g., in %) in other setups / pairs in comparison to baseline pair.

[0196] In some example embodiments, for each non baseline pair, if the relative performance degradation is less than the generalization threshold, then the generalization test of the AI / ML model or functionality 125 is reported as SUCCESSFUL. Otherwise, the generalization test is reported as NOT SUCCESSFUL. The test equipment 110 may verify or check if the relative performance degradation in Pair ‘j’ is less than the threshold ‘y / , for j G {1, ..., N} or not. If it is less or equal to the threshold, this implies that the AI / ML model or functionality generalizes to the scenario and configuration of Pair £j’, otherwise, it indicates that the AI / ML model or functionality does not generalize to the scenario and configuration of Pair ‘j’.

[0197] It is noted that the thresholds ‘y / , ‘y2’, ..., ‘yw’ will be defined by RAN4 using either mathematical analysis or simulations or using real field data.

[0198] In some example embodiments, for each non-baseline pair with relative performance degradation lower than the generalization threshold, in the successful case, at 646, the test equipment 110 may report that validation of generalization is successful for the corresponding non-baseline pair. In the failure case, at 648, the test equipment 110 may report that validation of generalization is not successful for the corresponding nonbaseline pair.

[0199] In some example embodiments, at 652, the test equipment 110 may calculate an overall generalization score of the AI / ML model or functionality 125 across the nonbaseline scenarios.

[0200] An overall generalization score can be calculated as a function of individual pair verdicts, e.g., as simple average or weighted average of the individual generalization verdict of each Pair ‘j’ (i.e., Pair ‘1’ to Pair ‘N), where the verdict of Pair ‘j’ = 1 if the functionality / model generalize to Pair ‘j’ else it is equal to 0.

[0201] In an example, the overall generalization score may be calculated using simple average as follows: n n r t <-■ c (Verdictpair +- + VerdictPairN) Overall Generalization Score = --------1-----------— x 100%. N

[0202] In an example, the overall generalization score may be calculated using weighted 5 average: Overall Generalization Score = * VerdictPairi + —I- PN * VerdiclPairN) x 100%, where Pj = probability of selection / occurrence of scenario and configuration of Pair £j’.

[0203] The example generalization score framework is summarized in following Table 10 9. Table 9 Scenario &Configuration Pair Baseline Non-Baseline (Others) Pair 1 Pair 2 Pair £N’ Measured KPI KPIbase KPIi kpi2 KPIn Performance Difference 0 KPIbase^ KPI, | KPIbase ^KPI2| | KPIbase - KPIn| Performance Difference (in %) 0 6± d=f 1 KPIbase “ KP1J '---—-----11 x 100o / o KPIbase <S2 1 KPIbase "KPI2 I J--—----- x 100% KPIbase 8n d=f I KPIbase - KPlN| I---2252-----— X 100% KPIbase Generalization Threshold (in %) N / A Yi K2 Yn Pair Verdict N / A If <5i <Yi, Ver dietPair± = 1 Else Ver diet Pairi = 0 If S2 <y2, VerdictPalr = 1 Else Ver diet Pair2 = 0 If $N — YN. VerdictPairN — 1 Else Ver diet PairN = 0

[0204] It would be appreciated that there may be various other ways to calculate the generalization score based on the KPIs of the baseline and non-baseline pairs. In some embodiments, the generalization score may be calculated in a way to represent the 15 performance degradation instead of the performance level of the AI / ML model or functionality.

[0205] At 654, the test equipment 110 may generate a validation report with the generalization score. The validation report may be transmitted to the terminal device 120 or other entity (e.g., a CN node or a RAN node).

[0206] The generalization score is a new metric to evaluate the generalization aspect of the AI / ML model or functionality. To verify / check whether the AI / ML model or functionality (which may be developed for a specific scenario and / or configuration) generalize well to other scenario(s) and / or configuration(s), the above procedure is proposed to calculate the generalization score.

[0207] FIG. 7 illustrates a detailed signaling flow 700 for application and verification of AI / ML model capability in an in-field stage in accordance with some further example embodiments of the present disclosure. The signaling flow 700 involves the terminal device 120 and the network device 130. The signaling flow 700 is considered as an example implementation of the signaling flow 500.

[0208] In the signaling flow 700, at 706, the terminal device 120 transmits capability information to the test equipment 110. The capability information including generalization capability of an AI / ML model or functionality that supports the testing AI / ML enabled feature. The generalization capability at least includes pair information which indicates a plurality of pairs of scenarios and communication configurations in enabling the AI / ML model or functionality 125.

[0209] In some example embodiments, the generalization capability of the AI / ML model or functionality 125 at the terminal device 120 may further comprises a generalization score of the AI / ML model or functionality, or characteristics of the AI / ML model or functionality 125. In some example embodiments, the capability information may further indicate the AI / ML features and configurations supported by the terminal device 120.

[0210] At 708, the network device 130 extracts the generalization score related information and verifies whether the generalization score of the AI / ML model or functionality 125 at the terminal device 120 is less than a predefined level (e.g., a generalization threshold) or not.

[0211] In some example embodiments, the predefined level may be decided in 3GPP (Standardized) or solely decided by the network or can be a mutual agreement between the terminal device 120 and the network device 130.

[0212] The network device 130 and the terminal device 120 may cooperate to obtain alignment of generalization scores of the AI / ML model or functionality to be enabled or activated.

[0213] If the generalization score is less than a generalization threshold (in the example where the generalization score represents the model performance), then at 712, the network device 130 requests the terminal device 120 to select an AI / ML model or functionality with a higher generalization score. The objective of this request to the terminal device 120 about switching the AI / ML model or functionality to a more generic AI / ML model or functionality is to avoid frequent switching or ping-pongs.

[0214] In case a better functionality or model with better generalization score is available at the terminal device 120 (Case 1), then at 716, the terminal device 120 will send a confirmation to the network device 130 along with its generalization capability of the new AI / ML model or functionality.

[0215] At 718, the network device 130 extracts the generalization score and its related information for the new AI / ML functionality / model indicated by the terminal device 120 and verify if the generalization score is at least equal to a predefined level (a generalization threshold).

[0216] In an alternate embodiment, the network device 130 can signal the generalization threshold (pre-defined level for generalization) to the terminal device 120 using which the terminal device 120 will send a confirmation of whether it has a model or functionality matching the required generalization threshold signaled by the network device 130.

[0217] In case a better AI / ML model or functionality with better generalization score is not available at the terminal device 120 (Case 2), then at 722, the terminal device 120 will send a response that an AI / ML model or functionality with a higher generalization score is not available at the terminal device 120.

[0218] Now at 724, it is up to the network device 130 to decide whether to activate / enable the AI / ML model / functionality for the model currently selected by the terminal device 120 in step 706 or switch the terminal device 120 to a legacy (non-AI / ML) functionality.

[0219] At this point, at 726, the terminal device 120 and the network device 130 have aligned on a suitable model / functionality at the terminal device 120 based on the generalization score.

[0220] In some example embodiments, at 728, there may be adaptation of performance monitoring mechanism at the network device 130 side.

[0221] At 730, the network device 130 selects appropriate input and / or output dimensions of the network sided AI / ML model or functionality 135 for the corresponding AI / ML model or functionality 125 at the terminal device 120.

[0222] At 732, the network device 130 uses the generalization capability reported from the terminal device 120 (e.g., in the UE Capabilities message) to adapt parameters related to performance monitoring mechanism. One example of these parameters may be the time required before taking an LCM action (such as switching to another model / functionality or fallback to legacy) once certain threshold of performance degradation is detected. This would be helpful in avoiding transmission of unnecessary LCM commands to the terminal device 120, since the confidence level of the network on a more generalized terminal device 120 (based on generalization capability reported by the terminal device 120) would be higher. It would be more likely that the temporary performance degradation may be linked to some other on-field incidents (such as temporary blockage around the terminal device 120) and may not be linked to the changing environment, since the terminal device 120 has good generalization capabilities.

[0223] At 734, the network device 130 configures performance monitoring mechanism using selected parameters which are based on the generalization capability of the model / functionality aligned by the terminal device 120 and the network device 130 in step 726.

[0224] At 736, The network device 130 then configures the ground truth reporting frequency at the terminal device 120 based on the generalization score for the terminal device 120 performance monitoring, i.e., different monitoring configurations can be configured such as high monitoring periodicity for the terminal devices with bad generalization scores in the edge conditions.

[0225] It would be appreciated that the reporting frequency is one example of the parameters that can be adapted based on the generalization capability reported from the terminal device 120. In other examples, the network device 130 may determine other parameters related to the LCM decision or even the LCM action on the AI / ML models or functionalities at the terminal device 120.

[0226] According to various example embodiments of the present disclosure, generalization capability exchanged with the network device or the test equipment makes the device or the test equipment aware of the generalization capabilities of the AI / ML model or functionality at the terminal device. As the scenarios within the generalization capabilities are pre-identified / standardized, this will provide a comprehensive overview of the generalization capabilities of the model across varied scenarios.

[0227] In field, based on the generalization capabilities, the network device can take LCM decisions which can ensure fewer model switches at both the UE and the network based on the generalization capability of the model.

[0228] During conformance testing, based on the generalization capabilities, the test equipment can decide on specific scenarios that need to be tested for the model and thus can saving valuable test times.

[0229] FIG. 8 shows a flowchart of an example method 800 implemented at a first apparatus in an in-field stage in accordance with some example embodiments of the present disclosure. In some example embodiments, the first apparatus may be or be comprised in a terminal device. For the purpose of discussion, the method 800 will be described from the perspective of the terminal device 120 in FIG. 1.

[0230] At block 810, the first apparatus obtains capability information indicating generalization capability of a first artificial intelligence (AI) / machine learning (ML) model or functionality in the first apparatus, the generalization capability at least comprising a first generalization score of the first AI / ML model or functionality.

[0231] At block 820, the first apparatus transmits the capability information to a network device.

[0232] In some example embodiments, the method 800 further comprises: receiving, from the network device, a request for selecting an AI / ML model or functionality with a higher generalization score than the first generalization score of the first AI / ML model or functionality; and transmitting, to the network device, a response to the request indicating whether an AI / ML model or functionality with a higher generalization score is available at the first apparatus.

[0233] In some example embodiments, the method 800 further comprises: in accordance with a determination that a second AI / ML model or functionality with a higher generalization score than the first generalization score is available in the first apparatus, transmitting, to the network device, generalization capability of the second AI / ML model or functionality.

[0234] In some example embodiments, the generalization capability of the first AI / ML model or functionality further comprises at least one of the following: pair information, the pair information indicating a plurality of pairs of scenarios and communication configurations supported in enabling the first AI / ML model or functionality, or characteristics of the first AI / ML model or functionality.

[0235] In some example embodiments, the method 800 further comprises: receiving, from the network device, a configuration indicating a reporting frequency for performance monitoring on at least one AI / ML model or functionality in the first apparatus.

[0236] In some example embodiments, the method 800 further comprises: receiving, from the network device, an indication of enablement or activation of the first AI / ML model or functionality at the terminal device.

[0237] In some example embodiments, the first AI / ML model or functionality is configured for channel state information (CSI) compression and is paired with a further AI / ML model or functionality in the network device that is configured for CSI decompression.

[0238] In some example embodiments, the first apparatus is or is comprised in a terminal device.

[0239] FIG. 9 shows a flowchart of an example method 900 implemented at a second apparatus in an in-field stage in accordance with some example embodiments of the present disclosure. In some example embodiments, the second apparatus may be or be comprised in a network device. For the purpose of discussion, the method 900 will be described from the perspective of the network device 130 in FIG. 1.

[0240] At block 910, the second apparatus receives, from a terminal device, capability information indicating generalization capability of a first artificial intelligence (AI) / machine learning (ML) model or functionality in the terminal device, the generalization capability at least comprising a first generalization score of the first AI / ML model or functionality.

[0241] At block 920, the second apparatus determines enablement or activation of the first AI / ML model or functionality at the terminal device based on a determination of whether the first generalization score satisfies a generalization requirement.

[0242] In some example embodiments, the method 900 further comprises: in accordance with a determination that the first generalization score fails to satisfy the generalization requirement, transmitting to the terminal device, a request for selecting an AI / ML model or functionality with a higher generalization score than the first generalization score of the first AI / ML model or functionality; and receiving, from the terminal device, a response to the request indicating whether an AI / ML model or functionality with a higher generalization score is available at the terminal device.

[0243] In some example embodiments, the method 900 further comprises: in accordance with a determination that the response indicates that an AI / ML model or functionality with a higher generalization score is unavailable at the terminal device, determining to enable the first AI / ML model or functionality at the terminal device or switch to a non-AI / ML functionality at the terminal device.

[0244] In some example embodiments, the method 900 further comprises: receiving, from the terminal device, generalization capability of a second AI / ML model or functionality with a higher generalization score than the first generalization score.

[0245] In some example embodiments, the generalization capability of the first AI / ML model or functionality further comprises at least one of the following: pair information, the pair information indicating a plurality of pairs of scenarios and communication configurations supported in enabling the first AI / ML model or functionality, or characteristics of the first AI / ML model or functionality.

[0246] In some example embodiments, the method 900 further comprises: in accordance with a determination that the first AI / ML model or functionality is to be enabled or activated, determining based on characteristics of the first AI / ML model or functionality, a further AI / ML model or functionality at the network device.

[0247] In some example embodiments, the method 900 further comprises: determining, based at least one the generalization capability of the first AI / ML model or functionality, a reporting frequency for performance monitoring on at least one AI / ML model or functionality in the terminal device; and transmitting, to the terminal device, a configuration indicating the reporting frequency.

[0248] FIG. 10 shows a flowchart of an example method 1000 implemented at a first apparatus in a test stage in accordance with some example embodiments of the present disclosure. In some example embodiments, the first apparatus may be or be comprised in a terminal device. For the purpose of discussion, the method 1000 will be described from the perspective of the terminal device 120 in FIG. 1.

[0249] At block 1010, the first apparatus transmits, to a second apparatus, capability information indicating generalization capability of an artificial intelligence / machine learning (AI / ML) model or functionality in the first apparatus, the generalization capability at least comprising pair information, the pair information indicating a plurality of pairs of scenarios and communication configurations in enabling the AI / ML model or functionality.

[0250] At block 1020, the first apparatus receives, from the second apparatus, an indication of a generalization score of the AI / ML model or functionality.

[0251] In some example embodiments, the first apparatus may further receive, from the second apparatus, an indication of enabling or disabling the AI / ML model or functionality.

[0252] In some example embodiments, the generalization capability further comprises at least one of the following: a generalization score of the AI / ML model or functionality, or characteristics of the AI / ML model or functionality.

[0253] In some example embodiments, the method 1000 further comprises: receiving, from the second apparatus, an instruction of running the AI / ML model or functionality according to the plurality of pairs of scenarios and communication configurations; and in response to the instruction, causing the AI / ML model or functionality to be run according to the plurality of pairs of scenarios and communication configurations.

[0254] In some example embodiments, the AI / ML model or functionality is configured for channel state information (CSI) compression and is paired with a further AI / ML model or functionality in the second apparatus that is configured for CSI decompression.

[0255] In some example embodiments, the first apparatus is or is comprised in a terminal device and wherein the second apparatus is or is comprised in an equipment for testing the first apparatus.

[0256] FIG. 11 shows a flowchart of an example method 1100 implemented at a second apparatus in a test stage in accordance with some example embodiments of the present disclosure. In some example embodiments, the first apparatus may be or be comprised in a test equipment. For the purpose of discussion, the method 1100 will be described from the perspective of the test equipment 110 in FIG. 1.

[0257] At block 1110, the second apparatus receives, from a first apparatus capability information indicating generalization capability of an artificial intelligence (AI) / machine learning (ML) model or functionality in the first apparatus, the generalization capability at least comprising pair information, the pair information indicating a plurality of pairs of scenarios and communication configurations in enabling the AI / ML model or functionality.

[0258] At block 1120, the second apparatus performs generalization test on the AI / ML model or functionality in the first apparatus based on the generalization capability of the AI / ML model.

[0259] At block 1130, the second apparatus transmits, to the first apparatus and based on a result of the generalization test, an indication of a generalization score of the AI / ML model or functionality.

[0260] In some example embodiments, the second apparatus may further transmit, to the first apparatus, an indication of enabling or disabling the AI / ML model or functionality.

[0261] In some example embodiments, the generalization capability further comprises at least one of the following: a generalization score of the AI / ML model or functionality, or characteristics of the AI / ML model or functionality.

[0262] In some example embodiments, the method 1100 further comprises: transmitting, to the first apparatus, an instruction of running the AI / ML model or functionality according to a plurality of pairs of scenarios and communication configurations.

[0263] In some example embodiments, the method 1100 further comprises: determining, based on characteristics of the AI / ML model or functionality in the first apparatus, a further AI / ML model or functionality at the second apparatus; and performing generalization test on the AI / ML model or functionality by running the further AI / ML model or functionality at the second apparatus.

[0264] In some example embodiments, the second apparatus is further caused to perform the generalization test by: determining, from the generalization capability, a baseline pair of scenario and communication configuration and at least one non-baseline pair of scenario and communication configuration; and determining a baseline performance indicator for the AI / ML model or functionality by causing the AI / ML model or functionality at the first apparatus to run according to the baseline pair of scenario and communication configuration; determining at least one non-baseline performance indicator for the AI / ML model or functionality by causing the AI / ML model or functionality at the first apparatus to run according to the at least one non-baseline pair of scenario and communication configuration; and determining a test generalization score for the AI / ML model or functionality based on a performance degradation of the at least one non-baseline performance indicator from the baseline performance indicator.

[0265] In some example embodiments, the AI / ML model or functionality in the first apparatus is configured for channel state information (CSI) compression and is paired with a further AI / ML model or functionality in the second apparatus that is configured for CSI decompression.

[0266] In some example embodiments, the first apparatus is or is comprised in a terminal device, and wherein the second apparatus is or is comprised in an equipment for testing the first apparatus.

[0267] In some example embodiments, a first apparatus capable of performing any of the method 800 (for example, the terminal device 120 in FIG. 1) may comprise means for performing the respective operations of the method 800. The means may be implemented in any suitable form. For example, the means may be implemented in a circuitry or software module. The first apparatus may be implemented as or included in the terminal device 120 in FIG. 1.

[0268] In some example embodiments, the first apparatus comprises means for obtaining capability information indicating generalization capability of a first artificial intelligence (AI) / machine learning (ML) model or functionality in the first apparatus, the generalization capability at least comprising a first generalization score of the first AI / ML model or functionality; and means for transmitting the capability information to a network device.

[0269] In some example embodiments, the first apparatus further comprises: means for receiving, from the network device, a request for selecting an AI / ML model or functionality with a higher generalization score than the first generalization score of the first AI / ML model or functionality; and means for transmitting, to the network device, a response to the request indicating whether an AI / ML model or functionality with a higher generalization score is available at the first apparatus.

[0270] In some example embodiments, the first apparatus further comprises: in accordance with a determination that a second AI / ML model or functionality with a higher generalization score than the first generalization score is available in the first apparatus, means for transmitting, to the network device, generalization capability of the second AI / ML model or functionality.

[0271] In some example embodiments, the generalization capability of the first AI / ML model or functionality further comprises at least one of the following: pair information, the pair information indicating a plurality of pairs of scenarios and communication configurations supported in enabling the first AI / ML model or functionality, or characteristics of the first AI / ML model or functionality.

[0272] In some example embodiments, the first apparatus further comprises: means for receiving, from the network device, a configuration indicating a reporting frequency for performance monitoring on at least one AI / ML model or functionality in the first apparatus.

[0273] In some example embodiments, the first apparatus further comprises: means for receiving, from the network device, an indication of enablement or activation of the first AI / ML model or functionality at the terminal device.

[0274] In some example embodiments, the first AI / ML model or functionality is configured for channel state information (CSI) compression and is paired with a further AI / ML model or functionality in the network device that is configured for CSI decompression.

[0275] In some example embodiments, the first apparatus is or is comprised in a terminal device.

[0276] In some example embodiments, a second apparatus capable of performing any of the method 900 (for example, the network device 130 in FIG. 1) may comprise means for performing the respective operations of the method 900. The means may be implemented in any suitable form. For example, the means may be implemented in a circuitry or software module. The second apparatus may be implemented as or included in the network device 130 in FIG. 1.

[0277] In some example embodiments, the second apparatus comprises means for receiving, from a terminal device, capability information indicating generalization capability of a first artificial intelligence (AI) / machine learning (ML) model or functionality in the terminal device, the generalization capability at least comprising a first generalization score of the first AI / ML model or functionality; and means for determining enablement or activation of the first AI / ML model or functionality at the terminal device based on a determination of whether the first generalization score satisfies a generalization requirement.

[0278] In some example embodiments, the second apparatus further comprises: means for in accordance with a determination that the first generalization score fails to satisfy the generalization requirement, transmitting to the terminal device, a request for selecting an AI / ML model or functionality with a higher generalization score than the first generalization score of the first AI / ML model or functionality; and means for receiving, from the terminal device, a response to the request indicating whether an AI / ML model or functionality with a higher generalization score is available at the terminal device.

[0279] In some example embodiments, the second apparatus further comprises: means for in accordance with a determination that the response indicates that an AI / ML model or functionality with a higher generalization score is unavailable at the terminal device, determining to enable the first AI / ML model or functionality at the terminal device or switch to a non-AI / ML functionality at the terminal device.

[0280] In some example embodiments, the second apparatus further comprises: means for receiving, from the terminal device, generalization capability of a second AI / ML model or functionality with a higher generalization score than the first generalization score.

[0281] In some example embodiments, the generalization capability of the first AI / ML model or functionality further comprises at least one of the following: pair information, the pair information indicating a plurality of pairs of scenarios and communication configurations supported in enabling the first AI / ML model or functionality, or characteristics of the first AI / ML model or functionality.

[0282] In some example embodiments, the second apparatus further comprises: means for in accordance with a determination that the first AI / ML model or functionality is to be enabled or activated, determining based on characteristics of the first AI / ML model or functionality, a further AI / ML model or functionality at the network device.

[0283] In some example embodiments, the second apparatus further comprises: means for determining, based at least one the generalization capability of the first AI / ML model or functionality, a reporting frequency for performance monitoring on at least one AI / ML model or functionality in the terminal device; and means for transmitting, to the terminal device, a configuration indicating the reporting frequency.

[0284] In some example embodiments, a first apparatus capable of performing any of the method 1000 (for example, the terminal device 120 in FIG. 1) may comprise means for performing the respective operations of the method 1000. The means may be implemented in any suitable form. For example, the means may be implemented in a circuitry or software module. The first apparatus may be implemented as or included in the terminal device 120 in FIG. 1.

[0285] In some example embodiments, the first apparatus comprises means for transmitting, to a second apparatus, capability information indicating generalization capability of an artificial intelligence / machine learning (AI / ML) model or functionality in the first apparatus, the generalization capability at least comprising pair information, the pair information indicating a plurality of pairs of scenarios and communication configurations in enabling the AI / ML model or functionality; and means for receiving, from the second apparatus, an indication of a generalization score of the AI / ML model or functionality.

[0286] In some example embodiments, the first apparatus further comprises means for receiving, from the second apparatus, an indication of enabling or disabling the AI / ML model or functionality.

[0287] In some example embodiments, the generalization capability further comprises at least one of the following: a generalization score of the AI / ML model or functionality, or characteristics of the AI / ML model or functionality.

[0288] In some example embodiments, the first apparatus further comprises: means for receiving, from the second apparatus, an instruction of running the AI / ML model or functionality according to the plurality of pairs of scenarios and communication configurations; and means for in response to the instruction, causing the AI / ML model or functionality to be run according to the plurality of pairs of scenarios and communication configurations.

[0289] In some example embodiments, the AI / ML model or functionality is configured for channel state information (CSI) compression and is paired with a further AI / ML model or functionality in the second apparatus that is configured for CSI decompression.

[0290] In some example embodiments, the first apparatus is or is comprised in a terminal device and wherein the second apparatus is or is comprised in an equipment for testing the first apparatus.

[0291] In some example embodiments, a second apparatus capable of performing any of the method 1100 (for example, the test equipment 110 in FIG. 1) may comprise means for performing the respective operations of the method 1100. The means may be implemented in any suitable form. For example, the means may be implemented in a circuitry or software module. The second apparatus may be implemented as or included in the test equipment 110 in FIG. 1.

[0292] In some example embodiments, the second apparatus comprises means for receiving, from a first apparatus capability information indicating generalization capability of an artificial intelligence (AI) / machine learning (ML) model or functionality in the first apparatus, the generalization capability at least comprising pair information, the pair information indicating a plurality of pairs of scenarios and communication configurations in enabling the AI / ML model or functionality; means for performing generalization test on the AI / ML model or functionality in the first apparatus based on the generalization capability of the AI / ML model or functionality; and means for transmitting, to the first apparatus and based on a result of the generalization test, an indication of a generalization score of the AI / ML model or functionality.

[0293] In some example embodiments, the second apparatus further comprises means for transmitting, to the first apparatus, an indication of enabling or disabling the AI / ML model or functionality.

[0294] In some example embodiments, the generalization capability further comprises at least one of the following: a generalization score of the AI / ML model or functionality, or characteristics of the AI / ML model or functionality.

[0295] In some example embodiments, the second apparatus further comprises: means for transmitting, to the first apparatus, an instruction of running the AI / ML model or functionality according to a plurality of pairs of scenarios and communication configurations.

[0296] In some example embodiments, the second apparatus further comprises: means for determining, based on characteristics of the AI / ML model or functionality in the first apparatus, a further AI / ML model or functionality at the second apparatus; and means for performing generalization test on the AI / ML model or functionality by running the further AI / ML model or functionality at the second apparatus.

[0297] In some example embodiments, the second apparatus is further caused to perform the generalization test by: determining, from the generalization capability, a baseline pair of scenario and communication configuration and at least one non-baseline pair of scenario and communication configuration; and determining a baseline performance indicator for the AI / ML model or functionality by causing the AI / ML model or functionality at the first apparatus to run according to the baseline pair of scenario and communication configuration; determining at least one non-baseline performance indicator for the AI / ML model or functionality by causing the AI / ML model or functionality at the first apparatus to run according to the at least one non-baseline pair of scenario and communication configuration; and determining a test generalization score for the AI / ML model or functionality based on a performance degradation of the at least one non-baseline performance indicator from the baseline performance indicator.

[0298] In some example embodiments, the AI / ML model or functionality in the first apparatus is configured for channel state information (CSI) compression and is paired with a further AI / ML model or functionality in the second apparatus that is configured for CSI decompression.

[0299] In some example embodiments, the first apparatus is or is comprised in a terminal device, and wherein the second apparatus is or is comprised in an equipment for testing the first apparatus.

[0300] FIG. 12 is a simplified block diagram of a device 1200 that is suitable for implementing example embodiments of the present disclosure. The device 1200 may be provided to implement a communication device, for example, the test equipment 110, the terminal device 120 or the network device 130 as shown in FIG. 1. As shown, the device 1200 includes one or more processors 1210, one or more memories 1220 coupled to the processor 1210, and one or more communication modules 1240 coupled to the processor 1210.

[0301] The communication module 1240 is for bidirectional communications. The communication module 1240 has one or more communication interfaces to facilitate communication with one or more other modules or devices. The communication interfaces may represent any interface that is necessary for communication with other network elements. In some example embodiments, the communication module 1240 may include at least one antenna.

[0302] The processor 1210 may be of any type suitable to the local technical network and may include one or more of the following: general purpose computers, special purpose computers, microprocessors, digital signal processors (DSPs) and processors based on multicore processor architecture, as non-limiting examples. The device 1200 may have multiple processors, such as an application specific integrated circuit chip that is slaved in time to a clock which synchronizes the main processor.

[0303] The memory 1220 may include one or more non-volatile memories and one or more volatile memories. Examples of the non-volatile memories include, but are not limited to, a Read Only Memory (ROM) 1224, an electrically programmable read only memory (EPROM), a flash memory, a hard disk, a compact disc (CD), a digital video disk (DVD), an optical disk, a laser disk, and other magnetic storage and / or optical storage. Examples of the volatile memories include, but are not limited to, a random-access memory (RAM) 1222 and other volatile memories that will not last in the power-down duration.

[0304] A computer program 1230 includes computer executable instructions that are executed by the associated processor 1210. The instructions of the program 1230 may include instructions for performing operations / acts of some example embodiments of the present disclosure. The program 1230 may be stored in the memory, e.g., the ROM 1224. The processor 1210 may perform any suitable actions and processing by loading the program 1230 into the RAM 1222.

[0305] The example embodiments of the present disclosure may be implemented by means of the program 1230 so that the device 1200 may perform any process of the disclosure as discussed with reference to FIG. 4 to FIG. 11. The example embodiments of the present disclosure may also be implemented by hardware or by a combination of software and hardware.

[0306] In some example embodiments, the program 1230 may be tangibly contained in a computer readable medium which may be included in the device 1200 (such as in the memory 1220) or other storage devices that are accessible by the device 1200. The device 1200 may load the program 1230 from the computer readable medium to the RAM 1222 for execution. In some example embodiments, the computer readable medium may include any types of non-transitory storage medium, such as ROM, EPROM, a flash memory, a hard disk, CD, DVD, and the like. The term “non-transitory,” as used herein, is a limitation of the medium itself (i.e., tangible, not a signal) as opposed to a limitation on data storage persistency (e.g., RAM vs. ROM).

[0307] FIG. 13 shows an example of the computer readable medium 1300 which may be in form of CD, DVD or other optical storage disk. The computer readable medium 1300 has the program 1230 stored thereon.

[0308] Generally, various embodiments of the present disclosure may be implemented in hardware or special purpose circuits, software, logic or any combination thereof. Some aspects may be implemented in hardware, and other aspects may be implemented in firmware or software which may be executed by a controller, microprocessor or other computing device. Although various aspects of embodiments of the present disclosure are illustrated and described as block diagrams, flowcharts, or using some other pictorial representations, it is to be understood that the block, apparatus, system, technique or method described herein may be implemented in, as non-limiting examples, hardware, software, firmware, special purpose circuits or logic, general purpose hardware or controller or other computing devices, or some combination thereof.

[0309] Some example embodiments of the present disclosure also provide at least one computer program product tangibly stored on a computer readable medium, such as a non-transitory computer readable medium. The computer program product includes computerexecutable instructions, such as those included in program modules, being executed in a device on a target physical or virtual processor, to carry out any of the methods as described above. Generally, program modules include routines, programs, libraries, objects, classes, components, data structures, or the like that perform particular tasks or implement particular abstract data types. The functionality of the program modules may be combined or split between program modules as desired in various embodiments. Machine-executable instructions for program modules may be executed within a local or distributed device. In a distributed device, program modules may be located in both local and remote storage media.

[0310] Program code for carrying out methods of the present disclosure may be written in any combination of one or more programming languages. The program code may be provided to a processor or controller of a general purpose computer, special purpose computer, or other programmable data processing apparatus, such that the program code, when executed by the processor or controller, cause the functions / operations specified in the flowcharts and / or block diagrams to be implemented. The program code may execute entirely on a machine, partly on the machine, as a stand-alone software package, partly on the machine and partly on a remote machine or entirely on the remote machine or server.

[0311] In the context of the present disclosure, the computer program code or related data may be carried by any suitable carrier to enable the device, apparatus or processor to perform various processes and operations as described above. Examples of the carrier include a signal, computer readable medium, and the like.

[0312] The computer readable medium may be a computer readable signal medium or a computer readable storage medium. A computer readable medium may include but not limited to an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any suitable combination of the foregoing. More specific examples of the computer readable storage medium would include an electrical connection having one or more wires, a portable computer diskette, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or Flash memory), an optical fiber, a portable compact disc read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

[0313] Further, although operations are depicted in a particular order, this should not be understood as requiring that such operations be performed in the particular order shown or in sequential order, or that all illustrated operations be performed, to achieve desirable results. In certain circumstances, multitasking and parallel processing may be advantageous. Likewise, although several specific implementation details are contained in the above discussions, these should not be construed as limitations on the scope of the present disclosure, but rather as descriptions of features that may be specific to particular embodiments. Unless explicitly stated, certain features that are described in the context of separate embodiments may also be implemented in combination in a single embodiment. Conversely, unless explicitly stated, various features that are described in the context of a single embodiment may also be implemented in a plurality of embodiments separately or in any suitable sub-combination.

[0314] Although the present disclosure has been described in languages specific to structural features and / or methodological acts, it is to be understood that the present disclosure defined in the appended claims is not necessarily limited to the specific features or acts described above. Rather, the specific features and acts described above are disclosed as example forms of implementing the claims.

Claims

1. A first apparatus comprising:at least one processor; andat least one memory storing instructions that, when executed by the at least one processor, cause the first apparatus at least to:transmit, to a second apparatus, capability information indicating generalization capability of an artificial intelligence / machine learning (AI / ML) model or functionality in the first apparatus, the generalization capability at least comprising pair information, the pair information indicating a plurality of pairs of scenarios and communication configurations in enabling the AI / ML model or functionality; andreceive, from the second apparatus, an indication of a generalization score of the AI / ML model or functionality.

2. The first apparatus of claim 1, wherein the generalization capability further comprises at least one of the following:a generalization score of the AI / ML model or functionality, or characteristics of the AI / ML model or functionality.

3. The first apparatus of claim 1 or 2, wherein the first apparatus is further caused to:receive, from the second apparatus, an instruction of running the AI / ML model or functionality according to the plurality of pairs of scenarios and communication configurations; andin response to the instruction, cause the AI / ML model or functionality to be run according to the plurality of pairs of scenarios and communication configurations.

4. The first apparatus of any of claims 1 to 3, wherein the AI / ML model or functionality is configured for channel state information (CSI) compression and is paired with a further AI / ML model or functionality in the second apparatus that is configured for CSI decompression.

5. The first apparatus of any of claims 1 to 4, wherein the first apparatus is or is comprised in a terminal device and wherein the second apparatus is or is comprised in an equipment for testing the first apparatus.

6. A second apparatus comprising:at least one processor; andat least one memory storing instructions that, when executed by the at least one processor, cause the second apparatus at least to:receive, from a first apparatus capability information indicating generalization capability of an artificial intelligence (AI) / machine learning (ML) model or functionality in the first apparatus, the generalization capability at least comprising pair information, the pair information indicating a plurality of pairs of scenarios and communication configurations in enabling the AI / ML model or functionality;perform generalization test on the AI / ML model or functionality in the first apparatus based on the generalization capability of the AI / ML model or functionality; andtransmit, to the first apparatus and based on a result of the generalization test, an indication of a generalization score of the AI / ML model or functionality.

7. The second apparatus of claim 6, wherein the generalization capability further comprises at least one of the following:a generalization score of the AI / ML model or functionality, orcharacteristics of the AI / ML model or functionality.

8. The second apparatus of claim 6 or 7, wherein the second apparatus is further caused to:transmit, to the first apparatus, an instruction of running the AI / ML model or functionality according to a plurality of pairs of scenarios and communication configurations.

9. The second apparatus of any of claims 6 to 8, wherein the second apparatus is further caused to:determine, based on characteristics of the AI / ML model or functionality in the first apparatus, a further AI / ML model or functionality at the second apparatus; andperform generalization test on the AI / ML model or functionality by running the further AI / ML model or functionality at the second apparatus.

10. The second apparatus of any of claims 6 to 9, wherein the second apparatus is further caused to perform the generalization test by:determining, from the generalization capability, a baseline pair of scenario and communication configuration and at least one non-baseline pair of scenario and communication configuration; anddetermining a baseline performance indicator for the AI / ML model or functionality by causing the AI / ML model or functionality at the first apparatus to run according to the baseline pair of scenario and communication configuration;determining at least one non-baseline performance indicator for the AI / ML model or functionality by causing the AI / ML model or functionality at the first apparatus to run according to the at least one non-baseline pair of scenario and communication configuration; anddetermining a test generalization score for the AI / ML model or functionality based on a performance degradation of the at least one non-baseline performance indicator from the baseline performance indicator.

11. The second apparatus of any of claims 6 to 10, wherein the AI / ML model or functionality in the first apparatus is configured for channel state information (CSI) compression and is paired with a further AI / ML model or functionality in the second apparatus that is configured for CSI decompression.

12. The second apparatus of any of claims 6 to 11, wherein the first apparatus is or is comprised in a terminal device, and wherein the second apparatus is or is comprised in an equipment for testing the first apparatus.

13. A method comprising:transmitting, by a first apparatus and to a second apparatus, capability information indicating generalization capability of an artificial intelligence / machine learning (AI / ML) model or functionality in the first apparatus, the generalization capability at least comprising pair information, the pair information indicating a plurality of pairs of scenarios and communication configurations in enabling the AI / ML model or functionality.receiving, from the second apparatus, an indication of a generalization score of the AI / ML model or functionality.

14. A method comprising:receiving, by a second apparatus and from a first apparatus capability information indicating generalization capability of an artificial intelligence (AI) / machine learning (ML) model or functionality in the first apparatus, the generalization capability at least comprising 5 pair information, the pair information indicating a plurality of pairs of scenarios and communication configurations in enabling the AI / ML model or functionality.performing generalization test on the AI / ML model in the first apparatus based on the generalization capability of the AI / ML model.transmitting, to the first apparatus and based on a result of the generalization test, an 10 indication of a generalization score of the AI / ML model or functionality.

15. A computer readable medium comprising instructions stored thereon for causing an apparatus at least to perform the method of claim 13 or the method of claim 14.

Images