Latency testing of CSI feedback based on ai / ml

Abstract

Transmit, from a first apparatus to a second apparatus, a first indication of a first action to be performed for channel state information, CSI, feedback based on artificial intelligence / machine learning; receive, from the second apparatus, a first response about completion of the first action; and determine a result of a test on the first action based on the first response and a first time length indicating a latency requirement for the first action.

Description

LATENCY TESTING OF CSI FEEDBACK BASED ON AI / MLCROSS-REFERENCE TO RELATED APPLICATION

[0001] The present application claims priority to, and the benefit of, India Provisional Application No. 202441010950, filed February 16, 2024, the contents of which are hereby incorporated by reference in their entirety.FIELDS

[0002] Various example embodiments of the present disclosure generally relate to the field of telecommunication and in particular, to methods, devices, apparatuses and computer readable storage medium for latency testing of channel state information (CSI) feedback based on artificial intelligence / machine learning (AI / ML).BACKGROUND

[0003] Several technologies have been proposed to improve communication performances. For example, communication devices may employ an artificial intelligence (Al) / ML model to improve communication qualities. The Al / ML model can be applied to different use cases, for example CSI feedback. Therefore, performances of CSI feedback based on AL / ML are an importance issue.SUMMARY

[0004] In a first aspect of the present disclosure, there is provided a first apparatus. The first apparatus comprises at least one processor; and at least one memory storing instructions that, when executed by the at least one processor, cause the first apparatus at least to: transmit, to a second apparatus, a first indication of a first action to be performed for channel state information, CSI, feedback based on artificial intelligence / machine learning, AI / ML; receive, from the second apparatus, a first response about completion of the first action; and determine a result of a test on the first action based on the first response and a first time length indicating a latency requirement for the first action.

[0005] In a second aspect of the present disclosure, there is provided a second apparatus. The second apparatus comprises at least one processor; and at least one memory storing instructions that, when executed by the at least one processor, cause the second apparatus at least to: receive, from a first apparatus, a first indication of a first action to be performed ifor channel state information, CSI, feedback based on artificial intelligence / machine learning, AI / ML; and transmit, to the first apparatus, a first response about completion of the first action, the first response being used together with a first time length indicating a latency requirement for the first action to determine a result of a test on the first action.

[0006] In a third aspect of the present disclosure, there is provided a method. The method comprises: transmitting, at a first apparatus to a second apparatus, a first indication of a first action to be performed for channel state information, CSI, feedback based on artificial intelligence / machine learning, AI / ML; receiving, from the second apparatus, a first response about completion of the first action; and determining a result of a test on the first action based on the first response and a first time length indicating a latency requirement for the first action.

[0007] In a fourth aspect of the present disclosure, there is provided a method. The method comprises: receiving, at a second apparatus from a first apparatus, a first indication of a first action to be performed for channel state information, CSI, feedback based on artificial intelligence / machine learning, AI / ML; and transmitting, to the first apparatus, a first response about completion of the first action, the first response being used together with a first time length indicating a latency requirement for the first action to determine a result of a test on the first action.

[0008] In a fifth aspect of the present disclosure, there is provided a first apparatus. The first apparatus comprises means for transmitting, to a second apparatus, a first indication of a first action to be performed for channel state information, CSI, feedback based on artificial intelligence / machine learning, AI / ML; means for receiving, from the second apparatus, a first response about completion of the first action; and means for determining a result of a test on the first action based on the first response and a first time length indicating a latency requirement for the first action.

[0009] In a sixth aspect of the present disclosure, there is provided a second apparatus. The second apparatus comprises means for receiving, from a first apparatus, a first indication of a first action to be performed for channel state information, CSI, feedback based on artificial intelligence / machine learning, AI / ML; and means for transmitting, to the first apparatus, a first response about completion of the first action, the first response being used together with a first time length indicating a latency requirement for the first action to determine a result of a test on the first action.

[0010] In a seventh aspect of the present disclosure, there is provided a computer readable medium. The computer readable medium comprises instructions stored thereonfor causing an apparatus to perform at least the method according to the third aspect.

[0011] In an eighth aspect of the present disclosure, there is provided a computer readable medium. The computer readable medium comprises instructions stored thereon for causing an apparatus to perform at least the method according to the fourth aspect.

[0012] It is to be understood that the Summary section is not intended to identify key or essential features of embodiments of the present disclosure, nor is it intended to be used to limit the scope of the present disclosure. Other features of the present disclosure will become easily comprehensible through the following description.BRIEF DESCRIPTION OF THE DRAWINGS

[0013] Some example embodiments will now be described with reference to the accompanying drawings, where:

[0014] FIG. 1 illustrates an example communication environment in which example embodiments of the present disclosure can be implemented;

[0015] FIG. 2 illustrates a schematic diagram of a functional framework for AI / ML;

[0016] FIG. 3 illustrates an example of performance degradation of an AI / ML functionality;

[0017] FIG. 4 illustrates an example signaling chart for latency testing of CSI feedback based on AI / ML according to some example embodiments of the present disclosure;

[0018] FIG. 5 illustrates another example signaling chart for latency testing of CSI feedback based on AI / ML according to some example embodiments of the present disclosure;

[0019] FIG. 6 illustrates an example signaling chart for negotiating latency requirements according to some example embodiments of the present disclosure;

[0020] FIG. 7 illustrates another example signaling chart for negotiating latency requirements according to some example embodiments of the present disclosure;

[0021] FIG. 8 illustrates a flowchart of a method implemented at a first device according to some example embodiments of the present disclosure;

[0022] FIG. 9 illustrates a flowchart of a method implemented at a second device according to some example embodiments of the present disclosure;

[0023] FIG. 10 illustrates a simplified block diagram of a device that is suitable for implementing example embodiments of the present disclosure; and

[0024] Throughout the drawings, the same or similar reference numerals represent the same or similar element.DETAILED DESCRIPTION

[0025] Principle of the present disclosure will now be described with reference to some example embodiments. It is to be understood that these embodiments are described only for the purpose of illustration and help those skilled in the art to understand and implement the present disclosure, without suggesting any limitation as to the scope of the disclosure. Embodiments described herein can be implemented in various manners other than the ones described below.

[0026] In the following description and claims, unless defined otherwise, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skills in the art to which this disclosure belongs.

[0027] References in the present disclosure to “one embodiment,” “an embodiment,” “an example embodiment,” and the like indicate that the embodiment described may include a particular feature, structure, or characteristic, but it is not necessary that every embodiment includes the particular feature, structure, or characteristic. Moreover, such phrases are not necessarily referring to the same embodiment. Further, when a particular feature, structure, or characteristic is described in connection with an embodiment, it is submitted that it is within the knowledge of one skilled in the art to affect such feature, structure, or characteristic in connection with other embodiments whether or not explicitly described.

[0028] It shall be understood that although the terms “first,” “second,”..., etc. in front of noun(s) and the like may be used herein to describe various elements, these elements should not be limited by these terms. These terms are only used to distinguish one element from another and they do not limit the order of the noun(s). For example, a first element could be termed a second element, and similarly, a second element could be termed a first element, without departing from the scope of example embodiments. As used herein, the term “and / or” includes any and all combinations of one or more of the listed terms.

[0029] As used herein, “at least one of the following: ” and “at least one of ” and similar wording, where the list of two or more elements are joined by “and” or “or”, mean at least any one of the elements, or at least any two or more of the elements, or at least all the elements.

[0030] As used herein, unless stated explicitly, performing a step “in response to A” does not indicate that the step is performed immediately after “A” occurs and one or more intervening steps may be included.

[0031] The terminology used herein is for the purpose of describing particular embodiments only and is not intended to be limiting of example embodiments. As used herein, the singular forms “a”, “an” and “the” are intended to include the plural forms as well, unless the context clearly indicates otherwise. It will be further understood that the terms “comprises”, “comprising”, “has”, “having”, “includes” and / or “including”, when used herein, specify the presence of stated features, elements, and / or components etc., but do not preclude the presence or addition of one or more other features, elements, components and / or combinations thereof.

[0032] As used in this application, the term “circuitry” may refer to one or more or all of the following:(a) hardware-only circuit implementations (such as implementations in only analog and / or digital circuitry) and(b) combinations of hardware circuits and software, such as (as applicable):(i) a combination of analog and / or digital hardware circuit(s) with software / firmware and(ii) any portions of hardware processor(s) with software (including digital signal processor(s)), software, and memory(ies) that work together to cause an apparatus, such as a mobile phone or server, to perform various functions) and(c) hardware circuit(s) and or processor(s), such as a microprocessor(s) or a portion of a microprocessor(s), that requires software (e.g., firmware) for operation, but the software may not be present when it is not needed for operation.

[0033] This definition of circuitry applies to all uses of this term in this application, including in any claims. As a further example, as used in this application, the term circuitry also covers an implementation of merely a hardware circuit or processor (or multiple processors) or portion of a hardware circuit or processor and its (or their) accompanying software and / or firmware. The term circuitry also covers, for example and if applicable to the particular claim element, a baseband integrated circuit or processor integrated circuit for a mobile device or a similar integrated circuit in server, a cellular network device, or other computing or network device.

[0034] As used herein, the term “communication network” refers to a network following any suitable communication standards, such as New Radio (NR), Long Term Evolution (LTE), LTE-Advanced (LTE-A), Wideband Code Division Multiple Access (WCDMA),High-Speed Packet Access (HSPA), Narrow Band Internet of Things (NB-IoT) and so on. Furthermore, the communications between a terminal device and a network device in the communication network may be performed according to any suitable generation communication protocols, including, but not limited to, the first generation (1G), the second generation (2G), 2.5G, 2.75G, the third generation (3G), the fourth generation (4G), 4.5G, the fifth generation (5G), the sixth generation (6G) communication protocols, and / or any other protocols either currently known or to be developed in the future. Embodiments of the present disclosure may be applied in various communication systems. Given the rapid development in communications, there will of course also be future type communication technologies and systems with which the present disclosure may be embodied. It should not be seen as limiting the scope of the present disclosure to only the aforementioned system.

[0035] As used herein, the term “network device” refers to a node in a communication network via which a terminal device accesses the network and receives services therefrom. The network device may refer to a base station (BS) or an access point (AP), for example, a node B (NodeB or NB), an evolved NodeB (eNodeB or eNB), an NR NB (also referred to as a gNB), a Remote Radio Unit (RRU), a radio header (RH), a remote radio head (RRH), a relay, an Integrated Access and Backhaul (IAB) node, a low power node such as a femto, a pico, a non-terrestrial network (NTN) or non-ground network device such as a satellite network device, a low earth orbit (LEO) satellite and a geosynchronous earth orbit (GEO) satellite, an aircraft network device, and so forth, depending on the applied terminology and technology. In some example embodiments, radio access network (RAN) split architecture comprises a Centralized Unit (CU) and a Distributed Unit (DU) at an IAB donor node. An IAB node comprises a Mobile Terminal (IAB-MT) part that behaves like a UE toward the parent node, and a DU part of an IAB node behaves like a base station toward the next-hop IAB node.

[0036] The term “terminal device” refers to any end device that may be capable of wireless communication. By way of example rather than limitation, a terminal device may also be referred to as a communication device, user equipment (UE), a Subscriber Station (SS), a Portable Subscriber Station, a Mobile Station (MS), or an Access Terminal (AT). The terminal device may include, but not limited to, a mobile phone, a cellular phone, a smart phone, voice over IP (VoIP) phones, wireless local loop phones, a tablet, a wearable terminal device, a personal digital assistant (PDA), portable computers, desktop computer, image capture terminal devices such as digital cameras, gaming terminal devices, musicstorage and playback appliances, vehicle-mounted wireless terminal devices, wireless endpoints, mobile stations, laptop-embedded equipment (LEE), laptop-mounted equipment (LME), USB dongles, smart devices, wireless customer-premises equipment (CPE), an Internet of Things (loT) device, a watch or other wearable, a head-mounted display (HMD), a vehicle, a drone, a medical device and applications (e.g., remote surgery), an industrial device and applications (e.g., a robot and / or other wireless devices operating in an industrial and / or an automated processing chain contexts), a consumer electronics device, a device operating on commercial and / or industrial wireless networks, and the like. The terminal device may also correspond to a Mobile Termination (MT) part of an IAB node (e.g., a relay node). In the following description, the terms “terminal device”, “communication device”, “terminal”, “user equipment” and “UE” may be used interchangeably.

[0037] As used herein, the term “resource,” “transmission resource,” “resource block,” “physical resource block” (PRB), “uplink resource,” or “downlink resource” may refer to any resource for performing a communication, for example, a communication between a terminal device and a network device, such as a resource in time domain, a resource in frequency domain, a resource in space domain, a resource in code domain, or any other combination of the time, frequency, space and / or code domain resource enabling a communication, and the like. In the following, unless explicitly stated, a resource in both frequency domain and time domain will be used as an example of a transmission resource for describing some example embodiments of the present disclosure. It is noted that example embodiments of the present disclosure are equally applicable to other resources in other domains.

[0038] As used herein, the term “AI / ML model” may refer to a data driven algorithm that applies AI / ML techniques to generate a set of outputs based on a set of inputs. In the context of the present disclosure, the term “AI / ML model” may be used interchangeably with the terms “model”, “Al model” and “ML model”. The term “AI / ML” may be used interchangeably with the terms “Al” and “ML”. In the present disclosure, the term “AI / ML model” and the term “AI / ML functionality” can be used interchangeably.

[0039] FIG. 1 illustrates an example communication environment 100 in which example embodiments of the present disclosure may be implemented. As shown in FIG. 1, the communication network 100 may include a first apparatus 110 and a second apparatus 120. The first apparatus 110 may communicate with the second apparatus 120. The second apparatus 120 may be configured to perform CSI feedback based on AI / ML, which maybe also referred to as “AI / ML enabled CSI feedback”. CSI feedback may include but not limited to CSI prediction, CSI compression, etc. Monitoring or testing the performance of the CSI feedback based on AI / ML may be needed.

[0040] It is to be understood that the number of second apparatus and first apparatus shown in FIG. 1 is given for the purpose of illustration without suggesting any limitations. The communication network 100 may include any suitable number of second apparatus and first apparatus.

[0041] In some example embodiments, the first apparatus 110 may comprise a test equipment (TE), and the second apparatus 120 may comprise a device under test (DUT). For example, the CSI feedback based on AI / ML at the second apparatus 120 may need to be tested.

[0042] In some example embodiments, the first apparatus 110 may comprise a network device (for example, a gNB), and the second apparatus 120 may comprise a terminal device (for example, a UE). For example, the performance of the CSI feedback based on AI / ML at the UE may need to be monitored.

[0043] In the following, for the purpose of illustration, some example embodiments are described with the first apparatus 110 operating as a network device and the second apparatus 120 operating as a terminal device. However, in some example embodiments, operations described in connection with a terminal device may be implemented at a network device or other device, and operations described in connection with a network device may be implemented at a terminal device or other device.

[0044] In some example embodiments, if the first apparatus 110 is a network device and the second apparatus 120 is a terminal device, a link from the second apparatus 120 to the first apparatus 110 is referred to as an uplink (UL), and a link from the first apparatus 110 to the second apparatus 120 is referred to as a downlink (DL). In UL, the second apparatus 120 is a transmitting (TX) device (or a transmitter) and the first apparatus 110 is a receiving (RX) device (or a receiver). In DL, the first apparatus 110 is a TX device (or a transmitter) and the second apparatus 120 is a RX device (or a receiver).

[0045] Communications in the communication environment 100 may be implemented according to any proper communication protocol(s), comprising, but not limited to, cellular communication protocols of the first generation (1G), the second generation (2G), the third generation (3G), the fourth generation (4G), the fifth generation (5G), the sixth generation (6G), and the like, wireless local network communication protocols such as Institute for Electrical and Electronics Engineers (IEEE) 802.11 and the like, and / or anyother protocols currently known or to be developed in the future. Moreover, the communication may utilize any proper wireless communication technology, comprising but not limited to: Code Division Multiple Access (CDMA), Frequency Division Multiple Access (FDMA), Time Division Multiple Access (TDMA), Frequency Division Duplex (FDD), Time Division Duplex (TDD), Multiple-Input Multiple-Output (MIMO), Orthogonal Frequency Division Multiple (OFDM), Discrete Fourier Transform spread OFDM (DFT-s-OFDM) and / or any other technologies currently known or to be developed in the future.

[0046] As briefly mentioned above, AI / ML for New Radio (NR) air interface is now being studied. One of the objectives of the study is to cover the interoperability and testability aspect of the newly defined AI / ML enabled features as listed below.Table 1

[0047] AI / ML enabled positioning is one of the selected use-cases for the study item in Release (Rel) 18 Study Item. The work on this use case may potentially continue in terms of the work item (WI) in Release 19 and also in the upcoming next generations (e.g., in 6G). In Release 18, UE based direct AI / ML positioning is one of the selected sub-use cases for positioning.

[0048] With regard to life cycle management (LCM) procedures for AI / ML functionalities and models, the functional framework for AI / ML for NR air interface is illustrated in FIG. 2. Some example aspects regarding LCM are listed as below.Table 2_

[0049] Based on the Rel-18 LCM discussion in 3rd Generation Partnership Project (3GPP), LCM actions may be triggered by the network (NW) as listed in the table below.Table 3

[0050] For evaluation of performance monitoring approaches, the following model monitoring KPIs as listed below are considered as general guidance.Table 4

[0051] The following table 5 and table 6 list some aspects with regard to LCM and AI / ML CSI feedback in RAN4.Table 5Table 6

[0052] As it can be seen from above tables, LCM aspects are an important topic currently being discussed for AI / ML air interface and there are potential impacts in RAN4 requirements as well.

[0053] Furthermore, AI / ML enabled CSI feedback enhancement is one of the selected use-case for further studies.

[0054] The aspects of inference performance evaluation have been addressed for different selected use cases for AI / ML enabled features as listed below.Table 7

[0055] Table 8 lists KPIs selected for AI / ML based CSI feedback enhancements.Table 8

[0056] Furthermore, in RANI discussions, as mentioned in the TR38.843, performance monitoring is one of the core components of AI / ML based functionalities mainly due to indeterministic nature of AI / ML based solutions. For performance monitoring in case of AI / ML based CSI feedback enhancements use case following text is captured in the TR.Table 9

[0057] As can be seen from above, LCM related impacts for AI / ML based CSI feedbackenhancement use case is a significantly important topic currently under discussion. Therefore, core requirements and testing mechanism for these core requirements to validate the correct functionality of the device is required.

[0058] As discussed above, there is lot of interest in the requirements and testability of LCM aspects for AIML enabled functionalities. One of the factors that influence the performance of the AIML enabled functionality is the latency of the LCM actions between the Network and the UE.

[0059] If performance monitoring detects a performance degradation to a point where a decision to either switch this model / functionality with another model / functionality is taken or a fallback decision is taken, it means that the AI / ML functionality is degrading the system performance and if this functionality, with detected performance degradation, keeps running then the impact on system performance may result in catastrophic consequences.

[0060] Therefore, it is crucial to stop this model / functionality, either by falling back to legacy method or by switching to another model / functionality, within a specified time. This specified time would depend upon the use case since different use cases would need different level of urgency to stop / switch to different model / functionality. For instance, it would be very urgent to stop in case of CSI feedback enhancement because a wrong channel information would lead to wrong link adaptation and scheduling decisions impacting the throughput.

[0061] The specified time allowed to switch / di sable the model / functionality should guarantee that system performance would not be allowed to be degraded to unacceptable levels. Therefore, the values of these specified times may be calculated based on simulation results and / or on field data.

[0062] FIG. 3 depicts an example of performance degradation, where around 10% of degradation is shown for a 40ms delay in the execution of switching / disabling command of LCM action. If the scenario shown in FIG. 3 is not stopped / switched timely the performance would be degraded quickly and a 20% degradation could be observed for a delay of 80ms.

[0063] Currently, there is no RAN4 requirement that guarantees the latency of the LCM to DUT action for AI / ML-enabled CSI feedback, and there is no test mechanism to help validate the LCM performance latency of the feedback.

[0064] An example LCM workflow is illustrated next with respect to the CSI Feedback (for example, compression / prediction) use case. At step 1, AI / ML enabled CSI feedbackis activated in the UE. At step 2, performance monitoring as part of LCM is activated at the network in this example. Then at step 3, the network detects a performance degradation from the monitored KPIs. At step 4, the network decides to take an appropriate LCM action to not allow further degradation in the performance (KPIs). It should be noted that these LCM actions may include but not limited to switch to another model / functionality or switch back to legacy mechanism. Then at step 5, the network initiates the LCM action at the UE. At step 6, UE executes the LCM action. Finally, at step 7, the network either observes improvements in the performance KPIs and if it is not the case then the network takes another suitable LCM action.

[0065] In the above LCM workflow, as soon as these LCM action commands are sent to the UE (at step 5), the UE needs to execute them in a timely manner. Any delay in the execution of these commands would further deteriorate the performance (KPIs). Especially in the case of functionalities that have stringent time bounds like CSI feedback, this becomes even more critical since crucial decisions of link adaptation and scheduling are based on CSI feedback.

[0066] Therefore, it becomes imperative to have some requirements and test methodologies to test and validate the behavior of LCM from the performance monitoring perspective to ensure minimum latency of LCM actions as a performance guarantee.

[0067] According to some example embodiments of the present disclosure, there is provided a solution for latency testing of AI / ML enable CSI feedback. In this solution, a first apparatus may transmit, to a second apparatus, an indication to perform an action for CSI feedback based on AI / ML. The second apparatus upon receiving the indication may try to perform the action and transmit a response concerning completion of the action. Then, the first apparatus may determine a test result for the action based on the response and a time length indicating a latency requirement for the action.

[0068] In this way, the latency of LCM actions can be guaranteed within the test framework for AI / ML enabled CSI feedback.

[0069] Example embodiments of the present disclosure will be described in detail below with reference to the accompanying drawings.

[0070] In the present disclosure, the time length indicating a latency requirement for an action (which is also referred to as LCM action) related to CSI feedback based on AI / ML is proposed. The LCM action may include any suitable action related to CSI feedback based on AI / ML. For example, the LCM action may include activation, deactivation, fallback, switching of AI / ML functionality, and / or activation, deactivation, selection,switching of individual AI / ML models. It is noted that the above actions may given as examples without any limitation. The LCM action may include any action currently defined or defined in future development.

[0071] The time length may be implemented as an LCM response timer value. In some example embodiments, each LCM action may have one or more corresponding time lengths, for example one or more LCM response timer values. In some example embodiments, the latency requirement may depend on a load condition or load level for AI / ML processing at the second apparatus, for example at DUT or UE. For the same LCM action, the timer values under different load levels may be different. To this end, a mapping between different time lengths and different load levels of the AI / ML processing may be employed.

[0072] An example core requirement for LCM action latency is provided in table 10 as an example of the mapping.Table 10

[0073] It should be noted that the timer values may be derived from the field tests and simulation results. In the above, load levels of “No load” and “Full Load” are shown as examples. Any other load levels may be employed.

[0074] In some example embodiments, the mapping such as Table 10 may be predefined, for example, defined in a specification. In some example embodiments, the mapping may be exchanged between the UE and network, for example as part of UE capability information.

[0075] The present disclosure also provides a mechanism to notify the LCM actions between the DUT and the TE. This mechanism can be generically applied to be available on field as part of LCM. In addition, this can be made specific to testing by introducingthis as a part of the test mode. Example processes for the mechanism are now described.

[0076] Reference is first made to FIG. 4. FIG. 4 illustrates an example signaling chart 400 for latency testing of CSI feedback based on AI / ML according to some example embodiments of the present disclosure. For the purpose of discussion, the chart 400 will be described with reference to FIG. 1.

[0077] As shown in FIG. 4, in some example embodiments, the first apparatus 110 may configure (405) the second apparatus 120 to test the AI / ML enabled functionality. The LCM procedures may be turned on for performance monitoring at the network side. Alternatively, the first apparatus 110 may notify or cause (410) the second apparatus 120 to enter into a test mode of testing the AI / ML enabled functionality.

[0078] During the LCM procedure or in the test mode, the first apparatus 110 determines (425) a first action to be perform for CSI feedback based on AI / ML. The first action may be any of the LCM actions described above.

[0079] In some example embodiments, the first action may be determined in response to that a performance degradation is detected during performance monitoring of the CSI feedback based on AI / ML. For example, the first apparatus 110 may monitor the performance of the CSI feedback based on AI / ML and if a performance degradation is detected, an LCM action may be determined accordingly. Alternatively, the first action may be determined in response to that a performance degradation is triggered under control of the first apparatus. For example, the first apparatus 110 may a controlled performance degradation. Alternatively, the first action may be determined in response to operating in the test mode. For example, in the test mode, the first apparatus 110 may autonomously determine to take an LCM action so as to test that LCM action.

[0080] The first apparatus 110 transmits (430) to the second apparatus 120, a first indication of the first action. That is, the first apparatus 110 indicate the second apparatus 120 to perform the first action for the CSI feedback based on AI / ML. In some example embodiments, the first indication may be transmitted via a medium access control (MAC) layer.

[0081] The second apparatus 120 receives the first indication of the first action from the first apparatus 110, and performs (440) the first action. The second apparatus 120 transmits (445) to the first apparatus 110, a first response about completion of the first action. In the case that the first action is completed, the first response may indicate successful completion of the first action. In the case that the first action is not completed, the first response may indicate a failure in completing the first action and optionally acause of the failure, for example lack of AI / ML processing resources.

[0082] The first apparatus 110 receives the first response and determines (450) a result of a test on the first action based on the first response and a first time length indicating a latency requirement for the first action. For example, the first time length is a timer value corresponding to the first action, as described above.

[0083] The first time length may be obtained in any suitable manner. In some example embodiments, the first apparatus 110 may receive (415) from the second apparatus 120, capability information about at least one action supported by the second apparatus for the CSI feedback based on AI / ML. For each action of the at least one action, the capability information may comprise at least one time length indicating a latency requirement for the action. For example, the mapping between different actions and timer values as shown in Table 10 may be reported to the first apparatus 110 as capability information of the second apparatus 120.

[0084] In some example embodiments, the first apparatus 110 may receive (420) from the second apparatus 120, load information indicating a load level of AI / ML processing at the second apparatus 120. The first apparatus 110 may determine the first time length based on the indicated load level and a mapping between different time lengths and different load levels of the AI / ML processing. For example, Table 10 may be predefined or reported to the first apparatus 110. Thus, based on the indicated load level, the first apparatus 110 may determine the timer value for the first action under the indicated load level.

[0085] To compare the reception time of the first response and the first time length, in some example embodiments, the first apparatus 110 may start (435) a first timer with the first timer length upon transmitting the first indication. If the first response indicates successful completion of the first action and is received before the first timer expires, the first apparatus 110 may determine that the test on the first action is passed.

[0086] In some example embodiments, if the first response indicates successful completion of the first action but is received after the first timer expires, the first apparatus 110 may determine that the test on the first action is failed.

[0087] In some example embodiments, based on the first response and performance monitoring, another decision may be made by the first apparatus 110. For example, if the first response indicates successful completion of the first action and no performance improvement is monitored after completion of the first action, the first apparatus 110 may determine (455) a second action which is to be performed for CSI feedback based onAI / ML. The second action is different from the first action.

[0088] The first action and second action may be any suitable LCM actions, including those having been defined or those to be defined in future. In some example embodiments, the first and second actions may be of the same action type but with different parameters. For example, the first action may be switching to a first model and the second action may be switching to a different second action.

[0089] In some example embodiments, the first and second actions may be of different action types. In an example, the first action may comprise a change to an AI / ML model used for the CSI feedback based on AI / ML, for example, model switch or model update. The second action may comprise a switch from the CSI feedback based on AI / ML to CSI feedback without AI / ML, for example fallback to legacy CSI feedback mode.

[0090] Then, the first apparatus 110 may transmit (460) to the second apparatus 120, a second indication of a second action. That is, the first apparatus 110 indicate the second apparatus 120 to perform the second action for the CSI feedback based on AI / ML. In some example embodiments, the second indication may be transmitted via the MAC layer.

[0091] The second apparatus 120 receives the second indication of the second action from the first apparatus 110, and performs (470) the second action. The second apparatus 120 transmits (475) to the first apparatus 110, a second response about completion of the second action. In the case that the second action is completed, the second response may indicate successful completion of the second action. In the case that the second action is not completed, the second response may indicate a failure in completing the second action and optionally a cause of the failure, for example lack of AI / ML processing resources.

[0092] The first apparatus 110 receives the second response and determines (480) a result of a test on the second action based on the second response and a second time length indicating a latency requirement for the second action. For example, the second time length is a timer value corresponding to the first action, as described above. The second time length may be obtained similarly as the first action and description thereof is not repeated here.

[0093] To compare the reception time of the second response and the second time length, in some example embodiments, the first apparatus 110 may start (465) a second timer with the second timer length upon transmitting the second indication. If the second response indicates successful completion of the second action and is received before the second timer expires, the first apparatus 110 may determine that the test on the second action is passed. If the second response indicates successful completion of the second action but isreceived after the second timer expires, the first apparatus 110 may determine that the test on the second action is failed.

[0094] FIG. 5 illustrates another example signaling chart 500 for latency testing of CSI feedback based on AI / ML according to some example embodiments of the present disclosure. In this example, the signaling chart 500 may be an example implementation of the signaling chart 400. The TE 501 is an example of the first apparatus 110 and the DUT 502 is an example of the second apparatus 120. The solution can be further illustrated with the signaling flow as described in FIG. 5.

[0095] As shown in FIG. 5, in some example embodiments, as an initial step 510, the DUT may be configured to test the AI / ML enabled CSI feedback functionality. The LCM procedures are turned on for network side performance monitoring. Alternatively, the TE 501 may cause (512) the DUT 502 to enter into the AI / ML test mode to test the AI / ML- enabled functionality.

[0096] The DUT 502 may send (514) the LCM response timer values and indications of LCM actions supported at the MAC layer to the TE 501 as part of capability information of the DUT 502. The TE 501 stores (516) the LCM response timer values to be used later.

[0097] The TE 501 starts (518) the CSI feedback testing. In some example embodiments, the TE 501 may start (520) a controlled performance degradation to trigger the detection of performance degradation and consequently an LCM action. Alternatively, in the due course of the testing, the TE 501 may detect (522) performance degradation during performance monitoring as part of the LCM procedure. Alternatively, in the case of operating in the test mode, performance degradation may not be detected to trigger an LCM action.

[0098] The LCM entity at the TE side decides (524) that there should be a model / functionality switch at the DUT 502 to improve the performance. The TE 501 sends (526) an indication of the model / functionality switch to the DUT 502.

[0099] The TE 501 starts (528) the LCM response timer and sends an indication of an LCM action to the DUT 502 to switch the model / functionality. It should be noted that this LCM action message may be implemented as part of MAC layer for fast processing of the request.

[0100] The DUT 502 switches (530) the model / functionality. The DUT 502 acknowledges (532) the TE 501 about the switching of the model / functionality by transmitting the response of the LCM action completion.

[0101] Next, based on the kind of the response and reception time of the response, theremay be various alternatives to act at the TE.

[0102] In the alternative 1, the TE 501 may determine (534) that the response of the LCM action completion is received before the LCM action timer expires. The TE may determine (536) that the test for switch to another model / functionality is considered passed.

[0103] In the alternative 2, the TE 501 may determine (538) that the response of the LCM action completion is received after the LCM action timer expires. The TE may determine (540) that the test for switch to another model / functionality is considered failed.

[0104] In the alternative 3, the TE 501 may determine (542) that the response of the LCM action completion is received before the LCM action timer expires, however no improvement in the performance is observed after the change. The LCM entity at TE side decides to switch the AI / ML enabled CSI feedback functionality to the legacy mode of the CSI feedback. TE 501 sends (544) an indication of the LCM action to switch to the legacy mode and may start (545) the LCM response timer.

[0105] The DUT 502 switches (546) the CSI feedback functionality to the legacy mode (without the usage of AI / ML). The DUT 502 informs (548) the TE 501 about the LCM action completion.

[0106] The TE 501 determines (550) that the LCM action response is received on time, for example before expiration of the LCM response timer. There is an observable change in the performance. The TE 501 determines (552) that the test is considered as passed.

[0107] In some example embodiments, a negotiation mechanism may be employed to negotiate the latency requirement based on the load level for AI / ML processing. Reference is first made to FIG. 6. FIG. 6 illustrates an example signaling chart for negotiating latency requirements according to some example embodiments of the present disclosure. For the purpose of discussion, the process 600 will be described with reference to FIG. 1. Acts with the same reference number as those of FIG. 4 are similar, and thus description thereof is not repeated.

[0108] As shown in FIG. 6, the first apparatus 110 determines (605) that it is unable to complete the first action within a required latency. In other words, a current load level of AI / ML processing is too heavy to complete the first action within the required latency. For example, the heavy AI / ML processing load cause a lack in processing resource to complete the first action.

[0109] Accordingly, the second apparatus 120 transmits (610) to the first apparatus 110, negotiation information for updating the first time length. In this case, the first responseindicates a failure in completing the first action.

[0110] In some example embodiments, the negotiation information indicates at least one of: an updated value of the first time length, or a current load level of AI / ML processing at the second apparatus 120.

[0111] Upon receiving the negotiation information, the first apparatus 110 updates (615) the first time length based on the negotiation information. The first time length may be updated to the updated value or a value corresponding to the current load level, for example by checking a table similar to Table 10.

[0112] Then, the first apparatus 110 transmits (620), to the second apparatus 120, a third indication to perform the first action, and may start (625) a timer with the updated time length.

[0113] The second apparatus 120 receives the third indication from the first apparatus 110, and perform (630) the first action again. Then, the second apparatus 120 may transmit (635) a third response about completion of the first action to the first apparatus 110. The first apparatus 110 receives (635) the third response from the second apparatus 120.

[0114] Then, the first apparatus 110 determines (640) the result of the test on the first action based on reception time of the third response and the updated first time length. Act 640 is similar to the act 450 and thus is not repeated here.

[0115] For the above example process, it can be seen that there can be a scenario where the AI / ML engine of the DUT is loaded, and it cannot respect the LCM action requested in the allocated time. Then DUT can negotiate for a new timer value based on its load. In this case a modified test in terms of the LCM action timer values is needed that takes UE load into account.

[0116] A further example in the timer value negotiation scenario is described in the FIG.7. FIG. 7 illustrates another example signaling chart for negotiating latency requirements according to some example embodiments of the present disclosure. In this example, the signaling chart 700 may be an example implementation of the signaling chart 600. The TE 501 is an example of the first apparatus 110 and the DUT 502 is an example of the second apparatus 120. Acts with the same reference number as those of FIG. 5 are similar, and thus description thereof is not repeated.

[0117] As shown in FIG. 7, the DUT 502 determines (705) that it is unable to switch the model / functionality due to non-availability of resources. The DUT 502 responds (710) to the TE 501 that resources are too busy to perform the switch.

[0118] The DUT 502 also responds (715) the TE 501 with a new timer value (greaterthan the previous value) that can be used for following test.

[0119] Correspondingly, the TE 501 sends (720) an indication of model / functionality switch to the DUT 501. Then the TE 501 starts (725) a timer with the new timer value.

[0120] The DUT 501 performs (730) the LCM action to switch the model / functionality. This time, the DUT 501 switches the model. The DUT 501 responds (735) the TE 501 with model switch confirmation.

[0121] The TE 501 observes (740) an improvement in the performance. The TE 501 determines (745) that the test is considered passed.

[0122] In the present disclosure, LCM action response timer values may be reported as part of UE capability message that will be exchanged with the network. The LCM action timer values may be derived based on field data / simulation data. It should be noted that this can be as a starting point. Fine tuning of the values can be done based on the load of the AI / ML computation engine in the DUT. Additionally, a mechanism to notify the LCM actions between the DUT and the TE is proposed. This mechanism can be generically applied to be available on field as part of LCM. In addition, this can be made specific to testing by introducing this as a part of the test mode.

[0123] In some example embodiments, the mechanism can be envisaged to be in the MAC layer keeping in mind the latency requirements of CSI feedback as a use case. LCM actions implemented in the MAC layer can be considered an important and distinguishing part for the following reasons. In case of CSI feedback, if LCM operations are not handled in a timely manner, performance will degrade rapidly. Therefore, in the CSI feedback use case, there is no choice but to implement the LCM operation between the DUT and the TE at a lower layer (e.g., MAC). Also, in the case of AI / ML based CSI compression, since it is a two-sided model, changing channel conditions at the DUT can force frequent updates to the models / functionality at both the DUT and the TE, triggering more frequent LCM actions and related signaling exchanges between the DUT and the TE., which also need to be handled in a timely manner.

[0124] To sum up, the solution to the above problem is provided herein, using which the latency of LCM actions can be guaranteed within the framework of AI / ML-enabled CSI feedback testing.

[0125] FIG. 8 shows a flowchart of an example method 800 implemented at a first device in accordance with some example embodiments of the present disclosure. For the purpose of discussion, the method 800 will be described from the perspective of the first apparatus 110 in FIG. 1.

[0126] At block 810, the first apparatus transmits, to a second apparatus, a first indication of a first action to be performed for channel state information, CSI, feedback based on artificial intelligence / machine learning, AI / ML.

[0127] At block 820, the first apparatus receives, from the second apparatus, a first response about completion of the first action.

[0128] At block 830, the first apparatus determines a result of a test on the first action based on the first response and a first time length indicating a latency requirement for the first action.

[0129] In some example embodiments, the method 800 further comprises: receiving, from the second apparatus, capability information about at least one action supported by the second apparatus for the CSI feedback based on AI / ML, and for an action of the at least one action, the capability information comprises at least one time length indicating a latency requirement for the action.

[0130] In some example embodiments, the method 800 further comprises: receiving, from the second apparatus, load information indicating a load level of AI / ML processing at the second apparatus; and determining the first time length based on the indicated load level and a mapping between different time lengths and different load levels of the AI / ML processing.

[0131] In some example embodiments, the method 800 comprises: starting a first timer with the first timer length upon transmitting the first indication; and in accordance with a determination that the first response indicates successful completion of the first action and is received before the first timer expires, determining that the test on the first action is passed.

[0132] In some example embodiments, the method 800 comprises: in accordance with a determination that the first response indicates successful completion of the first action and is received after the first timer expires, determining that the test on the first action is failed.

[0133] In some example embodiments, the method 800 comprises: in accordance with a determination that the first response indicates successful completion of the first action and no performance improvement is monitored after completion of the first action, transmitting, to the second apparatus, a second indication of a second action to be performed for the CSI feedback based on AI / ML, the second action being different from the first action; receiving, from the second apparatus, a second response about completion of the second action; and determining a result of a test on the second action based on thesecond response and a second time length indicating a latency requirement for the second action.

[0134] In some example embodiments, the first action comprises a change to an AI / ML model used for the CSI feedback based on AI / ML, and the second action comprises a switch from the CSI feedback based on AI / ML to CSI feedback without AI / ML.

[0135] In some example embodiments, the method 800 further comprises: receiving, from the second apparatus, negotiation information for updating the first time length; in accordance with a determination that the first response indicates a failure in completing the first action, updating the first time length based on the negotiation information; transmitting, to the second apparatus, a third indication to perform the first action; receiving, from the second apparatus, a third response about completion of the first action; and determining the result of the test on the first action based on reception time of the third response and the updated first time length.

[0136] In some example embodiments, the negotiation information indicates at least one of an updated value of the first time length, or a current load level of AI / ML processing at the second apparatus.

[0137] In some example embodiments, the first indication is transmitted via a medium access control layer.

[0138] In some example embodiments, the first action is determined in response to at least one of that a performance degradation is detected during performance monitoring of the CSI feedback based on AI / ML, that a performance degradation is triggered under control of the first apparatus, or operating in a test mode for testing the CSI feedback based on AI / ML.

[0139] In some example embodiments, the first apparatus comprises a test equipment, and the second apparatus comprises a device under test, or the first apparatus comprises a network device, and the second apparatus comprises a terminal device.

[0140] FIG. 9 shows a flowchart of an example method 900 implemented at a second device in accordance with some example embodiments of the present disclosure. For the purpose of discussion, the method 900 will be described from the perspective of the second apparatus 120 in FIG. 1.

[0141] At block 910, the second apparatus receives, from a first apparatus, a first indication of a first action to be performed for channel state information, CSI, feedback based on artificial intelligence / machine learning, AI / ML.

[0142] At block 920, the second apparatus transmits, to the first apparatus, a firstresponse about completion of the first action, the first response being used together with a first time length indicating a latency requirement for the first action to determine a result of a test on the first action.

[0143] In some example embodiments, the method 900 further comprises: transmitting, to the first apparatus, capability information about at least one action supported by the second apparatus for the CSI feedback based on AI / ML, and for an action of the at least one action, the capability information comprises at least one time length indicating a latency requirement for the action.

[0144] In some example embodiments, the method 900 further comprises: transmitting, to the first apparatus, load information indicating a load level of AI / ML processing at the second apparatus.

[0145] In some example embodiments, the method 900 further comprises: receiving, from the first apparatus, a second indication of a second action to be performed for the CSI feedback based on AI / ML, the second action being different from the first action; and transmitting, to the first apparatus, a second response about completion of the second action, the second response be used together with a second time length indicating a latency requirement for the second action to determine a result of a test on the second action.

[0146] In some example embodiments, the first action comprises a change to an AI / ML model used for the CSI feedback based on AI / ML, and the second action comprises a switch from the CSI feedback based on AI / ML to CSI feedback without AI / ML.

[0147] In some example embodiments, the method 900 further comprises: in accordance with a determination that a current load level of AI / ML processing is too heavy to complete the first action within a required latency, transmitting, to the first apparatus, negotiation information for updating the first time length and the first response indicating a failure in completing the first action; receiving, from the first apparatus, a third indication to perform the first action; and transmitting, to the first apparatus, a third response about completion of the first action, a reception time of the third response being used together with the updated first time length to determine the result of the test on the first action.

[0148] In some example embodiments, the negotiation information indicates at least one of: an updated value of the first time length, or a current load level of AI / ML processing at the second apparatus.

[0149] In some example embodiments, the first indication is received via a medium access control layer.

[0150] In some example embodiments, the first apparatus comprises a test equipment, and the second apparatus comprises a device under test, or the first apparatus comprises a network device, and the second apparatus comprises a terminal device.

[0151] In some example embodiments, a first apparatus capable of performing any of the method 800 (for example, the first apparatus 110 in FIG. 1) may comprise means for performing the respective operations of the method 800. The means may be implemented in any suitable form. For example, the means may be implemented in a circuitry or software module. The first apparatus may be implemented as or included in the first apparatus 110 in FIG. 1.

[0152] In some example embodiments, the first apparatus comprises means for transmitting, to a second apparatus, a first indication of a first action to be performed for channel state information, CSI, feedback based on artificial intelligence / machine learning, AI / ML; means for receiving, from the second apparatus, a first response about completion of the first action; and means for determining a result of a test on the first action based on the first response and a first time length indicating a latency requirement for the first action.

[0153] In some example embodiments, the first apparatus further comprises: means for receiving, from the second apparatus, capability information about at least one action supported by the second apparatus for the CSI feedback based on AI / ML, and for an action of the at least one action, the capability information comprises at least one time length indicating a latency requirement for the action.

[0154] In some example embodiments, the first apparatus further comprises: means for receiving, from the second apparatus, load information indicating a load level of AI / ML processing at the second apparatus; and means for determining the first time length based on the indicated load level and a mapping between different time lengths and different load levels of the AI / ML processing.

[0155] In some example embodiments, the first apparatus comprises: means for starting a first timer with the first timer length upon transmitting the first indication; and means for in accordance with a determination that the first response indicates successful completion of the first action and is received before the first timer expires, determining that the test on the first action is passed.

[0156] In some example embodiments, the first apparatus comprises: means for in accordance with a determination that the first response indicates successful completion of the first action and is received after the first timer expires, determining that the test on thefirst action is failed.

[0157] In some example embodiments, the first apparatus comprises: means for in accordance with a determination that the first response indicates successful completion of the first action and no performance improvement is monitored after completion of the first action, transmitting, to the second apparatus, a second indication of a second action to be performed for the CSI feedback based on AI / ML, the second action being different from the first action; means for receiving, from the second apparatus, a second response about completion of the second action; and means for determining a result of a test on the second action based on the second response and a second time length indicating a latency requirement for the second action.

[0158] In some example embodiments, the first action comprises a change to an AI / ML model used for the CSI feedback based on AI / ML, and the second action comprises a switch from the CSI feedback based on AI / ML to CSI feedback without AI / ML.

[0159] In some example embodiments, the first apparatus comprises: means for receiving, from the second apparatus, negotiation information for updating the first time length; means for in accordance with a determination that the first response indicates a failure in completing the first action, updating the first time length based on the negotiation information; means for transmitting, to the second apparatus, a third indication to perform the first action; means for receiving, from the second apparatus, a third response about completion of the first action; and means for determining the result of the test on the first action based on reception time of the third response and the updated first time length.

[0160] In some example embodiments, the negotiation information indicates at least one of: an updated value of the first time length, or a current load level of AI / ML processing at the second apparatus.

[0161] In some example embodiments, the first indication is transmitted via a medium access control layer.

[0162] In some example embodiments, the first action is determined in response to at least one of: that a performance degradation is detected during performance monitoring of the CSI feedback based on AI / ML, that a performance degradation is triggered under control of the first apparatus, or operating in a test mode for testing the CSI feedback based on AI / ML.

[0163] In some example embodiments, the first apparatus comprises a test equipment, and the second apparatus comprises a device under test, or the first apparatus comprises anetwork device, and the second apparatus comprises a terminal device.

[0164] In some example embodiments, the first apparatus further comprises means for performing other operations in some example embodiments of the method 800 or the first apparatus 110. In some example embodiments, the means comprises at least one processor; and at least one memory storing instructions that, when executed by the at least one processor, cause the performance of the first apparatus.

[0165] In some example embodiments, a second apparatus capable of performing any of the method 900 (for example, the second apparatus 120 in FIG. 1) may comprise means for performing the respective operations of the method 900. The means may be implemented in any suitable form. For example, the means may be implemented in a circuitry or software module. The second apparatus may be implemented as or included in the second apparatus 120 in FIG. 1.

[0166] In some example embodiments, the second apparatus comprises means for receiving, from a first apparatus, a first indication of a first action to be performed for channel state information, CSI, feedback based on artificial intelligence / machine learning, AI / ML; and means for transmitting, to the first apparatus, a first response about completion of the first action, the first response being used together with a first time length indicating a latency requirement for the first action to determine a result of a test on the first action.

[0167] In some example embodiments, the second apparatus further comprises: means for transmitting, to the first apparatus, capability information about at least one action supported by the second apparatus for the CSI feedback based on AI / ML, and for an action of the at least one action, the capability information comprises at least one time length indicating a latency requirement for the action.

[0168] In some example embodiments, the second apparatus further comprises: means for transmitting, to the first apparatus, load information indicating a load level of AI / ML processing at the second apparatus.

[0169] In some example embodiments, the second apparatus further comprises: means for receiving, from the first apparatus, a second indication of a second action to be performed for the CSI feedback based on AI / ML, the second action being different from the first action; and means for transmitting, to the first apparatus, a second response about completion of the second action, the second response be used together with a second time length indicating a latency requirement for the second action to determine a result of a test on the second action.

[0170] In some example embodiments, the first action comprises a change to an AI / ML model used for the CSI feedback based on AI / ML, and the second action comprises a switch from the CSI feedback based on AI / ML to CSI feedback without AI / ML.

[0171] In some example embodiments, the second apparatus further comprises: means for in accordance with a determination that a current load level of AI / ML processing is too heavy to complete the first action within a required latency, transmitting, to the first apparatus, negotiation information for updating the first time length and the first response indicating a failure in completing the first action; means for receiving, from the first apparatus, a third indication to perform the first action; and means for transmitting, to the first apparatus, a third response about completion of the first action, a reception time of the third response being used together with the updated first time length to determine the result of the test on the first action.

[0172] In some example embodiments, the negotiation information indicates at least one of an updated value of the first time length, or a current load level of AI / ML processing at the second apparatus.

[0173] In some example embodiments, the first indication is received via a medium access control layer.

[0174] In some example embodiments, the first apparatus comprises a test equipment, and the second apparatus comprises a device under test, or the first apparatus comprises a network device, and the second apparatus comprises a terminal device.

[0175] In some example embodiments, the second apparatus further comprises means for performing other operations in some example embodiments of the method 900 or the second apparatus 120. In some example embodiments, the means comprises at least one processor; and at least one memory storing instructions that, when executed by the at least one processor, cause the performance of the second apparatus.

[0176] FIG. 10 is a simplified block diagram of a device 1000 that is suitable for implementing example embodiments of the present disclosure. The device 1000 may be provided to implement a communication device, for example, the first apparatus 110 or the second apparatus 120 as shown in FIG. 1. As shown, the device 1000 includes one or more processors 1010, one or more memories 1020 coupled to the processor 1010, and one or more communication modules 1040 coupled to the processor 1010.

[0177] The communication module 1040 is for bidirectional communications. The communication module 1040 has one or more communication interfaces to facilitate communication with one or more other modules or devices. The communication interfacesmay represent any interface that is necessary for communication with other network elements. In some example embodiments, the communication module 1040 may include at least one antenna.

[0178] The processor 1010 may be of any type suitable to the local technical network and may include one or more of the following: general purpose computers, special purpose computers, microprocessors, digital signal processors (DSPs) and processors based on multicore processor architecture, as non-limiting examples. The device 1000 may have multiple processors, such as an application specific integrated circuit chip that is slaved in time to a clock which synchronizes the main processor.

[0179] The memory 1020 may include one or more non-volatile memories and one or more volatile memories. Examples of the non-volatile memories include, but are not limited to, a Read Only Memory (ROM) 1024, an electrically programmable read only memory (EPROM), a flash memory, a hard disk, a compact disc (CD), a digital video disk (DVD), an optical disk, a laser disk, and other magnetic storage and / or optical storage. Examples of the volatile memories include, but are not limited to, a random access memory (RAM) 1022 and other volatile memories that will not last in the power-down duration.

[0180] A computer program 1030 includes computer executable instructions that are executed by the associated processor 1010. The instructions of the program 1030 may include instructions for performing operations / acts of some example embodiments of the present disclosure. The program 1030 may be stored in the memory, e.g., the ROM 1024. The processor 1010 may perform any suitable actions and processing by loading the program 1030 into the RAM 1022.

[0181] The example embodiments of the present disclosure may be implemented by means of the program 1030 so that the device 1000 may perform any process of the disclosure as discussed with reference to FIG. 2 to FIG. 9. The example embodiments of the present disclosure may also be implemented by hardware or by a combination of software and hardware.

[0182] In some example embodiments, the program 1030 may be tangibly contained in a computer readable medium which may be included in the device 1000 (such as in the memory 1020) or other storage devices that are accessible by the device 1000. The device 1000 may load the program 1030 from the computer readable medium to the RAM 1022 for execution. In some example embodiments, the computer readable medium may include any types of non-transitory storage medium, such as ROM, EPROM, a flash memory, ahard disk, CD, DVD, and the like. The term “non-transitory,” as used herein, is a limitation of the medium itself (i.e., tangible, not a signal) as opposed to a limitation on data storage persistency (e.g., RAM vs. ROM).

[0183] Generally, various embodiments of the present disclosure may be implemented in hardware or special purpose circuits, software, logic or any combination thereof. Some aspects may be implemented in hardware, and other aspects may be implemented in firmware or software which may be executed by a controller, microprocessor or other computing device. Although various aspects of embodiments of the present disclosure are illustrated and described as block diagrams, flowcharts, or using some other pictorial representations, it is to be understood that the block, apparatus, system, technique or method described herein may be implemented in, as non-limiting examples, hardware, software, firmware, special purpose circuits or logic, general purpose hardware or controller or other computing devices, or some combination thereof.

[0184] Some example embodiments of the present disclosure also provide at least one computer program product tangibly stored on a computer readable medium, such as a non- transitory computer readable medium. The computer program product includes computerexecutable instructions, such as those included in program modules, being executed in a device on a target physical or virtual processor, to carry out any of the methods as described above. Generally, program modules include routines, programs, libraries, objects, classes, components, data structures, or the like that perform particular tasks or implement particular abstract data types. The functionality of the program modules may be combined or split between program modules as desired in various embodiments. Machine-executable instructions for program modules may be executed within a local or distributed device. In a distributed device, program modules may be located in both local and remote storage media.

[0185] Program code for carrying out methods of the present disclosure may be written in any combination of one or more programming languages. The program code may be provided to a processor or controller of a general purpose computer, special purpose computer, or other programmable data processing apparatus, such that the program code, when executed by the processor or controller, cause the functions / operations specified in the flowcharts and / or block diagrams to be implemented. The program code may execute entirely on a machine, partly on the machine, as a stand-alone software package, partly on the machine and partly on a remote machine or entirely on the remote machine or server.

[0186] In the context of the present disclosure, the computer program code or relateddata may be carried by any suitable carrier to enable the device, apparatus or processor to perform various processes and operations as described above. Examples of the carrier include a signal, computer readable medium, and the like.

[0187] The computer readable medium may be a computer readable signal medium or a computer readable storage medium. A computer readable medium may include but not limited to an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any suitable combination of the foregoing. More specific examples of the computer readable storage medium would include an electrical connection having one or more wires, a portable computer diskette, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or Flash memory), an optical fiber, a portable compact disc read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

[0188] Further, although operations are depicted in a particular order, this should not be understood as requiring that such operations be performed in the particular order shown or in sequential order, or that all illustrated operations be performed, to achieve desirable results. In certain circumstances, multitasking and parallel processing may be advantageous. Likewise, although several specific implementation details are contained in the above discussions, these should not be construed as limitations on the scope of the present disclosure, but rather as descriptions of features that may be specific to particular embodiments. Unless explicitly stated, certain features that are described in the context of separate embodiments may also be implemented in combination in a single embodiment. Conversely, unless explicitly stated, various features that are described in the context of a single embodiment may also be implemented in a plurality of embodiments separately or in any suitable sub-combination.

[0189] Although the present disclosure has been described in languages specific to structural features and / or methodological acts, it is to be understood that the present disclosure defined in the appended claims is not necessarily limited to the specific features or acts described above. Rather, the specific features and acts described above are disclosed as example forms of implementing the claims.

Claims

WHAT IS CLAIMED IS:

1. A first apparatus comprising: at least one processor; and at least one memory storing instructions that, when executed by the at least one processor, cause the first apparatus at least to: transmit, to a second apparatus, a first indication of a first action to be performed for channel state information, CSI, feedback based on artificial intelligence / machine learning, AI / ML; receive, from the second apparatus, a first response about completion of the first action; and determine a result of a test on the first action based on the first response and a first time length indicating a latency requirement for the first action.

2. The first apparatus of claim 1, wherein the first apparatus is further caused to: receive, from the second apparatus, capability information about at least one action supported by the second apparatus for the CSI feedback based on AI / ML, and for an action of the at least one action, the capability information comprises at least one time length indicating a latency requirement for the action.

3. The first apparatus of claim 1, wherein the first apparatus is further caused to: receive, from the second apparatus, load information indicating a load level of AI / ML processing at the second apparatus; and determine the first time length based on the indicated load level and a mapping between different time lengths and different load levels of the AI / ML processing.

4. The first apparatus of claim 1, wherein the first apparatus is caused to: start a first timer with the first timer length upon transmitting the first indication; and in accordance with a determination that the first response indicates successful completion of the first action and is received before the first timer expires, determine that the test on the first action is passed.

5. The first apparatus of claim 4, wherein the first apparatus is caused to: in accordance with a determination that the first response indicates successful completion of the first action and is received after the first timer expires, determine that the test on the first action is failed.

6. The first apparatus of claim 1, wherein the first apparatus is caused to: in accordance with a determination that the first response indicates successful completion of the first action and no performance improvement is monitored after completion of the first action, transmit, to the second apparatus, a second indication of a second action to be performed for the CSI feedback based on AI / ML, the second action being different from the first action; receive, from the second apparatus, a second response about completion of the second action; and determine a result of a test on the second action based on the second response and a second time length indicating a latency requirement for the second action.

7. The first apparatus of claim 6, wherein the first action comprises a change to an AI / ML model used for the CSI feedback based on AI / ML, and the second action comprises a switch from the CSI feedback based on AI / ML to CSI feedback without AI / ML.

8. The first apparatus of claim 1, wherein the first apparatus is further caused to: receive, from the second apparatus, negotiation information for updating the first time length; in accordance with a determination that the first response indicates a failure in completing the first action, update the first time length based on the negotiation information; transmit, to the second apparatus, a third indication to perform the first action; receive, from the second apparatus, a third response about completion of the first action; and determine the result of the test on the first action based on reception time of the third response and the updated first time length.

9. The first apparatus of claim 8, wherein the negotiation information indicates at least one of an updated value of the first time length, or a current load level of AI / ML processing at the second apparatus.

10. The first apparatus of claim 1, wherein the first indication is transmitted via a medium access control layer.

11. The first apparatus of claim 1, wherein the first action is determined in response to at least one of that a performance degradation is detected during performance monitoring of the CSI feedback based on AI / ML, that a performance degradation is triggered under control of the first apparatus, or operating in a test mode for testing the CSI feedback based on AI / ML.

12. The first apparatus of claim 1, wherein the first apparatus comprises a test equipment, and the second apparatus comprises a device under test, or the first apparatus comprises a network device, and the second apparatus comprises a terminal device.

13. A second apparatus comprising: at least one processor; and at least one memory storing instructions that, when executed by the at least one processor, cause the second apparatus at least to: receive, from a first apparatus, a first indication of a first action to be performed for channel state information, CSI, feedback based on artificial intelligence / machine learning, AI / ML; and transmit, to the first apparatus, a first response about completion of the first action, the first response being used together with a first time length indicating a latency requirement for the first action to determine a result of a test on the first action.

14. The second apparatus of claim 13, wherein the second apparatus is further caused to: transmit, to the first apparatus, capability information about at least one action supportedby the second apparatus for the CSI feedback based on AI / ML, and for an action of the at least one action, the capability information comprises at least one time length indicating a latency requirement for the action.

15. The second apparatus of claim 13, wherein the second apparatus is further caused to: transmit, to the first apparatus, load information indicating a load level of AI / ML processing at the second apparatus.

16. The second apparatus of claim 13, wherein the first response indicates successful completion of the first action and no performance improvement is monitored after completion of the first action and the second apparatus is caused to: receive, from the first apparatus, a second indication of a second action to be performed for the CSI feedback based on AI / ML, the second action being different from the first action; and transmit, to the first apparatus, a second response about completion of the second action, the second response be used together with a second time length indicating a latency requirement for the second action to determine a result of a test on the second action.

17. The second apparatus of claim 16, wherein the first action comprises a change to an AI / ML model used for the CSI feedback based on AI / ML, and the second action comprises a switch from the CSI feedback based on AI / ML to CSI feedback without AI / ML.

18. The second apparatus of claim 13, wherein the second apparatus is further caused to: in accordance with a determination that a current load level of AI / ML processing is too heavy to complete the first action within a required latency, transmit, to the first apparatus, negotiation information for updating the first time length and the first response indicating a failure in completing the first action; receive, from the first apparatus, a third indication to perform the first action; and transmit, to the first apparatus, a third response about completion of the first action, a reception time of the third response being used together with the updated first time length to determine the result of the test on the first action.

19. The second apparatus of claim 18, wherein the negotiation information indicates at least one of: an updated value of the first time length, or a current load level of AI / ML processing at the second apparatus.

20. The second apparatus of claim 13, wherein the first indication is received via a medium access control layer.

21. The second apparatus of claim 13, wherein the first apparatus comprises a test equipment, and the second apparatus comprises a device under test, or the first apparatus comprises a network device, and the second apparatus comprises a terminal device.

22. A method comprising: transmitting, to a second apparatus, a first indication of a first action to be performed for channel state information, CSI, feedback based on artificial intelligence / machine learning, AI / ML; receiving, from the second apparatus, a first response about completion of the first action; and determining a result of a test on the first action based on the first response and a first time length indicating a latency requirement for the first action.

23. A method comprising: receiving, at a second apparatus from a first apparatus, a first indication of a first action to be performed for channel state information, CSI, feedback based on artificial intelligence / machine learning, AI / ML; and transmitting, to the first apparatus, a first response about completion of the first action, the first response being used together with a first time length indicating a latency requirement for the first action to determine a result of a test on the first action.

24. A first apparatus comprising: means for transmitting, to a second apparatus, a first indication of a first action to be performed for channel state information, CSI, feedback based on artificial intelligence / machine learning, AI / ML; means for receiving, from the second apparatus, a first response about completion of the first action; and means for determining a result of a test on the first action based on the first response and a first time length indicating a latency requirement for the first action.

25. A second apparatus comprising: means for receiving, from a first apparatus, a first indication of a first action to be performed for channel state information, CSI, feedback based on artificial intelligence / machine learning, AI / ML; and means for transmitting, to the first apparatus, a first response about completion of the first action, the first response being used together with a first time length indicating a latency requirement for the first action to determine a result of a test on the first action.

26. A computer readable medium comprising instructions stored thereon for causing an apparatus at least to perform the method of claim 22 or the method of claim 23.

Images