Performance monitoring for ai / ML functionality or ai / ML model
Patent Information
- Application Number
- PCT/CN2024/070851
- Authority / Receiving Office
- WO · WO
- Patent Type
- Applications
- Current Assignee / Owner
- Filing Date
- 2024-01-05
- Publication Date
- 2024-11-28
Smart Images

Figure CN2024070851_28112024_PF_FP_ABST
Abstract
Description
PERFORMANCE MONITORING FOR AI / ML FUNCTIONALITY OR AI / ML MODELTECHNICAL FIELD
[0001] The present disclosure relates to wireless communications, and more specifically to a user equipment (UE) , a network device, processors for wireless communication, methods, and non-transitory computer readable media for performance monitoring for artificial intelligence / machine learning (AI / ML) functionality or AI / ML model.BACKGROUND
[0002] A wireless communications system may include one or multiple network communication devices, such as base stations, which may be otherwise known as an eNodeB (eNB) , a next-generation NodeB (gNB) , or other suitable terminology. Each network communication devices, such as a base station may support wireless communications for one or multiple user communication devices, which may be otherwise known as user equipment (UE) , or other suitable terminology. The wireless communications system may support wireless communications with one or multiple user communication devices by utilizing resources of the wireless communication system (e.g., time resources (e.g., symbols, slots, subframes, frames, or the like) or frequency resources (e.g., subcarriers, carriers) . Additionally, the wireless communications system may support wireless communications across various radio access technologies including third generation (3G) radio access technology, fourth generation (4G) radio access technology, fifth generation (5G) radio access technology, among other suitable radio access technologies beyond 5G (e.g., sixth generation (6G) ) .
[0003] A life cycle management (LCM) procedure manages AI / ML model in the entire life cycle. There are two types of LCM methods with different management units, namely, model-identification (ID) -based LCM and functionality-based LCM. Further study on functionality / model-ID based LCM is still needed.SUMMARY
[0004] The present disclosure relates to methods, apparatuses, and systems that support performance monitoring for AI / ML functionality or AI / ML model. By aligning a parameter configuration for monitoring performance of an AI / ML functionality or performance of an AI / ML model between the UE and the base station and transmitting a result of the monitoring of the performance of the AI / ML functionality or the performance of the AI / ML model based on the parameter configuration from the UE to the base station, a framework of performance monitoring for AI / ML functionality or AI / ML model may be designed.
[0005] In a first aspect of the solution, a UE receives, from a base station, a parameter configuration for monitoring performance of an artificial intelligence / machine learning (AI / ML) functionality or performance of an AI / ML model. The UE transmits, to the base station, a result of the monitoring of the performance of the AI / ML functionality or the performance of the AI / ML model based on the parameter configuration. In this way, a framework of performance monitoring for AI / ML functionality or AI / ML model may be designed.
[0006] In some implementations of the methods and apparatuses described herein, the parameter configuration may include at least one of the following: a condition for triggering the monitoring; a report quantity for specifying a content type of the result to be transmitted; a time requirement for reporting the result; an indication of a level of the monitoring in the case that the parameter configuration is associated with the AI / ML functionality; a performance metric target associated with the AI / ML model in the case that the parameter configuration is associated with the AI / ML model; at least one performance metric target associated with at least one AI / ML model to be assessed in the case that the parameter configuration is associated with the AI / ML functionality, wherein the at least one AI / ML model to be assessed is associated with the AI / ML functionality; a resource configuration for reference signal (RS) transmissions for the monitoring; a time duration for RS transmissions for the monitoring; the number of performance metric samples needed for the monitoring in the case that the parameter configuration is associated with the AI / ML model; at least one number of performance metric samples needed for the monitoring corresponding to at least one AI / ML model to be assessed in the case that the parameter configuration is associated with the AI / ML functionality, wherein the at least one AI / ML model to be assessed is associated with the AI / ML functionality; or information of an area in which the UE is located.
[0007] In some implementations of the methods and apparatuses described herein, the condition may include an occurrence of an event associated with potential degradation of the performance of the AI / ML functionality or the performance of the AI / ML model.
[0008] Some implementations of the method and apparatuses described herein may further include: transmitting, to the base station, an indication that the condition is fulfilled in the case that the event occurs for a time period reaching a threshold, wherein the threshold is predefined or configured.
[0009] In some implementations of the methods and apparatuses described herein, the condition may include a periodicity and an offset. Some implementations of the method and apparatuses described herein may further include: starting the monitoring at a time instance associated with the periodicity and the offset.
[0010] Some implementations of the method and apparatuses described herein may further include:
[0011] receiving, from the base station, an indication for triggering the monitoring.
[0012] In some implementations of the methods and apparatuses described herein, the content type is one of the following: an event type associated with the performance of the AI / ML functionality or the performance of the AI / ML model; a performance metric type; or none.
[0013] In some implementations of the methods and apparatuses described herein, transmitting the result of the monitoring may include: transmitting, to the base station, at least one performance metric of the AI / ML model.
[0014] In some implementations of the methods and apparatuses described herein, transmitting the result of the monitoring may include: transmitting, to the base station, an indication of an event that the AI / ML model is not applicable in the case of one of the following: a ratio of sample failures of the AI / ML model in the time duration for RS transmissions for the monitoring or among the number of performance metric samples needed for the monitoring reaches a maximum ratio of sample failures; the number of sample failures of the AI / ML model before end of the time duration for RS transmissions for the monitoring reaches a maximum number of sample failures; the number of sample failures of the AI / ML model among performance metric samples obtained for the monitoring reaches a maximum number of sample failures, wherein the number of the obtained performance metric samples is less than or equal to the number of performance metric samples needed for the monitoring; the number of consecutive sample failures of the AI / ML model before end of the time duration for RS transmissions for the monitoring reaches a maximum number of consecutive sample failures; the number of consecutive sample failures of the AI / ML model among performance metric samples obtained for the monitoring reaches a maximum number of consecutive sample failures, wherein the number of the obtained performance metric samples is less than or equal to the number of performance metric samples needed for the monitoring; a time duration with sample failures of the AI / ML model before end of the time duration for RS transmissions for monitoring reaches a maximum duration of model failure; or a time duration with sample failures of the AI / ML model among performance metric samples obtained for the monitoring reaches a maximum duration of model failure, wherein the number of obtained the performance metric samples is less than or equal to the number of performance metric samples needed for the monitoring.
[0015] In some implementations of the methods and apparatuses described herein, transmitting the result of the monitoring may include: transmitting, to the base station, an indication of an event that the AI / ML model is applicable in the case of one of the following: a ratio of sample failures of the AI / ML model in the time duration for RS transmissions for the monitoring or among the number of performance metric samples needed for the monitoring does not reach a maximum ratio of sample failures; the number of sample failures of the AI / ML model in the time duration for RS transmissions for the monitoring or among the number of performance metric samples needed for the monitoring does not reach a maximum number of sample failures; the number of consecutive sample failures of the AI / ML model in the time duration for RS transmissions for the monitoring or among the number of performance metric samples needed for the monitoring does not reach a maximum number of consecutive sample failures; or a time duration with sample failures of the AI / ML model in the time duration for RS transmissions for the monitoring or among the number of performance metric samples needed for the monitoring does not reach a maximum duration of model failure.
[0016] In some implementations of the methods and apparatuses described herein, the parameter configuration may include the performance metric target associated with the AI / ML model, and the sample failure of the AI / ML model is determined by that a performance metric of the AI / ML model for a performance metric sample is worse than the performance metric target.
[0017] In some implementations of the methods and apparatuses described herein, the performance of the AI / ML model is monitored. Transmitting the result of the monitoring may include: transmitting, to the base station, the result of the monitoring at one of the following time instances: a first time instance before end of the time duration for RS transmissions for the monitoring or when the number of performance metric samples obtained for the monitoring is less than the number of performance metric samples needed for the monitoring in the case of one of the following: the number of sample failures of the AI / ML model reaches a maximum number of sample failures at the first time instance, the number of consecutive sample failures of the AI / ML model reaches a maximum number of consecutive sample failures at the first time instance, or a time duration with sample failures of the AI / ML model reaches a maximum duration of model failure at the first time instance; a second time instance after the end of the time duration for RS transmissions for the monitoring in the case that the result of the monitoring has not been transmitted or in the case that a ratio of sample failures of the AI / ML model reaches a maximum ratio of sample failures; or a third time instance when the number of performance metric samples needed for the monitoring are obtained in the case that the result of the monitoring has not been transmitted or in the case that a ratio of sample failures of the AI / ML model reaches a maximum ratio of sample failures.
[0018] In some implementations of the methods and apparatuses described herein, the parameter configuration further may include one of the following: the maximum ratio of sample failures; the maximum number of sample failures; the maximum number of consecutive sample failures; or the maximum duration of model failure.
[0019] In some implementations of the methods and apparatuses described herein, the performance of the AI / ML functionality is based on at least one performance of at least one AI / ML model to be assessed among one or more AI / ML models associated with the AI / ML functionality.
[0020] In some implementations of the methods and apparatuses described herein, the indication of the level is indicative of one of the following: a first monitoring level in which one AI / ML model associated with the AI / ML functionality is to be assessed during the monitoring; a second monitoring level in which a default AI / ML model and an AI / ML model associated with the AI / ML functionality are to be assessed during the monitoring; or a third monitoring level in which a plurality of AI / ML models associated with the AI / ML functionality are to be assessed during the monitoring.
[0021] In some implementations of the methods and apparatuses described herein, the indication of the level is indicative of the third monitoring level. Some implementations of the method and apparatuses described herein may further include: receiving, from the base station, one of the following: identifications of the plurality of AI / ML models; the number of AI / ML models associated with the AI / ML functionality that are to be assessed; or a maximum number of AI / ML models associated with the AI / ML functionality that are to be assessed. The result of the monitoring is based on the performances of the plurality of AI / ML models.
[0022] In some implementations of the methods and apparatuses described herein, the parameter configuration may include the at least one performance metric target associated with the at least one AI / ML model to be assessed. Some implementations of the method and apparatuses described herein may further include: determining a sample failure of a respective AI / ML model among the at least one AI / ML model in the case that a performance metric for a performance metric sample of the respective AI / ML model is worse than a performance metric target associated with the respective AI / ML model.
[0023] Some implementations of the method and apparatuses described herein may further include: determining that the respective AI / ML model is not applicable in the case of one of the following: a ratio of sample failures of the respective AI / ML model in the time duration for RS transmissions for the monitoring or among a respective number of performance metric samples needed for the monitoring corresponding to the respective AI / ML model reaches a respective maximum ratio of sample failures; the number of sample failures of the respective AI / ML model before end of the time duration for RS transmissions for the monitoring reaches a respective maximum number of sample failures; the number of sample failures of the respective AI / ML model among performance metric samples of the respective AI / ML model obtained for the monitoring reaches a respective maximum number of sample failures, wherein the number of the obtained performance metric samples of the respective AI / ML model is less than or equal to the respective number of performance metric samples needed for the monitoring corresponding to the respective AI / ML model; the number of consecutive sample failures of the respective AI / ML model before end of the time duration for RS transmissions for the monitoring reaches a respective maximum number of consecutive sample failures; the number of consecutive sample failures of the respective AI / ML model among performance metric samples of the respective AI / ML model obtained for the monitoring reaches a respective maximum number of consecutive sample failures, wherein the number of the obtained performance metric samples of the respective AI / ML model is less than or equal to the respective number of performance metric samples needed for the monitoring corresponding to the respective AI / ML model; a time duration with sample failures of the respective AI / ML model before end of the time duration for RS transmissions for the monitoring reaches a respective maximum duration of model failure; or a time duration with sample failures of the respective AI / ML model among performance metric samples of the respective AI / ML model obtained for the monitoring reaches a respective maximum duration of model failure, wherein the number of the obtained performance metric samples of the respective AI / ML model is less than or equal to the respective number of performance metric samples needed for the monitoring corresponding to the respective AI / ML model.
[0024] Some implementations of the method and apparatuses described herein may further include: determining that the respective AI / ML model is applicable in the case of one of the following: a ratio of sample failures of the respective AI / ML model in the time duration for RS transmissions for the monitoring or among a respective number of performance metric samples needed for the monitoring does not reach a respective maximum ratio of sample failures; the number of sample failures of the respective AI / ML model in the time duration for RS transmissions for the monitoring or among a respective number of performance metric samples needed for the monitoring does not reach a respective maximum number of sample failures; the number of consecutive sample failures of the respective AI / ML model in the time duration for RS transmissions for the monitoring or among a respective number of performance metric samples needed for the monitoring does not reach a respective maximum number of consecutive sample failures; or a time duration with sample failures of the respective AI / ML model in the time duration for RS transmissions for the monitoring or among a respective number of performance metric samples needed for the monitoring does not reach a respective maximum duration of model failure.
[0025] In some implementations of the methods and apparatuses described herein, transmitting the result of the monitoring may include: transmitting, to the base station, the result of the monitoring at one of the following time instances: a first time instance before end of the time duration for RS transmissions for the monitoring or when the number of performance metric samples obtained for the monitoring for a respective AI / ML model among the at least one AI / ML model is less than a respective number of performance metric samples needed for the monitoring in the case of one of the following: for each of the at least one AI / ML model, the number of sample failures of the AI / ML model reaches a respective maximum number of sample failures at the first time instance; for each of the at least one AI / ML model, the number of consecutive sample failures of the AI / ML model reaches a respective maximum number of consecutive sample failures at the first time instance; for each of the at least one AI / ML model, a time duration with sample failures of the AI / ML model reaches a respective maximum duration of model failure at the first time instance; a second time instance after the end of the time duration for RS transmissions for the monitoring in the case that the result of the monitoring has not been transmitted or in the case that a ratio of sample failures of a respective AI / ML model among the at least one AI / ML model reaches a respective maximum ratio of sample failures; or a third time instance when a respective number of performance metric samples needed for the monitoring are obtained for each of the at least one AI / ML model in the case that the result of the monitoring has not been transmitted or in the case that a ratio of sample failures of a respective AI / ML model among the at least one AI / ML model reaches a respective maximum ratio of sample failures.
[0026] In some implementations of the methods and apparatuses described herein, the parameter configuration further may include one of the following: at least one maximum ratio of sample failures for the at least one AI / ML model; at least one maximum number of sample failures for the at least one AI / ML model; at least one maximum number of consecutive sample failures for the at least one AI / ML model; or at least one maximum duration of model failure for the at least one AI / ML model.
[0027] In some implementations of the methods and apparatuses described herein, the indication of the level is indicative of the first monitoring level, the content type is a performance metric type. Transmitting the result of the monitoring may include: transmitting, to the base station, at least one performance metric of the AI / ML model associated with the AI / ML functionality.
[0028] In some implementations of the methods and apparatuses described herein, the indication of the level is indicative of the second monitoring level, the content type is a performance metric type. Transmitting the result of the monitoring may include: transmitting, to the base station, at least one performance metric of the default AI / ML model and at least one performance metric of the AI / ML model associated with the AI / ML functionality.
[0029] In some implementations of the methods and apparatuses described herein, the indication of the level is indicative of the third monitoring level, the content type is a performance metric type. Transmitting the result of the monitoring may include: transmitting, to the base station, a plurality of performance metrics of the plurality of AI / ML models associated with the AI / ML functionality.
[0030] In some implementations of the methods and apparatuses described herein, the indication of the level is indicative of the first monitoring level, the content type is an event type associated with the performance of the AI / ML functionality. Transmitting the result of the monitoring may include: transmitting, to the base station, one of the following: an indication of an event that the AI / ML functionality is applicable in the case that the AI / ML model associated with the AI / ML functionality is applicable; or an indication of an event that the AI / ML functionality is not applicable in the case that the AI / ML model associated with the AI / ML functionality is not applicable.
[0031] In some implementations of the methods and apparatuses described herein, the indication of the level is indicative of the second monitoring level, the content type is an event type associated with the performance of the AI / ML functionality. Transmitting the result of the monitoring may include: transmitting, to the base station, one of the following: an indication of an event that the AI / ML functionality is applicable in the case that the default AI / ML model and the AI / ML model associated with the AI / ML functionality are applicable; an indication of an event that the AI / ML functionality is applicable with the default AI / ML model in the case that the default AI / ML model is applicable while the AI / ML model associated with the AI / ML functionality is not applicable; or an indication of an event that the AI / ML functionality is not applicable in the case that both the default AI / ML model and the AI / ML model associated with the AI / ML functionality are not applicable.
[0032] In some implementations of the methods and apparatuses described herein, the indication of the level is indicative of the third monitoring level, the content type is an event type associated with the performance of the AI / ML functionality. Transmitting the result of the monitoring may include: transmitting, to the base station, one of the following: an indication of an event that the AI / ML functionality is applicable in the case that at least one of the plurality of AI / ML models associated with the AI / ML functionality is applicable; an indication of an event that the AI / ML functionality is applicable but requires for a configuration for inference in the case that at least one of the plurality of AI / ML models associated with the AI / ML functionality is applicable but requires for a configuration for inference; or an indication of an event that the AI / ML functionality is not applicable in the case that none of the plurality of AI / ML models associated with the AI / ML functionality is applicable.
[0033] In a second aspect, a base station transmits, to a user equipment (UE) , a parameter configuration for monitoring performance of an artificial intelligence / machine learning (AI / ML) functionality or performance of an AI / ML model. The base station receives, from the UE, a result of the monitoring. In this way, a framework of performance monitoring for AI / ML functionality or AI / ML model may be designed.
[0034] In some implementations of the methods and apparatuses described herein, the parameter configuration may include at least one of the following: a condition for triggering the monitoring; a report quantity for specifying a content type of the result; a time requirement for reporting the result; an indication of a level of the monitoring in the case that the parameter configuration is associated with the AI / ML functionality; a performance metric target associated with the AI / ML model in the case that the parameter configuration is associated with the AI / ML model; at least one performance metric target associated with at least one AI / ML model to be assessed in the case that the parameter configuration is associated with the AI / ML functionality, wherein the at least one AI / ML model to be assessed is associated with the AI / ML functionality; a resource configuration for RS transmissions for the monitoring; a time duration for RS transmissions for the monitoring; the number of performance metric samples needed for the monitoring in the case that the parameter configuration is associated with the AI / ML model; at least one number of performance metric samples needed for the monitoring corresponding to at least one AI / ML model to be assessed in the case that the parameter configuration is associated with the AI / ML functionality, wherein the at least one AI / ML model to be assessed is associated with the AI / ML functionality; or information of an area in which the UE is located.
[0035] In some implementations of the methods and apparatuses described herein, the condition may include an occurrence of an event associated with potential degradation of the performance of the AI / ML functionality or the performance of the AI / ML model.
[0036] Some implementations of the method and apparatuses described herein may further include: receiving, from the UE, an indication that the condition is fulfilled.
[0037] In some implementations of the methods and apparatuses described herein, the condition may include a periodicity and an offset. Some implementations of the method and apparatuses described herein may further include: starting transmitting reference signals (RSs) for the monitoring at a time instance associated with the periodicity and the offset.
[0038] Some implementations of the method and apparatuses described herein may further include: transmitting, to the UE, an indication for triggering the monitoring.
[0039] In some implementations of the methods and apparatuses described herein, the content type is one of the following: an event type associated with the performance of the AI / ML functionality or the performance of the AI / ML model; a performance metric type; or none.
[0040] In some implementations of the methods and apparatuses described herein, the result of the monitoring may include: at least one performance metric of the AI / ML model.
[0041] In some implementations of the methods and apparatuses described herein, the parameter configuration may include a performance metric target associated with the AI / ML model, and the result of the monitoring may include one of the following: an indication of an event that the AI / ML model is not applicable; or an indication of an event that the AI / ML model is applicable.
[0042] In some implementations of the methods and apparatuses described herein, the performance of the AI / ML model is monitored, the parameter configuration may include the performance metric target associated with the AI / ML model, and the parameter configuration further may include one of the following: a maximum ratio of sample failures; a maximum number of sample failures; a maximum number of consecutive sample failures; or a maximum duration of model failure.
[0043] In some implementations of the methods and apparatuses described herein, the performance of the AI / ML functionality is monitored, the parameter configuration may include the at least one performance metric target associated with the at least one AI / ML model to be assessed, and the parameter configuration further may include one of the following: at least one maximum ratio of sample failures for the at least one AI / ML model to be assessed; at least one maximum number of sample failures for the at least one AI / ML model to be assessed; at least one maximum number of consecutive sample failures for the at least one AI / ML model to be assessed; or at least one maximum duration of model failure for the at least one AI / ML model to be assessed.
[0044] In some implementations of the methods and apparatuses described herein, the indication of the level is indicative of one of the following: a first monitoring level in which one AI / ML model associated with the AI / ML functionality is to be assessed during the monitoring; a second monitoring level in which a default AI / ML model and an AI / ML model associated with the AI / ML functionality are to be assessed during the monitoring; or a third monitoring level in which a plurality of AI / ML models associated with the AI / ML functionality are to be assessed during the monitoring.
[0045] In some implementations of the methods and apparatuses described herein, the indication of the level is indicative of the third monitoring level. Some implementations of the method and apparatuses described herein may further include: transmitting, to the UE, one of the following: identifications of the plurality of AI / ML models; the number of AI / ML models associated with the AI / ML functionality that are to be assessed; or a maximum number of AI / ML models associated with the AI / ML functionality that are to be assessed. The result of the monitoring is based on the performances of the plurality of AI / ML models.
[0046] In some implementations of the methods and apparatuses described herein, the indication of the level is indicative of the first monitoring level, the content type is a performance metric type, and the result of the monitoring may include: at least one performance metric of the AI / ML model associated with the AI / ML functionality.
[0047] In some implementations of the methods and apparatuses described herein, the indication of the level is indicative of the second monitoring level, the content type is a performance metric type, and the result of the monitoring may include: at least one performance metric of the default AI / ML model and at least one performance metric of the AI / ML model associated with the AI / ML functionality.
[0048] In some implementations of the methods and apparatuses described herein, the indication of the level is indicative of the third monitoring level, the content type is a performance metric type, and the result of the monitoring may include: a plurality of performance metrics of the plurality of AI / ML models associated with the AI / ML functionality.
[0049] In some implementations of the methods and apparatuses described herein, the indication of the level is indicative of the first monitoring level, the content type is an event type associated with the performance of the AI / ML functionality, and the result of the monitoring may include one of the following: an indication of an event that the AI / ML functionality is applicable; or an indication of an event that the AI / ML functionality is not applicable.
[0050] In some implementations of the methods and apparatuses described herein, the indication of the level is indicative of the second monitoring level, the content type is an event type associated with the performance of the AI / ML functionality, and the result of the monitoring may include one of the following: an indication of an event that the AI / ML functionality is applicable; an indication of an event that the AI / ML functionality is applicable with the default AI / ML model; or an indication of an event that the AI / ML functionality is not applicable.
[0051] In some implementations of the methods and apparatuses described herein, the indication of the level is indicative of the third monitoring level, the content type is an event type associated with the performance of the AI / ML functionality, and the result of the monitoring may include one of the following: an indication of an event that the AI / ML functionality is applicable; an indication of an event that the AI / ML functionality is applicable but requires for a configuration for inference; or an indication of an event that the AI / ML functionality is not applicable.BRIEF DESCRIPTION OF THE DRAWINGS
[0052] FIG. 1 illustrates an example of a wireless communications system that supports performance monitoring for AI / ML functionality or AI / ML model in accordance with aspects of the present disclosure.
[0053] FIG. 2 illustrates an example signaling chart of an example process that supports performance monitoring for AI / ML functionality or AI / ML model in accordance with aspects of the present disclosure.
[0054] FIG. 3 illustrates an example signaling chart of a general procedure of performance monitoring of an AI / ML functionality or an AI / ML model in accordance with aspects of the present disclosure.
[0055] FIG. 4 illustrates an example signaling chart of a specific procedure of performance monitoring of an AI / ML functionality in accordance with aspects of the present disclosure.
[0056] FIG. 5 illustrates an example of a device that supports performance monitoring for AI / ML functionality or AI / ML model in accordance with aspects of the present disclosure.
[0057] FIG. 6 illustrates an example of a processor that support performance monitoring for AI / ML functionality or AI / ML model in accordance with aspects of the present disclosure.
[0058] FIGS. 7 through 10 illustrate flowcharts of methods that support performance monitoring for AI / ML functionality or AI / ML model in accordance with aspects of the present disclosure.DETAILED DESCRIPTION
[0059] Principles of the present disclosure will now be described with reference to some embodiments. It is to be understood that these embodiments are described only for the purpose of illustration and help those skilled in the art to understand and implement the present disclosure, without suggesting any limitation as to the scope of the disclosure. The disclosure described herein may be implemented in various manners other than the ones described below.
[0060] In the following description and claims, unless defined otherwise, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skills in the art to which this disclosure belongs.
[0061] References in the present disclosure to “one embodiment, ” “an example embodiment, ” “an embodiment, ” “some embodiments, ” and the like indicate that the embodiment (s) described may include a particular feature, structure, or characteristic, but it is not necessary that every embodiment includes the particular feature, structure, or characteristic. Moreover, such phrases do not necessarily refer to the same embodiment (s) . Further, when a particular feature, structure, or characteristic is described in connection with an embodiment, it is submitted that it is within the knowledge of one skilled in the art to affect such feature, structure, or characteristic in connection with other embodiments whether or not explicitly described.
[0062] It shall be understood that although the terms “first” and “second” or the like may be used herein to describe various elements, these elements should not be limited by these terms. These terms are only used to distinguish one element from another element. For example, a first element could also be termed as a second element, and similarly, a second element could also be termed as a first element, without departing from the scope of embodiments. As used herein, the term “and / or” includes any and all combinations of one or more of the listed terms.
[0063] The terminology used herein is for the purpose of describing particular embodiments only and is not intended to be limiting of example embodiments. As used herein, the singular forms “a” , “an” and “the” are intended to include the plural forms as well, unless the context clearly indicates otherwise. It will be further understood that the terms “comprises” , “comprising” , “has” , “having” , “includes” and / or “including” , when used herein, specify the presence of stated features, elements, and / or components etc., but do not preclude the presence or addition of one or more other features, elements, components and / or combinations thereof. For example, the term “includes” and its variants are to be read as open terms that mean “includes, but is not limited to. ” The term “based on” is to be read as “based at least in part on. ” The term “one embodiment” and “an embodiment” are to be read as “at least one embodiment. ” The term “another embodiment” is to be read as “at least one other embodiment. ” The use of an expression such as “A and / or B” can mean either “only A” or “only B” or “both A and B. ” Other definitions, explicit and implicit, may be included below.
[0064] As used herein, the term “communication network” refers to a network following any suitable communication standards, such as, 5G NR, long term evolution (LTE) , LTE-advanced (LTE-A) , wideband code division multiple access (WCDMA) , high-speed packet access (HSPA) , narrow band internet of things (NB-IoT) , and so on. Further, the communications between a terminal device and a network device in the communication network may be performed according to any suitable generation communication protocols, including but not limited to, the first generation (1G) , the second generation (2G) , 2.5G, 2.75G, the third generation (3G) , the fourth generation (4G) , 4.5G, the fifth generation (5G) communication protocols, and / or any other protocols either currently known or to be developed in the future. Embodiments of the present disclosure may be applied in various communication systems. Given the rapid development in communications, there will also be future type communication technologies and systems in which the present disclosure may be embodied. It should not be seen as limiting the scope of the present disclosure to only the aforementioned systems.
[0065] As used herein, the term “network device” generally refers to a node in a communication network via which a terminal device can access the communication network and receive services therefrom. The network device may refer to a base station (BS) or an access point (AP) , for example, a node B (NodeB or NB) , a radio access network (RAN) node, an evolved NodeB (eNodeB or eNB) , an NR NB (also referred to as a gNB) , a remote radio unit (RRU) , a radio header (RH) , an infrastructure device for a V2X (vehicle-to-everything) communication, a transmission and reception point (TRP) , a reception point (RP) , a remote radio head (RRH) , a relay, an integrated access and backhaul (IAB) node, a low power node such as a femto BS, a pico BS, and so forth, depending on the applied terminology and technology.
[0066] As used herein, the term “terminal device” generally refers to any end device that may be capable of wireless communications. By way of example rather than a limitation, a terminal device may also be referred to as a communication device, a user equipment (UE) , an end user device, a subscriber station (SS) , an unmanned aerial vehicle (UAV) , a portable subscriber station, a mobile station (MS) , or an access terminal (AT) . The terminal device may include, but is not limited to, a mobile phone, a cellular phone, a smart phone, a voice over IP (VoIP) phone, a wireless local loop phone, a tablet, a wearable terminal device, a personal digital assistant (PDA) , a portable computer, a desktop computer, an image capture terminal device such as a digital camera, a gaming terminal device, a music storage and playback appliance, a vehicle-mounted wireless terminal device, a wireless endpoint, a mobile station, laptop-embedded equipment (LEE) , laptop-mounted equipment (LME) , a USB dongle, a smart device, wireless customer-premises equipment (CPE) , an internet of things (loT) device, a watch or other wearable, a head-mounted display (HMD) , a vehicle, a drone, a medical device (for example, a remote surgery device) , an industrial device (for example, a robot and / or other wireless devices operating in an industrial and / or an automated processing chain contexts) , a consumer electronics device, a device operating on commercial and / or industrial wireless networks, and the like. In the following description, the terms: “terminal device, ” “communication device, ” “terminal, ” “user equipment” and “UE, ” may be used interchangeably.
[0067] As mentioned above, model-ID-based LCM and functionality-based LCM are two types of management methods with different management units for UE side model. The LCM procedure is currently studied for the case that an AI / ML model has a model ID with associated information and / or for the case that a given functionality is provided by some AI / ML operations.
[0068] In functionality-based LCM, the network indicates activation / deactivation / fallback / switching of AI / ML functionality via 3GPP signalling (e.g., RRC, MAC-CE, DCI) . Models may not be identified at the network, and UE may perform model-level LCM. Whether and how much awareness / interaction the network should have about model-level LCM requires further study. For functionality identification, there may be either one or more than one Functionalities defined within an AI / ML-enabled feature, whereby AI / ML-enabled Feature refers to a Feature where AI / ML may be used. It should be noted that the UE may have one AI / ML model for the functionality, or the UE may have multiple AI / ML models for the functionality.
[0069] For AI / ML functionality identification and functionality-based LCM of UE-side models and / or UE-part of two-sided models, functionality refers to an AI / ML-enabled Feature / FG enabled by configuration (s) , where configuration (s) is (are) supported based on conditions indicated by UE capability. Correspondingly, functionality-based LCM operates at least based on one configuration of AI / ML-enabled Feature / FG or specific configurations of an AI / ML-enabled Feature / FG.
[0070] After functionality identification, necessity and mechanisms for the UE to report updates on applicable functionality (ies) among all functionality (ies) are studied, where the applicable functionalities may be a subset of all functionalities. Applicable functionalities can be reported by the UE.
[0071] In model-ID-based LCM, models are identified at the network, and the network or the UE may activate / deactivate / select / switch individual AI / ML models via model ID. For AI / ML model identification and model-ID-based LCM of UE-side models and / or UE-part of two-sided models, model-ID-based LCM operates based on identified models, where a model may be associated with specific configurations / conditions associated with UE capability of an AI / ML-enabled Feature / FG and additional conditions (e.g., scenarios, sites, and datasets) as determined / identified between UE-side and network -side.
[0072] After model identification, necessity and mechanisms for the UE to report updates on applicable UE part / UE-side model (s) are studied, where the applicable models may be a subset of all identified models. Applicable models can be reported by the UE.
[0073] For functionality / model-ID based LCM, once functionalities / models are identified, the same or similar procedures may be used for their activation, deactivation, switching, fallback, and monitoring. Model ID, if needed, can be used in a Functionality (defined in functionality-based LCM) for LCM operations.
[0074] For AI / ML model identification of UE-side or UE-part of two-sided models, model identification may be categorized in Type A and Type B including Type B1 and Type B2.
[0075] In Type A model identification, a model is identified to the network (if applicable) and the UE (if applicable) without over-the-air signalling. The model may be assigned with a model ID during the model identification, which may be referred / used in over-the-air signalling after model identification.
[0076] In Type B model identification, a model is identified via over-the-air signalling. Specifically, in Type B1 model identification, model identification is initiated by the UE, and the network assists the remaining steps (if any) of the model identification. The model may be assigned with a model ID during the model identification. In Type B2 model identification, model identification is initiated by the network, and the UE responds (if applicable) for the remaining steps (if any) of the model identification. The model may be assigned with a model ID during the model identification.
[0077] One example use case for Type B1 and Type B2 is model identification in model transfer from the network to the UE. Another example is model identification with data collection related configuration (s) and / or indication (s) and / or dataset transfer.
[0078] Once models are identified, at least for Type A, the UE can indicate supported AI / ML model IDs for a given AI / ML-enabled Feature / FG in a UE capability report as starting point. Model ID may or may not be globally unique, and different types of model IDs may be created for a single model for various LCM purposes.
[0079] A functionality may include multiple AI / ML models of which each associates with a configuration from the base station for generalization performance. For example, channel status information (CSI) predication functionality may contain multiple models adapting UE speed changing, in which different CSI-RS configurations (e.g., CSI-RS transmission occasions and time-interval of the occasions) may be needed for data collection for performance monitoring. For beam prediction functionality, various AI / ML models may be used to predict the best beam in case of different gNB beam patterns, respectively.
[0080] In one specific aspect, a framework for performance monitoring for AI / ML model should be studied. In another specific aspect, a framework for performance monitoring for AI / ML functionality should be studied with the consideration of multiple AI / ML models within an AI / ML functionality.
[0081] In view of the above and other aspects, embodiments of the present disclosure provide solutions for performance monitoring for AI / ML functionality or AI / ML model. Aspects of the present disclosure are described in the context of a wireless communications system.
[0082] FIG. 1 illustrates an example of a wireless communications system 100 that supports performance monitoring for AI / ML functionality or AI / ML model in accordance with aspects of the present disclosure. The wireless communications system 100 may include one or more network entities 102 (also referred to as network equipment (NE) ) , one or more UEs 104, a core network 106, and a packet data network 108. The wireless communications system 100 may support various radio access technologies. In some implementations, the wireless communications system 100 may be a 4G network, such as an LTE network or an LTE-Advanced (LTE-A) network. In some other implementations, the wireless communications system 100 may be a 5G network, such as an NR network. In other implementations, the wireless communications system 100 may be a combination of a 4G network and a 5G network, or other suitable radio access technology including Institute of Electrical and Electronics Engineers (IEEE) 802.11 (Wi-Fi) , IEEE 802.16 (WiMAX) , IEEE 802.20. The wireless communications system 100 may support radio access technologies beyond 5G. Additionally, the wireless communications system 100 may support technologies, such as time division multiple access (TDMA) , frequency division multiple access (FDMA) , or code division multiple access (CDMA) , etc.
[0083] The one or more network entities 102 may be dispersed throughout a geographic region to form the wireless communications system 100. One or more of the network entities 102 described herein may be or include or may be referred to as a network node, a base station, a network element, a radio access network (RAN) , a base transceiver station, an access point, a NodeB, an eNodeB (eNB) , a next-generation NodeB (gNB) , or other suitable terminology. A network entity 102 and a UE 104 may communicate via a communication link 110, which may be a wireless or wired connection. For example, a network entity 102 and a UE 104 may perform wireless communication (e.g., receive signaling, transmit signaling) over a Uu interface.
[0084] A network entity 102 may provide a geographic coverage area 112 for which the network entity 102 may support services (e.g., voice, video, packet data, messaging, broadcast, etc. ) for one or more UEs 104 within the geographic coverage area 112. For example, a network entity 102 and a UE 104 may support wireless communication of signals related to services (e.g., voice, video, packet data, messaging, broadcast, etc. ) according to one or multiple radio access technologies. In some implementations, a network entity 102 may be moveable, for example, a satellite associated with a non-terrestrial network. In some implementations, different geographic coverage areas 112 associated with the same or different radio access technologies may overlap, but the different geographic coverage areas 112 may be associated with different network entities 102. Information and signals described herein may be represented using any of a variety of different technologies and techniques. For example, data, instructions, commands, information, signals, bits, symbols, and chips that may be referenced throughout the description may be represented by voltages, currents, electromagnetic waves, magnetic fields or particles, optical fields or particles, or any combination thereof.
[0085] The one or more UEs 104 may be dispersed throughout a geographic region of the wireless communications system 100. A UE 104 may include or may be referred to as a mobile device, a wireless device, a remote device, a remote unit, a handheld device, or a subscriber device, or some other suitable terminology. In some implementations, the UE 104 may be referred to as a unit, a station, a terminal, or a client, among other examples. Additionally, or alternatively, the UE 104 may be referred to as an Internet-of-Things (IoT) device, an Internet-of-Everything (IoE) device, or machine-type communication (MTC) device, among other examples. In some implementations, a UE 104 may be stationary in the wireless communications system 100. In some other implementations, a UE 104 may be mobile in the wireless communications system 100.
[0086] The one or more UEs 104 may be devices in different forms or having different capabilities. Some examples of UEs 104 are illustrated in FIG. 1. A UE 104 may be capable of communicating with various types of devices, such as the network entities 102, other UEs 104, or network equipment (e.g., the core network 106, the packet data network 108, a relay device, an integrated access and backhaul (IAB) node, or another network equipment) , as shown in FIG. 1. Additionally, or alternatively, a UE 104 may support communication with other network entities 102 or UEs 104, which may act as relays in the wireless communications system 100.
[0087] A UE 104 may also be able to support wireless communication directly with other UEs 104 over a communication link 114. For example, a UE 104 may support wireless communication directly with another UE 104 over a device-to-device (D2D) communication link. In some implementations, such as vehicle-to-vehicle (V2V) deployments, vehicle-to-everything (V2X) deployments, or cellular-V2X deployments, the communication link 114 may be referred to as a sidelink. For example, a UE 104 may support wireless communication directly with another UE 104 over a PC5 interface.
[0088] A network entity 102 may support communications with the core network 106, or with another network entity 102, or both. For example, a network entity 102 may interface with the core network 106 through one or more backhaul links 116 (e.g., via an S1, N2, N2, or another network interface) . The network entities 102 may communicate with each other over the backhaul links 116 (e.g., via an X2, Xn, or another network interface) . In some implementations, the network entities 102 may communicate with each other directly (e.g., between the network entities 102) . In some other implementations, the network entities 102 may communicate with each other or indirectly (e.g., via the core network 106) . In some implementations, one or more network entities 102 may include subcomponents, such as an access network entity, which may be an example of an access node controller (ANC) . An ANC may communicate with the one or more UEs 104 through one or more other access network transmission entities, which may be referred to as a radio heads, smart radio heads, or transmission-reception points (TRPs) .
[0089] In some implementations, a network entity 102 may be configured in a disaggregated architecture, which may be configured to utilize a protocol stack physically or logically distributed among two or more network entities 102, such as an integrated access backhaul (IAB) network, an open RAN (O-RAN) (e.g., a network configuration sponsored by the O-RAN Alliance) , or a virtualized RAN (vRAN) (e.g., a cloud RAN (C-RAN) ) . For example, a network entity 102 may include one or more of a central unit (CU) , a distributed unit (DU) , a radio unit (RU) , a RAN Intelligent Controller (RIC) (e.g., a Near-Real Time RIC (Near-RT RIC) , a Non-Real Time RIC (Non-RT RIC) ) , a Service Management and Orchestration (SMO) system, or any combination thereof.
[0090] An RU may also be referred to as a radio head, a smart radio head, a remote radio head (RRH) , a remote radio unit (RRU) , or a transmission reception point (TRP) . One or more components of the network entities 102 in a disaggregated RAN architecture may be co-located, or one or more components of the network entities 102 may be located in distributed locations (e.g., separate physical locations) . In some implementations, one or more network entities 102 of a disaggregated RAN architecture may be implemented as virtual units (e.g., a virtual CU (VCU) , a virtual DU (VDU) , a virtual RU (VRU) ) .
[0091] Split of functionality between a CU, a DU, and an RU may be flexible and may support different functionalities depending upon which functions (e.g., network layer functions, protocol layer functions, baseband functions, radio frequency functions, and any combinations thereof) are performed at a CU, a DU, or an RU. For example, a functional split of a protocol stack may be employed between a CU and a DU such that the CU may support one or more layers of the protocol stack and the DU may support one or more different layers of the protocol stack. In some implementations, the CU may host upper protocol layer (e.g., a layer 3 (L3) , a layer 2 (L2) ) functionality and signaling (e.g., Radio Resource Control (RRC) , service data adaption protocol (SDAP) , Packet Data Convergence Protocol (PDCP) ) . The CU may be connected to one or more DUs or RUs, and the one or more DUs or RUs may host lower protocol layers, such as a layer 1 (L1) (e.g., physical (PHY) layer) or an L2 (e.g., radio link control (RLC) layer, medium access control (MAC) layer) functionality and signaling, and may each be at least partially controlled by the CU 160.
[0092] Additionally, or alternatively, a functional split of the protocol stack may be employed between a DU and an RU such that the DU may support one or more layers of the protocol stack and the RU may support one or more different layers of the protocol stack. The DU may support one or multiple different cells (e.g., via one or more RUs) . In some implementations, a functional split between a CU and a DU, or between a DU and an RU may be within a protocol layer (e.g., some functions for a protocol layer may be performed by one of a CU, a DU, or an RU, while other functions of the protocol layer are performed by a different one of the CU, the DU, or the RU) .
[0093] A CU may be functionally split further into CU control plane (CU-CP) and CU user plane (CU-UP) functions. A CU may be connected to one or more DUs via a midhaul communication link (e.g., F1, F1-c, F1-u) , and a DU may be connected to one or more RUs via a fronthaul communication link (e.g., open fronthaul (FH) interface) . In some implementations, a midhaul communication link or a fronthaul communication link may be implemented in accordance with an interface (e.g., a channel) between layers of a protocol stack supported by respective network entities 102 that are in communication via such communication links.
[0094] The core network 106 may support user authentication, access authorization, tracking, connectivity, and other access, routing, or mobility functions. The core network 106 may be an evolved packet core (EPC) , or a 5G core (5GC) , which may include a control plane entity that manages access and mobility (e.g., a mobility management entity (MME) , an access and mobility management functions (AMF) ) and a user plane entity that routes packets or interconnects to external networks (e.g., a serving gateway (S-GW) , a Packet Data Network (PDN) gateway (P-GW) , or a user plane function (UPF) ) . In some implementations, the control plane entity may manage non-access stratum (NAS) functions, such as mobility, authentication, and bearer management (e.g., data bearers, signal bearers, etc. ) for the one or more UEs 104 served by the one or more network entities 102 associated with the core network 106.
[0095] In some embodiments, the core network 106 may further include a location server, e.g., a location management function (LMF) . The LMF may receive measurements and assistance information from the network entity 102 and the UE 104 via the AMF to compute the position of the UE 104. A NR positioning protocol A (NRPPa) protocol was introduced to carry the positioning information between RAN and LMF over the next generation control plane interface (NG-C) . The LMF and the network entity 102 may communicate using the NRPPa defined in 3GPP TS 38.455, where NRPPa messages are communicated between the network entity 102 and the LMF via an AMF. The LMF and the UE 104 may communicate using the LTE Positioning Protocol (LPP) defined in 3GPP TS 36.355, where LPP messages are communicated between the UE 104 and the LMF via a serving AMF and a serving network entity for the UE. For example, LPP messages may be communicated between the LMF and the AMF using hypertext transfer protocol (HTTP) -based service operations, and LPP messages may be communicated between the AMF and the UE using a 5G non-access stratum (NAS) protocol. The LPP protocol may be used to support positioning of the UE using UE-assisted and / or UE-based positioning methods, such as assisted GNSS (a-GNSS) , Real Time Kinematics (RTK) , Wireless Local Area Network (WLAN) , observed time difference of arrival (OTDOA) , and / or Enhanced Cell Identity (ECID) . The NRPPa protocol may be used to support positioning of UE using network-based positioning methods, such as ECID (when used with measurements obtained by the network entity 102) , and / or the NRPPa protocol may be used by the LMF to obtain location-related information from the network entity 102, such as parameters defining Positioning Reference Signal (PRS) transmissions from the network entity 102 and the location of the network entity 102, to support OTDOA and ECID.
[0096] The core network 106 may communicate with the packet data network 108 over one or more backhaul links 116 (e.g., via an S1, N2, N2, or another network interface) . The packet data network 108 may include an application server 118. In some implementations, one or more UEs 104 may communicate with the application server 118. A UE 104 may establish a session (e.g., a protocol data unit (PDU) session, or the like) with the core network 106 via a network entity 102. The core network 106 may route traffic (e.g., control information, data, and the like) between the UE 104 and the application server 118 using the established session (e.g., the established PDU session) . The PDU session may be an example of a logical connection between the UE 104 and the core network 106 (e.g., one or more network functions of the core network 106) .
[0097] In the wireless communications system 100, the network entities 102 and the UEs 104 may use resources of the wireless communications system 100 (e.g., time resources (e.g., symbols, slots, subframes, frames, or the like) or frequency resources (e.g., subcarriers, carriers) ) to perform various operations (e.g., wireless communications) . In some implementations, the network entities 102 and the UEs 104 may support different resource structures. For example, the network entities 102 and the UEs 104 may support different frame structures. In some implementations, such as in 4G, the network entities 102 and the UEs 104 may support a single frame structure. In some other implementations, such as in 5G and among other suitable radio access technologies, the network entities 102 and the UEs 104 may support various frame structures (i.e., multiple frame structures) . The network entities 102 and the UEs 104 may support various frame structures based on one or more numerologies.
[0098] One or more numerologies may be supported in the wireless communications system 100, and a numerology may include a subcarrier spacing and a cyclic prefix. A first numerology (e.g., μ=0) may be associated with a first subcarrier spacing (e.g., 15 kHz) and a normal cyclic prefix. In some implementations, the first numerology (e.g., μ=0) associated with the first subcarrier spacing (e.g., 15 kHz) may utilize one slot per subframe. A second numerology (e.g., μ=1) may be associated with a second subcarrier spacing (e.g., 30 kHz) and a normal cyclic prefix. A third numerology (e.g., μ=2) may be associated with a third subcarrier spacing (e.g., 60 kHz) and a normal cyclic prefix or an extended cyclic prefix. A fourth numerology (e.g., μ=3) may be associated with a fourth subcarrier spacing (e.g., 120 kHz) and a normal cyclic prefix. A fifth numerology (e.g., μ=4) may be associated with a fifth subcarrier spacing (e.g., 240 kHz) and a normal cyclic prefix.
[0099] A time interval of a resource (e.g., a communication resource) may be organized according to frames (also referred to as radio frames) . Each frame may have a duration, for example, a 10 millisecond (ms) duration. In some implementations, each frame may include multiple subframes. For example, each frame may include 10 subframes, and each subframe may have a duration, for example, a 1 ms duration. In some implementations, each frame may have the same duration. In some implementations, each subframe of a frame may have the same duration.
[0100] Additionally or alternatively, a time interval of a resource (e.g., a communication resource) may be organized according to slots. For example, a subframe may include a number (e.g., quantity) of slots. The number of slots in each subframe may also depend on the one or more numerologies supported in the wireless communications system 100. For instance, the first, second, third, fourth, and fifth numerologies (i.e., μ=0, μ=1, μ=2, μ=3, μ=4) associated with respective subcarrier spacings of 15 kHz, 30 kHz, 60 kHz, 120 kHz, and 240 kHz may utilize a single slot per subframe, two slots per subframe, four slots per subframe, eight slots per subframe, and 16 slots per subframe, respectively. Each slot may include a number (e.g., quantity) of symbols (e.g., OFDM symbols) . In some implementations, the number (e.g., quantity) of slots for a subframe may depend on a numerology. For a normal cyclic prefix, a slot may include 14 symbols. For an extended cyclic prefix (e.g., applicable for 60 kHz subcarrier spacing) , a slot may include 12 symbols. The relationship between the number of symbols per slot, the number of slots per subframe, and the number of slots per frame for a normal cyclic prefix and an extended cyclic prefix may depend on a numerology. It should be understood that reference to a first numerology (e.g., μ=0) associated with a first subcarrier spacing (e.g., 15 kHz) may be used interchangeably between subframes and slots.
[0101] In the wireless communications system 100, an electromagnetic (EM) spectrum may be split, based on frequency or wavelength, into various classes, frequency bands, frequency channels, etc. By way of example, the wireless communications system 100 may support one or multiple operating frequency bands, such as frequency range designations FR1 (410 MHz –7.125 GHz) , FR2 (24.25 GHz –52.6 GHz) , FR3 (7.125 GHz –24.25 GHz) , FR4 (52.6 GHz –114.25 GHz) , FR4a or FR4-1 (52.6 GHz –71 GHz) , and FR5 (114.25 GHz –300 GHz) . In some implementations, the network entities 102 and the UEs 104 may perform wireless communications over one or more of the operating frequency bands. In some implementations, FR1 may be used by the network entities 102 and the UEs 104, among other equipment or devices for cellular communications traffic (e.g., control information, data) . In some implementations, FR2 may be used by the network entities 102 and the UEs 104, among other equipment or devices for short-range, high data rate capabilities.
[0102] FR1 may be associated with one or multiple numerologies (e.g., at least three numerologies) . For example, FR1 may be associated with a first numerology (e.g., μ=0) , which includes 15 kHz subcarrier spacing; a second numerology (e.g., μ=1) , which includes 30 kHz subcarrier spacing; and a third numerology (e.g., μ=2) , which includes 60 kHz subcarrier spacing. FR2 may be associated with one or multiple numerologies (e.g., at least 2 numerologies) . For example, FR2 may be associated with a third numerology (e.g., μ=2) , which includes 60 kHz subcarrier spacing; and a fourth numerology (e.g., μ=3) , which includes 120 kHz subcarrier spacing.
[0103] FIG. 2 illustrates an example signaling chart of an example process 200 that supports performance monitoring for AI / ML functionality or AI / ML model in accordance with aspects of the present disclosure. For the purpose of discussion, the process 200 will be described with reference to FIG. 1, and the process 200 may involve a UE 104 and a network entity 102 as shown in FIG. 1. The network entity 102 may be implemented as a base station. It is to be understood that the steps and the order of the steps in FIG. 2 are merely for illustration, and not for limitation. It is to be understood that process 200 may further include additional blocks not shown and / or omit some shown blocks, and the scope of the present disclosure is not limited in this regard.
[0104] As shown in FIG. 2, the base station 102 transmits 201, to the UE 104, a parameter configuration 202 for monitoring performance of an AI / ML functionality or performance of an AI / ML model. Accordingly, the UE 104 receives 203 the parameter configuration 202 from the base station 102. The UE 104 may monitor the performance of the AI / ML functionality or the performance of the AI / ML model based on the parameter configuration 202. The UE 104 then transmits 204 a result 205 of the monitoring to the base station 102. Accordingly, the base station 102 receives 206 the result 205 of the monitoring from the UE 104. In this way, a framework of performance monitoring for AI / ML functionality or AI / ML model may be designed.
[0105] The parameter configuration 202 may include various information for performing the performance monitoring. Based on the parameter configuration 202, information associated with performance monitoring of AI / ML functionalities / models may be aligned between the base station 102 and the UE 104.
[0106] In some embodiments, the parameter configuration 202 may include a condition for triggering the monitoring. For example, the monitoring may be triggered or started or resumed based on the condition. In an example implementation, the condition may include an occurrence of an event associated with potential degradation of the performance of the AI / ML functionality or the performance of the AI / ML model. As an example, the event may be that a strength (e.g., signal to interference plus noise ratio (SINR) , reference signal receiving power (RSRP) , or reference signal received quality (RSRQ) ) of reference signals (RSs) received by the UE 104 from the base station 102 is lower than a strength threshold. As another example, the event may be that the UE is moving to the cell edge or moving to a new cell. The UE 104 is aware of the event occurrence. If the event occurs for a time period reaching a duration threshold, the UE 104 may transmit an indication that the condition is fulfilled to the base station 102. The duration threshold may be configured by the base station 102 or predefined. The performance monitoring may thus be triggered or started or resumed by the UE 104. In another example implementation, the condition may include a periodicity and an offset. The base station 102 may start transmitting RSs for the monitoring at a time instance associated with the periodicity and the offset. Accordingly, the UE 104 may start monitoring the performance of the AI / ML functionality / model at a time instance associated with the periodicity and the offset. The performance monitoring may thus be started periodically based on the periodicity and the offset.
[0107] Alternatively, the base station 102 may transmit an indication for triggering the monitoring to the UE 104. In other words, the monitoring may be triggered or started or resumed by the base station 102 via a downlink (DL) signaling, e.g., a DCI signaling, a MAC CE signaling or a RRC signaling.
[0108] In some embodiments, the parameter configuration 202 may include information of an area in which the UE 104 is located. In some examples, the parameter configuration 202 may include information of a cell, a site, a TRP or a geographical area in which the UE 104 is located. For example, the RS sequence, the RS ID, the cell ID or the area ID may be considered.
[0109] In some embodiments, the parameter configuration 202 may include a report quantity for specifying a content type of the result 205 to be transmitted. In an example implementation, the content type may be an event type associated with the performance of the AI / ML functionality or the performance of the AI / ML model. For example, if the parameter configuration 202 is associated with the AI / ML model, the performance of the AI / ML model is monitored, and the content of the result 205 may include an indication of an event that the AI / ML model is applicable or an indication of an event that the AI / ML model is not applicable. In another example implementation, the content type may be a performance metric type. In other words, the content of the result 205 may include the performance metric (s) of the AI / ML functionality / model. In a further example implementation, the content type may be none. In other words, nothing would be reported from the UE 104 to the base station 102 for the functionality monitoring. In this case, the UE 104 may make decisions based on the performance monitoring outcome.
[0110] In some embodiments, the parameter configuration 202 may include a resource configuration for RS transmissions for the monitoring. The UE 104 may generate a dataset for the AI / ML functionality / model by monitoring RSs based on the resource configuration and calculate the result 205 of the performance monitoring based on the dataset.
[0111] In some embodiments, the parameter configuration 202 may include a time duration for RS transmissions for the monitoring. For example, the dataset for the AI / ML functionality / model may be generated by monitoring RSs during the time duration.
[0112] Alternatively, the parameter configuration 202 may include the number (s) of samples needed for performance monitoring. When calculating the result 205 of the performance monitoring, more than one performance metric samples may be obtained so as to improve the reliability of the performance monitoring. The number (s) of the performance metric samples for the performance monitoring may be indicated by the base station 102 to the UE 104. For example, if the parameter configuration 202 is associated with the AI / ML model, the parameter configuration 202 may include the number of performance metric samples needed for the monitoring. If the parameter configuration 202 is associated with the AI / ML functionality, at least one AI / ML model associated with the AI / ML functionality may be assessed during the performance monitoring, and the parameter configuration 202 may include at least one number of performance metric samples needed for the monitoring corresponding to the at least one AI / ML model to be assessed.
[0113] In some embodiments, the result 205 may be transmitted based on the parameter configuration 202. In other words, the parameter configuration 202 may include report configurations for the result of the monitoring. In some implementations, the parameter configuration 202 may include a time requirement for reporting the result 205. For example, the time requirement of report may indicate that the time requirement of report of the performance monitoring is tight or relax. In a specific implementation, a value of 1 means a tight time requirement and a value of 0 means a relax time requirement. The UE 102.
[0114] In some embodiments, the parameter configuration 202 may include performance metric target (s) for performance metric samples obtained during the performance monitoring. In an example implementation, if the parameter configuration 202 is associated with the AI / ML model, the parameter configuration 202 may include a performance metric target associated with the AI / ML model. A sample failure of the AI / ML model may be determined by that a performance metric of the AI / ML model for a performance metric sample is worse than the performance metric target. In another example implementation, if the parameter configuration 202 is associated with the AI / ML functionality, the parameter configuration 202 may include at least one performance metric target associated with at least one AI / ML model to be assessed. The at least one AI / ML model to be assessed is associated with the AI / ML functionality. A sample failure of a respective AI / ML model among the at least one AI / ML model may be determined by that a performance metric for a performance metric sample of the respective AI / ML model is worse than a performance metric target associated with the respective AI / ML model.
[0115] In some embodiments, if the parameter configuration 202 may be associated with the AI / ML functionality, the parameter configuration 202 may include an indication of a level of the monitoring. The performance of the AI / ML functionality may be monitored by monitoring at least one performance of at least one AI / ML model to be assessed among one or more AI / ML models associated with the AI / ML functionality.
[0116] For example, the indication of the level of the monitoring may be indicative of one among a first monitoring level, a second monitoring level and a third monitoring level. In the first monitoring level, one AI / ML model associated with the AI / ML functionality is to be assessed during the monitoring. In other words, when the first monitoring level is indicated for the performance monitoring of the AI / ML functionality, the performance of one model associated with the AI / ML functionality would be assessed and the monitoring result of the performance of the model would be regarded as the performance monitoring result of the AI / ML functionality. The model may be determined by the UE 104. In the second monitoring level, a default AI / ML model and an AI / ML model associated with the AI / ML functionality are to be assessed during the monitoring. In other words, when the second monitoring level is indicated for the performance monitoring of the AI / ML functionality, the performance of a default model and a specific model of the AI / ML functionality would be assessed, where the default model has general performance in conditions relevant to the AI / ML functionality and the specific model may be selected by UE within models associated with the AI / ML functionality. The default model should be identified between the base station 102 and the UE 104 through type A, type B1 or type B2 model identification. It should be understood that these monitoring levels are merely examples for illustrations. Other monitoring levels are also possible.
[0117] In the third monitoring level, a plurality of AI / ML models associated with the AI / ML functionality are to be assessed during the monitoring. In a more specific example, if the indication of the level is indicative of the third monitoring level, the UE 104 may further receive, from the base station 102, one of the following: identifications of the plurality of AI / ML models, the number of AI / ML models associated with the AI / ML functionality that are to be assessed, or a maximum number of AI / ML models associated with the AI / ML functionality that are to be assessed. The result 205 of the monitoring is based on the performances of the plurality of AI / ML models. In other words, when the third monitoring level is indicated for the performance monitoring of the AI / ML functionality, the performances of candidate models belonging to the AI / ML functionality would be assessed, where the number or maximum number of the candidate models or the combinations of candidate models should be aligned by the base station 102 and the UE 104.
[0118] In some embodiments, the UE 104 may report information associated with the performance monitoring of the AI / ML functionality / model through UE capability or UE assistance information (UAI) . The base station 102 may transmit the parameter configuration 202 based on the reported information.
[0119] After the performance monitoring for the AI / ML functionality / model is triggered, the base station 102 may transmit RSs to enable the UE 104 to generate a dataset for the performance monitoring. The dataset may include input data for an AI / ML model (e.g., the AI / ML model associated with the parameter configuration 202 is associated with the AI / ML model, or an AI / ML model belonging to the AI / ML functionality associated with the parameter configuration 202) and labeled data. The input data may be e.g., beam measurement results, CSI measurement results, or UE measurement results for positioning (reference signal time difference (RSTD) , RSRP, received signal code power (RSCP) , etc. ) . The labeled data may be e.g., identifiers and / or L1-RSRP of top N beams, CSI measurement result or UE position. The UE 104 may calculate the result 205 of the performance monitoring based on the dataset.
[0120] Hereinafter, example embodiments are illustrated in detail for the case where the parameter configuration 202 is associated with the AI / ML model and for the case where the parameter configuration 202 is associated with the AI / ML functionality, respectively.
[0121] For the case where the parameter configuration 202 is associated with the AI / ML model, the UE 104 may monitor the performance of the AI / ML model. For example, the UE 104 may measure RSs from the base station 102 and generate a dataset based on the measured RSs. The dataset may include input data for the AI / ML model and corresponding labeled data. The UE 104 may calculate at least one inferencing result based on the input data and the AI / ML model, and determine at least one performance metric of the AI / ML model for at least one performance metric sample by a comparison of the at least one inferencing result and the corresponding labeled data. The result 205 of the monitoring may be based on the at least one performance metric.
[0122] In some embodiments, the UE 104 may determine a sample failure of the AI / ML model by that a performance metric of the AI / ML model for a performance metric sample is worse than a performance metric target associated with the AI / ML model. The performance metric target may be configured to the UE 104 through e.g., the parameter configuration 202. In one example implementation, a counter may be maintained at the UE 104 to count the number of times that the performance metric of a sample is worse than the performance metric target.
[0123] In some embodiments, the parameter configuration 202 may include a time duration for RS transmissions for the monitoring. Before the end of the configured time duration for RS transmissions for the monitoring, the UE 104 may transmit the result 205 of the monitoring to the base station 102 if the number of sample failures of the AI / ML model has reached a maximum number of sample failures. Alternatively, before the end of the configured time duration for RS transmissions for the monitoring, the UE 104 may transmit the result 205 of the monitoring to the base station 102 if the number of consecutive sample failures of the AI / ML model has reached a maximum number of consecutive sample failures. Alternatively, before the end of the configured time duration for RS transmissions for the monitoring, the UE 104 may transmit the result 205 of the monitoring to the base station 102 if a time duration with sample failures of the AI / ML model has reached a maximum duration of model failure.
[0124] In some embodiments, the parameter configuration 202 may include the number of performance metric samples needed for the monitoring. Even if the number of the obtained performance metric samples is less than the number of performance metric samples needed for the monitoring, the UE 104 may transmit the result 205 of the monitoring to the base station 102 based on the obtained performance metric samples if the number of sample failures of the AI / ML model has reached a maximum number of sample failures. Alternatively, even if the number of the obtained performance metric samples is less than the number of performance metric samples needed for the monitoring, the UE 104 may transmit the result 205 of the monitoring to the base station 102 based on the obtained performance metric samples if the number of consecutive sample failures of the AI / ML model has reached a maximum number of consecutive sample failures. Alternatively, even if the number of the obtained performance metric samples is less than the number of performance metric samples needed for the monitoring, the UE 104 may transmit the result 205 of the monitoring to the base station 102 based on the obtained performance metric samples if a time duration with sample failures of the AI / ML model has reached a maximum duration of model failure.
[0125] In other words, the monitoring result may be reported once the UE 104 determines that the AI / ML model is not applicable and the RS transmissions and performance monitoring may be skipped after the reporting of the monitoring result. In this way, the efficiency for performance monitoring may be improved and the resource overhead may be reduced.
[0126] In some embodiments, if the result 205 of the monitoring has not been transmitted when the configured time duration for RS transmissions for the monitoring elapses, the UE 104 may transmit the result 205 of the monitoring to the base station 102. Alternatively, if the result 205 of the monitoring has not been transmitted when the number of performance metric samples needed for the monitoring are obtained, the UE 104 may transmit the result 205 of the monitoring to the base station 102.
[0127] In some embodiments, when the configured time duration for RS transmissions for the monitoring elapses, the UE 104 may transmit the result 205 of the monitoring to the base station 102 if a ratio of sample failures of the AI / ML model reaches a maximum ratio of sample failures. Alternatively, when the number of performance metric samples needed for the monitoring are obtained, the UE 104 may transmit the result 205 of the monitoring to the base station 102 if a ratio of sample failures of the AI / ML model reaches a maximum ratio of sample failures.
[0128] In some embodiments, the parameter configuration 202 may include one of the following: the maximum ratio of sample failures, the maximum number of sample failures, the maximum number of consecutive sample failures, or the maximum duration of model failure.
[0129] In one example implementation, when transmitting the result 205 of the monitoring, the UE 104 may transmit the at least one performance metric of the AI / ML model to the base station 102 if the content type of the result of the monitoring is a performance metric type.
[0130] In another example implementation, when transmitting the result 205 of the monitoring, the UE 104 may transmit an indication of an event that the AI / ML model is applicable or an indication of an event that the AI / ML model is not applicable if the content type of the result of the monitoring is an event type.
[0131] In one example, the UE 104 may transmit an indication of an event that the AI / ML model is not applicable if a ratio of sample failures of the AI / ML model reaches a maximum ratio of sample failures. The UE 104 may transmit an indication of an event that the AI / ML model is applicable if a ratio of sample failures of the AI / ML model does not reach a maximum ratio of sample failures. The ratio of sample failures of the AI / ML model may be based on the performance metrics obtained by the measurement of the RS transmitted in the time duration configured in the parameter configuration 202. Alternatively, the ratio of sample failures of the AI / ML model may be based on the obtained performance metrics whose number is the same as the number of performance metric samples needed for the monitoring configured in the parameter configuration 202. In some embodiments, the maximum ratio of sample failures may be configured to the UE 104 through e.g., the parameter configuration 202.
[0132] In one example, the UE 104 may transmit an indication of an event that the AI / ML model is not applicable if the number of sample failures of the AI / ML model before end of the time duration for RS transmissions for the monitoring reaches a maximum number of sample failures. The UE 104 may transmit an indication of an event that the AI / ML model is applicable if the number of sample failures of the AI / ML model in the time duration for RS transmissions for the monitoring does not reach the maximum number of sample failures. In some embodiments, before the configured time duration for RS transmission ends, the UE 104 may report that the AI / ML model is not applicable once the number of sample failures reaches the maximum number of sample failures. If the number of sample failures does not reach the maximum number of sample failures when the configured time duration for RS transmission has elapsed, the UE 104 may report that the AI / ML model is applicable. In some embodiments, the maximum number of sample failures may be configured to the UE 104 through e.g., the parameter configuration 202.
[0133] In one example, the UE 104 may transmit an indication of an event that the AI / ML model is not applicable if the number of sample failures of the AI / ML model among performance metric samples obtained for the monitoring reaches a maximum number of sample failures, wherein the number of the obtained performance metric samples is less than or equal to the number of performance metric samples needed for the monitoring. The UE 104 may transmit an indication of an event that the AI / ML model is applicable if the number of sample failures of the AI / ML model among the number of performance metric samples needed for the monitoring does not reach the maximum number of sample failures. In some embodiments, before the UE 104 obtains the number of performance metric samples needed for the monitoring, the UE 104 may report that the AI / ML model is not applicable once the number of sample failures reaches the maximum number of sample failures. If the number of sample failures does not reach the maximum number of sample failures when the UE 104 obtains the number of performance metric samples needed for the monitoring, the UE 104 may report that the AI / ML model is applicable.
[0134] In one example, the UE 104 may transmit an indication of an event that the AI / ML model is not applicable if the number of consecutive sample failures of the AI / ML model before end of the time duration for RS transmissions for the monitoring reaches a maximum number of consecutive sample failures. The UE 104 may transmit an indication of an event that the AI / ML model is applicable if the number of consecutive sample failures of the AI / ML model in the time duration for RS transmissions for the monitoring does not reach the maximum number of consecutive sample failures. In some embodiments, before the configured time duration for RS transmission ends, the UE 104 may report that the AI / ML model is not applicable once the number of consecutive sample failures reaches the maximum number of consecutive sample failures. If the number of consecutive sample failures does not reach the maximum number of consecutive sample failures when the configured time duration for RS transmission has elapsed, the UE 104 may report that the AI / ML model is applicable. In some embodiments, the maximum number of consecutive sample failures may be configured to the UE 104 through e.g., the parameter configuration 202.
[0135] In one example, the UE 104 may transmit an indication of an event that the AI / ML model is not applicable if the number of consecutive sample failures of the AI / ML model among performance metric samples obtained for the monitoring reaches a maximum number of consecutive sample failures, wherein the number of the obtained performance metric samples is less than or equal to the number of performance metric samples needed for the monitoring. The UE 104 may transmit an indication of an event that the AI / ML model is applicable if the number of consecutive sample failures of the AI / ML model among the number of performance metric samples needed for the monitoring does not reach the maximum number of consecutive sample failures. In some embodiments, before the UE 104 obtains the number of performance metric samples needed for the monitoring, the UE 104 may report that the AI / ML model is not applicable once the number of consecutive sample failures reaches the maximum number of consecutive sample failures. If the number of consecutive sample failures does not reach the maximum number of consecutive sample failures when the UE 104 obtains the number of performance metric samples needed for the monitoring, the UE 104 may report that the AI / ML model is applicable.
[0136] In one example, the UE 104 may transmit an indication of an event that the AI / ML model is not applicable if the time duration with sample failures of the AI / ML model before end of the time duration for RS transmissions for the monitoring reaches a maximum duration of model failure. The UE 104 may transmit an indication of an event that the AI / ML model is applicable if the time duration with sample failures of the AI / ML model in the time duration for RS transmissions for the monitoring does not reach the maximum duration of model failure. In some embodiments, before the configured time duration for RS transmission ends, the UE 104 may report that the AI / ML model is not applicable once the time duration with sample failures reaches the maximum duration of model failure. If the time duration with sample failures does not reach the maximum duration of model failure when the configured time duration for RS transmission has elapsed, the UE 104 may report that the AI / ML model is applicable. In some embodiments, the maximum duration of model failure may be configured to the UE 104 through e.g., the parameter configuration 202.
[0137] In one example, the UE 104 may transmit an indication of an event that the AI / ML model is not applicable if the time duration with sample failures of the AI / ML model among performance metric samples obtained for the monitoring reaches a maximum duration of model failure, wherein the number of the obtained performance metric samples is less than or equal to the number of performance metric samples needed for the monitoring. The UE 104 may transmit an indication of an event that the AI / ML model is applicable if the time duration with sample failures of the AI / ML model among the number of performance metric samples needed for the monitoring does not reach the maximum duration of model failure. In some embodiments, before the UE 104 obtains the number of performance metric samples needed for the monitoring, the UE 104 may report that the AI / ML model is not applicable once the time duration with sample failures reaches the maximum duration of model failure. If the time duration with sample failures does not reach the maximum duration of model failure when the UE 104 obtains the number of performance metric samples needed for the monitoring, the UE 104 may report that the AI / ML model is applicable.
[0138] For the case where the parameter configuration 202 is associated with the AI / ML functionality, in order to obtain the performance of the AI / ML functionality, the UE 104 may monitor at least one performance of at least one AI / ML model among one or more AI / ML models associated with the AI / ML functionality. When monitoring the performance of each of the at least one AI / ML model, the UE 104 may measure RSs from the base station 102 and generate a dataset based on the measured RSs. The dataset may include input data for the AI / ML model and corresponding labeled data. The UE 104 may calculate at least one inferencing result based on the input data and the AI / ML model, and determine at least one performance metric of the AI / ML model for at least one performance metric sample by a comparison of the at least one inferencing result and the corresponding labeled data. The performance of the AI / ML model may be based on the at least one performance metric of the AI / ML model. The result 205 of the performance monitoring of the AI / ML functionality may be based on at least one performance of the at least one AI / ML model.
[0139] For example, the parameter configuration 202 may include an indication of a level of the monitoring. In a more specific implementation, the indication of the level of the monitoring may be indicative of one among a first monitoring level, a second monitoring level and a third monitoring level. In the first monitoring level, one AI / ML model associated with the AI / ML functionality is to be assessed during the monitoring. The model may be determined by the UE 104. In the second monitoring level, a default AI / ML model and an AI / ML model associated with the AI / ML functionality are to be assessed during the monitoring. In the third monitoring level, a plurality of AI / ML models associated with the AI / ML functionality are to be assessed during the monitoring.
[0140] In some embodiments, if the content type of the result of the monitoring is a performance metric type, when transmitting the result 205 of the performance monitoring of the AI / ML functionality, the UE 104 may transmit the performance of the assessed AI / ML model (s) to the base station 102. For example, if the level of the monitoring is the first monitoring level, the result 205 of the monitoring may include at least one performance metric of one AI / ML model associated with the AI / ML functionality. If the level of the monitoring is the second monitoring level, the result 205 of the monitoring may include at least one performance metric of the default AI / ML model and at least one performance metric of the AI / ML model associated with the AI / ML functionality. If the level of the monitoring is the third monitoring level, the result 205 of the monitoring may include a plurality of performance metrics of the plurality of AI / ML models associated with the AI / ML functionality.
[0141] In some embodiments, if the content type of the result of the monitoring is an event type, the UE 104 may determine the event to be indicated in the result 205 based on determining whether each of the at least one AI / ML model is applicable. The events for each monitoring level may be predefined and aligned between the base station 102 and the UE 104.
[0142] For example, the first monitoring level may be associated with two events, namely, an event that the AI / ML functionality is applicable, and an event that the AI / ML functionality is not applicable. The result 205 of the monitoring may include an indication of an event that the AI / ML functionality is applicable if the AI / ML model associated with the AI / ML functionality is applicable. The result 205 of the monitoring may include an indication of an event that the AI / ML functionality is not applicable if the AI / ML model associated with the AI / ML functionality is not applicable.
[0143] The second monitoring level may be associated with three events, namely, an event that the AI / ML functionality is applicable, an event that the AI / ML functionality is applicable with the default AI / ML model, and an event that the AI / ML functionality is not applicable. The result 205 of the monitoring may include an indication of an event that the AI / ML functionality is applicable if the default AI / ML model and the AI / ML model associated with the AI / ML functionality are applicable. The result 205 of the monitoring may include an indication of an event that the AI / ML functionality is applicable with the default AI / ML model if the default AI / ML model is applicable while the AI / ML model associated with the AI / ML functionality is not applicable. The result 205 of the monitoring may include an indication of an event that the AI / ML functionality is not applicable if both the default AI / ML model and the AI / ML model associated with the AI / ML functionality are not applicable.
[0144] The third monitoring level may be associated with three events, namely, an event that the AI / ML functionality is applicable, an event that the AI / ML functionality is applicable but requires for a configuration for inference, and an event that the AI / ML functionality is not applicable. The result 205 of the monitoring may include an indication of an event that the AI / ML functionality is applicable if at least one of the plurality of AI / ML models associated with the AI / ML functionality is applicable. The result 205 of the monitoring may include an indication of an event that the AI / ML functionality is applicable but requires for a configuration for inference if at least one of the plurality of AI / ML models associated with the AI / ML functionality is applicable but requires for a configuration for inference. The result 205 of the monitoring may include an indication of an event that the AI / ML functionality is not applicable if none of the plurality of AI / ML models associated with the AI / ML functionality is applicable.
[0145] It should be understood that these monitoring levels and associated events for reporting are merely examples for illustrations. Other monitoring levels and events for reporting are also possible.
[0146] In some embodiments, the UE 104 may determine whether each of the assessed AI / ML model (s) is applicable. In one example, the UE 104 may determine that an AI / ML model is not applicable if a ratio of sample failures of the AI / ML model reaches a respective maximum ratio of sample failures. The UE 104 may determine that the AI / ML model is applicable if a ratio of sample failures of the AI / ML model does not reach a respective maximum ratio of sample failures. The ratio of sample failures of the AI / ML model may be based on the performance metrics obtained by the measurement of the RS transmitted in the time duration configured in the parameter configuration 202. Alternatively, the ratio of sample failures of the AI / ML model may be based on the obtained performance metrics whose number is the same as the respective number of performance metric samples needed for the monitoring corresponding to the AI / ML model. In some embodiments, the respective maximum ratio of sample failures of the AI / ML model may be configured to the UE 104 through e.g., the parameter configuration 202.
[0147] In one example, the UE 104 may determine that the AI / ML model is not applicable if the number of sample failures of the AI / ML model before end of the time duration for RS transmissions for the monitoring reaches a respective maximum number of sample failures. The UE 104 may determine that the AI / ML model is applicable if the number of sample failures of the AI / ML model in the time duration for RS transmissions for the monitoring does not reach the respective maximum number of sample failures.
[0148] In one example, the UE 104 may determine that the AI / ML model is not applicable if the number of sample failures of the AI / ML model among performance metric samples of the AI / ML model obtained for the monitoring reaches a respective maximum number of sample failures, wherein the number of the obtained performance metric samples of the AI / ML model is less than or equal to the respective number of performance metric samples needed for the monitoring corresponding to the AI / ML model. The UE 104 may determine that the AI / ML model is applicable if the number of sample failures of the AI / ML model among the number of performance metric samples needed for the monitoring does not reach the respective maximum number of sample failures.
[0149] In one example, the UE 104 may determine that the AI / ML model is not applicable if the number of consecutive sample failures of the AI / ML model before end of the time duration for RS transmissions for the monitoring reaches a respective maximum number of consecutive sample failures. The UE 104 may determine that the AI / ML model is applicable if the number of consecutive sample failures of the AI / ML model in the time duration for RS transmissions for the monitoring does not reach the respective maximum number of consecutive sample failures.
[0150] In one example, the UE 104 may determine that the AI / ML model is not applicable if the number of consecutive sample failures of the AI / ML model among performance metric samples of the AI / ML model obtained for the monitoring reaches a respective maximum number of consecutive sample failures, wherein the number of the obtained performance metric samples is less than or equal to the respective number of performance metric samples needed for the monitoring corresponding to the AI / ML model. The UE 104 may determine that the AI / ML model is applicable if the number of consecutive sample failures of the AI / ML model among the number of performance metric samples needed for the monitoring does not reach the respective maximum number of consecutive sample failures.
[0151] In one example, the UE 104 may determine that the AI / ML model is not applicable if the time duration with sample failures of the AI / ML model before end of the time duration for RS transmissions for the monitoring reaches a respective maximum duration of model failure. The UE 104 may determine that the AI / ML model is applicable if the time duration with sample failures of the AI / ML model in the time duration for RS transmissions for the monitoring does not reach the respective maximum duration of model failure.
[0152] In one example, the UE 104 may determine that the AI / ML model is not applicable if the time duration with sample failures of the AI / ML model among performance metric samples of the AI / ML model obtained for the monitoring reaches a respective maximum duration of model failure, wherein the number of the obtained performance metric samples is less than or equal to the respective number of performance metric samples needed for the monitoring corresponding to the AI / ML model. The UE 104 may determine that the AI / ML model is applicable if the time duration with sample failures of the AI / ML model among the number of performance metric samples needed for the monitoring does not reach the respective maximum duration of model failure.
[0153] In some embodiments, the parameter configuration 202 may include a time duration for RS transmissions for the monitoring. Before the end of the configured time duration for RS transmissions for the monitoring, the UE 104 may transmit the result 205 of the monitoring to the base station 102 if for each of the assessed AI / ML model (s) , the number of sample failures of the AI / ML model has reached a respective maximum number of sample failures for the AI / ML model. Alternatively, before the end of the configured time duration for RS transmissions for the monitoring, the UE 104 may transmit the result 205 of the monitoring to the base station 102 if for each of the assessed AI / ML model (s) , the number of consecutive sample failures of the AI / ML model has reached a respective maximum number of consecutive sample failures for the AI / ML model. Alternatively, before the end of the configured time duration for RS transmissions for the monitoring, the UE 104 may transmit the result 205 of the monitoring to the base station 102 if for each of the assessed AI / ML model (s) , a time duration with sample failures of the AI / ML model has reached a respective maximum duration of model failure for the AI / ML model.
[0154] In some embodiments, the parameter configuration 202 may include at least one number of performance metric samples needed for the monitoring corresponding to at least one AI / ML model to be assessed. Even if the number of the obtained performance metric samples for a respective AI / ML model among the assessed AI / ML model (s) is less than the respective number of performance metric samples needed for the monitoring, the UE 104 may transmit the result 205 of the monitoring to the base station 102 based on the obtained performance metric samples if for each of the assessed AI / ML model (s) , the number of sample failures of the AI / ML model has reached a respective maximum number of sample failures for the AI / ML model. Alternatively, even if the number of the obtained performance metric samples for a respective AI / ML model among the assessed AI / ML model (s) is less than the respective number of performance metric samples needed for the monitoring, the UE 104 may transmit the result 205 of the monitoring to the base station 102 based on the obtained performance metric samples if for each of the assessed AI / ML model (s) , the number of consecutive sample failures of the AI / ML model has reached a respective maximum number of consecutive sample failures for the AI / ML model. Alternatively, even if the number of the obtained performance metric samples for a respective AI / ML model among the assessed AI / ML model (s) is less than the respective number of performance metric samples needed for the monitoring, the UE 104 may transmit the result 205 of the monitoring to the base station 102 based on the obtained performance metric samples if for each of the assessed AI / ML model (s) , a time duration with sample failures of the AI / ML model has reached a respective maximum duration of model failure for the AI / ML model.
[0155] In other words, the monitoring result may be reported once the UE 104 determines that the none of the assessed AI / ML model (s) is applicable and the RS transmissions and performance monitoring may be skipped after the reporting of the monitoring result. In this way, the efficiency for performance monitoring may be improved and the resource overhead may be reduced.
[0156] In some embodiments, if the result 205 of the monitoring has not been transmitted when the configured time duration for RS transmissions for the monitoring elapses, the UE 104 may transmit the result 205 of the monitoring to the base station 102. Alternatively, if the result 205 of the monitoring has not been transmitted when the respective number of performance metric samples needed for the monitoring for each of the assessed AI / ML model (s) are obtained, the UE 104 may transmit the result 205 of the monitoring to the base station 102.
[0157] In some embodiments, when the configured time duration for RS transmissions for the monitoring elapses, the UE 104 may transmit the result 205 of the monitoring to the base station 102 if a ratio of sample failures of a respective AI / ML model among the assessed AI / ML model (s) reaches a respective maximum ratio of sample failures. Alternatively, when the respective number of performance metric samples needed for the monitoring for each of the assessed AI / ML model (s) are obtained, the UE 104 may transmit the result 205 of the monitoring to the base station 102 if a ratio of sample failures of a respective AI / ML model among the assessed AI / ML model (s) reaches a respective maximum ratio of sample failures.
[0158] In some embodiments, the parameter configuration 202 for the performance monitoring of the AI / ML functionality may include one of the following: at least one maximum ratio of sample failures for the AI / ML model (s) to be assessed, at least one maximum number of sample failures for the AI / ML model (s) to be assessed, at least one maximum number of consecutive sample failures for the AI / ML model (s) to be assessed, or at least one maximum duration of model failure for the AI / ML model (s) to be assessed.
[0159] FIG. 3 illustrates an example signaling chart of a general procedure of performance monitoring of an AI / ML functionality or an AI / ML model in accordance with aspects of the present disclosure. For the purpose of discussion, the communication process 300 will be described with reference to FIG. 1. It would be appreciated that although the communication process 300 has been described referring to the network environment 100 of FIG. 1, this communication process 300 may be likewise applied to other similar communication scenarios. The communication process 300 may be regarded as a specific example implementation of the process 200 of FIG. 2. The process 300 may involve the UE 104 and the base station 102.
[0160] As shown in FIG. 3, at 301, a configuration associated with a performance monitoring may be aligned between the UE 104 and the base station 102 during or after a functionality identification procedure for an AI / ML functionality or a model identification procedure for an AI / ML model. The configuration may include at least one of the following: a triggering condition, a report quantity, or other associated information. In the case that the procedure is for monitoring performance of an AI / ML functionality, the configuration may further include at least one of the following: an indication for performance monitoring of inactive functionality or an indication for functionality monitoring level. The configuration can be reported to the base station 102 by UE capability or UAI e.g., as a kind of functionality-related information or can be sent from the base station 102 to the UE 104 by DL signaling.
[0161] At 302, the performance monitoring may be triggered e.g., based on the triggering condition in the configuration. At 303, the UE 102 may perform measurement and calculation for monitoring result of the AI / ML functionality / model. For example, after the triggering of the performance monitoring, the base station 102 may transmit RSs in a time duration for UE measurement for performance monitoring. The UE 104 may generate a dataset for the performance monitoring and calculate the monitoring result of assessed model (s) based on the dataset. At 304, the UE 104 may report the monitoring result of the AI / ML functionality / model with a report format determined based on the configuration.
[0162] At 305, the base station 102 may make decisions on operation for the monitored AI / ML functionality and / or operation for the models of the monitored AI / ML functionality or make decisions on operation for the monitored AI / ML model. In the case that the procedure is for monitoring performance of an AI / ML functionality, based on the reported monitoring output of the AI / ML functionality and corresponding performance monitoring level, the base station 102 may make decision on the functionality operation or model operation and indicates the decision to the UE 104.
[0163] FIG. 4 illustrates an example signaling chart of a specific procedure of performance monitoring of an AI / ML functionality in accordance with aspects of the present disclosure. For the purpose of discussion, the communication process 400 will be described with reference to FIG. 1. It would be appreciated that although the communication process 400 has been described referring to the network environment 100 of FIG. 1, this communication process 400 may be likewise applied to other similar communication scenarios. The communication process 400 may be regarded as a specific example implementation of the process 200 of FIG. 2. The process 400 may involve the UE 104 and the base station 102. FIG. 4 illustrates the interactions between the base station 102 and the UE 104 over the air interface during the procedure of the performance monitoring method of an AI / ML functionality. It should be noted a similar communication process may also apply to performance monitoring of an AI / ML model.
[0164] As shown in FIG. 4, information associated with performance monitoring of AI / ML functionalities may be aligned between the base station 102 and the UE 104. There may be two alternative ways for the alignment. At 401-2, the base station 102 may indicate the information associated with the performance monitoring of AI / ML functionality by a higher-level signaling, e.g., RRC signaling. Optionally, at 401-1, the UE 104 may report the information associated with the performance monitoring of the AI / ML functionality by UE capability or UAI. For example, the information may be reported as a kind of condition or additional condition associated with AI / ML-enabled Feature / FG.
[0165] In some embodiments, prior to the alignment of the information for an AI / ML functionality between the base station 102 and the UE 104, information about performance monitoring of inactive functionality may be reported by the UE 104 through UE capability. The information about performance monitoring of inactive functionality may include a first indication about whether the performance monitoring of inactive functionality is supported, and / or a maximum number of functionalities that can be monitored at the same time.
[0166] The aligned information for the AI / ML functionality may include at least one of following: a triggering condition of a performance monitoring of the functionality, a report quantity that specifies a report content of the performance monitoring of the functionality, a time requirement of report, a second indication to indicate which level (s) of performance monitoring is (are) supported, at least one performance metric target (s) , a RS resource configuration for the performance monitoring, a time duration during which the dataset is generated for performance monitoring or the number of samples needed for performance monitoring, or Cell / site / TRP / area information.
[0167] In some examples, the aligned information for the AI / ML functionality may include the triggering condition used to trigger / start / resume a performance monitoring of the functionality. There may be three alternatives of the triggering condition.
[0168] In a first alternative, the triggering condition may be associated with event occurrence implying performance degradation of the functionality. In this case, the UE 104 is aware of the event occurrence, then the performance monitoring will be triggered by the UE 104. The entry condition of the event may include that a reference signal strength (e.g., SINR, RSRP, RSRQ) is lower than a threshold configured by higher layers or that the UE 104 is moving to the cell edge or moving to a new cell. The UE 104 may trigger a L1 or L2 report when the entry condition for the event is fulfilled and time of duration for this event is larger than a configured timer.
[0169] In a second alternative, the performance monitoring may be triggered with a DL signaling. In this case, the base station 102 signals the UE 104 to perform the functionality monitoring. This may be done by an aperiodic CSI or beam report.
[0170] In a third alternative, the triggering condition may be associated with a performance monitoring periodicity and an offset. In this case, the performance monitoring will be started periodically at time instances determined by the periodicity and the offset. In a specific implementation, this may be done by a periodic CSI or beam report by introducing a new report quantity. In another specific implementation, a timer-based performance monitoring may be supported in L2.
[0171] In some examples, the aligned information for the AI / ML functionality may include the report quantity that specifies a report content of the performance monitoring of the functionality. In a first alternative, the report content of the performance monitoring may include an event indicating performance monitoring result of the functionality. In a second alternative, the report content of the performance monitoring may include performance metric (s) based on the model inference output and labeled data. For example, the report content of the performance monitoring may include the differential RSRP of a same beam pair or statistical results of some measured metrics. In a third alternative, the report content of the performance monitoring may include none. In this case, nothing would be reported for the functionality monitoring, and the UE 104 may select a model transparently to the base station 102 based on performance monitoring outcome.
[0172] In some examples, the aligned information for the AI / ML functionality may include the time requirement of report indicating that the time requirement of report of the performance monitoring is tight or relax. For example, value of 1 means a tight time requirement and value of 0 means a relax time requirement.
[0173] In some examples, the aligned information for the AI / ML functionality may include the second indication to indicate which level (s) of performance monitoring is (are) supported. There may be three potential performance monitoring levels, i.e., Level a, Level b and Level c.
[0174] For Level a, the performance of a model determined by the UE 104 would be assessed and the result would be regarded as the performance monitoring result of the functionality.
[0175] For Level b, the performance of a default model and a specific model of the functionality would be assessed, where the default model has general performance in conditions relevant to the functionality and the specific model is selected by the UE 104 within the functionality. The default model should be identified between the base station 102 and the UE 104 through e.g., type A, type B1 or type B2 model identification.
[0176] For Level c, the performance of candidate models belonging to the functionality would be assessed, where the number or maximum number of the candidate models or the combinations of candidate models should be aligned by the base station 102 and the UE 104.
[0177] In some examples, the aligned information for the AI / ML functionality may include at least one performance metric target (s) as the reference to performance metric of assessed model (s) of the functionality. In addition, the aligned information for the AI / ML functionality may further include one of the following: at least one corresponding maximum failure rate / ratio or at least one corresponding maximum failure sample number or at least one corresponding maximum consecutive number of failure sample for the assessed model (s) of the functionality.
[0178] In some examples, the aligned information for the AI / ML functionality may include the RS resource configuration for the performance monitoring. The RS resource configuration may be used to generate a dataset for performance monitoring of the AI / ML functionality.
[0179] In some examples, the aligned information for the AI / ML functionality may include the Cell / site / TRP / area information. For example, RS sequence, RS ID, cell ID or area ID of the UE 104 may be considered.
[0180] The performance monitoring may be triggered / started / resumed based on the triggering condition in the aligned information for the AI / ML functionality with three potential ways corresponding to three alternatives of the triggering condition. In the first alternative, the UE 104 may be aware of the event occurrence implying performance monitoring degradation of the functionality, e.g., SINR is lower than threshold. Then, at 402-1, the UE 104 may report the event occurrence to the base station 102 for triggering the performance monitoring. In the second alternative, at 402-2, the base station 102 may trigger the performance monitoring by signaling directly, e.g., DCI, MAC CE or RRC signaling. In the third alternative, at 402-3, the performance monitoring may be triggered periodically based on the performance monitoring periodicity and offset.
[0181] At 403-1, after the performance monitoring is triggered, the base station 102 may transmit RSs in a time duration for measurement to enable the UE 104 to generate a dataset for the performance monitoring. The dataset may include input data for model and labeled data, where the input data may be beam measurement results, CSI measurement results, or UE measurement result for positioning (e.g., RSTD, RSRP, RSCP, etc. ) . The labeled data may be identifiers and / or L1-RSRP of top N beams, the CSI measurement result or the UE position. The time duration and the resource configuration for the RS transmission may be indicated by the aligned information for the AI / ML functionality.
[0182] At 403-2, based on the dataset, the UE 104 may calculate the monitoring result of the functionality, where the model of the functionality used for the monitoring may be determined based on the second indication. In detail, the second indication may indicate one among Level a, Level b and Level c as the level of performance monitoring of the functionality. If the Level a is indicated, the UE 104 may determine a model of the functionality by UE implementation to assess performance of the functionality. If the Level b is indicated, a default model and a specific model of the functionality would be assessed, where the default model has general performance in conditions relevant to the functionality and was identified between the base station 102 and the UE 104, and the specific model is selected by the UE 104 within the functionality. If the Level c is indicated, information about models to be assessed would be further indicated to the UE 104, e.g., IDs of models to be assessed. The UE 104 may assess performance of these indicated models.
[0183] A performance metric may be calculated by the UE 104 based on the dataset for performance monitoring. The performance metric may be a per sample metric or a statistical metric. For example, a bias between an inference result and corresponding labeled data is a per sample metric, where the inference result is obtained using the model with input data of the dataset. The bias may be different for different use cases. For example, the bias may be the difference between the predicted L1-RSRP and the actual L1-RSRP of the same beam for a beam management use case, or the squared generalized cosine similarity (SGCS) between the predicted CSI and the actual CSI for a CSI prediction use case, or the position difference for a direct or AI assisted positioning accuracy enhancement use case. Another example of a per sample metric may be whether the predicted top-N beams from AI / ML model includes the actual best beam from the labeled data. The statistical metric is a statistical value of a per sample metric, e.g., 95%-tile of the bias, top-N / 1beam prediction accuracy, a statistical value of the position difference, a statistical value of time-of-arrival (ToA) , line-of-sight (LOS) information, etc.
[0184] At 404, the UE 104 may report the monitoring result of the functionality. If the performance metric target is configured and the report quantity is configured with “event” or “performance metric” , a counter may be maintained at the UE 104 to count the number of times that the performance metric is worse than the performance metric target in the time duration.
[0185] In one example, if the maximum failure sample number is configured, and the value of the counter reaches the maximum failure sample number, the UE 104 may be triggered to report monitoring result of the functionality, so that the LCM operation may be performed accordingly, e.g., functionality activation, deactivation, switching and fallback. The time duration for measurement would be terminated as well.
[0186] In another example, if the maximum consecutive number of failure sample is configured, and the value of the counter reaches the maximum consecutive number of failure sample, the UE 104 may be triggered to report monitoring result of the functionality, so that LCM operation may be performed accordingly. The time duration for measurement would be terminated as well. The counter may be reset when the performance metric is better than the performance metric target before the counter reaches the maximum consecutive number of failure sample.
[0187] If the UE 104 measured all RSs transmitted in the time duration and completed calculation of corresponding performance metrics, the UE 104 may report the monitoring result of the functionality.
[0188] If the maximum duration of functionality failure is configured, and the time duration during which the performance metric is worse than the performance metric target reaches the maximum duration of functionality failure, the UE 104 may be triggered to report monitoring result of the functionality.
[0189] To make the LCM operation timely, the report may be carried by L1 signaling, e.g., UCI. To reduce air interface overhead, the report may be carried by higher layer signaling, e.g., MAC CE or RRC signaling.
[0190] If the performance metric target is not configured and the report quantity is configured with “performance metric” , if the UE 104 measured all RSs transmitted in the time duration and completed calculation of corresponding performance metrics, the UE 104 may report the monitoring result of the functionality using L1 / L2 signaling or higher layer signaling.
[0191] It is worth noting that the case that the maximum failure sample number is not configured can be regarded to a special case that the maximum failure sample number is configured to zero.
[0192] The monitoring result is determined at 403 based on the calculated performance metric. Based on the alignment of performance monitoring configuration between the UE 104 and the base station 102, there may be three types of report quantity for performance monitoring of functionality, i.e., “event” , “performance metric” and “none” . Hereinafter, the report format for the cases of “event” and “performance metric” will be described in detail.
[0193] For the Level a monitoring, if the report quantity is “performance metric” , the UE 104 would report a performance metric of the functionality, where a model used to generate the performance metric is selected by the UE 104 and is transparent to the base station 102.
[0194] For the Level a monitoring, if the report quantity is “event” , one among two events would be reported. These two events should be predefined and aligned between the base station 102 and the UE 104: Event C1 (functionality is suitable) and Event C2 (functionality is not suitable) .
[0195] After the measurement and calculation of monitoring result for the functionality, the UE 104 may report an event associated with the monitoring result using one bit. For example, if the value of the counter at the end of the time duration is less than the maximum failure sample number which means performance of the functionality is suitable, the UE 104 would report the Event C1 using a bit with “1” ; otherwise, the UE 104 would report the Event C2 using a bit with “0” .
[0196] For Level b monitoring, if the report quantity is “performance metric” , the UE 104 would report two performance metrics of the functionality, i.e., performance metrics of the default model and the specific model. To distinguish thes two performance metrics at the base station 102, the order of them in a report should be predefined, e.g., the performance metric of the default model may be followed by the performance metric of the special model. In some embodiments, corresponding model IDs may be also reported in the report.
[0197] For Level b monitoring, if the report quantity is “event” , one among three events would be reported. These three events should be predefined and aligned between the base station 102 and the UE 104: Event D1 (functionality is suitable) , Event D2 (functionality is suitable with default model) , or Event D3 (functionality is not suitable) .
[0198] After measurement and calculation of monitoring result for the functionality, the UE 104 may report and event associated with the monitoring result using a bit field. In detail, if the values of the counter of the default model and the specific model at the end of the time duration are less than corresponding maximum failure sample numbers, respectively, the UE 104 would report the Event D1 using a bit field with “01” , which means that the performance of the functionality is suitable;
[0199] If the value of the counter of the default model is less than corresponding maximum failure sample number but the value of the counter of the specific model is not less than corresponding maximum failure sample number at the end of the time duration, the UE 104 would report the Event D2 using a bit field with “10” , which means that the performance of the functionality would be suitable if a configuration for inference of the default model is configured;
[0200] If the values of the counter of the default model and the specific model at the end of the time duration are not less than corresponding maximum failure sample numbers, respectively, the UE 104 would report the Event D3 using a bit field with “00” , which means that the performance of the functionality would not be suitable even with a new configuration for inference.
[0201] With Level b monitoring, the performance of the functionality may be monitored through two models, i.e., a default model and a specific model. For example, when the environment / condition in which the UE 104 is located has been changed slightly, the specific model is no longer suitable but the default model is still able to work, so the functionality doesn’t have to be deactivated or switched. Unnecessary overhead for functionality deactivation / switching may be avoided.
[0202] For Level c monitoring, if the report quantity is “event” , one among three events would be reported. These three events should be predefined and aligned between the base station 102 and the UE 104: Event E1 (functionality is suitable) , Event E2 (functionality is suitable with new configuration for inference) , and Event E3 (functionality is not suitable) .
[0203] After measurement and calculation of monitoring result for the functionality, the UE 104 may report an event associated with the monitoring result using a bit field. In detail, if the value of the counter of any model is less than corresponding maximum failure sample number, the UE 104 would report the Event E1 using a bit field with “01” , which means that the performance of the functionality is suitable; the content of report may include Event E1, e.g., {Event E1} .
[0204] If the value of the counter of any model is less than corresponding maximum failure sample number but a new configuration is necessary for inference of the model, the UE 104 would report the Event E2 using a bit field with “10” , which means that the performance of the functionality is suitable with a new configuration for inference; the content of report may include Event E2, and model ID or RS resource configuration ID associated with the model, e.g., {Event E2, CSI-RS resource ID / model ID} .
[0205] If the value of the counter of any model is not less than corresponding maximum failure sample number, the UE 104 would report the Event E3 using a bit field with “00” , which means that the performance of the functionality is not suitable; the content of report may include Event E3, e.g., {Event E3} .
[0206] For Level c monitoring, if the report quantity is “performance metric” , the UE 104 would report performance metrics of all assessed models, along with identifiers of assessed models e.g., model ID or identifier of associated configuration for performance monitoring (e.g., CSI-RS resource set ID) . As an example, the report format may be {model ID_1, Performance metric_1} , {model ID_2, Performance metric_2} , …, {model ID_n, Performance metric_n} } . As another example, the report format may be { {CSI-RS set ID_1, Performance metric_1} , {CSI-RS set ID_2, Performance metric_2} , …, {CSI-RS set ID_n, Performance metric_n} } .
[0207] At 405, the base station 102 may make decisions on functionality operation or model operation for the AI / ML functionality.
[0208] For the case that the report quantity is “event” , the base station 102 may make decisions on operations of the functionality or model of the functionality based on the reported event associated with the functionality. For event C1, event D1 or event E1, the base station 102 may not indicate a LCM operation to the UE 104. For event C2, event D3 or event E3, the base station 102 may indicate the UE 104 to perform a functionality deactivation, switching or fallback. For event D2, the base station 102 may indicate a configuration for inference of the default model to the UE 104. For event E2 along with model ID or configuration ID, the base station 102 may indicate a configuration for inference of the corresponding model to the UE 104.
[0209] For the case that the report quantity is “performance metric” , for Level a monitoring, if the performance metric of the active model belonging to the functionality satisfies the corresponding performance metric target, the functionality would be maintained; otherwise, a functionality operation would be executed, e.g., switching / updating / deactivation / fallback.
[0210] For Level b monitoring, if both the default model and the specific model cannot reach the corresponding performance target (which may be the same or different for the two models) , respectively, the functionality may be claimed to fail. If the default model can reach the corresponding performance target but the specific model cannot, the functionality may be maintained and the UE 104 may perform inference using the default model. If both the default model and the specific model can reach the corresponding performance target, respectively, the functionality would be maintained as before.
[0211] For Level c monitoring, if the performance metric of at least one assessed model (s) satisfies the performance metric threshold, information to maintain the functionality and, if need, activation of one of the models together with associated RS resource configuration for inference would be indicated to the UE 104 from the base station 102; otherwise, functionality operations would be executed, e.g., switching / updating / deactivation / fallback.
[0212] Hereinafter, an example of performance monitoring for an AI / ML beam prediction functionality will be described in detail in accordance with aspects of the present disclosure.
[0213] The UE may report, by UE capability report, a first indication of supporting performance monitoring of inactive functionality and / or a maximum number of functionalities that can be monitored at the same time. The UE may also report, by UE capability report or UAI, a second indication of supporting Level a / b / c performance monitoring for an AI / ML beam prediction functionality.
[0214] The base station may configure, by a higher layer signaling, e.g., RRC (re-) configuration, one or more than one triggering event (s) as a triggering condition used to trigger a performance monitoring of the AI / ML beam prediction functionality, where the one or more than one triggering event (s) may include at least one of following: that measured L1-RSRP value or predicted L1-RSRP value of top-K (K>=1) beam (s) from AI / ML model corresponding to the functionality is lower than a threshold; and / or that the UE is accessed to a cell whose cell ID is different from pervious serving cell, and / or that the SINR is lower than a threshold. In addition to the triggering event, the base station may configure a performance monitoring periodicity and an offset for triggering performance monitoring.
[0215] The base station may configure a report quantity for performance monitoring of the functionality. In one implementation, the report quantity may be “performance metric” , e.g., 95%-tile of difference between predicted L1-RSRP and measured L1-RSRP of the same beam, or top-N / 1 beam prediction accuracy. In another implementation, the report quantity may be “event” .
[0216] The base station may configure at least one performance metric target (s) as the reference to performance metric of the functionality. For example, in case of level b as the monitoring level and top-N / 1 beam prediction accuracy, the corresponding performance metric targets can be 90%and 95%for the default model and the specific model, respectively.
[0217] The base station may configure RS resource configuration for performance monitoring used to generate dataset of beam measurement. Beam set for inference result and model input are denoted Set A and Set B, respectively. In the case that Set B is a subset of Set A and both are sets of narrow beams, a CSI-RS resource set is configured for beam measurement of Set A and Set B. In the case that Set B is different from Set A, and Set A and Set B are a set of wider beams and a set of narrow beams, respectively, a SSB-CSI resource set may be configured for beam measurement of Set B and a CSI-RS resource set may be configured for beam measurement of Set A.
[0218] The base station may configure a time duration for collecting data used for performance monitoring, e.g., 100ms.
[0219] After the UE detects occurrence of at least one of the triggering events, a performance monitoring for the corresponding functionality will be triggered by the UE reporting the event to the base station. Alternatively, a performance monitoring for the corresponding functionality will be triggered periodically at time instances determined by the periodicity and the offset. The UE may receive RS based on the RS resource configuration for performance monitoring in a time duration. Starting point of the time duration is associated with the time instance when the UE reports the event to the base station. For example, the RS resource configuration may include a periodic CSI-RS resource set for Set A and Set B beam measurement, and the time duration with length of 100ms starts after x slots from the time instance when the UE reports the event to the base station. The UE may perform beam measurements based on the received RSs and calculate the performance metrics based on results of the beam measurements. The calculation of performance metrics is up to UE implementation.
[0220] The UE may report monitoring results of the functionality based on the calculated performance metrics. For different cases of report quantity and performance monitoring level, there are different contents and formats of report. The details of contents and formats of report may follow step 404 in FIG. 4.
[0221] FIG. 5 illustrates an example of a device 500 that supports performance monitoring for AI / ML functionality or AI / ML model in accordance with aspects of the present disclosure. The device 500 may be an example of a UE 104 or a network entity 102 as described herein. The device 500 may support wireless communication with one or more network entities 102, UEs 104, or any combination thereof. The device 500 may include components for bi-directional communications including components for transmitting and receiving communications, such as a processor 502, a memory 504, a transceiver 506, and, optionally, an I / O controller 508. These components may be in electronic communication or otherwise coupled (e.g., operatively, communicatively, functionally, electronically, electrically) via one or more interfaces (e.g., buses) .
[0222] The processor 502, the memory 504, the transceiver 506, or various combinations thereof or various components thereof may be examples of means for performing various aspects of the present disclosure as described herein. For example, the processor 502, the memory 504, the transceiver 506, or various combinations or components thereof may support a method for performing one or more of the operations described herein.
[0223] In some implementations, the processor 502, the memory 504, the transceiver 506, or various combinations or components thereof may be implemented in hardware (e.g., in communications management circuitry) . The hardware may include a processor, a digital signal processor (DSP) , an application-specific integrated circuit (ASIC) , a field-programmable gate array (FPGA) or other programmable logic device, a discrete gate or transistor logic, discrete hardware components, or any combination thereof configured as or otherwise supporting a means for performing the functions described in the present disclosure. In some implementations, the processor 502 and the memory 504 coupled with the processor 502 may be configured to perform one or more of the functions described herein (e.g., executing, by the processor 502, instructions stored in the memory 504) .
[0224] In an example in which the device 500 is implemented as a UE 104, the processor 502 may support wireless communication at the device 500 in accordance with examples as disclosed herein. The processor 502 may be configured to operable to support a means for receiving, from a network entity, a parameter configuration for monitoring performance of an AI / ML functionality or performance of an AI / ML model; and a means for transmitting, to the network entity, a result of the monitoring of the performance of the AI / ML functionality or the performance of the AI / ML model based on the parameter configuration.
[0225] In an example in which the device 500 is implemented as a network entity 102, the processor 502 may support wireless communication at the device 500 in accordance with examples as disclosed herein. The processor 502 may be configured to operable to support a means for transmitting, to a UE, a parameter configuration for monitoring performance of an AI / ML functionality or performance of an AI / ML model; and a means for receiving, from the UE, a result of the monitoring.
[0226] The processor 502 may include an intelligent hardware device (e.g., a general-purpose processor, a DSP, a CPU, a microcontroller, an ASIC, an FPGA, a programmable logic device, a discrete gate or transistor logic component, a discrete hardware component, or any combination thereof) . In some implementations, the processor 502 may be configured to operate a memory array using a memory controller. In some other implementations, a memory controller may be integrated into the processor 502. The processor 502 may be configured to execute computer-readable instructions stored in a memory (e.g., the memory 504) to cause the device 500 to perform various functions of the present disclosure such that the device 500 may perform any process of the disclosure as discussed with reference to FIGS. 2 to 10C.
[0227] The memory 504 may include random access memory (RAM) and read-only memory (ROM) . The memory 504 may store computer-readable, computer-executable code including instructions that, when executed by the processor 502 cause the device 500 to perform various functions described herein. The code may be stored in a non-transitory computer-readable medium such as system memory or another type of memory. In some implementations, the code may not be directly executable by the processor 502 but may cause a computer (e.g., when compiled and executed) to perform functions described herein. In some implementations, the memory 504 may include, among other things, a basic I / O system (BIOS) which may control basic hardware or software operation such as the interaction with peripheral components or devices.
[0228] The I / O controller 508 may manage input and output signals for the device 500. The I / O controller 508 may also manage peripherals not integrated into the device M02. In some implementations, the I / O controller 508 may represent a physical connection or port to an external peripheral. In some implementations, the I / O controller 508 may utilize an operating system such as or another known operating system. In some implementations, the I / O controller 508 may be implemented as part of a processor, such as the processor 506. In some implementations, a user may interact with the device 500 via the I / O controller 508 or via hardware components controlled by the I / O controller 508.
[0229] In some implementations, the device 500 may include a single antenna 510. However, in some other implementations, the device 500 may have more than one antenna 510 (i.e., multiple antennas) , including multiple antenna panels or antenna arrays, which may be capable of concurrently transmitting or receiving multiple wireless transmissions. The transceiver 506 may communicate bi-directionally, via the one or more antennas 510, wired, or wireless links as described herein. For example, the transceiver 506 may represent a wireless transceiver and may communicate bi-directionally with another wireless transceiver. The transceiver 506 may also include a modem to modulate the packets, to provide the modulated packets to one or more antennas 510 for transmission, and to demodulate packets received from the one or more antennas 510. The transceiver 506 may include one or more transmit chains, one or more receive chains, or a combination thereof.
[0230] A transmit chain may be configured to generate and transmit signals (e.g., control information, data, packets) . The transmit chain may include at least one modulator for modulating data onto a carrier signal, preparing the signal for transmission over a wireless medium. The at least one modulator may be configured to support one or more techniques such as amplitude modulation (AM) , frequency modulation (FM) , or digital modulation schemes like phase-shift keying (PSK) or quadrature amplitude modulation (QAM) . The transmit chain may also include at least one power amplifier configured to amplify the modulated signal to an appropriate power level suitable for transmission over the wireless medium. The transmit chain may also include one or more antennas 510 for transmitting the amplified signal into the air or wireless medium.
[0231] A receive chain may be configured to receive signals (e.g., control information, data, packets) over a wireless medium. For example, the receive chain may include one or more antennas 510 for receive the signal over the air or wireless medium. The receive chain may include at least one amplifier (e.g., a low-noise amplifier (LNA) ) configured to amplify the received signal. The receive chain may include at least one demodulator configured to demodulate the receive signal and obtain the transmitted data by reversing the modulation technique applied during transmission of the signal. The receive chain may include at least one decoder for decoding the processing the demodulated signal to receive the transmitted data.
[0232] FIG. 6 illustrates an example of a processor 600 that supports performance monitoring for AI / ML functionality or AI / ML model in accordance with aspects of the present disclosure. The processor 600 may be an example of a processor configured to perform various operations in accordance with examples as described herein. The processor 600 may be implemented in a device or its components as described herein. For example, the device may be an example of a UE 104 or a network entity 102 as described herein. The processor 600 may include a controller 602 configured to perform various operations in accordance with examples as described herein. The processor 600 may optionally include at least one memory 604, such as L1 / L2 / L3 cache. Additionally, or alternatively, the processor 600 may optionally include one or more arithmetic-logic units (ALUs) 600. One or more of these components may be in electronic communication or otherwise coupled (e.g., operatively, communicatively, functionally, electronically, electrically) via one or more interfaces (e.g., buses) .
[0233] The processor 600 may be a processor chipset and include a protocol stack (e.g., a software stack) executed by the processor chipset to perform various operations (e.g., receiving, obtaining, retrieving, transmitting, outputting, forwarding, storing, determining, identifying, accessing, writing, reading) in accordance with examples as described herein. The processor chipset may include one or more cores, one or more caches (e.g., memory local to or included in the processor chipset (e.g., the processor 600) or other memory (e.g., random access memory (RAM) , read-only memory (ROM) , dynamic RAM (DRAM) , synchronous dynamic RAM (SDRAM) , static RAM (SRAM) , ferroelectric RAM (FeRAM) , magnetic RAM (MRAM) , resistive RAM (RRAM) , flash memory, phase change memory (PCM) , and others) .
[0234] The controller 602 may be configured to manage and coordinate various operations (e.g., signaling, receiving, obtaining, retrieving, transmitting, outputting, forwarding, storing, determining, identifying, accessing, writing, reading) of the processor 600 to cause the processor 600 to support various operations in accordance with examples as described herein. For example, the controller 602 may operate as a control unit of the processor 600, generating control signals that manage the operation of various components of the processor 600. These control signals include enabling or disabling functional units, selecting data paths, initiating memory access, and coordinating timing of operations.
[0235] The controller 602 may be configured to fetch (e.g., obtain, retrieve, receive) instructions from the memory 604 and determine subsequent instruction (s) to be executed to cause the processor 600 to support various operations in accordance with examples as described herein. The controller 602 may be configured to track memory address of instructions associated with the memory 604. The controller 602 may be configured to decode instructions to determine the operation to be performed and the operands involved. For example, the controller 602 may be configured to interpret the instruction and determine control signals to be output to other components of the processor 600 to cause the processor 600 to support various operations in accordance with examples as described herein. Additionally, or alternatively, the controller 602 may be configured to manage flow of data within the processor 600. The controller 602 may be configured to control transfer of data between registers, arithmetic logic units (ALUs) , and other functional units of the processor 600.
[0236] The memory 604 may include one or more caches (e.g., memory local to or included in the processor 600 or other memory, such RAM, ROM, DRAM, SDRAM, SRAM, MRAM, flash memory, etc. In some implementation, the memory 604 may reside within or on a processor chipset (e.g., local to the processor 600) . In some other implementations, the memory 604 may reside external to the processor chipset (e.g., remote to the processor 600) .
[0237] The memory 604 may store computer-readable, computer-executable code including instructions that, when executed by the processor 600, cause the processor 600 to perform various functions described herein. The code may be stored in a non-transitory computer-readable medium such as system memory or another type of memory. The controller 602 and / or the processor 600 may be configured to execute computer-readable instructions stored in the memory 604 to cause the processor 600 to perform various functions. For example, the processor 600 and / or the controller 602 may be coupled with or to the memory 604, and the processor 600, the controller 602, and the memory 604 may be configured to perform various functions described herein. In some examples, the processor 600 may include multiple processors and the memory 604 may include multiple memories. One or more of the multiple processors may be coupled with one or more of the multiple memories, which may, individually or collectively, be configured to perform various functions herein.
[0238] The one or more ALUs 600 may be configured to support various operations in accordance with examples as described herein. In some implementation, the one or more ALUs 600 may reside within or on a processor chipset (e.g., the processor 600) . In some other implementations, the one or more ALUs 600 may reside external to the processor chipset (e.g., the processor 600) . One or more ALUs 600 may perform one or more computations such as addition, subtraction, multiplication, and division on data. For example, one or more ALUs 600 may receive input operands and an operation code, which determines an operation to be executed. One or more ALUs 600 be configured with a variety of logical and arithmetic circuits, including adders, subtractors, shifters, and logic gates, to process and manipulate the data according to the operation. Additionally, or alternatively, the one or more ALUs 600 may support logical operations such as AND, OR, exclusive-OR (XOR) , not-OR (NOR) , and not-AND (NAND) , enabling the one or more ALUs 600 to handle conditional operations, comparisons, and bitwise operations.
[0239] The processor 600 may support wireless communication in accordance with examples as disclosed herein. In an example in which the device 500 is implemented as a UE 104, the processor 600 may be configured to or operable to support a means for receiving, from a network entity, a parameter configuration for monitoring performance of an AI / ML functionality or performance of an AI / ML model; and a means for transmitting, to the network entity, a result of the monitoring of the performance of the AI / ML functionality or the performance of the AI / ML model based on the parameter configuration.
[0240] In an example in which the device 500 is implemented as a network entity 102, the processor 600 may be configured to or operable to support a means for transmitting, to a UE, a parameter configuration for monitoring performance of an AI / ML functionality or performance of an AI / ML model; and a means for receiving, from the UE, a result of the monitoring.
[0241] FIG. 7 illustrates a flowchart of a method 700 that supports performance monitoring for AI / ML functionality or AI / ML model in accordance with aspects of the present disclosure. The operations of the method 700 may be implemented by a device or its components as described herein. For example, the operations of the method 700 may be performed by a UE 104 as described herein. In some implementations, the device may execute a set of instructions to control the function elements of the device to perform the described functions. Additionally, or alternatively, the device may perform aspects of the described functions using special-purpose hardware.
[0242] At 705, the method may include receiving, from a base station, a parameter configuration for monitoring performance of an AI / ML functionality or performance of an AI / ML model. The operations of 705 may be performed in accordance with examples as described herein. In some implementations, aspects of the operations of 705 may be performed by a device as described with reference to FIG. 1.
[0243] At 710, the method may include transmitting, to the base station, a result of the monitoring of the performance of the AI / ML functionality or the performance of the AI / ML model based on the parameter configuration. The operations of 710 may be performed in accordance with examples as described herein. In some implementations, aspects of the operations of 710 may be performed by a device as described with reference to FIG. 1.
[0244] FIG. 8 illustrates a flowchart of a method 800 that supports performance monitoring for AI / ML functionality or AI / ML model in accordance with aspects of the present disclosure. The operations of the method 800 may be implemented by a device or its components as described herein. For example, the operations of the method 800 may be performed by a UE 104 as described herein. In some implementations, the device may execute a set of instructions to control the function elements of the device to perform the described functions. Additionally, or alternatively, the device may perform aspects of the described functions using special-purpose hardware. The method 800 may be implemented as a specific example of the step 710 in FIG. 7.
[0245] At 805, the method may include transmitting, to the base station, one of the following: an indication of an event that the AI / ML functionality is applicable in the case that the default AI / ML model and the AI / ML model associated with the AI / ML functionality are applicable; an indication of an event that the AI / ML functionality is applicable with the default AI / ML model in the case that the default AI / ML model is applicable while the AI / ML model associated with the AI / ML functionality is not applicable; or an indication of an event that the AI / ML functionality is not applicable in the case that both the default AI / ML model and the AI / ML model associated with the AI / ML functionality are not applicable. The operations of 805 may be performed in accordance with examples as described herein. In some implementations, aspects of the operations of 805 may be performed by a device as described with reference to FIG. 1.
[0246] FIG. 900 illustrates a flowchart of a method 900 that supports performance monitoring for AI / ML functionality or AI / ML model in accordance with aspects of the present disclosure. The operations of the method 900 may be implemented by a device or its components as described herein. For example, the operations of the method 900 may be performed by a network entity 102 as described herein. In some implementations, the device may execute a set of instructions to control the function elements of the device to perform the described functions. Additionally, or alternatively, the device may perform aspects of the described functions using special-purpose hardware.
[0247] At 905, the method may include transmitting, to a UE, a parameter configuration for monitoring performance of an AI / ML functionality or performance of an AI / ML model. The operations of 905 may be performed in accordance with examples as described herein. In some implementations, aspects of the operations of 905 may be performed by a device as described with reference to FIG. 1.
[0248] At 910, the method may include receiving, from the UE, a result of the monitoring. The operations of 910 may be performed in accordance with examples as described herein. In some implementations, aspects of the operations of 910 may be performed by a device as described with reference to FIG. 1.
[0249] FIG. 10 illustrates a flowchart of a method 1000 that supports performance monitoring for AI / ML functionality or AI / ML model in accordance with aspects of the present disclosure. The operations of the method 1000 may be implemented by a device or its components as described herein. For example, the operations of the method 1000 may be performed by a network entity 102 as described herein. In some implementations, the device may execute a set of instructions to control the function elements of the device to perform the described functions. Additionally, or alternatively, the device may perform aspects of the described functions using special-purpose hardware. In some embodiments, the method 1000 may be implemented as a specific example of the step 910 in FIG. 9.
[0250] At 1005, the method may include receiving, to a UE, one of the following: an indication of an event that the AI / ML functionality is applicable; an indication of an event that the AI / ML functionality is applicable but requires for a configuration for inference; or an indication of an event that the AI / ML functionality is not applicable. The operations of 1005 may be performed in accordance with examples as described herein. In some implementations, aspects of the operations of 1005 may be performed by a device as described with reference to FIG. 1.
[0251] It should be noted that the methods described herein describes possible implementations, and that the operations and the steps may be rearranged or otherwise modified and that other implementations are possible. Further, aspects from two or more of the methods may be combined.
[0252] The various illustrative blocks and components described in connection with the disclosure herein may be implemented or performed with a general-purpose processor, a DSP, an ASIC, a CPU, an FPGA or other programmable logic device, discrete gate or transistor logic, discrete hardware components, or any combination thereof designed to perform the functions described herein. A general-purpose processor may be a microprocessor, but in the alternative, the processor may be any processor, controller, microcontroller, or state machine. A processor may also be implemented as a combination of computing devices (e.g., a combination of a DSP and a microprocessor, multiple microprocessors, one or more microprocessors in conjunction with a DSP core, or any other such configuration.
[0253] The functions described herein may be implemented in hardware, software executed by a processor, firmware, or any combination thereof. If implemented in software executed by a processor, the functions may be stored on or transmitted over as one or more instructions or code on a computer-readable medium. Other examples and implementations are within the scope of the disclosure and appended claims. For example, due to the nature of software, functions described herein may be implemented using software executed by a processor, hardware, firmware, hardwiring, or combinations of any of these. Features implementing functions may also be physically located at various positions, including being distributed such that portions of functions are implemented at different physical locations.
[0254] Computer-readable media includes both non-transitory computer storage media and communication media including any medium that facilitates transfer of a computer program from one place to another. A non-transitory storage medium may be any available medium that may be accessed by a general-purpose or special-purpose computer. By way of example, non-transitory computer-readable media may include RAM, ROM, electrically erasable programmable ROM (EEPROM) , flash memory, compact disk (CD) ROM or other optical disk storage, magnetic disk storage or other magnetic storage devices, or any other non-transitory medium that may be used to carry or store desired program code means in the form of instructions or data structures and that may be accessed by a general-purpose or special-purpose computer, or a general-purpose or special-purpose processor.
[0255] As used herein, including in the claims, an article “a” before an element is unrestricted and understood to refer to “at least one” of those elements or “one or more” of those elements. The terms “a, ” “at least one, ” “one or more, ” and “at least one of one or more” may be interchangeable. As used herein, including in the claims, “or” as used in a list of items (e.g., a list of items prefaced by a phrase such as “at least one of” or “one or more of” or “one or both of” ) indicates an inclusive list such that, for example, a list of at least one of A, B, or C means A or B or C or AB or AC or BC or ABC (i.e., A and B and C) . Also, as used herein, the phrase “based on” shall not be construed as a reference to a closed set of conditions. For example, an example step that is described as “based on condition A” may be based on both a condition A and a condition B without departing from the scope of the present disclosure. In other words, as used herein, the phrase “based on” shall be construed in the same manner as the phrase “based at least in part on. Further, as used herein, including in the claims, a “set” may include one or more elements.
[0256] The description herein is provided to enable a person having ordinary skill in the art to make or use the disclosure. Various modifications to the disclosure will be apparent to a person having ordinary skill in the art, and the generic principles defined herein may be applied to other variations without departing from the scope of the disclosure. Thus, the disclosure is not limited to the examples and designs described herein but is to be accorded the broadest scope consistent with the principles and novel features disclosed herein.
Claims
1.A user equipment (UE) , comprising:a processor; anda transceiver coupled to the processor,wherein the processor is configured to:receive, from a base station via the transceiver, a parameter configuration for monitoring performance of an artificial intelligence / machine learning (AI / ML) functionality or performance of an AI / ML model; andtransmit, to the base station via the transceiver, a result of the monitoring of the performance of the AI / ML functionality or the performance of the AI / ML model based on the parameter configuration.2.The UE of claim 1, wherein the parameter configuration comprises at least one of the following:a condition for triggering the monitoring;a report quantity for specifying a content type of the result to be transmitted;a time requirement for reporting the result;an indication of a level of the monitoring in the case that the parameter configuration is associated with the AI / ML functionality;a performance metric target associated with the AI / ML model in the case that the parameter configuration is associated with the AI / ML model;at least one performance metric target associated with at least one AI / ML model to be assessed in the case that the parameter configuration is associated with the AI / ML functionality, wherein the at least one AI / ML model to be assessed is associated with the AI / ML functionality;a resource configuration for reference signal (RS) transmissions for the monitoring;a time duration for RS transmissions for the monitoring;the number of performance metric samples needed for the monitoring in the case that the parameter configuration is associated with the AI / ML model;at least one number of performance metric samples needed for the monitoring corresponding to at least one AI / ML model to be assessed in the case that the parameter configuration is associated with the AI / ML functionality, wherein the at least one AI / ML model to be assessed is associated with the AI / ML functionality; orinformation of an area in which the UE is located.3.The UE of claim 2, wherein the condition comprises an occurrence of an event associated with potential degradation of the performance of the AI / ML functionality or the performance of the AI / ML model.4.The UE of claim 2, wherein the condition comprises a periodicity and an offset, and the processor is further configured to:start the monitoring at a time instance associated with the periodicity and the offset.5.The UE of claim 2, wherein the content type is one of the following:an event type associated with the performance of the AI / ML functionality or the performance of the AI / ML model;a performance metric type; ornone.6.The UE of claim 2, wherein the performance of the AI / ML functionality is based on at least one performance of at least one AI / ML model to be assessed among one or more AI / ML models associated with the AI / ML functionality.7.The UE of claim 6, wherein the indication of the level is indicative of one of the following:a first monitoring level in which one AI / ML model associated with the AI / ML functionality is to be assessed during the monitoring;a second monitoring level in which a default AI / ML model and an AI / ML model associated with the AI / ML functionality are to be assessed during the monitoring; ora third monitoring level in which a plurality of AI / ML models associated with the AI / ML functionality are to be assessed during the monitoring.8.The UE of claim 6, wherein the parameter configuration comprises the at least one performance metric target associated with the at least one AI / ML model to be assessed, and the processor is further configured to:determine a sample failure of a respective AI / ML model among the at least one AI / ML model in the case that a performance metric for a performance metric sample of the respective AI / ML model is worse than a performance metric target associated with the respective AI / ML model.9.The UE of claim 8, wherein the processor is further configured to:determine that the respective AI / ML model is applicable in the case of one of the following:a ratio of sample failures of the respective AI / ML model in the time duration for RS transmissions for the monitoring or among a respective number of performance metric samples needed for the monitoring does not reach a respective maximum ratio of sample failures;the number of sample failures of the respective AI / ML model in the time duration for RS transmissions for the monitoring or among a respective number of performance metric samples needed for the monitoring does not reach a respective maximum number of sample failures;the number of consecutive sample failures of the respective AI / ML model in the time duration for RS transmissions for the monitoring or among a respective number of performance metric samples needed for the monitoring does not reach a respective maximum number of consecutive sample failures; ora time duration with sample failures of the respective AI / ML model in the time duration for RS transmissions for the monitoring or among a respective number of performance metric samples needed for the monitoring does not reach a respective maximum duration of model failure.10.The UE of claim 6, wherein transmitting the result of the monitoring comprises:transmitting, to the base station via the transceiver, the result of the monitoring at one of the following time instances:a first time instance before end of the time duration for RS transmissions for the monitoring or when the number of performance metric samples obtained for the monitoring for a respective AI / ML model among the at least one AI / ML model is less than a respective number of performance metric samples needed for the monitoring in the case of one of the following:for each of the at least one AI / ML model, the number of sample failures of the AI / ML model reaches a respective maximum number of sample failures at the first time instance;for each of the at least one AI / ML model, the number of consecutive sample failures of the AI / ML model reaches a respective maximum number of consecutive sample failures at the first time instance;for each of the at least one AI / ML model, a time duration with sample failures of the AI / ML model reaches a respective maximum duration of model failure at the first time instance;a second time instance after the end of the time duration for RS transmissions for the monitoring in the case that the result of the monitoring has not been transmitted or in the case that a ratio of sample failures of a respective AI / ML model among the at least one AI / ML model reaches a respective maximum ratio of sample failures; ora third time instance when a respective number of performance metric samples needed for the monitoring are obtained for each of the at least one AI / ML model in the case that the result of the monitoring has not been transmitted or in the case that a ratio of sample failures of a respective AI / ML model among the at least one AI / ML model reaches a respective maximum ratio of sample failures.11.The UE of any of claims 9 or 10, wherein the parameter configuration further comprises one of the following:at least one maximum ratio of sample failures for the at least one AI / ML model;at least one maximum number of sample failures for the at least one AI / ML model;at least one maximum number of consecutive sample failures for the at least one AI / ML model; orat least one maximum duration of model failure for the at least one AI / ML model.12.The UE of claim 7, wherein the indication of the level is indicative of the first monitoring level, the content type is a performance metric type, and transmitting the result of the monitoring comprises:transmitting, to the base station via the transceiver, at least one performance metric of the AI / ML model associated with the AI / ML functionality.13.The UE of claim 7, wherein the indication of the level is indicative of the second monitoring level, the content type is a performance metric type, and transmitting the result of the monitoring comprises:transmitting, to the base station via the transceiver, at least one performance metric of the default AI / ML model and at least one performance metric of the AI / ML model associated with the AI / ML functionality.14.The UE of claim 7, wherein the indication of the level is indicative of the third monitoring level, the content type is a performance metric type, and transmitting the result of the monitoring comprises:transmitting, to the base station via the transceiver, a plurality of performance metrics of the plurality of AI / ML models associated with the AI / ML functionality.15.The UE of claim 7, wherein the indication of the level is indicative of the first monitoring level, the content type is an event type associated with the performance of the AI / ML functionality, and transmitting the result of the monitoring comprises:transmitting, to the base station via the transceiver, one of the following:an indication of an event that the AI / ML functionality is applicable in the case that the AI / ML model associated with the AI / ML functionality is applicable; oran indication of an event that the AI / ML functionality is not applicable in the case that the AI / ML model associated with the AI / ML functionality is not applicable.16.The UE of claim 7, wherein the indication of the level is indicative of the second monitoring level, the content type is an event type associated with the performance of the AI / ML functionality, and transmitting the result of the monitoring comprises:transmitting, to the base station via the transceiver, one of the following:an indication of an event that the AI / ML functionality is applicable in the case that the default AI / ML model and the AI / ML model associated with the AI / ML functionality are applicable;an indication of an event that the AI / ML functionality is applicable with the default AI / ML model in the case that the default AI / ML model is applicable while the AI / ML model associated with the AI / ML functionality is not applicable; oran indication of an event that the AI / ML functionality is not applicable in the case that both the default AI / ML model and the AI / ML model associated with the AI / ML functionality are not applicable.17.The UE of claim 7, wherein the indication of the level is indicative of the third monitoring level, the content type is an event type associated with the performance of the AI / ML functionality, and transmitting the result of the monitoring comprises:transmitting, to the base station via the transceiver, one of the following:an indication of an event that the AI / ML functionality is applicable in the case that at least one of the plurality of AI / ML models associated with the AI / ML functionality is applicable;an indication of an event that the AI / ML functionality is applicable but requires for a configuration for inference in the case that at least one of the plurality of AI / ML models associated with the AI / ML functionality is applicable but requires for a configuration for inference; oran indication of an event that the AI / ML functionality is not applicable in the case that none of the plurality of AI / ML models associated with the AI / ML functionality is applicable.18.A base station, comprising:a processor; anda transceiver coupled to the processor,wherein the processor is configured to:transmit, to a user equipment (UE) via the transceiver, a parameter configuration for monitoring performance of an artificial intelligence / machine learning (AI / ML) functionality or performance of an AI / ML model; andreceive, from the UE via the transceiver, a result of the monitoring.19.A method performed by a user equipment, comprising:receiving, from a base station, a parameter configuration for monitoring performance of an artificial intelligence / machine learning (AI / ML) functionality or performance of an AI / ML model; andtransmitting, to the base station, a result of the monitoring of the performance of the AI / ML functionality or the performance of the AI / ML model based on the parameter configuration.20.A method performed by a base station, comprising:transmitting, to a user equipment (UE) , a parameter configuration for monitoring performance of an artificial intelligence / machine learning (AI / ML) functionality or performance of an AI / ML model; andreceiving, from the UE, a result of the monitoring.
Citation Information
Patent Citations
Supplementing artificial intelligence (AI) / machine learning (ML) model via action center, AI / ML model retraining hardware control, and AI / ML model setup management
CN114930356A
Method and apparatus for transmitting and receiving signal in wireless communication system
WO2023211041A1
User equipment machine learning functionality monitoring
WO2023211356A1
Cited By
Methods and devices for life cycle management of two side artificial intelligence machine learning model
WO2026118184A1
A method for monitoring and reporting performance of ai models in predicting measurements in wireless networks
WO2026118214A1
Mobility operation execution method and apparatus, mobility operation triggering method and apparatus, and terminal and network device
WO2026152312A1
Technologies for reporting applicable artificial intelligence functionality or model
WO2026165720A1
Channel state information prediction method and wireless communication device
WO2026165810A1
