Hierarchical monitoring of artificial intelligence or machine learning models for air interface

Abstract

Certain aspects of the present disclosure provide techniques for performing wireless communication. In some aspects, the techniques include obtaining a first metric associated with a model that is associated with wireless communication; and obtaining, in response to a trigger condition associated with the first metric being satisfied, a second metric associated with the model, the second metric providing a different measure of the model than the first metric.

Description

Qualcomm Ref. No.: 2404871WO1HIERARCHICAL MONITORING OF ARTIFICIAL INTELLIGENCE OR MACHINE LEARNING MODELS FOR AIR INTERFACECROSS REFERENCE TO RELATED APPLICATION

[0001] The present Application for Patent claims priority to and benefit of Greek Patent Application No. 20240100658, filed September 26, 2024, which is hereby expressly incorporated by reference herein in its entirety.INTRODUCTIONField of the Disclosure

[0002] Aspects of the present disclosure relate to wireless communications, and more particularly, to techniques for performing hierarchical monitoring of artificial intelligence (Al) models in wireless communications systems.Description of Related Art

[0003] Wireless communications systems are widely deployed to provide various telecommunication services such as telephony, video, data, messaging, broadcasts, or other similar types of services. These wireless communications systems may employ multiple-access technologies capable of supporting communications with multiple users by sharing available wireless communications system resources with those users.

[0004] Although wireless communications systems have made great technological advancements over many years, challenges still exist. For example, complex and dynamic environments can still attenuate or block signals between wireless transmitters and wireless receivers. Accordingly, there is a continuous desire to improve the technical performance of wireless communications systems, including, for example: improving speed and data carrying capacity of communications, improving efficiency of the use of shared communications mediums, reducing power used by transmitters and receivers while performing communications, improving reliability of wireless communications, avoiding redundant transmissions and / or receptions and related processing, improving the coverage area of wireless communications, increasing the number and types of devices that can access wireless communications systems, increasing the ability for different types of devices to intercommunicate, increasing the number and type of wireless communications mediums available for use, and the like. Consequently, there exists aD&S Ref. No.: QCM2404871WOQualcomm Ref. No.: 2404871WO2 need for further improvements in wireless communications systems to overcome the aforementioned technical challenges and others.SUMMARY

[0005] Artificial Intelligence (Al) and Machine Learning (ML) techniques are increasingly being utilized in wireless communication systems to enhance various aspects such as channel estimation, beam management, and resource allocation. However, the performance of AI / ML models can be affected by factors like changes in the wireless environment, user mobility, and network load. Continuously monitoring the performance of these models is beneficial to ensure their effective and efficient operation over time, but it can be resource-intensive in terms of power consumption, processing overhead, and communication bandwidth.

[0006] Indiscriminately performing monitoring using resource-intensive approaches like full ground truth-based monitoring can lead to significant resource expenditure at the UE or network entity for limited benefit. On the other hand, always using lightweight ground truth- free approaches may miss important model performance degradation. There is a need for monitoring techniques that can efficiently monitor AI / ML model performance in wireless systems based on the tradeoff between monitoring benefits and resource overhead.

[0007] The present disclosure addresses these challenges by providing techniques for hierarchical monitoring of AI / ML models in wireless communication systems. The techniques involve monitoring a first metric associated with the performance of an AI / ML model using a first set of resources. When a trigger condition is satisfied based on the first metric, the techniques involve monitoring a second metric associated with the performance of the AI / ML model using a second set of resources. The second metric provides a different measure of the model’s performance than the first metric. The second set of resources may be larger than the first set of resources.

[0008] By using a hierarchical monitoring approach, the techniques enable efficient monitoring of AI / ML models in wireless communication systems. The first metric serves as a low-overhead indicator of the AI / ML model’s performance, allowing for continuous monitoring without consuming significant resources. When the first metric indicates a potential issue with the AI / ML model’s performance, the second metric is monitored using additional resources to provide a more comprehensive assessment of the AI / MLD&S Ref. No.: QCM2404871WOQualcomm Ref. No.: 2404871WO3 model’s performance. This hierarchical approach maintains a balance between monitoring granularity and resource efficiency.

[0009] The techniques described herein offer several benefits, including early detection of performance issues in AI / MT models, reduced overall resource overhead associated with monitoring, and flexibility in adapting the monitoring process based on factors such as network conditions, device capabilities, and user preferences.

[0010] Some aspects provide a method for wireless communication by a UE. The method includes obtaining a first metric associated with a model that is associated with wireless communication; and obtaining, in response to a trigger condition associated with the first metric being satisfied, a second metric associated with the model, the second metric providing a different measure of the model than the first metric.

[0011] Some other aspects provide a method for wireless communication by a network entity. The method includes obtaining a first metric associated with a model that is associated with wireless communication; and obtaining, in response to a trigger condition associated with the first metric being satisfied, a second metric associated with the model, the second metric providing a different measure of the model than the first metric.

[0012] Some other aspects provide a method for wireless communications by an apparatus. The method may include monitoring a first metric associated with a machine learning model using a first level of monitoring granularity, wherein the machine learning model is associated with wireless communication, and wherein the first level of monitoring granularity comprises a first set of one or more parameters associated with a performance of the machine learning model; monitoring, in response to a trigger condition associated with the first metric being satisfied, a second metric associated with the machine learning model using a second level of monitoring granularity, wherein the second metric is configured to provide a different measure of the machine learning model than the first metric, wherein the second level of monitoring granularity comprises a second set of one or more parameters associated with the performance of the machine learning model, and wherein the second set of one or more parameters includes at least one parameter that is not included in the first set of one or more parameters; and signaling, in response to the second metric satisfying a reporting condition that is not satisfied by the first metric, an indication associated with the performance of the model.D&S Ref. No.: QCM2404871WOQualcomm Ref. No.: 2404871WO4

[0013] Other aspects provide: one or more apparatuses operable, configured, or otherwise adapted to perform any portion of any method described herein (e.g., such that performance may be by only one apparatus or in a distributed fashion across multiple apparatuses); one or more non-transitory, computer-readable media comprising instructions that, when executed by one or more processors of one or more apparatuses, cause the one or more apparatuses to perform any portion of any method described herein (e.g., such that instructions may be included in only one computer-readable medium or in a distributed fashion across multiple computer-readable media, such that instructions may be executed by only one processor or by multiple processors in a distributed fashion, such that each apparatus of the one or more apparatuses may include one processor or multiple processors, and / or such that performance may be by only one apparatus or in a distributed fashion across multiple apparatuses); one or more computer program products embodied on one or more computer-readable storage media comprising code for performing any portion of any method described herein (e.g., such that code may be stored in only one computer-readable medium or across computer-readable media in a distributed fashion); and / or one or more apparatuses comprising one or more means for performing any portion of any method described herein (e.g., such that performance would be by only one apparatus or by multiple apparatuses in a distributed fashion). By way of example, an apparatus may comprise a processing system, a device with a processing system, or processing systems cooperating over one or more networks. An apparatus may comprise one or more memories; and one or more processors configured to cause the apparatus to perform any portion of any method described herein. In some examples, one or more of the processors may be preconfigured to perform various functions or operations described herein without requiring configuration by software.

[0014] The following description and the appended figures set forth certain features for purposes of illustration.BRIEF DESCRIPTION OF DRAWINGS

[0015] The appended figures depict certain features of the various aspects described herein and are not to be considered limiting of the scope of this disclosure.

[0016] FIG. 1 depicts an example wireless communications network.

[0017] FIG. 2 depicts an example disaggregated base station architecture.D&S Ref. No.: QCM2404871WOQualcomm Ref. No.: 2404871WO5

[0018] FIG. 3 depicts aspects of an example base station and an example user equipment (UE).

[0019] FIGS. 4A, 4B, 4C, and 4D depict various example aspects of data structures for a wireless communications network.

[0020] FIG. 5 depicts a diagram illustrating an example artificial intelligence (Al) architecture that may be used for Al-enhanced wireless communications.

[0021] FIG. 6 depicts aspects of an example monitoring architecture in accordance with aspects of the present disclosure.

[0022] FIG. 7 depicts an example hierarchical monitoring process for an Al or machine learning (AI / ML) model in accordance with aspects of the present disclosure.

[0023] FIG. 8 depicts aspects of an example monitoring architecture in accordance with aspects of the present disclosure.

[0024] FIG. 9 depicts an example monitoring agent in accordance with aspects of the present disclosure.

[0025] FIG. 10 depicts an example monitoring agent in accordance with aspects of the present disclosure.

[0026] FIG. 11 depicts an example communication system for monitoring the performance of an AI / ML model in a wireless network, in accordance with aspects of the present disclosure

[0027] FIG. 12 depicts a process flow for communications in a network between devices that may include a network entity or UE.

[0028] FIG. 13 depicts a process flow for communications in a network between devices that may include a network entity or UE.

[0029] FIG. 14 depicts a process flow for communications in a network between devices that may include a network entity or UE.

[0030] FIG. 15 depicts another method for wireless communications.

[0031] FIG. 16 depicts aspects of an example communications device.

[0032] FIG. 17 depicts aspects of an example communications device.D&S Ref. No.: QCM2404871WOQualcomm Ref. No.: 2404871WO6DETAILED DESCRIPTION

[0033] Aspects of the present disclosure provide apparatuses, methods, processing systems, and computer-readable mediums for performing hierarchical monitoring of artificial intelligence (Al) models in wireless communications systems.

[0034] Wireless communication systems are evolving to meet the increasing demands for high-speed, reliable, and efficient data transmission. Artificial intelligence (Al) and machine learning (ML) techniques can be utilized to enhance various aspects of wireless communication, such as channel estimation, beam management, and resource allocation. However, the performance of AI / ML models in wireless communication systems can be affected by various factors, such as changes in the wireless environment, user mobility, and network load.

[0035] Monitoring the performance of AI / ML models helps to ensure that the AI / ML models continue to operate effectively and efficiently over time. Such monitoring can include metrics based on inference accuracy (related to intermediate key performance indicators (KPIs), for example), system performance (related to system performance KPIs for example), data distribution (e.g., out-of-distribution detection or drift detection of input / output data, as examples), and applicable conditions (e.g., environmental conditions in which a wireless communication system operates). The monitoring metric calculation may be done at the network or UE.

[0036] Approaches for AI / ML model monitoring include, but are not limited to, ground truth (GT) based monitoring and ground truth- free monitoring. In GT-based monitoring, the model input measurements and expected model output (e.g., ground truth beam or channel state information (CSI) measurements for example) may be utilized to validate an AI / ML model. In contrast, GT-free monitoring may be based on model input measurements (e.g., statistics of measurements) and actual model output(s) (e.g., statistics of model output over time). Indirect KPIs such as block error rate (BLER), throughput, and negative acknowledgements (NACKs) can also be used to assess model performance, as these indirect KPIs can be affected by model failure.

[0037] However, continuously monitoring the performance of these models can be resource-intensive, particularly in terms of power consumption, processing overhead, and communication bandwidth. For example, GT-based monitoring may utilize more data collection, labeling, and comparison than GT-free approaches, leading to higher powerD&S Ref. No.: QCM2404871WOQualcomm Ref. No.: 2404871WO7 consumption, processing overhead, and bandwidth usage. Even within a particular form of monitoring like GT-free monitoring, different configurations of metrics, sampling rates, and analysis can have different resource footprints.

[0038] If monitoring is indiscriminately performed using a more burdensome approach like full GT-based monitoring, then significant resources may be expended at the UE or network entity. On the other hand, if monitoring always uses a lightweight GT- free approach, model performance degradation may be missed. Therefore, there is a need for monitoring techniques that can efficiently monitor AI / ME model performance in wireless systems based on the tradeoff between monitoring benefits and resource overhead.

[0039] Aspects of the present disclosure address these and other challenges by providing techniques for hierarchical monitoring of AI / ME models in wireless communication systems. In some aspects, such techniques may involve monitoring a first metric associated with the performance of an AI / ML model using a first set of resources. When a trigger condition is satisfied based on the first metric, in some aspects, the techniques may involve monitoring a second metric associated with the performance of the AI / ML model using a second set of resources. In some aspects, the second metric provides a different measure of the model’s performance than the first metric, and the second set of resources is larger than the first set of resources. For instance, the first metric might be based on a limited set of reference signals or beam measurements, while the second metric may involve a more comprehensive set of reference signals or finer-grained beam measurements, thereby providing a more detailed assessment of the AI / ML model’s performance than the first metric.

[0040] By using a hierarchical monitoring approach, techniques described herein can enable efficient monitoring of AI / ML models in wireless communication systems. In some aspects, the first metric provides a low-overhead indicator of the AI / ML model’s performance, allowing for continuous monitoring without consuming significant resources. In some aspects, when the first metric indicates a potential issue with the model’s performance, the second metric may be monitored using additional resources to provide a more comprehensive assessment of the model’s performance. Accordingly, this hierarchical approach maintains a balance between monitoring granularity and resource efficiency.D&S Ref. No.: QCM2404871WOQualcomm Ref. No.: 2404871WO8

[0041] The techniques described herein offer several technical benefits. In some aspects, the techniques described herein enable early detection of performance issues in AI / ML models, allowing for timely corrective actions to be taken. In some aspects, the techniques described herein reduce the overall resource overhead associated with monitoring the performance of AI / ML models, leading to improved system efficiency and battery life in wireless devices. In some aspects, the techniques described herein provide flexibility in adapting the monitoring process based on factors such as network conditions, device capabilities, and / or user preferences.

[0042] It should be noted that the techniques described herein can be applied for AI / ML functionalities as well as individual AI / ML models. An AI / ML functionality may be implemented using one or more AI / ML models. For example, an AI / ML functionality may include “CSI generation,” and the CSI may be generated using one or more AI / ML models.Introduction to Wireless Communications Networks

[0043] The techniques and methods described herein may be used for various wireless communications networks. While aspects may be described herein using terminology commonly associated with 3G, 4G, 5G, 6G, and / or other generations of wireless technologies, aspects of the present disclosure may likewise be applicable to other communications systems and standards not explicitly mentioned herein.

[0044] FIG. 1 depicts an example of a wireless communications network 100, in which aspects described herein may be implemented.

[0045] Generally, wireless communications network 100 includes various network entities (alternatively, network elements or network nodes). A network entity is generally a communications device and / or a communications function performed by a communications device (e.g., a user equipment (UE), a base station (BS), a component of a BS, a server, etc.). As such communications devices are part of wireless communications network 100, and facilitate wireless communications, such communications devices may be referred to as wireless communications devices. For example, various functions of a network as well as various devices associated with and interacting with a network may be considered network entities. Further, wireless communications network 100 includes terrestrial aspects, such as ground-based network entities (e.g., BSs 102), and non-terrestrial aspects (also referred to herein as non-D&S Ref. No.: QCM2404871WOQualcomm Ref. No.: 2404871WO9 terrestrial network entities), such as satellite 140 and / or aerial or spaceborne platform(s), which may include network entities on-board (e.g., one or more BSs) capable of communicating with other network elements (e.g., terrestrial BSs) and UEs.

[0046] In the depicted example, wireless communications network 100 includes BSs 102, UEs 104, and one or more core networks, such as an Evolved Packet Core (EPC) 160 and 5G Core (5GC) network 190, which interoperate to provide communications services over various communications links, including wired and wireless links.

[0047] FIG. 1 depicts various example UEs 104, which may more generally include: a cellular phone, smart phone, session initiation protocol (SIP) phone, laptop, personal digital assistant (PDA), satellite radio, global positioning system, multimedia device, video device, digital audio player, camera, game console, tablet, smart device, wearable device, vehicle, electric meter, gas pump, large or small kitchen appliance, healthcare device, implant, sensor / actuator, display, internet of things (loT) devices, always on (AON) devices, edge processing devices, data centers, or other similar devices. UEs 104 may also be referred to more generally as a mobile device, a wireless device, a station, a mobile station, a subscriber station, a mobile subscriber station, a mobile unit, a subscriber unit, a wireless unit, a remote unit, a remote device, an access terminal, a mobile terminal, a wireless terminal, a remote terminal, a handset, and others.

[0048] BSs 102 wirelessly communicates with (e.g., transmit signals to or receive signals from) UEs 104 via communications links 120. The communications links 120 between BSs 102 and UEs 104 may include uplink (UL) (also referred to as reverse link) transmissions from a UE 104 to a BS 102 and / or downlink (DL) (also referred to as forward link) transmissions from a BS 102 to a UE 104. The communications links 120 may use multiple-input and multiple-output (MIMO) antenna technology, including spatial multiplexing, beamforming, and / or transmit diversity in various aspects.

[0049] BSs 102 may generally include: a NodeB, enhanced NodeB (eNB), next generation enhanced NodeB (ng-eNB), next generation NodeB (gNB or gNodeB), access point, base transceiver station, radio base station, radio transceiver, transceiver function, transmission reception point, and / or others. Each of BSs 102 may provide communications coverage for a respective coverage area 110, which may sometimes be referred to as a cell, and which may overlap in some cases (e.g., small cell 102’ may have a coverage area 110’ that overlaps the coverage area 110 of a macro cell). A BS may, forD&S Ref. No.: QCM2404871WOQualcomm Ref. No.: 2404871WO10 example, provide communications coverage for a macro cell (covering relatively large geographic area), a pico cell (covering relatively smaller geographic area, such as a sports stadium), a femto cell (relatively smaller geographic area (e.g., a home)), and / or other types of cells.

[0050] Generally, a cell may refer to a portion, partition, or segment of wireless communication coverage served by a network entity within a wireless communication network. A cell may have geographic characteristics, such as a geographic coverage area, as well as radio frequency characteristics, such as time and / or frequency resources dedicated to the cell. For example, a specific geographic coverage area may be covered by multiple cells employing different frequency resources (e.g., bandwidth parts) and / or different time resources. As another example, a specific geographic coverage area may be covered by a single cell. In some contexts (e.g., a carrier aggregation scenario and / or multi-connectivity scenario), the terms “cell” or “serving cell” may refer to or correspond to a specific carrier frequency (e.g., a component carrier) used for wireless communications, and a “cell group” may refer to or correspond to multiple carriers used for wireless communications. As examples, in a carrier aggregation scenario, a UE may communicate on multiple component carriers corresponding to multiple (serving) cells in the same cell group, and in a multi-connectivity (e.g., dual connectivity) scenario, a UE may communicate on multiple component carriers corresponding to multiple cell groups.

[0051] While BSs 102 are depicted in various aspects as unitary communications devices, BSs 102 may be implemented in various configurations. For example, one or more components of a base station may be disaggregated, including a central unit (CU), one or more distributed units (DUs), one or more radio units (RUs), a Near-Real Time (Near-RT) RAN Intelligent Controller (RIC), or a Non-Real Time (Non-RT) RIC, to name a few examples. In another example, various aspects of a base station may be virtualized. More generally, a base station (e.g., BS 102) may include components that are located at a single physical location or components located at various physical locations. In examples in which a base station includes components that are located at various physical locations, the various components may each perform functions such that, collectively, the various components achieve functionality that is similar to a base station that is located at a single physical location. In some aspects, a base station including components that are located at various physical locations may be referred to as a disaggregated radio access network architecture, such as an Open RAN (O-RAN) orD&S Ref. No.: QCM2404871WOQualcomm Ref. No.: 2404871WO11Virtualized RAN (VRAN) architecture. FIG. 2 depicts and describes an example disaggregated base station architecture.

[0052] Different BSs 102 within wireless communications network 100 may also be configured to support different radio access technologies, such as 3G, 4G, and / or 5G. For example, BSs 102 configured for 4G LTE (collectively referred to as Evolved Universal Mobile Telecommunications System (UMTS) Terrestrial Radio Access Network (E- UTRAN)) may interface with the EPC 160 through first backhaul links 132 (e.g., an SI interface). BSs 102 configured for 5G (e.g., 5G NR or Next Generation RAN (NG-RAN)) may interface with 5GC 190 through second backhaul links 184. BSs 102 may communicate directly or indirectly (e.g., through the EPC 160 or 5GC 190) with each other over third backhaul links 134 (e.g., X2 interface), which may be wired or wireless.

[0053] Wireless communications network 100 may subdivide the electromagnetic spectrum into various classes, bands, channels, or other features. In some aspects, the subdivision is provided based on wavelength and frequency, where frequency may also be referred to as a carrier, a subcarrier, a frequency channel, a tone, or a subband. For example, 3GPP currently defines Frequency Range 1 (FR1) as including 410 MHz - 7125 MHz, which is often referred to (interchangeably) as “Sub-6 GHz”. Similarly, 3GPP currently defines Frequency Range 2 (FR2) as including 24,250 MHz - 71,000 MHz, which is sometimes referred to (interchangeably) as a “millimeter wave” (“mmW” or “mmWave”). In some cases, FR2 may be further defined in terms of sub-ranges, such as a first sub-range FR2-1 including 24,250 MHz - 52,600 MHz and a second sub-range FR2-2 including 52,600 MHz - 71,000 MHz. A base station configured to communicate using mmWave / near mmWave radio frequency bands (e.g., a mmWave base station such as BS 180) may utilize beamforming (e.g., 182) with a UE (e.g., 104) to improve path loss and range.

[0054] The communications links 120 between BSs 102 and, for example, UEs 104, may be through one or more carriers, which may have different bandwidths (e.g., 5, 10, 15, 20, 100, 400, and / or other MHz), and which may be aggregated in various aspects. Carriers may or may not be adjacent to each other. Allocation of carriers may be asymmetric with respect to DE and UL (e.g., more or fewer carriers may be allocated for DL than for UL).D&S Ref. No.: QCM2404871WOQualcomm Ref. No.: 2404871WO12

[0055] Communications using higher frequency bands may have higher path loss and a shorter range compared to lower frequency communications. Accordingly, certain base stations (e.g., 180 in FIG. 1) may utilize beamforming 182 with a UE 104 to improve path loss and range. For example, BS 180 and the UE 104 may each include a plurality of antennas, such as antenna elements, antenna panels, and / or antenna arrays to facilitate the beamforming. In some cases, BS 180 may transmit a beamformed signal to UE 104 in one or more transmit directions 182’. UE 104 may receive the beamformed signal from the BS 180 in one or more receive directions 182”. UE 104 may also transmit a beamformed signal to the BS 180 in one or more transmit directions 182”. BS 180 may also receive the beamformed signal from UE 104 in one or more receive directions 182’. BS 180 and UE 104 may then perform beam training to determine the best receive and transmit directions for each of BS 180 and UE 104. Notably, the transmit and receive directions for BS 180 may or may not be the same. Similarly, the transmit and receive directions for UE 104 may or may not be the same.

[0056] Wireless communications network 100 further includes a Wi-Fi AP 150 in communication with Wi-Fi stations (STAs) 152 via communications links 154 in, for example, a 2.4 GHz and / or 5 GHz unlicensed frequency spectrum.

[0057] Certain UEs 104 may communicate with each other using device-to-device (D2D) communications link 158. D2D communications link 158 may use one or more sidelink channels, such as a physical sidelink broadcast channel (PSBCH), a physical sidelink discovery channel (PSDCH), a physical sidelink shared channel (PSSCH), a physical sidelink control channel (PSCCH), and / or a physical sidelink feedback channel (PSFCH).

[0058] EPC 160 may include various functional components, including: a Mobility Management Entity (MME) 162, other MMEs 164, a Serving Gateway 166, a Multimedia Broadcast Multicast Service (MBMS) Gateway 168, a Broadcast Multicast Service Center (BM-SC) 170, and / or a Packet Data Network (PDN) Gateway 172, such as in the depicted example. MME 162 may be in communication with a Home Subscriber Server (HSS) 174. MME 162 is the control node that processes the signaling between the UEs 104 and the EPC 160. Generally, MME 162 provides bearer and connection management.

[0059] Generally, user Internet protocol (IP) packets are transferred through Serving Gateway 166, which itself is connected to PDN Gateway 172. PDN Gateway 172D&S Ref. No.: QCM2404871WOQualcomm Ref. No.: 2404871WO13 provides UE IP address allocation as well as other functions. PDN Gateway 172 and the BM-SC 170 are connected to IP Services 176, which may include, for example, the Internet, an intranet, an IP Multimedia Subsystem (IMS), a Packet Switched (PS) streaming service, and / or other IP services.

[0060] BM-SC 170 may provide functions for MBMS user service provisioning and delivery. BM-SC 170 may serve as an entry point for content provider MBMS transmission, may be used to authorize and initiate MBMS Bearer Services within a public land mobile network (PLMN), and / or may be used to schedule MBMS transmissions. MBMS Gateway 168 may be used to distribute MBMS traffic to the BSs 102 belonging to a Multicast Broadcast Single Frequency Network (MBSFN) area broadcasting a particular service, and / or may be responsible for session management (start / stop) and for collecting eMBMS related charging information.

[0061] 5GC 190 may include various functional components, including: an Access and Mobility Management Function (AMF) 192, other AMFs 193, a Session Management Function (SMF) 194, and a User Plane Function (UPF) 195. AMF 192 may be in communication with Unified Data Management (UDM) 196.

[0062] AMF 192 is a control node that processes signaling between UEs 104 and 5GC 190. AMF 192 provides, for example, quality of service (QoS) flow and session management.

[0063] Internet protocol (IP) packets are transferred through UPF 195, which is connected to the IP Services 197, and which provides UE IP address allocation as well as other functions for 5GC 190. IP Services 197 may include, for example, the Internet, an intranet, an IMS, a PS streaming service, and / or other IP services.

[0064] In various aspects, a network entity or network node can be implemented as an aggregated base station, as a disaggregated base station, a component of a base station, an integrated access and backhaul (TAB) node, a relay node, a sidelink node, to name a few examples.

[0065] FIG. 2 depicts an example disaggregated base station 200 architecture. The disaggregated base station 200 architecture may include one or more central units (CUs) 210 that can communicate directly with a core network 220 via a backhaul link, or indirectly with the core network 220 through one or more disaggregated base station units (such as a Near-Real Time (Near-RT) RAN Intelligent Controller (RIC) 225 via an E2D&S Ref. No.: QCM2404871WOQualcomm Ref. No.: 2404871WO14 link, or aNon-Real Time (Non-RT) RIC 215 associated with a Service Management and Orchestration (SMO) Framework 205, or both). A CU 210 may communicate with one or more distributed units (DUs) 230 via respective midhaul links, such as an Fl interface. The DUs 230 may communicate with one or more radio units (RUs) 240 via respective fronthaul links. The RUs 240 may communicate with respective UEs 104 via one or more radio frequency (RF) access links. In some implementations, the UE 104 may be simultaneously served by multiple RUs 240.

[0066] Each of the units, e.g., the CUs 210, the DUs 230, the RUs 240, as well as the Near-RT RICs 225, the Non-RT RICs 215 and the SMO Framework 205, may include one or more interfaces or be coupled to one or more interfaces configured to receive or transmit signals, data, or information (collectively, signals) via a wired or wireless transmission medium. Each of the units, or an associated processor or controller providing instructions to the communications interfaces of the units, can be configured to communicate with one or more of the other units via the transmission medium. For example, the units can include a wired interface configured to receive or transmit signals over a wired transmission medium to one or more of the other units. Additionally or alternatively, the units can include a wireless interface, which may include a receiver, a transmitter or transceiver (such as a radio frequency (RF) transceiver), configured to receive or transmit signals, or both, over a wireless transmission medium to one or more of the other units.

[0067] In some aspects, the CU 210 may host one or more higher layer control functions. Such control functions can include radio resource control (RRC), packet data convergence protocol (PDCP), service data adaptation protocol (SDAP), or the like. Each control function can be implemented with an interface configured to communicate signals with other control functions hosted by the CU 210. The CU 210 may be configured to handle user plane functionality (e.g., Central Unit - User Plane (CU-UP)), control plane functionality (e.g., Central Unit - Control Plane (CU-CP)), or a combination thereof. In some implementations, the CU 210 can be logically split into one or more CU-UP units and one or more CU-CP units. The CU-UP unit can communicate bidirectionally with the CU-CP unit via an interface, such as the El interface when implemented in an O-RAN configuration. The CU 210 can be implemented to communicate with the DU 230, as necessary, for network control and signaling.D&S Ref. No.: QCM2404871WOQualcomm Ref. No.: 2404871WO15

[0068] The DU 230 may correspond to a logical unit that includes one or more base station functions to control the operation of one or more RUs 240. In some aspects, the DU 230 may host one or more of a radio link control (RLC) layer, a medium access control (MAC) layer, and one or more high physical (PHY) layers (such as modules for forward error correction (FEC) encoding and decoding, scrambling, modulation and demodulation, or the like) depending, at least in part, on a functional split, such as those defined by the 3rdGeneration Partnership Project (3GPP). In some aspects, the DU 230 may further host one or more low PHY layers. Each layer (or module) can be implemented with an interface configured to communicate signals with other layers (and modules) hosted by the DU 230, or with the control functions hosted by the CU 210.

[0069] Lower-layer functionality can be implemented by one or more RUs 240. In some deployments, an RU 240, controlled by a DU 230, may correspond to a logical node that hosts RF processing functions, or low-PHY layer functions (such as performing fast Fourier transform (FFT), inverse FFT (iFFT), digital beamforming, physical random access channel (PRACH) extraction and filtering, or the like), or both, based at least in part on the functional split, such as a lower layer functional split. In such an architecture, the RU(s) 240 can be implemented to handle over the air (OTA) communications with one or more UEs 104. In some implementations, real-time and non-real-time aspects of control and user plane communications with the RU(s) 240 can be controlled by the corresponding DU 230. In some scenarios, this configuration can enable the DU(s) 230 and the CU 210 to be implemented in a cloud-based RAN architecture, such as a vRAN architecture.

[0070] The SMO Framework 205 may be configured to support RAN deployment and provisioning of non- virtualized and virtualized network elements. For non- virtualized network elements, the SMO Framework 205 may be configured to support the deployment of dedicated physical resources for RAN coverage requirements which may be managed via an operations and maintenance interface (such as an 01 interface). For virtualized network elements, the SMO Framework 205 may be configured to interact with a cloud computing platform (such as an open cloud (O-Cloud) 290) to perform network element life cycle management (such as to instantiate virtualized network elements) via a cloud computing platform interface (such as an 02 interface). Such virtualized network elements can include, but are not limited to, CUs 210, DUs 230, RUs 240 and Near-RT RICs 225. In some implementations, the SMO Framework 205 canD&S Ref. No.: QCM2404871WOQualcomm Ref. No.: 2404871WO16 communicate with a hardware aspect of a 4G RAN, such as an open eNB (O-eNB) 211, via an 01 interface. Additionally, in some implementations, the SMO Framework 205 can communicate directly with one or more DUs 230 and / or one or more RUs 240 via an 01 interface. The SMO Framework 205 also may include aNon-RT RIC 215 configured to support functionality of the SMO Framework 205.

[0071] The Non-RT RIC 215 may be configured to include a logical function that enables non-real-time control and optimization of RAN elements and resources, Artificial Intelligence / Machine Teaming (AI / MF) workflows including model training and updates, or policy-based guidance of applications / features in the Near-RT RIC 225. The Non-RT RIC 215 may be coupled to or communicate with (such as via an Al interface) the Near-RT RIC 225. The Near-RT RIC 225 may be configured to include a logical function that enables near-real-time control and optimization of RAN elements and resources via data collection and actions over an interface (such as via an E2 interface) connecting one or more CUs 210, one or more DUs 230, or both, as well as an O-eNB, with the Near-RT RIC 225.

[0072] In some implementations, to generate AI / MF models to be deployed in the Near-RT RIC 225, the Non-RT RIC 215 may receive parameters or external enrichment information from external servers. Such information may be utilized by the Near-RT RIC 225 and may be received at the SMO Framework 205 or the Non-RT RIC 215 from nonnetwork data sources or from network functions. In some examples, the Non-RT RIC 215 or the Near-RT RIC 225 may be configured to tune RAN behavior or performance. For example, the Non-RT RIC 215 may monitor long-term trends and patterns for performance and employ AI / ML models to perform corrective actions through the SMO Framework 205 (such as reconfiguration via 01) or via creation of RAN management policies (such as Al policies).

[0073] FIG. 3 depicts aspects of an example BS 102 and a UE 104.

[0074] Generally, BS 102 includes various processors (e.g., 318, 320, 330, 338, and 340), antennas 334a-t (collectively 334), transceivers 332a-t (collectively 332), which include modulators and demodulators, and other aspects, which enable wireless transmission of data (e.g., data source 312) and wireless reception of data (e.g., data sink 314). For example, BS 102 may send and receive data between BS 102 and UE 104. BS 102 includes controller / processor 340, which may be configured to implement variousD&S Ref. No.: QCM2404871WOQualcomm Ref. No.: 2404871WO17 functions described herein related to wireless communications. Note that the BS 102 may have a disaggregated architecture as described herein with respect to FIG. 2.

[0075] Generally, UE 104 includes various processors (e.g., 358, 364, 366, 370, and 380), antennas 352a-r (collectively 352), transceivers 354a-r (collectively 354), which include modulators and demodulators, and other aspects, which enable wireless transmission of data (e.g., retrieved from data source 362) and wireless reception of data (e.g., provided to data sink 360). UE 104 includes controller / processor 380, which may be configured to implement various functions described herein related to wireless communications.

[0076] In regards to an example downlink transmission, BS 102 includes a transmit processor 320 that may receive data from a data source 312 and control information from a controller / processor 340. The control information may be for the physical broadcast channel (PBCH), physical control format indicator channel (PCFICH), physical hybrid automatic repeat request (HARQ) indicator channel (PHICH), physical downlink control channel (PDCCH), group common PDCCH (GC PDCCH), and / or others. The data may be for the physical downlink shared channel (PDSCH), in some examples.

[0077] Transmit processor 320 may process (e.g., encode and symbol map) the data and control information to obtain data symbols and control symbols, respectively. Transmit processor 320 may also generate reference symbols, such as for the primary synchronization signal (PSS), secondary synchronization signal (SSS), PBCH demodulation reference signal (DMRS), and channel state information reference signal (CSI-RS).

[0078] Transmit (TX) multiple-input multiple-output (MIMO) processor 330 may perform spatial processing (e.g., precoding) on the data symbols, the control symbols, and / or the reference symbols, if applicable, and may provide output symbol streams to the modulators (MODs) in transceivers 332a-332t. Each modulator in transceivers 332a- 332t may process a respective output symbol stream to obtain an output sample stream. Each modulator may further process (e.g., convert to analog, amplify, fdter, and upconvert) the output sample stream to obtain a downlink signal. Downlink signals from the modulators in transceivers 332a-332t may be transmitted via the antennas 334a-334t, respectively.D&S Ref. No.: QCM2404871WOQualcomm Ref. No.: 2404871WO18

[0079] In order to receive the downlink transmission, UE 104 includes antennas 352a-352r that may receive the downlink signals from the BS 102 and may provide received signals to the demodulators (DEMODs) in transceivers 354a-354r, respectively. Each demodulator in transceivers 354a-354r may condition (e.g., fdter, amplify, downconvert, and digitize) a respective received signal to obtain input samples. Each demodulator may further process the input samples to obtain received symbols.

[0080] RX MIMO detector 356 may obtain received symbols from all the demodulators in transceivers 354a-354r, perform MIMO detection on the received symbols if applicable, and provide detected symbols. Receive processor 358 may process (e.g., demodulate, deinterleave, and decode) the detected symbols, provide decoded data for the UE 104 to a data sink 360, and provide decoded control information to a controller / processor 380.

[0081] In regards to an example uplink transmission, UE 104 further includes a transmit processor 364 that may receive and process data (e.g., for the PUSCH) from a data source 362 and control information (e.g., for the physical uplink control channel (PUCCH)) from the controller / processor 380. Transmit processor 364 may also generate reference symbols for a reference signal (e.g., for the sounding reference signal (SRS)). The symbols from the transmit processor 364 may be precoded by a TX MIMO processor 366 if applicable, further processed by the modulators in transceivers 354a-354r (e.g., for SC-FDM), and transmitted to BS 102.

[0082] At BS 102, the uplink signals from UE 104 may be received by antennas 334a- t, processed by the demodulators in transceivers 332a-332t, detected by a RX MIMO detector 336 if applicable, and further processed by a receive processor 338 to obtain decoded data and control information sent by UE 104. Receive processor 338 may provide the decoded data to a data sink 314 and the decoded control information to the controller / processor 340.

[0083] Memories 342 and 382 may store data and program codes for BS 102 and UE 104, respectively.

[0084] Scheduler 344 may schedule UEs for data transmission on the downlink and / or uplink.

[0085] In various aspects, BS 102 may be described as transmitting and receiving various types of data associated with the methods described herein. In these contexts,D&S Ref. No.: QCM2404871WOQualcomm Ref. No.: 2404871WO19“transmiting” may refer to various mechanisms of outputting data, such as outputting data from data source 312, scheduler 344, memory 342, transmit processor 320, controller / processor 340, TX MIMO processor 330, transceivers 332a-t, antenna 334a-t, and / or other aspects described herein. Similarly, “receiving” may refer to various mechanisms of obtaining data, such as obtaining data from antennas 334a-t, transceivers 332a-t, RX MIMO detector 336, controller / processor 340, receive processor 338, scheduler 344, memory 342, and / or other aspects described herein.

[0086] In various aspects, UE 104 may likewise be described as transmitting and receiving various types of data associated with the methods described herein. In these contexts, “transmitting” may refer to various mechanisms of outputting data, such as outputting data from data source 362, memory 382, transmit processor 364, controller / processor 380, TX MIMO processor 366, transceivers 354a-t, antenna 352a-t, and / or other aspects described herein. Similarly, “receiving” may refer to various mechanisms of obtaining data, such as obtaining data from antennas 352a-t, transceivers 354a-t, RX MIMO detector 356, controller / processor 380, receive processor 358, memory 382, and / or other aspects described herein.

[0087] In some aspects, a processor may be configured to perform various operations, such as those associated with the methods described herein, and transmit (output) to or receive (obtain) data from another interface that is configured to transmit or receive, respectively, the data.

[0088] In various aspects, artificial intelligence (Al) processors 318 and 370 may perform Al processing for BS 102 and / or UE 104, respectively. The Al processor 318 may include Al accelerator hardware or circuitry such as one or more neural processing units (NPUs), one or more neural network processors, one or more tensor processors, one or more deep learning processors, etc. The Al processor 370 may likewise include Al accelerator hardware or circuitry. As an example, the Al processor 370 may perform AI- based beam management, Al-based channel state feedback (CSF), Al-based antenna tuning, and / or Al-based positioning (e.g., non-line of sight positioning prediction). In some cases, the Al processor 318 may process feedback from the UE 104 (e.g., CSF) using hardware accelerated Al inferences and / or Al training. The Al processor 318 may decode compressed CSF from the UE 104, for example, using a hardware accelerated Al inference associated with the CSF. In certain cases, the Al processor 318 may performD&S Ref. No.: QCM2404871WOQualcomm Ref. No.: 2404871WO20 certain RAN-based functions including, for example, network planning, network performance management, energy-efficient network operations, etc.

[0089] FIGS. 4A, 4B, 4C, and 4D depict aspects of data structures for a wireless communications network, such as wireless communications network 100 of FIG. 1.

[0090] In particular, FIG. 4A is a diagram 400 illustrating an example of a first subframe within a 5G (e.g., 5GNR) frame structure, FIG. 4B is a diagram 430 illustrating an example of DL channels within a 5G subframe, FIG. 4C is a diagram 450 illustrating an example of a second subframe within a 5G frame structure, and FIG. 4D is a diagram 480 illustrating an example of UL channels within a 5G subframe.

[0091] Wireless communications systems may utilize orthogonal frequency division multiplexing (OFDM) with a cyclic prefix (CP) on the uplink and downlink. Such systems may also support half-duplex operation using time division duplexing (TDD). OFDM and single-carrier frequency division multiplexing (SC-FDM) partition the system bandwidth (e.g., as depicted in FIGS. 4B and 4D) into multiple orthogonal subcarriers. Each subcarrier may be modulated with data. Modulation symbols may be sent in the frequency domain with OFDM and / or in the time domain with SC-FDM.

[0092] A wireless communications frame structure may be frequency division duplex (FDD), in which, for a particular set of subcarriers, subframes within the set of subcarriers are dedicated for either DL or UL. Wireless communications frame structures may also be time division duplex (TDD), in which, for a particular set of subcarriers, subframes within the set of subcarriers are dedicated for both DL and UL.

[0093] In FIG. 4A and 4C, the wireless communications frame structure is TDD where D is DL, U is UL, and X is flexible for use between DL / UL. UEs may be configured with a slot format through a received slot format indicator (SFI) (dynamically through DL control information (DCI), or semi-statically / statically through radio resource control (RRC) signaling). In the depicted examples, a 10 ms frame is divided into 10 equally sized 1 ms subframes. Each subframe may include one or more time slots. In some examples, each slot may include 12 or 14 symbols, depending on the cyclic prefix (CP) type (e.g., 12 symbols per slot for an extended CP or 14 symbols per slot for a normal CP). Subframes may also include mini-slots, which generally have fewer symbols than an entire slot. Other wireless communications technologies may have a different frame structure and / or different channels.D&S Ref. No.: QCM2404871WOQualcomm Ref. No.: 2404871WO21

[0094] In certain aspects, the number of slots within a subframe (e.g., a slot duration in a subframe) is based on a numerology, which may define a frequency domain subcarrier spacing and symbol duration as further described herein. In certain aspects, given a numerology p, there are 2gslots per subframe. Thus, numerologies (p) 0 to 6 may allow for 1, 2, 4, 8, 16, 32, and 64 slots, respectively, per subframe. In some cases, the extended CP (e.g., 12 symbols per slot) may be used with a specific numerology, e.g., numerology 2 allowing for 4 slots per subframe. The subcarrier spacing and symbol length / duration are a function of the numerology. The subcarrier spacing may be equal to 211x 15 kHz, where p is the numerology 0 to 6. As an example, the numerology p = 0 corresponds to a subcarrier spacing of 15 kHz, and the numerology p = 6 corresponds to a subcarrier spacing of 960 kHz. The symbol length / duration is inversely related to the subcarrier spacing. FIGS. 4A, 4B, 4C, and 4D provide an example of a slot format having 14 symbols per slot (e.g., a normal CP) and a numerology p = 2 with 4 slots per subframe. In such a case, the slot duration is 0.25 ms, the subcarrier spacing is 60 kHz, and the symbol duration is approximately 16.67 ps.

[0095] As depicted in FIGS. 4A, 4B, 4C, and 4D, a resource grid may be used to represent the frame structure. Each time slot includes a resource block (RB) (also referred to as physical RBs (PRBs)) that extends, for example, 12 consecutive subcarriers. The resource grid is divided into multiple resource elements (REs). The number of bits carried by each RE depends on the modulation scheme including, for example, quadrature phase shift keying (QPSK) or quadrature amplitude modulation (QAM).

[0096] As illustrated in FIG. 4A, some of the REs carry reference (pilot) signals (RS) for a UE (e.g., UE 104 of FIGS. 1 and 3). The RS may include demodulation RS (DMRS) and / or channel state information reference signals (CSI-RS) for channel estimation at the UE. The RS may also include beam measurement RS (BRS), beam refinement RS (BRRS), and / or phase tracking RS (PT-RS).

[0097] FIG. 4B illustrates an example of various DE channels within a subframe of a frame. The physical downlink control channel (PDCCH) carries DCI within one or more control channel elements (CCEs), each CCE including, for example, nine RE groups (REGs), each REG including, for example, four consecutive REs in an OFDM symbol.D&S Ref. No.: QCM2404871WOQualcomm Ref. No.: 2404871WO22

[0098] A primary synchronization signal (PSS) may be within symbol 2 of particular subframes of a frame. The PSS is used by a UE (e.g., 104 of FIGS. 1 and 3) to determine subframe / symbol timing and a physical layer identity.

[0099] A secondary synchronization signal (SSS) may be within symbol 4 of particular subframes of a frame. The SSS is used by a UE to determine a physical layer cell identity group number and radio frame timing.

[0100] Based on the physical layer identity and the physical layer cell identity group number, the UE can determine a physical cell identifier (PCI). Based on the PCI, the UE can determine the locations of the aforementioned DMRS. The physical broadcast channel (PBCH), which carries a master information block (MIB), may be logically grouped with the PSS and SSS to form a synchronization signal (SS) / PBCH block (SSB), and in some cases, referred to as a synchronization signal block (SSB). The MIB provides a number of RBs in the system bandwidth and a system frame number (SFN). The physical downlink shared channel (PDSCH) carries user data, broadcast system information not transmitted through the PBCH such as system information blocks (SIBs), and / or paging messages.

[0101] As illustrated in FIG. 4C, some of the REs carry DMRS (indicated as R for one particular configuration, but other DMRS configurations are possible) for channel estimation at the base station. The UE may transmit DMRS for the PUCCH and DMRS for the PUS CH. The PUS CH DMRS may be transmitted, for example, in the first one or two symbols of the PUSCH. The PUCCH DMRS may be transmitted in different configurations depending on whether short or long PUCCHs are transmitted and depending on the particular PUCCH format used. UE 104 may transmit sounding reference signals (SRS). The SRS may be transmitted, for example, in the last symbol of a subframe. The SRS may have a comb structure, and a UE may transmit SRS on one of the combs. The SRS may be used by a base station for channel quality estimation to enable frequency-dependent scheduling on the UE.

[0102] FIG. 4D illustrates an example of various UL channels within a subframe of a frame. The PUCCH may be located as indicated in one configuration. The PUCCH carries uplink control information (UCI), such as scheduling requests, a channel quality indicator (CQI), a precoding matrix indicator (PMI), a rank indicator (RI), and HARQD&S Ref. No.: QCM2404871WOQualcomm Ref. No.: 2404871WO23ACK / NACK feedback. The PUSCH carries data, and may additionally be used to carry a buffer status report (BSR), a power headroom report (PHR), and / or UCI.Example Artificial Intelligence for Wireless Communications

[0103] Certain aspects described herein may be implemented, at least in part, using some form of artificial intelligence (Al), e.g., the process of using a machine learning (ML) model to infer or predict output data based on input data. An example ML model may include a mathematical representation of one or more relationships among various objects to provide an output representing one or more predictions or inferences. Once an ML model has been trained, the AI / ML model may be deployed to process data that may be similar to, or associated with, all or part of the training data and provide an output representing one or more predictions or inferences based on the input data.

[0104] ML is often characterized in terms of types of learning that generate specific types of learned models that perform specific types of tasks. For example, different types of machine learning include supervised learning, unsupervised learning, semi-supervised learning, and reinforcement learning.

[0105] Supervised learning algorithms generally model relationships and dependencies between input features (e.g., a feature vector) and one or more target outputs. Supervised learning uses labeled training data, which are data including one or more inputs and a desired output. Supervised learning may be used to train models to perform tasks like classification, where the goal is to predict discrete values, or regression, where the goal is to predict continuous values. Some example supervised learning algorithms include nearest neighbor, naive Bayes, decision trees, linear regression, support vector machines (SVMs), and artificial neural networks (ANNs).

[0106] Unsupervised learning algorithms work on unlabeled input data and train models that take an input and transform it into an output to solve a practical problem. Examples of unsupervised learning tasks are clustering, where the output of the model may be a cluster identification, dimensionality reduction, where the output of the model is an output feature vector that has fewer features than the input feature vector, and outlier detection, where the output of the model is a value indicating how the input is different from a typical example in the dataset. An example unsupervised learning algorithm is k- Means.D&S Ref. No.: QCM2404871WOQualcomm Ref. No.: 2404871WO24

[0107] Semi-supervised learning algorithms work on datasets containing both labeled and unlabeled examples, where often the quantity of unlabeled examples is much higher than the number of labeled examples. However, the goal of a semi-supervised learning is that of supervised learning. Often, a semi-supervised model includes a model trained to produce pseudo-labels for unlabeled data that is then combined with the labeled data to train a second classifier that leverages the higher quantity of overall training data to improve task performance.

[0108] Reinforcement learning algorithms use observations gathered by an agent from an interaction with an environment to take actions that may maximize a reward or minimize a risk. Reinforcement learning is a continuous and iterative process in which the agent learns from its experiences with the environment until it explores, for example, a full range of possible states. An example type of reinforcement learning algorithm is an adversarial network. Reinforcement learning may be particularly beneficial when used to improve or attempt to optimize a behavior of a model deployed in a dynamically changing environment, such as a wireless communication network.

[0109] AI / ML models may be deployed in one or more devices (e.g., network entities such as base station(s) and / or user equipment(s)) to support various wired and / or wireless communication aspects of a communication system. For example, an AI / ML model may be trained to identify patterns and relationships in data corresponding to a network, a device, an air interface, or the like. An AI / ML model may improve operations relating to one or more aspects, such as transceiver circuitry controls, frequency synchronization, timing synchronization, channel state estimation, channel equalization, channel state feedback, modulation, demodulation, device positioning, transceiver tuning, beamforming, signal coding / decoding, network routing, load balancing, and energy conservation (to name just a few) associated with communications devices, services, and / or networks. Al-enhanced transceiver circuitry controls may include, for example, filter tuning, transmit power controls, gain controls (including automatic gain controls), phase controls, power management, and the like.

[0110] Aspects described herein may describe the performance of certain tasks and the technical solution of various technical problems by application of a specific type of AI / ML model, such as an ANN. It should be understood, however, that other type(s) of AI / ML models may be used in addition to or instead of an ANN. An AI / ML model may be an example of an AI / ML model, and any suitable AI / ML model may be used inD&S Ref. No.: QCM2404871WOQualcomm Ref. No.: 2404871WO25 addition to or instead of any of the AI / ML models described herein. Hence, unless expressly recited, subject matter regarding an AI / ML model is not necessarily intended to be limited to just an ANN solution or machine learning. Further, it should be understood that, unless otherwise specifically stated, terms such “Al model,” “ML model,” “AI / ML model,” “trained ML model,” and the like are intended to be interchangeable.

[0111] FIG. 5 is a diagram illustrating an example Al architecture 500 that may be used for Al-enhanced wireless communications. As illustrated, the Al architecture 500 includes multiple logical entities, such as a model training host 502, a model inference host 504, one or more data sources 506, and an agent 508. The Al architecture 500 may be used in any of various use cases or functionalities for wireless communications, such as channel state information (CSI) feedback compression, CSI feedback prediction, or beam management.

[0112] The model inference host 504, in the Al architecture 500, is configured to run an ML model based on inference data 512 provided by data source(s) 506. The model inference host 504 may produce an output 514 (e.g., a prediction or inference, such as predicted or compressed CSI, or recommended beams) based on the inference data 512. The output 514 is then provided as input to the agent 508.

[0113] The agent 508 may be an element or an entity of a wireless communication system including, for example, a RAN, a wireless local area network, a D2D communications system, etc. As an example, the agent 508 may be a user equipment (e.g., UE 104 in FIG. 1), abase station (e.g., the BS 102 in FIG. 1 or any disaggregated network entity thereof including a CU, a DU, and / or a RU), an access point, a wireless station, or a RIC in a cloud-based RAN, among some examples. In some aspects, the type of agent 508 may depend on the type of tasks performed by the model inference host 504, the type of inference data 512 provided to model inference host 504, and / or the type of output 514 produced by model inference host 504. For example, if output 514 from the model inference host 504 is associated with CSI prediction or compression, the agent 508 may be or include a UE or a BS. If the output 514 is associated with beam management, the agent 508 may be a UE, BS, or a specific part of the BS such as the RU or DU.

[0114] After the agent 508 receives output 514 from the model inference host 504, agent 508 may determine whether to act based on the output. For example, if agent 508D&S Ref. No.: QCM2404871WOQualcomm Ref. No.: 2404871WO26 is a UE and the output from model inference host 504 is compressed CSI, the agent 508 may transmit the compressed CSI to a subject of action 510 that is a BS, which reduces uplink signaling overhead. As another example, if the agent 508 is a BS and the output 514 is a recommended beam or beam pair for communication with a UE, the agent 508 may send a beam switch command to a subject of action 510 that is the UE, enabling the UE to modify the transmit and / or receive beams.

[0115] The data sources 506 may be configured for collecting data that is used as training data 516 for training an AI / ML model, or as inference data 512 for feeding an AI / ML model inference operation. In particular, the data sources 506 may collect data from any of various entities (e.g., the UE and / or the BS), which may include the subject of action 510, and provide the collected data to a model training host 502 for AI / ML model training. For example, the data sources 506 may collect beam measurement reports and CSI measurements from UEs and BSs to be used as training data 516 for beam management and CSI compression / prediction models. The model training host 502 may use this data to train and update the models, which are then deployed at the model inference host 504.

[0116] In certain aspects, the model training host 502 may be deployed at or with a different entity than that in which the model inference host 504 is deployed. In certain aspects, the model training host 502 may be deployed at or with the same entity than that in which the model inference host 504 is deployed. For example, in order to offload model training processing, which can impact the performance of the model inference host 504, the model training host 502 may be deployed at a model server as further described herein. Further, in some cases, training and / or inference may be distributed amongst devices in a decentralized or federated fashion.

[0117] In some aspects, an AI / ML model is deployed at or on a network entity (e.g., such as BS 102 in FIG. 1) for beam management. More specifically, a model inference host, such as model inference host 504 in FIG. 5, may be deployed at or on the network entity for predicting the best beam(s) for communication with a UE. For example, the UE (subject of action 510) may measure reference signals on different beams and report measurements for a subset of beams (e.g., the best M out of N beams) to the BS (agent 508). The model inference host 504 at the BS then uses an AI / ML model to predict the quality of all N beams based on the reported measurements, and selects the best beam(s) for subsequent communication with the UE.D&S Ref. No.: QCM2404871WOQualcomm Ref. No.: 2404871WO 1

[0118] In some other aspects, an AI / ML model is deployed at or on a UE (e.g., such as UE 104 in FIG. 1) for beam management. More specifically, a model inference host, such as model inference host 504 in FIG. 5, may be deployed at or on the UE for selecting the best beam(s) for communication with a BS. For instance, the UE (agent 508) measures reference signals on different beams and the model inference host 504 uses an AI / ML model to predict the quality of the beams. The UE then selects the best beam(s) based on the predictions and sends a beam recommendation or request to the BS (subject of action 510) to modify transmit beams and / or receive beams.

[0119] The choice between the different beam management approaches (e.g., beam selection at the BS vs. beam recommendation from the UE) depends on various factors such as the available computational resources at the UE and BS, the signaling overhead, and the required speed and flexibility of beam adaptation. The Al architecture 500 supports both approaches by deploying the model inference host 504 at either the UE or BS as needed.

[0120] Furthermore, the Al architecture 500 allows for different types of AI / ML models to be used for beam management. For example, the model inference host 504 may employ a deep neural network (DNN) to learn the mapping between beam measurements and beam qualities, or a reinforcement learning (RL) model to learn beam adaptation policies based on the observed network state and performance. The specific model architecture and training approach can be customized based on the scenario and requirements.Aspects of Artificial Intelligence Model Training

[0121] There are a variety of model training techniques and processes that may be used prior to, or at some point following, deployment of an AI / ML model.

[0122] As part of a model development process, information in the form of applicable training data may be gathered or otherwise created for use in training an AI / ML model accordingly. For example, training data may be gathered or otherwise created regarding information associated with received / transmitted signal strengths, interference, and resource usage data, as well as any other relevant data that might be useful for training a model to address one or more problems or issues in a communication system. In certain instances, all or part of the training data may originate in one or more UEs, one or more network entities, or one or more other devices in a wireless communication system. InD&S Ref. No.: QCM2404871WOQualcomm Ref. No.: 2404871WO28 some cases, all or part of the training data may be aggregated from multiple sources (e.g., one or more UEs, one or more network entities, the Internet, etc.). For example, selforganizing network (SON) techniques or minimization of drive test (MDT) techniques may be adapted to support collection of data for AI / ML model applications. In another example, training data may be generated or collected online, offline, or both online and offline by a UE, network entity, or other device(s). All or part of such training data may be transferred or shared (in real or near-real time), such as through store and forward functions or the like. “Offline training” may refer to creating and using a static training dataset, e.g., in a batched manner, whereas “online training” may refer to a real-time or near-real-time collection and use of training data. For example, an AI / ML model at a network device (e.g., a UE) may be trained and / or fine-tuned using online or offline training. For offline training, data collection and training can occur in an offline manner at the network side (e.g., at a base station or other network entity) or at the UE side. For online training, the training of a UE-side AI / ML model may be performed locally at the UE or by a server device (e.g., a server hosted by a UE vendor) in a real-time or near- real-time manner based on data provided to the server device from the UE.

[0123] In certain instances, all or part of the training data may be shared within a wireless communication system, or even shared (or obtained from) outside of the wireless communication system.

[0124] Once an AI / ML model has been trained with training data, its performance may be evaluated. In some scenarios, evaluation / verification tests may use a validation dataset, which may include data not in the training data, to compare the model’s performance to baseline or other benchmark information. If model performance is deemed unsatisfactory, it may be beneficial to fine-tune the model, e.g., by changing its architecture, re-training it on the data, or using different optimization techniques. Once a model’s performance is deemed satisfactory, the model may be deployed. In certain instances, a model may be updated in some manner. For example, all or part of the model may be changed or replaced, or undergo further training, just to name a few examples.

[0125] The model management host 518 may oversee the operation and monitoring the performance of the AI / ML models or functionalities. Using monitoring data, the model management host 518 makes decisions to ensure proper inference operation. In some aspects, model management host 518 can provide management instructions 520 to control the model inference host 504. These instructions may include selecting,D&S Ref. No.: QCM2404871WOQualcomm Ref. No.: 2404871WO29 activating, deactivating or switching between different AI / ML models or functionalities. The model management host 518 can also decide to fall back to non- AI / ML operation if needed based on the monitored performance. Additionally, model management host 518 may send performance feedback or retraining requests to the model training host 502 (via model inference host 504 for example) to initiate retraining or updating of the models when necessary to improve performance.

[0126] As part of a training process for a model, parameters affecting the functioning of the artificial neurons and layers may be adjusted. For example, backpropagation techniques may be used to train the model by iteratively adjusting weights and / or biases of certain artificial neurons associated with errors between a predicted output of the model and a desired output that may be known or otherwise deemed acceptable. Backpropagation may include a forward pass, a loss function, a backward pass, and a parameter update that may be performed in training iteration. The process may be repeated for a certain number of iterations for each set of training data until the weights of the artificial neurons / layers are adequately tuned.

[0127] Backpropagation techniques associated with a loss function may measure how well a model is able to predict a desired output for a given input. An optimization algorithm may be used during a training process to adjust weights and / or biases to reduce or minimize the loss function which should improve the performance of the model. There are a variety of optimization algorithms that may be used along with backpropagation techniques or other training techniques. Some initial examples include a gradient descent based optimization algorithm and a stochastic gradient descent based optimization algorithm. A stochastic gradient descent (or ascent) technique may be used to adjust weights / biases in order to minimize or otherwise reduce a loss function. A mini-batch gradient descent technique, which is a variant of gradient descent, may involve updating weights / biases using a small batch of training data rather than the entire dataset. A momentum technique may accelerate an optimization process by adding a momentum term to update or otherwise affect certain weights / biases.

[0128] An adaptive learning rate technique may adjust a learning rate of an optimization algorithm associated with one or more characteristics of the training data. A batch normalization technique may be used to normalize inputs to a model in order to stabilize a training process and potentially improve the performance of the model.D&S Ref. No.: QCM2404871WOQualcomm Ref. No.: 2404871WO30

[0129] A “dropout” technique may be used to randomly drop out some of the artificial neurons from a model during a training process, e.g., in order to reduce overfitting and potentially improve the generalization of the model.

[0130] An “early stopping” technique may be used to stop an on-going training process early, such as when a performance of the model using a validation dataset starts to degrade.

[0131] Another example technique includes data augmentation to generate additional training data by applying transformations to all or part of the training information.

[0132] A transfer learning technique may be used which involves using a pre-trained model as a starting point for training a new model, which may be useful when training data is limited or when there are multiple tasks that are related to each other.

[0133] A multi-task learning technique may be used which involves training a model to perform multiple tasks simultaneously to potentially improve the performance of the model on one or more of the tasks. Hyperparameters or the like may be input and applied during a training process in certain instances.

[0134] Another example technique that may be useful with regard to an AI / ML model is a “pruning” technique. A pruning technique, which may be performed during a training process or after a model has been trained, involves the removal of unnecessary (e.g., because they have no impact on the output) or less necessary (e.g., because they have negligible impact on the output), or possibly redundant features from a model. In certain instances, a pruning technique may reduce the complexity of a model or improve efficiency of a model without undermining the intended performance of the model.Pruning techniques may be particularly useful in the context of wireless communication, where the available resources (such as power and bandwidth) may be limited. Some example pruning techniques include a weight pruning technique, a neuron pruning technique, a layer pruning technique, a structural pruning technique, and a dynamic pruning technique. Pruning techniques may, for example, reduce the amount of data corresponding to a model that may need to be transmitted or stored.

[0135] Weight pruning techniques may involve removing some of the weights from a model. Neuron pruning techniques may involve removing some neurons from a model. Layer pruning techniques may involve removing some layers from a model. StructuralD&S Ref. No.: QCM2404871WOQualcomm Ref. No.: 2404871WO31 pruning techniques may involve removing some connections between neurons in a model. Dynamic pruning techniques may involve adapting a pruning strategy of a model associated with one or more characteristics of the data or the environment. For example, in certain wireless communication devices, a dynamic pruning technique may more aggressively prune a model for use in a low-power or low-bandwidth environment, and less aggressively prune the AI / ML model for use in a high-power or high-bandwidth environment. In certain aspects, pruning techniques also may be applied to training data, e.g., to remove outliers, etc. In some implementations, pre-processing techniques directed to all or part of a training dataset may improve model performance or promote faster convergence of a model. For example, training data may be pre-processed to change or remove unnecessary data, extraneous data, incorrect data, or otherwise identifiable data. Such pre-processed training data may, for example, lead to a reduction in potential overfitting, or otherwise improve the performance of the trained model.

[0136] One or more of the example training techniques presented above may be employed as part of a training process. As above, some example training processes that may be used to train an AI / ML model include supervised learning, unsupervised learning, semi-supervised learning, and reinforcement learning technique.

[0137] Decentralized, distributed, or shared learning, such as federated learning, may enable training on data distributed across multiple devices or organizations, without the need to centralize data or the training. Federated learning may be particularly useful in scenarios where data is sensitive or subject to privacy constraints, or where it is impractical, inefficient, or expensive to centralize data. In the context of wireless communication, for example, federated learning may be used to improve performance by allowing an AI / ML model to be trained on data collected from a wide range of devices and environments. For example, an AI / ML model may be trained on data collected from a large number of wireless devices in a network, such as distributed wireless communication nodes, smartphones, or internet-of-things (loT) devices, to improve the network’s performanceAspects Related to Performing Hierarchical Monitoring of Al Models in Wireless Communications Systems

[0138] FIG. 6 illustrates an example of a monitoring architecture 600 in accordance with aspects of the present disclosure. The monitoring architecture 600 may include aD&S Ref. No.: QCM2404871WOQualcomm Ref. No.: 2404871WO32 model 602 that receives a model input 604 and generates a model output 606. The monitoring architecture 600 may include one or more monitoring agents 608, 610, and 612, each monitoring agent implementing a different monitoring configuration 614, 616, and 618 respectively. Accordingly, the monitoring agents 608, 610, and 612 may monitor a performance of the model 602 according to the respective monitoring configuration 614, 616, and / or 618.

[0139] In some aspects, and as will be described further with respect to FIGS. 7-10, one or more of the monitoring agents 608, 610, 612 may be invoked in a stepwise manner based on first metric 620, second metric 622, and third metric 624. For example, monitoring agent 608 may be initiated first to monitor the model 602 using the monitoring configuration 614. Based on a value of the first metric 620 computed by the monitoring agent 608, a trigger condition may be satisfied to invoke monitoring agent 610 to perform different monitoring of the model 602 using the monitoring configuration 616. Similarly, based on a value of the second metric 622 computed by the monitoring agent 610, a different trigger condition may be satisfied to invoke monitoring agent 612 to perform different monitoring of the model 602 using the monitoring configuration 616. In some aspects, based on a value of the third metric 624 computed by the monitoring agent 612, a trigger condition may be satisfied to invoke additional monitoring and / or a monitoring termination process. For example, if a value of the third metric 624 exceeds a threshold value, the monitoring of the model 602 may stop and a notification may be generated to alert one or more users.

[0140] In certain aspects, the first metric 620, second metric 622, and third metric 624 allow the monitoring of the model 602 to be performed in increasing levels of complexity and / or granularity. For example, monitoring of the model 602 may start with a ground truth- free approach implemented by the monitoring agent 608. A complexity of the ground truth- free approach may be increased by monitoring agent 610 based on the trigger condition being satisfied by the first metric 620. In some aspects, the complexity of monitoring the model 602 may be increased to a ground truth-based approach by monitoring agent 612 based on a trigger condition being satisfied by the second metric 622. In some aspects, the metrics 620, 622, and 624 may provide different measures of the model 602, each capturing a distinct aspect of the model's performance or behavior. For example, metric 620 may focus on the model's accuracy, while metric 622 may evaluate the model's resource utilization, and metric 624 may assess the model'sD&S Ref. No.: QCM2404871WOQualcomm Ref. No.: 2404871WO33 robustness to input variations. In a ground truth- free approach, a monitoring agent, such as monitoring agent 608 and / or 610, does not rely on ground truth data for monitoring the model 602. Rather, the monitoring agent 608 and / or 610 may generate a first metric 620 and / or second metric 622 based on at least one of the model input 604 and the model output 606, or other observable metrics. In some aspects, the ground truth-free approach may be less computationally expensive than a ground truth-based approach. The ground truth-free approach may be used to detect anomalies from an expected performance pattern associated with the model 602. The ground truth- free approach implemented by the monitoring agent 608 may involve analyzing statistical properties of the model input 604 and / or model output 606, providing a different measure of the model's performance compared to ground truth-based approaches. For example, the monitoring agent 608 may evaluate the model's performance based on domain-specific KPIs, such as BFER, packet loss rate, BER, throughput, spectral efficiency, latency, or jitter, without relying on ground truth data.

[0141] In some aspects, the first level of monitoring granularity and the second level of monitoring granularity refer to the level of detail and complexity used in monitoring the machine learning model 602. For example, the first level of monitoring granularity may be associated with the first set of one or more parameters, which may include basic or high-level metrics that provide an overview of the model's performance. For example, the first set of parameters may include metrics such as overall accuracy, error rate, or throughput. In contrast, the second level of monitoring granularity may be associated with the second set of one or more parameters, which may include at least one parameter that is not included in the first set. The second set of parameters may provide a more detailed and comprehensive analysis of the model's performance, considering factors such as resource utilization, robustness to input variations, or domain-specific metrics. By using different levels of monitoring granularity, the system can adaptively monitor the machine learning model 602 based on its performance and the trigger conditions, allowing for efficient and effective monitoring.

[0142] In some aspects, the first metric 620 and the second metric 622 may be configured to provide different measures of the machine learning model 602. The first metric 620 may provide a measure of the model's performance based on the first set of one or more parameters, which may be associated with the first level of monitoring granularity. In contrast, the second metric 622 may provide a different measure of theD&S Ref. No.: QCM2404871WOQualcomm Ref. No.: 2404871WO34 model's performance based on the second set of one or more parameters, which may be associated with the second level of monitoring granularity. The second metric 622 may capture aspects of the model's performance that are not captured by the first metric 620, providing a more comprehensive and detailed analysis of the model's behavior. For example, while the first metric 620 may focus on overall accuracy or error rate, the second metric 622 may evaluate the model's resource utilization, robustness to input variations, or domain-specific metrics such as BLER, packet loss rate, BER, throughput, spectral efficiency, latency, or jitter. By providing different measures of the model's performance, the first metric 620 and the second metric 622 may enable a more effective and adaptive monitoring of the machine learning model 602.

[0143] As an example, the monitoring configuration 614 implemented by the monitoring agent 608 may implement a ground truth-free approach that involves analyzing a statistical property of the model input 604 and / or model output 606, such as a data distribution or a range, to detect an anomaly without using ground truth data. As another example, the monitoring configuration 616 implemented by the monitoring agent 610 may implement a more computationally complex ground truth- free approach that may involve analyzing the additional input data 628. In some aspects additional input data 628 may be the same as or similar to additional input data 828 of FIG. 8. In some aspects, the monitoring agent 610 may analyze additional input data 628 as part of the more computationally complex ground truth-free approach specified by the monitoring configuration 616. The additional input data 628 may include supplementary information relevant to the model 602 and its performance, such as domain-specific data, historical data, or contextual information not directly used as the model input 604. By considering this additional input data 628, the monitoring agent 610 can perform a more comprehensive evaluation of the model 602 without relying on ground truth data.In a ground truth-based approach, a monitoring agent, such as monitoring agent 612, may rely on ground truth data, such as expected model output 626, for monitoring the model 602. The monitoring agent 612 may generate a third metric 624 based on a comparison of the model output 606 and the expected model output 626. In some aspects, the ground truthbased approach may be more computationally expensive than the ground truth-free approach. The ground truth-based approach may be used to detect deviations in the performance of the model 602 from an expected performance represented by the expected model output 626. In contrast to the ground truth-free approach, the ground truth-basedD&S Ref. No.: QCM2404871WOQualcomm Ref. No.: 2404871WO35 approach implemented by the monitoring agent 610 relies on the expected model output 626, providing a different measure of the model's performance that directly compares the model output 606 with the expected output.

[0144] In some aspects, the monitoring of the model 602 may be followed by a model adaptation process based on the obtained metrics. For example, if the second metric 622 indicates a degradation in the model's performance or a deviation from the expected behavior, it may trigger a retraining of the model 602 or an update of its parameters. The retraining process may involve using the second metric 622 as a feedback signal to adjust the model's internal weights, thresholds, or other learnable parameters. Alternatively, the model's parameters may be directly updated based on the second metric 622, without a full retraining. This adaptation process aims to improve the model's performance and bring it closer to the desired behavior. The updated or retrained model can then be deployed back into the system, and the monitoring process can continue to assess its performance using the available metrics.

[0145] In some aspects, the ground truth-free approach used by the monitoring agent 608 operates as an initial, lightweight monitoring step. If the second metric 622 computed by the monitoring agent 608 indicates a potential issue, one or more monitoring agents 610 and / or 612 may be initiated and a more computationally complex and monitoring approach can be employed.

[0146] In certain aspects, the first metric 620 and the second metric 622 may be different from each other. That is, inputs used by the monitoring agents 608 and 610 may be the same (or different), however, the actual values or subsets of the inputs used by the monitoring agents 608 and 610 may be different. For example, the monitoring agent 608 may evaluate a first subset of the model input 604, while the monitoring agent 610 may evaluate a second subset such as a larger subset of the model input 604.

[0147] For example, where the model 602 is used to predict the channel quality indicator (CQI) in a wireless communication system, the monitoring agent 608 may monitor the performance of the model 602 using a basic set of metrics, such as an SNR and / or a block error rate (BFER), which may be specified in the monitoring configuration 614. Alternatively, the monitoring agent 610 may monitor the performance of the model 602 using an expanded set of metrics, including a spectral efficiency, throughput, and latency, as specified in the monitoring configuration 616.D&S Ref. No.: QCM2404871WOQualcomm Ref. No.: 2404871WO36

[0148] In some aspects, by combining the monitoring agents 608 and 610, a more comprehensive assessment of the performance of the model 602 can be achieved. That is, the monitoring agent 608 may be based on metrics that directly impact CQI prediction, while the monitoring agent 610 may be based on additional metrics that provide a broader perspective on the model’s effectiveness in the overall wireless communication system.

[0149] The combination of monitoring agents 608 and 610 offers several benefits over using the monitoring agent 608 and monitoring agent 610 individually. First, the combination of monitoring agents 608 and 610 enables an evaluation of the performance of the model 602, considering both basic and additional metrics. This allows for a more thorough understanding of how well the model 602 is functioning in different aspects of the system. Second, the combination of monitoring agents 608 and 610 can help identify potential issues that may not be apparent when relying on a single set of metrics. For instance, if the basic metrics monitored by monitoring agent 608 indicate satisfactory performance, but the advanced metrics monitored by a monitoring agent 610 reveal reduction in throughput or an increase in latency, the first metric 620 and second metric 622 may suggest that the model 602 may be optimizing for the basic metrics at the expense of overall system performance.

[0150] Furthermore, the monitoring process can be customized by adjusting the relative importance or frequency of each monitoring agent. For example, in scenarios where computational resources are limited, the monitoring agent 608 can be used for more frequent evaluations, while the monitoring agent 610 can be used less frequently to provide periodic in-depth assessments.

[0151] Furthermore, the monitoring architecture 600 may utilize the monitoring agent 612 in combination with either or both of the monitoring agents 608 and 610. For example, the monitoring agent 612 can be used in conjunction with the monitoring agent 610, where the second metric 622 may invoke the monitoring agent 612 for ground truthbased monitoring (e.g., based on satisfying a trigger condition). In some aspects, all three monitoring agents 608, 610, and 612 can be used together, providing a comprehensive monitoring approach that combines ground truth- free and ground truth-based monitoring techniques.

[0152] As an example, suppose the model 602 predicts an optimal beamforming weights for a multi-antenna base station. In some aspects, the monitoring agent 608 mayD&S Ref. No.: QCM2404871WOQualcomm Ref. No.: 2404871WO37 be initially employed to monitor the performance of the model 602 using a basic set of metrics, such as an SNR and a received signal strength indicator (RSSI). If the first metric 620 satisfies a threshold (indicating potential issues with the performance of the model 602), the monitoring agent 612 may be invoked for a more comprehensive evaluation using a ground truth-based approach.

[0153] In this case, the monitoring agent 612 may compare predicted beamforming weights (e.g., model output 606) and optimal beamforming weights (e.g., expected model output 626) obtained through channel measurements and optimization algorithms. The monitoring agent 612 may calculate the third metric 624, which could be a mean squared error (MSE) between the predicted beamforming weights and the optimal beamforming weights. If the MSE exceeds a threshold, it may indicate that the model 602’ s predictions are deviating significantly from the optimal values, suggesting a potential problem with the performance of the model 602.

[0154] FIG. 7 illustrates an example hierarchical monitoring process 700 for an AI / ML model in accordance with aspects of the present disclosure. The operations of the monitoring process 700 may be performed by an apparatus, such as a BS 102, a UE 104, a device or function of FIG. 2, a model training host 502, a model management host 518, a model inference host 504, or the like. In some aspects, the monitoring process 700 may begin with a start or terminate monitoring step 702. In this step, the apparatus may initiate monitoring (such as based on a trigger 704) or terminate monitoring (such as according to a timer). In some aspects, the monitoring process 700 may be triggered at 704 or initiated based on various conditions, such as expiration of a timer, receipt of a monitoring request from another apparatus, or detection of a metric or key performance indicator (KPI) exceeding a threshold.

[0155] In some aspects, upon initiation, the monitoring process 700 proceeds to a first monitoring step 706. The first monitoring step 706 may be based on a first metric 708. For example, the first monitoring step 706 may involve evaluation of a model’s performance according to the first metric 708. In some examples, the first monitoring step 706 may be performed by a monitoring agent, such as the monitoring agent 608 described with reference to FIG. 6. In some aspects, the first metric is associated with a first level of monitoring granularity. For example, the first level of monitoring granularity may refer to a level of detail or complexity used in monitoring the machine learning model, and it may be associated with a first set of one or more parameters that provide a high-D&S Ref. No.: QCM2404871WOQualcomm Ref. No.: 2404871WO38 level or basic understanding of the model’s performance. For example, the first set of parameters may include metrics such as, but not limited to, an overall accuracy, error rate, or throughput. In some aspects, the monitoring process 700 continues within the first monitoring step 706 until at least one of a trigger condition 712 is satisfied or a first timer expires.

[0156] In some aspects, if the first timer expires without the trigger condition 712 being met, the monitoring process 700 may fall back to the start / terminate monitoring step 702, as shown at 710. This allows the monitoring process 700 to gracefully exit if no issues are detected within a predefined time period. In some aspects, if the trigger condition 712 is satisfied, the monitoring process 700 advances to a second monitoring step 714. The second monitoring step 714 may be based on a second metric 716. For example, the second monitoring step 714 may involve evaluation of a model’s performance according to the second metric 716. Similar to the first monitoring step 706, the second monitoring step 714 may be performed by a monitoring agent, such as the monitoring agent 610 of FIG. 6. In some aspects, the second metric may be monitoring using a second level of monitoring granularity. In some aspects, the second level of monitoring granularity refers to a higher level of detail or complexity compared to the first level of monitoring granularity and may be associated with a second set of one or more parameters that provide a more comprehensive and in-depth analyses of the model’s performance. The second set of parameters may include at least one parameters that is not included in the first set, allowing for a more fine-grained understanding or evaluation of the model’s behavior. For example, the second set of parameters may include metrics such as, but not limited to, resource utilization, robustness to input variations, or domainspecific metrics like BFER, packet loss rate, or latency. The monitoring process 700 may continue within the second monitoring step 714 until a subsequent trigger condition 720 is satisfied or a second timer expires, causing a fallback to the first monitoring step 706 as indicated by 718.

[0157] The process 700 can be performed for any number of monitoring steps. In FIG. 7, there are N monitoring steps. In some aspects, if the trigger condition 720 is satisfied, the monitoring process 700 progresses to an TVth monitoring step 722 based on an TVth metric 724. The monitoring process 700 continues within the TVth monitoring step 722 until a final trigger condition 732 is met or an TVth timer expires. If the TVth timer expires, the monitoring process 700 may fall back to a previous monitoring step, such asD&S Ref. No.: QCM2404871WOQualcomm Ref. No.: 2404871WO39 the second monitoring step 714, as shown at 726. In some aspects, the monitoring process 700 allows for cross-coupling between different monitoring steps. For example, the monitoring process 700 may skip from the first monitoring step 706 directly to the Nth monitoring step 722 via the trigger condition 730 if a condition is satisfied. For example, the trigger condition 730 may be different than the trigger condition 712 and / or the trigger condition 720. In some aspects, the trigger condition 730 may include both trigger condition 712 and trigger condition 720. Alternatively, or in addition, the monitoring process 700 may skip from the Ath monitoring step 722 to the first monitoring step 706 in accordance with another timer, illustrated at 928.

[0158] In some aspects, if the final trigger condition 732 is met, indicating that all monitoring steps have been exhausted without resolving any detected issues, the monitoring process 700 may proceed to an AI / ML model failure step 734. This step may involve generating a notification, alert, or report indicating the failure of the model being monitored.

[0159] In some aspects, if the final trigger condition 732 is met, indicating that all monitoring steps have been exhausted without resolving any detected issues, the monitoring process 700 may proceed to an AI / ML model failure step 734. This step may involve generating a notification, alert, or report indicating the failure of the model being monitored.

[0160] Alternatively, or in addition, the monitoring process 700 may also utilize a reporting condition. The reporting condition may refer to a criterion or set of criteria that determines when an indication associated with the performance of the machine learning model should be signaled. The reporting condition may be the same as or different from the trigger conditions, which may be used to determine when to switch between different levels of monitoring granularity (e.g., from monitoring the first metric to monitoring the second metric).

[0161] While the trigger conditions may be based on the metrics associated with the current level of monitoring granularity, the reporting condition may be based on the metric associated with the highest level of monitoring granularity that has been reached in the monitoring process. For example, if the monitoring process has progressed to monitoring the second metric (step 706), the reporting condition may be based on the second metric, which is associated with the second level of monitoring granularity.D&S Ref. No.: QCM2404871WOQualcomm Ref. No.: 2404871WO40

[0162] In some aspects, the reporting condition may be configured to identify deviations or anomalies in the model's performance that warrant further attention or action. For example, the reporting condition may be satisfied when the metric associated with the highest level of monitoring granularity exceeds a predetermined threshold, indicating a severe degradation in the model's performance. Alternatively, the reporting condition may be satisfied when this metric exhibits a sudden change or an unusual pattern that suggests a potential issue with the model.

[0163] In some aspects, when the reporting condition is satisfied, an indication is signaled to alert relevant parties (e.g., the model developer, the system administrator, or other monitoring components, etc.) about the model's performance issue. This indication may be in the form of a notification, alert, or report, similar to the one generated in the AI / ML model failure step 734. The indication may enable the relevant parties to take appropriate actions, such as model retraining, parameter adjustment, or further investigation, to address the identified performance issue. In some aspects, the indication may be provided to another apparatus such that the another apparatus can perform some action. In aspects, when the another apparatus performs an action, the action may influence or cause one or more parameters associated with monitoring a machine learning model to change.

[0164] The hierarchical nature of the monitoring process 700 allows for progressively more complex and resource-intensive monitoring techniques to be employed as needed. By starting with simpler monitoring metrics and only advancing to more complex metrics when necessary, the monitoring process 700 can efficiently identify and diagnose issues with the model while minimizing unnecessary resource utilization.

[0165] FIG. 8 depicts aspects of an example monitoring architecture 800 in accordance with aspects of the present disclosure. In some aspects, the example monitoring architecture 800 may include a monitoring agent 808 (e.g., monitoring agent 608) configured to receive model input 804 (e.g., model input 604) and model output 806 (e.g., model output 606) and monitor the performance of a model 802 (e.g., model 602).

[0166] In some aspects, the model 802 may be the same as or similar to a model described with respect to the model inference host 504 in FIG. 5. In some examples, the model 802 may be an artificial neural network (ANN), convolutional neural network (CNN), recurrent neural network (RNN), transformer model, or any other suitable AI / MLD&S Ref. No.: QCM2404871WOQualcomm Ref. No.: 2404871WO41 model architecture. In some examples, the model 802 may be used for CSI prediction. A CSI prediction model may utilize learned mappings between past CSI measurements and future CSI values to predict one or more of the following: raw channel matrices, eigenvectors, or other feedback CSI information for future time instances. A CSI prediction model may utilize techniques such as sequence-to-sequence learning or autoregressive modeling to capture temporal dependencies in the CSI data. As another example, the model 802 may be used for performing beam management. A beam management model may utilize learned relationships between beam measurements and optimal beam selections to predict the most suitable beam configuration, such as the top N beams, their associated quality measurements, or beam angles. A beam management model may employ techniques such as classification or ranking to identify the top N beams based on the input measurements.

[0167] In some aspects, the model input 804 may be obtained from a data source, such as the data source(s) 506 described with reference to FIG. 5, and may be inference data 512 (FIG. 5). In some aspects, the model input 804 represents the data that is provided to the model 802 for inference or prediction purposes and may be a primary input used by the model 802 to generate the model output 806. For example, in the context of a CSI prediction model, the model input 804 may include historical CSI measurements, channel quality indicators, or other relevant features that the model uses to predict future CSI values.

[0168] In some aspects, the model output 806 may be the same as or similar to the output 514 described with reference to FIG. 5. That is, the model output 806 may represent the results or predictions generated by the model 802 based on the model input 804. In some aspects, and depending on the model 802, the model output 806 can take various forms.

[0169] In some aspects, the monitoring agent 808 is configured to monitor the performance of the model 802 by analyzing the model input 804, additional input data 828, and / or the model output 806. In some aspects, the monitoring agent 808 may be implemented as part of a BS 102 or UE 104 and / or as part of the EPC 160 and / or 5GC 190 as depicted in FIG. 1. In certain aspects, the monitoring agent 808 may be a standalone entity or integrated into other components of a wireless communication system.D&S Ref. No.: QCM2404871WOQualcomm Ref. No.: 2404871WO42

[0170] In some examples, the monitoring agent 808 may be configured to monitor the performance of the model 802 using a monitoring configuration 814. In some aspects, the monitoring configuration 814 may utilize an unsupervised evaluation metric to perform a ground truth-free evaluation, which may assess the performance of the model 802 without relying on ground truth labels. “Ground truth labels” refer to true or correct values associated with the input data, which are often used to train and evaluate supervised learning models. However, in many real-world scenarios, ground truth labels may not be readily available or may be costly to obtain. Unsupervised evaluation metrics, on the other hand, aim to assess the model’s performance based on the inherent characteristics and patterns in the input data and the model’s output, without requiring explicit labels. These unsupervised evaluation metrics can help detect changes, anomalies, and / or drift in an input data distribution or behavior of the model 802 by comparing the unsupervised evaluation metrics to historical patterns or statistical properties.

[0171] In some aspects, an unsupervised evaluation metric can be derived from the model input 804, the additional input data 828, and / or the model output 806. While the monitoring agent 808 and / or monitoring configuration 814 may be configured to use one or more unsupervised evaluation metrics, other monitoring agents (e.g., monitoring agent 612 of FIG. 6) and / or monitoring configurations (e.g., monitoring configuration 618 of FIG. 6) may utilize supervised evaluation metrics. “Supervised evaluation metrics” refers to ground truth labels that may enable assessment of the performance of a model 802 by comparing the model output 806 to expected model outputs (e.g., expected model output 826). In some aspects, supervised evaluation metrics, such as accuracy, precision, recall, or Fl score, can measure how well the model 802 generalizes to new, labeled data and can provide insights into the predictive capabilities of the model 802. Moreover, the monitoring agent 808 and / or monitoring configuration 814 may switch from an unsupervised evaluation metric (e.g., a metric without labels or that is ground truth-free) to a supervised evaluation metric (e.g., a metric with labels or that is ground truth-based) depending on the availability of labeled data and the specific monitoring requirements. This flexibility allows the monitoring agent 808 to adapt to different scenarios and leverage a most appropriate evaluation metric for assessing the model 802’s performance.

[0172] In some aspects, the monitoring agent 808 may be configured by a monitoring configuration 810 that provides configuration information indicating how the monitoring agent 808 is to perform a monitoring task. In some aspects, the monitoring configurationD&S Ref. No.: QCM2404871WOQualcomm Ref. No.: 2404871WO43810 may specify a type of monitoring (e.g., ground truth- free or ground truth-based) to be conducted and / or one or more metrics to be used and / or calculated to perform the monitoring. In some aspects, the monitoring configuration 810 may be predefined. For example, the monitoring configuration 810 may already be established and / or stored within the monitoring agent 808 or in an accessible database. In some aspects, a predefined monitoring configuration 810 may be based on default settings, historical data, and / or domain expertise for example.

[0173] In some aspects, the monitoring agent 808 may use the monitoring configuration 810 to determine which unsupervised evaluation metrics to compute and how to process the model input 804, additional input data 828, and / or model output 806. For example, if the monitoring configuration 814 specifies calculation of input data metrics, such as calculation of a distribution of CSI measurements, the monitoring agent 808 can analyze the model input 804 accordingly. Moreover, the monitoring configuration 810 may provide a threshold for the monitoring agent 808 to compare a metric, such as the first metric 820.

[0174] In some aspects, the monitoring configuration 810 may be dynamically generated or updated in response to signaling or triggers from other components of a wireless communication system. For example, the signaling could come from various sources, such as signaling from a network entity, signaling from a UE, and / or signaling associated with a model performance trigger. In some aspects, the signaling may indicate how monitoring is to be performed. In an example where signaling is obtained from a network entity, the monitoring configuration 810 may be received from the network entity, such as a base station (e.g., BS 102 of FIG. 1), a centralized controller (e.g., as part of EPC 160 and / or 5GC 190 of FIG. 1), or an entity of FIG. 2. For example, a network entity may send a message to the apparatus implementing monitoring agent 808, indicating a desired monitoring type and a metric. For instance, if a network entity detects a change in the channel conditions or user behavior, the network entity may signal the monitoring agent 808 to adjust the monitoring configuration 810 accordingly. In some instances, a UE (e.g., UE 104 of FIG. 1) may provide a signaling message to the monitoring agent 808 to request a monitoring configuration. For example, if the UE experiences degraded performance or identifies an issue with the prediction of the model 802, the UE may signal the monitoring agent 808 to initiate more detailed monitoring or switch to a different evaluation metric. In some aspects, a UE-driven signaling enablesD&S Ref. No.: QCM2404871WOQualcomm Ref. No.: 2404871WO44 personalized and adaptive monitoring based on individual user needs or conditions at a UE without burdening a network entity.

[0175] In some aspects, the monitoring agent 808 may trigger updates to the monitoring configuration 810 based on an observed model performance. For example, if the unsupervised evaluation metric indicates a significant deviation or anomaly in the model 802’s behavior, the monitoring agent 808 may automatically adjust the monitoring configuration 810 to perform more targeted or intensive monitoring. In some aspects, the self-triggered adaptation allows the monitoring process to respond quickly to detected performance issues by providing a monitoring configuration 810 different from a previous monitoring configuration.

[0176] In some aspects, the adaptation of the monitoring configuration 810 can occur in various ways and can be triggered by different types of signaling or events. For example, a network entity may send an indication to the apparatus implementing the monitoring agent 808 that indicates to update the monitoring configuration 810 based on changes in network conditions or requirements. For instance, if a network entity detects an increase in traffic load or a change in the interference pattern, the network entity may send an indication to the monitoring agent 808 to adjust the monitoring configuration 810 accordingly. The indication may include instructions on how to modify the monitoring configuration 810, such as updating a time window for a metric calculation or changing an anomaly detection threshold.

[0177] In some aspects, a UE may send a signaling message to the monitoring agent 808 to request a change in the monitoring configuration 810 based on a specific need of the UE. For example, if the UE experiences a degradation in a quality of service or identifies an issue with the prediction of the model 802, the UE may signal the monitoring agent 808 to increase the frequency of metric reporting or include additional metrics in the monitoring process. The UE’s signaling message may include changes to the monitoring configuration 810 which can be incorporated by the monitoring agent 808.

[0178] In some aspects, the monitoring agent 808 may trigger an update to the monitoring configuration 810 based on an observed performance of the model 802. For example, if one or more metrics indicate a deviation from an expected behavior of the model 802 or a degraded performance of the model 802, the monitoring agent 808 may automatically adjust the monitoring configuration 810. For instance, if first metric 820D&S Ref. No.: QCM2404871WOQualcomm Ref. No.: 2404871WO45 shows a sudden change in the distribution of CSI measurements, the monitoring agent 808 may adapt the monitoring configuration 810 to include more granular metrics or tighter anomaly detection thresholds to closely monitor the response of the model 802 to a change in the distribution of CSI measurements.

[0179] In some aspects, the monitoring configuration 810 may include information about a type of monitoring to be performed. For example, in the context of FIG. 8, the monitoring configuration 810 may specify one or more unsupervised metrics to be calculated for the model input 804 and / or the model output 806. In some aspects, the monitoring configuration 810 may include parameters such as a time window for metric calculation, a threshold value for anomaly detection, and / or a frequency of metric reporting. That is, in some aspects, the monitoring configuration 810 specifies the type of monitoring, metrics, and the associated parameters. In some aspects, the monitoring configuration 810 can enable a flexible and adaptive monitoring process that can respond to the dynamic nature of wireless communication systems.

[0180] In some aspects, the monitoring agent 808 may receive additional input data 828. The additional input data 828 may refer to data that is used for monitoring purposes. For example, the additional input data 828 may be separate from the model input 804. The additional input data 828 may include information that can be used to assess the performance of the model 802, detect anomalies, or provide context for interpreting the monitoring results.

[0181] Some examples of additional input data 828 may include contextual data, historical data, and / or external data. In some aspects, “contextual data” may refer to data that can influence the performance of the model 802 or help interpret the monitoring results. For example, in a beam management use case, the additional input data 828 may include information about a user’s location, mobility patterns, or network conditions. In some aspects, such contextual data can provide insights into the factors affecting the model’s beam selection decisions and aid in understanding observed anomalies or performance variations of the model 802.

[0182] In some aspects, the additional input data 828 may include historical data that can be used to establish baselines or detect deviations from past behavior. For example, historical data could include historical performance metrics, data distributions, or anomaly patterns specific to one or more metrics. Thus, by comparing a metric, such asD&S Ref. No.: QCM2404871WOQualcomm Ref. No.: 2404871WO46 first metric 820, against historical data, the monitoring agent 808 can identify changes or trends in the performance of the model 802. In some aspects, the additional input data 828 may include external data that is relevant to the performance of the model 802. For instance, in the context of a CSI prediction use case, the additional input data 828 may include weather data or information about external interference sources. Weather data and / or external interference sources can impact wireless channel conditions and, consequently, the prediction accuracy of the model 802. Generally, external data may include any data other than contextual data or historical data.

[0183] In some aspects, the monitoring agent 808 may receive a set of key performance indicators (KPIs) from a second communication device, such as a network node or a monitoring server. The set of KPIs may include various metrics that are relevant to the specific application domain or the communication system in which the model 802 is deployed. For example, in a wireless communication system, the KPIs may include a block error rate (BLER), which measures the ratio of the number of erroneous blocks to the total number of blocks received; a packet loss rate, which represents the percentage of packets that fail to reach their destination; a bit error rate (BER), which quantifies the number of bit errors per unit time; a throughput, which indicates the amount of data successfully transmitted per unit time; a spectral efficiency, which measures how efficiently the available frequency spectrum is utilized; a latency, which represents the delay between the transmission and reception of data; or a jitter, which measures the variation in the delay of received packets. The monitoring agent 808 may compare the obtained metrics, such as metric 820, to the received set of KPIs to assess the model's performance in the context of the specific application domain or communication system.

[0184] FIG. 9 illustrates an example monitoring agent 902 in accordance with aspects of the present disclosure. In some examples, the monitoring agent 902 may be an implementation of the monitoring agent 608 described with reference to FIG. 6 or the monitoring agent 808 described with reference to FIG. 8. FIG. 9 provides additional details of the monitoring functionality.

[0185] In certain aspects, the monitoring agent 902 may receive various inputs, including model input 604, additional input data 628, and / or model output 606, as previously described with respect to FIGs. 6 and 8. The monitoring agent 902 may process one or more of these inputs using a monitoring configuration 614 to generate a metric 912 that may provide an indication of the model’s performance.D&S Ref. No.: QCM2404871WOQualcomm Ref. No.: 2404871WO47

[0186] In some aspects, the monitoring agent 808 may include a measurement component 904 that may receive model input 604 and / or additional input data 628. The measurement component 904 may process and analyze the received data (model input 604 and / or additional input data 628) to extract measurements that may characterize the received data. Some examples of measurements that the measurement component 904 may derive include, but are not limited to, statistical properties such as mean, median, standard deviation, or other summary statistics of the received data; data distributions such as histograms, density estimates, or other representations of how the received data may be distributed; correlations that may measure the relationships or dependencies between different features of the received data; or temporal statistics (such as drift or stability measures) that may capture how the received data may change or evolve over time. These measurements can provide different measures of the model's input data, offering insights into the data characteristics and potential issues that may affect the model's performance. The measurements provided by the measurement component 904 may be based on the nature of the received data (e.g., model input 604 and / or additional input data 628) and / or monitoring objectives specified in the monitoring configuration 614.

[0187] Optionally, in certain aspects, the monitoring agent 902 may include a model output processor 908 that may receive the model output 606. The model output processor 908 may analyze the model predictions or outputs to derive processed output data 910. Processed output data 910 may characterize the behavior and / or performance of the model (e.g., model 602 of FIG. 6 or model 802 of FIG. 8). The specific processing performed by the model output processor 908 may vary depending on a type of model and monitoring requirements. Some examples of the processing that model output processor 908 could perform include aggregating model predictions over time to identify trends or patterns, calculating summary statistics of the model output 606 such as an average confidence score or the distribution of predicted classes, and / or applying post-processing techniques to the model output 606 such as thresholding, calibration, or domain-specific transformations. The processed output data 910 may provide a representation of the behavior of the model (e.g., model 602 of FIG. 6 or model 802 of FIG. 8) that can be used for further analysis and monitoring.

[0188] In certain aspects, the monitoring agent 902 may include an unsupervised evaluation metric generator 906. The unsupervised evaluation metric generator 906 mayD&S Ref. No.: QCM2404871WOQualcomm Ref. No.: 2404871WO48 assess characteristics of the input data (e.g., model input 604 and / or additional input data 628) and / or the model output 606 without relying on ground truth labels. The unsupervised evaluation metric generator 906 may receive data from different sources and compute unsupervised evaluation metrics based on that data. In some aspects, model output 606 may be provided to the unsupervised evaluation metric generator 906. The unsupervised evaluation metric generator 906 may compute unsupervised evaluation metrics based on the model output 606 itself, with little to no intermediate processing. This may be applicable when the model output 606 is in a suitable format or representation for the unsupervised evaluation metric generator 906 to process. In some aspects, the processed output data 910 may be provided to the unsupervised evaluation metric generator 906. This may be applicable when the model output 606 may be subject to additional processing or transformation before the model output 606 can be effectively used by the unsupervised evaluation metric generator 906.

[0189] In some aspects, measurements derived from the model input 602 by the measurement component 904 may be provided to the unsupervised evaluation metric generator 906. The unsupervised evaluation metric generator 906 may compute unsupervised evaluation metrics based on the input data characteristics, such as consistency, representativeness, and / or quality of the input data over time. In some aspects, the unsupervised evaluation metric generator 906 may use one or more of the model output 806, processed output data 910, and / or output of the measurement component 904 to calculate a metric 912. Some examples of the metric 912 include, but are not limited to, stability metrics, anomaly scores, representativeness metrics, and / or other relevant indicators.

[0190] FIG. 10 illustrates additional details of a monitoring agent 1002 in accordance with aspects of the present disclosure. In some examples, the monitoring agent 1002 may be the same as or similar to the monitoring agent 612 described in FIG. 6. In some aspects, the monitoring agent 1002 may include a supervised evaluation metric generator 1004 that is configured to compare an expected model output 626 to an actual model output 606 generated by a model such as the model 602 of FIG. 6.

[0191] In some examples, the monitoring agent 1002 receives or is configured with a monitoring configuration 618 that specifies how the monitoring agent 1002 is to monitor and analyze data or events. Based on the monitoring configuration 618, the monitoringD&S Ref. No.: QCM2404871WOQualcomm Ref. No.: 2404871WO49 agent 1002 may process an input, such as the expected model output 626 or the model output 606.

[0192] In some aspects, the supervised evaluation metric generator 1004 may receive the expected model output 626 and the model output 606 produced by the monitoring agent 1002. The supervised evaluation metric generator 1004 may compare the expected model output 626 and the model output 606 to generate the metric 1006, where the metric 1006 may indicate a performance of the model (e.g., model 602 of FIG. 6). In some aspects, the metric 1006 quantifies a similarity or difference between the expected model output 626 and the model output 606, providing an accuracy, reliability, or consistency measure of the model. Various metrics can be employed depending on the nature of the model and the specific monitoring objectives. For example, the metric 1006 may include, but is not limited to, precision, recall, Fl score, mean squared error, or a custom-defined metric tailored to an application domain. These different metrics can provide distinct measures of the model's performance, each focusing on a specific aspect of the model's behavior or output quality. By calculating the metric 1006, the monitoring agent 1002 can enable a quantitative assessment of the performance of a model to facilitate model validation, comparison, and optimization.

[0193] In the context of model evaluation, precision, recall, and F 1 score may refer to common metrics used to assess the performance of a model, particularly in classification tasks. In some aspects, precision measures the proportion of true positive predictions among all positive predictions, indicating a model's ability to avoid false positives. In some aspects, recall, also known as sensitivity, may measure the proportion of true positive predictions among all actual positive instances, indicating the model's ability to identify all positive instances. In some aspects, an Fl score may refer to a harmonic mean of precision and recall, providing a balanced measure of the model's performance, especially when the classes are imbalanced.

[0194] In certain aspects, the monitoring agent 1002 may utilize the metric 1006 to adapt a monitoring behavior or update the monitoring configuration 618. For instance, if the metric 1006 indicates a deviation between the expected model output 626 and the model output 606, the monitoring agent 1002 may trigger an alert, adjust a monitoring threshold, or initiate a further investigation. This allows the monitoring agent 1002 to continuously improve monitoring capabilities and respond to changes in the performance of the model over time.D&S Ref. No.: QCM2404871WOQualcomm Ref. No.: 2404871WO50

[0195] FIG. 11 illustrates an example communication system 1100 for monitoring the performance of an AI / ML model in a wireless network, in accordance with aspects of the present disclosure. The communication system 1100 includes a UE 1102 and a network entity 1104, such as a base station (e.g., gNB). In some aspects, the UE 1102 may be the same as or similar to the UE 104 described in FIG. 1. Similarly, the network entity 1104 may be the same as or similar to the BS 102 described in FIG. 1. The UE 1102 may implement a monitoring agent, such as the monitoring agent 608, 610, or 612 described in FIG. 6, to evaluate the performance of an AI / ML model used for various tasks, such as CSI feedback prediction or beam management.

[0196] A hierarchical monitoring process can be configured by a core entity or 0AM entity, by the network entity 1104 using RRC, MAC-CE, or DCI signaling, or autonomously selected by the UE 1102 and informed to the network entity 1104 using RRC, MAC-CE, or UCI signaling. In some aspects, the UE 1102 and network entity 1104 may exchange a recommended priority list of hierarchical monitoring, including an approach to use, metrics to include, or the like. The monitoring approach and metrics can be based on ground -truth measurements, model input / output statistics, or KPIs such as throughput, latency, BLER, or NACK rates.

[0197] In certain aspects, during a monitoring process (e.g., monitoring process 700), the monitoring agent at the UE 1102 may report monitoring metrics and other relevant information to the network entity 1104 or one or more other UEs 1124. The UEs 1124 may be the same as or similar to the UEs 104 described in FIG. 1. For example, the UE 1102 may send signaling messages 1108 to the network entity 1104 via a communication link 1106. The signaling messages 1108 may include a monitoring metric, such as a first metric 620, which could represent an unsupervised evaluation metric, a statistical property of input or output of a model, or a KPI (e.g., BLER or throughput).

[0198] The UE 1102 and network entity 1104 can exchange assistance information for hierarchical monitoring. The assistance information can include ground truth data measurements (measured at the UE 1102) to use at one or more hierarchical steps (e.g., at one or more monitoring agents). The ground truth data measurements could include, for example, a delay spread, a peak / path width, an SINR / SNR distribution, an RSRP / RSSI distribution, or a Rician factor distribution related to one or more signals such as an SSB or CSI-RS. In some aspects, the assistance information may indicate to increase RS resources, such as an RS bandwidth, an increased RS set, or an increasedD&S Ref. No.: QCM2404871WOQualcomm Ref. No.: 2404871WO51 number of beam measurements (e.g., finer beam measurements in space), from one step (e.g., monitoring agent) to another.

[0199] For example, in CSI-RS monitoring, the UE 1102 may start with a first RS set for monitoring and move to a second RS set based on a condition being satisfied, where the first RS set is a subset of the second RS set. In the second RS set (which may be used when the model is closer to failure), the UE 1102 may monitor a larger bandwidth or a larger set of beams. In the first RS set, which may be used in good conditions, a smaller set of resources or beams could be used, serving as a power-saving feature without compromising performance.

[0200] In some aspects, the assistance information can be hierarchical, with the UE 1102 or network entity 1104 requesting additional assistance information from each other as a device (e.g., UE 1102) moves from one monitoring step or agent to the next. Alternatively, or in addition, all assistance information for all steps (or a subset of steps) could be provided upfront, with partitioning of which assistance information is used at each step. The network entity 1104 may provide an indication of availability of assistance information to the UE 1102 using RRC, MAC-CE, or DCI signaling, while the UE 1102 may provide an indication of availability of assistance information to the network entity 1104 using RRC, MAC-CE, or UCI signaling.

[0201] In some aspects, when a triggering condition is satisfied, such as the first metric 620 (FIG. 6) exceeding a threshold, the UE 1102 may request assistance information from the network entity 1104 and / or other UEs 1124. In some examples, the UE 1102 may send a signaling message 1110 to the network entity 1104, requesting assistance, such as additional resources or guidance for more comprehensive monitoring. The network entity 1104 may respond with a communication 1114, which may include a monitoring configuration 1120 instructing the monitoring agent at the UE 1102 to increase the monitoring agent’s monitoring capability, such as initiating a more advanced monitoring agent (e.g., monitoring agent 610 and / or monitoring agent 612 of FIG. 6).

[0202] The network entity 1104 may acknowledge the assistance request with a message 1118 and allocate additional resources 1116 to support the monitoring process. These resources may include increased bandwidth for CSI-RS measurements, a larger set of beams for beam monitoring, or more frequent transmission opportunities for reporting monitoring results.D&S Ref. No.: QCM2404871WOQualcomm Ref. No.: 2404871WO52

[0203] In some aspects, the UE 1102 may instruct the network entity 1104, via signaling 1112, to monitor one or more metrics on behalf of the UE 1102. For instance, the UE 1102 may request the network entity 1104 to track a distribution of SINR or RSRP measurements related to specific reference signals or to monitor a Rician factor distribution associated with a set of beams. The monitoring process can also be distributed across multiple entities, such as the one or more UEs 1124, the network entity 1104, and the UE 1102. The one or more UEs 1124 may communicate with the network entity 1104 and the UE 1102 using signaling messages 1128 of communication link 1126 and 1132 of communication link 1130, respectively. These signaling messages 1128 and 1132 may include monitoring reports, requests for assistance, or indications regarding a type of monitoring or monitoring agent that has been initiated. In some aspects, the one or more UEs 1124 may provide a monitoring report directly to the UE 1102 via communication link 1130.

[0204] In some aspects, by enabling a hierarchical and collaborative monitoring approach, the communication system 1100 allows for efficient and adaptive evaluation of AI / ML model performance in wireless networks. The system 1100 can dynamically adjust the monitoring intensity and allocate resources based on the observed model behavior and the specific requirements of the use case, ensuring reliable and optimized operation of AI / ME-based functionalities.Aspects Related to Hierarchical Monitoring of Artificial Intelligence (Al) models in Wireless Communications Systems

[0205] FIG. 12 illustrates an example process flow 1200 for hierarchical monitoring of an AI / ME model in a wireless network, in accordance with aspects of the present disclosure. The process flow 1200 involves communications between a first device 1202, a second device 1204, and a third device 1206. In some aspects, first device 1202, second device 1204, and / or third device 1206 may be examples of the UEs 104 or network entities 102 depicted and described with respect to FIG. 1, or a disaggregated base station depicted and described with respect to FIG. 2. However, in other aspects, first device 1202, second device 1204, and / or third device 1206 may be other types of wireless communications devices or network entities, such as those described herein. Note that any operations or signaling illustrated with dashed lines may indicate that the operation or signaling is an optional or alternative example.D&S Ref. No.: QCM2404871WOQualcomm Ref. No.: 2404871WO53

[0206] First device 1202, second device 1204, and third device 1206 can exchange a recommended priority list of hierarchical monitoring, including an approach and metrics to include. The approach can be based on ground-truth measurements, model input / output statistics, or KPIs such as throughput, latency, BLER, or NACK rates. In some aspects, approach may indicate an order of steps (e.g., monitoring agents) for the hierarchical monitoring

[0207] In some aspects, first device 1202 may provide a monitoring configuration to second device 1204 via a communication 1208. The monitoring configuration may include information about the metrics to monitor, thresholds for triggering actions, and the hierarchy of monitoring steps to be performed. For example, the monitoring configuration may be the same as or similar to the monitoring configuration 614, 616, or 618 of FIG. 6. Second device 1204 may initiate a first monitoring 1210. The second device 1204 may provide a report via communication 1212, such as a monitoring report, based on a first monitoring, to the first device 1202. Alternatively, or in addition, second device 1204 may provide a report via communication 1214, such as a monitoring report, based on a first monitoring, to third device 1206.

[0208] In some aspects, at 1216, second device 1204 initiates a second monitoring step, which may be triggered by the results of the first monitoring step or by a separate trigger condition as described in connection with FIGs. 6 and 7. The second monitoring step may involve further changes to the metrics being monitored or the monitoring approach being used. In some aspects, second device 1204 sends to third device 1206 and / or first device 1202 a report via communication 1218 or communication 1220. These reports may contain information about the monitored metrics, such as the values of the metrics, any detected anomalies or deviations from expected behavior, and the overall performance of the model being monitored. The reports may also include details about the monitoring approach employed, such as the specific monitoring agents used, the monitoring configurations applied, and any adaptations made during the monitoring process. In some aspects, third device 1206 may provide a report back to second device 1204, which may include feedback, recommendations, or additional insights based on the received monitoring reports.

[0209] In response to the report 1220, second device 1204 may initiate an Ath monitoring step at 1222. In some examples, the Ath monitoring step may represent aD&S Ref. No.: QCM2404871WOQualcomm Ref. No.: 2404871WO54 further escalation of the monitoring process, which may involve more advanced monitoring techniques and / or a broader set of metrics.

[0210] First device 1202 receives a report from second device 1204 via communication 1224, which may include updated monitoring results based on the ongoing monitoring process. In some aspects, third device 1206 may receive a report from second device 1204 via communication 1226, where the report may include updated monitoring results based on the ongoing monitoring process.

[0211] Note that the process flow illustrated in FIG. 12 is an example of hierarchical monitoring in a wireless network, and aspects of the present disclosure may be applied to various monitoring scenarios and network configurations. The process flow illustrated in FIG. 12 is described herein to facilitate an understanding of the hierarchical monitoring approach, and aspects of the present disclosure may be performed in various manners via alternative or additional signaling and / or operations. In certain aspects, the operations and / or signaling of FIG. 12 may occur in an order different from that described or depicted, and various actions, operations, and / or signaling may be added, omitted, or combined.

[0212] FIG. 13 illustrates an example process flow 1300 for hierarchical monitoring of an AI / ML model in a wireless network, in accordance with aspects of the present disclosure. The process flow 1300 involves communications between a first device 1302, a second device 1304, and a third device 1306. In some aspects, first device 1302, second device 1304, and / or third device 1306 may be examples of the UEs 104 or network entities 102 depicted and described with respect to FIG. 1, or a disaggregated base station depicted and described with respect to FIG. 2. However, in other aspects, first device 1302, second device 1304, and / or third device 1306 may be other types of wireless communications devices or network entities, such as those described herein. Note that any operations or signaling illustrated with dashed lines may indicate that the operation or signaling is an optional or alternative example.

[0213] The hierarchical monitoring process can be configured by a core entity (e.g., EPC 160 of FIG. 1) and / or 0AM entity (e.g., SMO Framework 205 of FIG. 2), configured by a network entity using RRC, MAC-CE, or DCI signaling, or autonomously selected by a UE and informed to the network entity using RRC, MAC-CE, or UCI signaling. In some aspects, first device 1302 sends a monitoring configuration to secondD&S Ref. No.: QCM2404871WOQualcomm Ref. No.: 2404871WO55 device 1304 via communication 1308. The monitoring configuration may include information about the metrics to monitor, thresholds for triggering actions, and / or a hierarchy of monitoring steps to be performed. Second device 1304 initiates a first monitoring step at 1310 based on the received monitoring configuration.

[0214] Second device 1304 requests assistance from third device 1306 via communication 1312. Third device 1306 provides assistance to second device 1304 via communication 1314. The assistance can include ground truth data measurements (measured at the UE) to use on each step, conditions that the devices need to check for triggers between steps (e.g., delay spread, peak / path width, SINR / SNR distribution, RSRP / RSSI distribution, Rician factor distribution related to SSB / CSI-RS or other reference signals), or an explicit trigger between steps. The assistance can also involve increasing RS resources, such as increased bandwidth, increased RS set, or increased number of beam measurements (finer beam measurements in space), from one step to another.

[0215] For example, in CSI-RS monitoring, second device 1304 may start with setl assistance for monitoring and move to set2 based on sub-conditions, where setl is a subset of set2. Close to failure, second device 1304 may need to monitor a larger bandwidth or a larger set of beams. Under better conditions, only a small set of resources or beams can be used, serving as a power-saving feature without compromising performance.

[0216] Second device 1304 initiates a second monitoring step at 1316 based on the assistance received from third device 1306. Second device 1304 provides a report to third device 1306 via communication 1318. Alternatively, or in addition, second device 1304 provides the report to first device 1302 via communication 1320. In some aspects, second device 1304 requests additional assistance from third device 1306 via communication 1322. Third device 1306 provides assistance via communication 1324, which may include an acknowledgment or an indication of the type of assistance that will be provided, if assistance is provided by third device 1306.

[0217] Based on the assistance received from third device 1306, second device 1304 falls back to the first monitoring step at 1326. Second device 1304 provides a report to third device 1306 via communication 1328 and / or sends a report to first device 1302 via communication 1330. The assistance can be hierarchical, with each device requesting additional assistance from the others as it moves from one monitoring step to the next. AtD&S Ref. No.: QCM2404871WOQualcomm Ref. No.: 2404871WO56 least some of the devices 1302 / 1504 / 1506 can announce assistance availability to each other using RRC, MAC-CE, DCI, or UCI signaling, depending on their role in the network.

[0218] Note that the process flow illustrated in FIG. 13 is an example of hierarchical monitoring in a wireless network, and aspects of the present disclosure may be applied to various monitoring scenarios and network configurations. The process flow illustrated in FIG. 13 is described herein to facilitate an understanding of the hierarchical monitoring approach, and aspects of the present disclosure may be performed in various manners via alternative or additional signaling and / or operations. In certain aspects, the operations and / or signaling of FIG. 13 may occur in an order different from that described or depicted, and various actions, operations, and / or signaling may be added, omitted, or combined.

[0219] FIG. 14 illustrates an example process flow 1400 for distributed monitoring of an AI / ML model in a wireless network, in accordance with aspects of the present disclosure. The process flow 1400 involves communications between a first device 1402, a second device 1404, and a third device 1406. In some aspects, first device 1402, second device 1404, and third device 1406 may be examples of the UEs 104 or network entities 108 depicted and described with respect to FIG. 1, or a disaggregated base station depicted and described with respect to FIG. 2. However, in other aspects, first device 1402, second device 1404, and third device 1406 may be other types of wireless communications devices or network entities, such as those described herein. Note that any operations or signaling illustrated with dashed lines may indicate that the operation or signaling is an optional or alternative example.

[0220] The distributed monitoring process can be configured by a core entity or 0AM entity, configured by a gNB using RRC, MAC-CE, or DCI signaling, or autonomously selected by a UE and informed to the gNB using RRC, MAC-CE, or UCI signaling. The devices involved in the monitoring process (e.g., first device 1402, second device 1404, and third device 1406) can exchange a recommended priority list of hierarchical monitoring, including the approach and metrics to include. The approach and order of monitoring steps can be based on ground-truth measurements, model input / output statistics, or KPIs such as throughput, latency, BEER, or NACK rates.D&S Ref. No.: QCM2404871WOQualcomm Ref. No.: 2404871WO57

[0221] In some aspects, and in the distributed monitoring approach depicted in FIG. 14, multiple devices (e.g., first device 1402, second device 1404, and third device 1406) collaborate to monitor the performance of an AI / ML model. Each device may perform different monitoring tasks or evaluate different aspects of the model’s performance. In some aspects, the first device 1402, second device 1404, and third device 1406 may share their monitoring results with each other to create a comprehensive view of the model’s performance. This distributed approach allows for efficient resource utilization and enables the devices to adapt their monitoring strategies based on the collective knowledge gained from the shared results.

[0222] In some aspects, second device 1404 provides a monitoring configuration to third device 1406 via communication 1408. Third device 1406 may initiate a first monitoring step at 1410 based on the received monitoring configuration. Third device 1406 may send a report to second device 1404 via communication 1412, which may include the results of the first monitoring step. Second device 1404 may send a monitoring configuration to first device 1402 via communication 1414. First device 1402 may initiate a first monitoring step at 1416, which may be different from the first monitoring step 1410 performed by third device 1406. First device 1402 may send a report to second device 1404 via communication 1418, which may include the results of the first monitoring step performed by first device 1402.

[0223] First device 1402 may send a request for assistance to second device 1404 via communication 1420. Second device 1404 may provide assistance to first device 1402 via communication 1422. Additionally, or alternatively, second device 141 may provide assistance to third device 1406 via communication 1424. The assistance can include ground truth data measurements (measured at the UE side) to use on each step, conditions that the devices need to check for triggers between steps (e.g., delay spread, peak / path width, SINR / SNR distribution, RSRP / RSSI distribution, Rician factor distribution related to SSB / CSI-RS or other reference signals), or an explicit trigger between steps. The assistance can also involve increasing RS resources, such as an increased bandwidth, an increased RS set, or an increased number of beam measurements (such as finer beam measurements in the spatial domain), from one step to another.

[0224] First device 1402 may initiate an Ath monitoring step at 1426 based on the assistance received from second device 1404. First device 1402 may provide a report toD&S Ref. No.: QCM2404871WOQualcomm Ref. No.: 2404871WO58 second device 1404 via communication 1428, which may include monitoring results of the Nth monitoring step.

[0225] In some examples, second device 1404 may provide a report to third device 1406 via communication 1430, which may include the monitoring results received from first device 1402 or other relevant information. Third device 1406 may initiate an TVth monitoring step at 1432 based on the report received from second device 1404. Third device 1406 may provide a report to second device 1404 via communication 1434, which may include the results of the Nth monitoring step performed by third device 1406.

[0226] In some aspects, the assistance can be hierarchical, with each device requesting additional assistance from the others as it moves from one monitoring step to the next.

[0227] Note that the process flow illustrated in FIG. 14 is an example of distributed monitoring in a wireless network, and aspects of the present disclosure may be applied to various monitoring scenarios and network configurations. The process flow illustrated in FIG. 14 is described herein to facilitate an understanding of the distributed monitoring approach, and aspects of the present disclosure may be performed in various manners via alternative or additional signaling and / or operations. In certain aspects, the operations and / or signaling of FIG. 14 may occur in an order different from that described or depicted, and various actions, operations, and / or signaling may be added, omitted, or combined.Example Operations of a User Equipment and / or a Network Entity

[0228] FIG. 15 shows a method 1500 for wireless communications by a UE, such as UE 104 of FIGS. 1 and 3. In some aspects, FIG. 15 shows a method 1500 for wireless communications by a network entity, such as BS 102 of FIGS. 1 and 3, or a disaggregated base station as discussed with respect to FIG. 2.

[0229] Method 1500 begins at block 1510 with monitoring a first metric associated with a machine learning model using a first level of monitoring granularity. In some aspects, the machine learning model is associated with wireless communication. In some aspects, the first level of monitoring granularity comprises a first set of one or more parameters associated with a performance of the machine learning model. In some aspects, block 1510 may be performed by a monitoring agent 608 of FIG. 6. In someD&S Ref. No.: QCM2404871WOQualcomm Ref. No.: 2404871WO59 examples, a first metric 620 (e.g., FIG. 6) is generated by the monitoring agent 608 (e.g., FIG. 6) based on model input 604 (e.g., FIG. 6) and / or model output 606.

[0230] Method 1500 then proceeds to block 1520 with monitoring, in response to a trigger condition associated with the first metric being satisfied, a second metric associated with the machine learning model using a second level of monitoring granularity. In some aspects, the second metric is configured to provide a different measure of the machine learning model than the first metric. In some aspects, the second level of monitoring granularity comprises a second set of one or more parameters associated with the performance of the machine learning model. In some aspects, the second set of one or more parameters includes at least one parameter that is not included in the first set of one or more parameters. In some aspects, block 1520 may be performed by a monitoring agent, such as monitoring agent 608, 1902, and / or 1002 discussed in the context of FIGS. 6, 9, and 10. In some aspects, the second metric 622 (e.g., FIG. 6) may be generated by the monitoring agent 608 (e.g., FIG. 6) based on model input 604 (e.g., FIG. 6), model output 606 (e.g., FIG. 6), and / or expected model output 626 (e.g., FIG. 6).

[0231] Method 1500 may then proceed to block 1530, with signaling, in response to the second metric satisfying a reporting condition that is not satisfied by the first metric, an indication associated with the performance of the machine learning model.

[0232] In some examples, the first and second metrics may be used to trigger different monitoring configurations or to switch between ground truth-based and ground truth- free monitoring techniques, as discussed in the context of FIG. 11. For example, the second metric 622 may trigger the initialization of the monitoring agent 612 for ground truthbased monitoring, while another metric may trigger the initialization of the monitoring of another host.

[0233] By performing the steps of method 1500 in conjunction with the monitoring configurations and techniques described in FIGS. 6-11, a UE may be able to adaptively select and apply the most suitable monitoring approach based on the obtained metrics. This provides the technical benefits of improved model performance, efficient resource utilization, and enhanced flexibility in accommodating varying network conditions and requirements.D&S Ref. No.: QCM2404871WOQualcomm Ref. No.: 2404871WO60

[0234] Method 1500 provides a technical solution for efficiently monitoring a model associated with wireless communication by initially using a less resource-intensive first metric and switching to a more comprehensive second metric only when needed based on a trigger condition. This allows the UE to conserve resources while still maintaining effective model monitoring.

[0235] In one aspect, method 1500 further includes causing an adjustment of one or more operating parameters of at least one of the machine learning model or the apparatus based on the signaled indication.

[0236] In one aspect of method 1500, the first set of one or more parameters and the second set of one or more parameters are hierarchically related, such that the second set of one or more parameters includes at least one parameter that provides additional information about the performance of the machine learning model beyond information provided by the first set of one or more parameters.

[0237] In one aspect of method 1500, the trigger condition is configured to indicate a potential performance issue with the machine learning model based on the first metric.

[0238] In one aspect of method 1500, the apparatus is one of a user equipment (UE) or a network entity in a wireless communications network.

[0239] In one aspect of method 1500, the first set of one or more parameters and the second set of one or more parameters are based on different sets of resources, and the second set of one or more parameters is based on a larger set of resources than the first set of one or more parameters.

[0240] In one aspect of method 1500, the trigger condition is based on a comparison of the first metric to a threshold value.

[0241] In one aspect, method 1500 further includes monitoring, in response to a second trigger condition associated with the second metric being satisfied, a third metric associated with the machine learning model using a third level of monitoring granularity, wherein the third metric is configured to provide a different measure of the machine learning model than the first metric and the second metric.

[0242] In one aspect, method 1500 further includes causing the apparatus to receive a configuration from a second apparatus, the configuration including an indication of atD&S Ref. No.: QCM2404871WOQualcomm Ref. No.: 2404871WO61 least one of: the first metric, the trigger condition, a threshold value for the first metric, the second metric, or a time period to monitor the first metric.

[0243] In one aspect, method 1500 further includes sending, to a second apparatus, a request for assistance to monitor the machine learning model; and receiving, from the second apparatus, an indication of one or more monitoring parameters in response to the request, wherein the one or more monitoring parameters include at least one of: a threshold value for the first metric, a threshold value for the second metric, a time period in which to monitor the first metric, a time period in which to monitor the second metric, a set of resources on which to monitor the first metric, or a set of resources on which to monitor the second metric.

[0244] In one aspect, method 1500 further includes adjusting the monitoring of the first metric or the second metric based on at least one of the first set of one or more parameters or the second set of one or more parameters.

[0245] In one aspect, method 1500 further includes receiving, from a second apparatus, a request for assistance with monitoring the machine learning model; and sending, to the second apparatus, an indication of one or more monitoring parameters in response to the request.

[0246] In one aspect, method 1500 further includes reverting, in response to a third trigger condition being satisfied, to monitoring the first metric, wherein the third trigger condition is satisfied when a timer associated with monitoring the second metric expires.

[0247] In one aspect, method 1500 further includes sending, to a second apparatus, an indication that the apparatus has reverted to monitoring the first metric.

[0248] In one aspect, method 1500 further includes receiving, from a second apparatus, an instruction to revert to monitoring the first metric; and reverting to monitoring the first metric in response to the instruction.

[0249] In one aspect, method 1500 further includes performing ground truth based monitoring to monitor the first metric or the second metric by: receiving, from a second apparatus, a set of ground truth data; and comparing the first metric or the second metric to the set of ground truth data, wherein the set of ground truth data includes at least one of: a set of input data and corresponding expected output data for the machine learningD&S Ref. No.: QCM2404871WOQualcomm Ref. No.: 2404871WO62 model, a set of measurements of a wireless environment, or a set of parameters of a wireless communication in a wireless communications network.

[0250] In one aspect, method 1500 further includes performing ground truth- free monitoring to monitor the first metric or the second metric by analyzing the first metric or the second metric without ground truth data, wherein analyzing the first metric or the second metric includes at least one of: comparing the first metric or the second metric to a threshold, evaluating a statistical property of the first metric or the second metric, or detecting an anomaly in the first metric or the second metric.

[0251] In one aspect, method 1500 further includes performing key performance indicator (KPI)-based monitoring to monitor the first metric or the second metric by: receiving, from a second communication device, a set of KPIs; and comparing the first metric or the second metric to the set of KPIs, wherein the set of KPIs includes at least one of: a block error rate (BLER), a packet loss rate, a bit error rate (BER), a throughput, a spectral efficiency, a latency, or a jitter.

[0252] In one aspect, method 1500, or any aspect related to it, may be performed by an apparatus, such as communications device of FIG. 16.

[0253] Note that FIG. 15 is just one example of a method, and other methods including fewer, additional, or alternative operations are possible consistent with this disclosure.Example Communications Devices

[0254] FIG. 16 depicts aspects of an example communications device 1600. In some aspects, communications device 1600 is a user equipment, such as UE 104 described above with respect to FIGS. 1 and 3.

[0255] The communications device 1600 includes a processing system 1602 coupled to a transceiver 1608 (e.g., a transmitter and / or a receiver). The transceiver 1608 is configured to transmit and receive signals for the communications device 1800 via an antenna 1610, such as the various signals as described herein. The processing system 1602 may be configured to perform processing functions for the communications device 1600, including processing signals received and / or to be transmitted by the communications device 1600.D&S Ref. No.: QCM2404871WOQualcomm Ref. No.: 2404871WO63

[0256] The processing system 1602 includes one or more processors 1620. In various aspects, the one or more processors 1620 may be representative of one or more of receive processor 358, transmit processor 364, TX MIMO processor 366, and / or controller / processor 380, as described with respect to FIG. 3. The one or more processors 1620 are coupled to a computer-readable medium / memory 1630 via a bus 1606. In certain aspects, the computer-readable medium / memory 1630 is configured to store instructions (e.g., computer-executable code) that when executed by the one or more processors 1620, cause the one or more processors 1620 to perform the method 1500 described with respect to FIG. 15, or any aspect related to it, including any operations described in relation to FIG. 15. Note that reference to a processor performing a function of communications device 1600 may include one or more processors performing that function of communications device 1600, such as in a distributed fashion.

[0257] In the depicted example, computer-readable medium / memory 1630 stores code (e.g., executable instructions) for monitoring a first metric associated with a machine learning model 1631, code for monitoring a second metric associated with the machine learning model 1632, and code for signaling an indication 1633. Processing of the code 1631-1633 may enable and cause the communications device 1600 to perform the method 1500 described with respect to FIG. 15, or any aspect related to it.

[0258] The one or more processors 1620 include circuitry configured to implement (e.g., execute) the code stored in the computer-readable medium / memory 1630, including circuitry for monitoring a first metric associated with a machine learning model 1621, circuitry for monitoring a second metric associated with the machine learning model 1622, and circuitry for signaling an indication 1623. Processing with circuitry 1621-1623 may enable and cause the communications device 1600 to perform the method 1500 described with respect to FIG. 15, or any aspect related to it.

[0259] More generally, means for communicating, transmitting, sending or outputting for transmission may include the transceivers 354, antenna(s) 352, transmit processor 364, TX MIMO processor 366, Al processor 370, and / or controller / processor 380 of the UE 104 illustrated in FIG. 3, transceiver 1608 and / or antenna 1610 of the communications device 1600 in FIG. 16, and / or one or more processors 1620 of the communications device 1600 in FIG. 16. Means for communicating, receiving or obtaining may include the transceivers 354, antenna(s) 352, receive processor 358, Al processor 370, and / or controller / processor 380 of the UE 104 illustrated in FIG. 3,D&S Ref. No.: QCM2404871WOQualcomm Ref. No.: 2404871WO64 transceiver 1608 and / or antenna 1610 of the communications device 1600 in FIG. 16, and / or one or more processors 1620 of the communications device 1600 in FIG. 16.

[0260] FIG. 17 depicts aspects of an example communications device 1700. In some aspects, communications device 1700 is a network entity, such as BS 102 of FIGS. 1 and 3, or a disaggregated base station as discussed with respect to FIG. 2.

[0261] The communications device 1700 includes a processing system 1702 coupled to a transceiver 1708 (e.g., a transmitter and / or a receiver) and / or a network interface 1712. The transceiver 1708 is configured to transmit and receive signals for the communications device 1700 via an antenna 1710, such as the various signals as described herein. The network interface 1712 is configured to obtain and send signals for the communications device 1700 via communications link(s), such as a backhaul link, midhaul link, and / or fronthaul link as described herein, such as with respect to FIG. 2. The processing system 1702 may be configured to perform processing functions for the communications device 1700, including processing signals received and / or to be transmitted by the communications device 1700.

[0262] The processing system 1702 includes one or more processors 1720. In various aspects, one or more processors 1720 may be representative of one or more of receive processor 338, transmit processor 320, TX MIMO processor 330, and / or controller / processor 340, as described with respect to FIG. 3. The one or more processors 1720 are coupled to a computer-readable medium / memory 1730 via a bus 1706. In certain aspects, the computer-readable medium / memory 1730 is configured to store instructions (e.g., computer-executable code), including code aspects 1731-1732, that when executed by the one or more processors 1720, cause the one or more processors 1720 to perform the method 1500 described with respect to FIG. 15, or any aspect related to it, including any operations described in relation to FIG. 15. Note that reference to a processor of communications device 1700 performing a function may include one or more processors of communications device 1700 performing that function, such as in a distributed fashion.

[0263] In the depicted example, the computer-readable medium / memory 1730 stores code (e.g., executable instructions) for obtaining a first metric associated with a model 1731 and code for obtaining a second metric associated with a model 1732. Processing of the code 1731-1732 may enable and cause the communications device 1700 to perform the method 1500 described with respect to FIG. 15, or any aspect related to it.D&S Ref. No.: QCM2404871WOQualcomm Ref. No.: 2404871WO65

[0264] The one or more processors 1720 include circuitry configured to implement (e.g., execute) the code stored in the computer-readable medium / memory 1730, including circuitry for obtaining a first metric associated with a model 1721 and circuitry for obtaining a second metric associated with a model 1722. Processing with circuitry 1721- 1722 may enable and cause the communications device 1700 to perform the method 1500 as described with respect to FIG. 15, or any aspect related to it.

[0265] Various components of the communications device 1700 may provide means for performing the method 1500 as described with respect to FIG. 15, or any aspect related to it. Means for communicating, transmitting, sending or outputting for transmission may include the transceivers 332, antenna(s) 334, transmit processor 320, TX MIMO processor 330, Al processor 318, and / or controller / processor 340 of the BS 102 illustrated in FIG. 3, transceiver 1708, antenna 1710, and / or network interface 1712 of the communications device 1700 in FIG. 17, and / or one or more processors 1720 of the communications device 1700 in FIG. 17. Means for communicating, receiving or obtaining may include the transceivers 332, antenna(s) 334, receive processor 338, Al processor 318, and / or controller / processor 340 of the BS 102 illustrated in FIG. 3, transceiver 1708, antenna 1710, and / or network interface 1712 of the communications device 1700 in FIG. 17, and / or one or more processors 1720 of the communications device 1700 in FIG. 17. For example, means for obtaining metrics and selecting models based on the metrics of the method 1500 described with respect to FIG. 15, or any aspect related to it, may include the processing system 1702, processor(s) 1920, computer- readable medium / memory 1730, and associated code 1731-1732 and circuitry 1721-1722.Example Clauses

[0266] Implementation examples are described in the following numbered clauses:

[0267] Clause 1 : A method for machine learning model performance monitoring, comprising: monitoring a first metric associated with a machine learning model using a first level of monitoring granularity, wherein the machine learning model is associated with wireless communication, and wherein the first level of monitoring granularity comprises a first set of one or more parameters associated with a performance of the machine learning model; monitoring, in response to a trigger condition associated with the first metric being satisfied, a second metric associated with the machine learning model using a second level of monitoring granularity, wherein the second metric isD&S Ref. No.: QCM2404871WOQualcomm Ref. No.: 2404871WO66 configured to provide a different measure of the machine learning model than the first metric, wherein the second level of monitoring granularity comprises a second set of one or more parameters associated with the performance of the machine learning model, and wherein the second set of one or more parameters includes at least one parameter that is not included in the first set of one or more parameters; and signaling, in response to the second metric satisfying a reporting condition that is not satisfied by the first metric, an indication associated with the performance of the machine learning model.

[0268] Clause 2: The method of Clause 1, further comprising causing an adjustment of one or more operating parameters of at least one of the machine learning model or an apparatus performing the method based on the signaled indication.

[0269] Clause 3 : The method of Clause 1 or Clause 2, wherein the first set of one or more parameters and the second set of one or more parameters are hierarchically related, such that the second set of one or more parameters includes at least one parameter that provides additional information about the performance of the machine learning model beyond information provided by the first set of one or more parameters.

[0270] Clause 4: The method of any one of Clauses 1 -3, wherein the trigger condition is configured to indicate a potential performance issue with the machine learning model based on the first metric.

[0271] Clause 5: The method of any one of Clauses 1-4, wherein the method is performed by one of a user equipment (UE) or a network entity in a wireless communications network.

[0272] Clause 6: The method of any one of Clauses 1-5, wherein the first set of one or more parameters and the second set of one or more parameters are based on different sets of resources, and wherein the second set of one or more parameters is based on a larger set of resources than the first set of one or more parameters.

[0273] Clause 7: The method of any one of Clauses 1-6, further comprising determining that the trigger condition is satisfied based on the first metric.

[0274] Clause 8: The method of any one of Clauses 1-7, wherein the trigger condition is based on a comparison of the first metric to a threshold value.

[0275] Clause 9: The method of any one of Clauses 1-8, further comprising monitoring, in response to a second trigger condition associated with the second metricD&S Ref. No.: QCM2404871WOQualcomm Ref. No.: 2404871WO67 being satisfied, a third metric associated with the machine learning model using a third level of monitoring granularity, wherein the third metric is configured to provide a different measure of the machine learning model than the first metric and the second metric.

[0276] Clause 10: The method of any one of Clauses 1-9, further comprising receiving a configuration from a second apparatus, the configuration including an indication of at least one of: the first metric, the trigger condition, a threshold value for the first metric, the second metric, or a time period to monitor the first metric.

[0277] Clause 11 : The method of any one of Clauses 1-10, further comprising: sending, to a second apparatus, a request for assistance to monitor the machine learning model; and receiving, from the second apparatus, an indication of one or more monitoring parameters in response to the request, wherein the one or more monitoring parameters include at least one of: a threshold value for the first metric, a threshold value for the second metric, a time period in which to monitor the first metric, a time period in which to monitor the second metric, a set of resources on which to monitor the first metric, or a set of resources on which to monitor the second metric.

[0278] Clause 12: The method of Cause 11, further comprising adjusting the monitoring of the first metric or the second metric based on the one or more monitoring parameters.

[0279] Clause 13: The method of any one of Clauses 1-12, further comprising: receiving, from a second apparatus, a request for assistance with monitoring the machine learning model; and sending, to the second apparatus, an indication of one or more monitoring parameters in response to the request.

[0280] Clause 14: The method of any one of Clauses 1-13, further comprising reverting, in response to a third trigger condition being satisfied, to monitoring the first metric, wherein the third trigger condition is satisfied when a timer associated with monitoring the second metric expires.

[0281] Clause 15: The method of Clause 14, further comprising sending, to a second apparatus, an indication that the monitoring has reverted to monitoring the first metric.D&S Ref. No.: QCM2404871WOQualcomm Ref. No.: 2404871WO68

[0282] Clause 16: The method of any one of Clauses 1-15, further comprising: receiving, from a second apparatus, an instruction to revert to monitoring the first metric; and reverting to monitoring the first metric in response to the instruction.

[0283] Clause 17: The method of any one of Clauses 1-16, wherein monitoring the first metric or the second metric comprises performing ground truth based monitoring, the method further comprising: receiving, from a second apparatus, a set of ground truth data; and comparing the first metric or the second metric to the set of ground truth data, wherein the set of ground truth data includes at least one of: a set of input data and corresponding expected output data for the machine learning model, a set of measurements of a wireless environment, or a set of parameters of the wireless communication.

[0284] Clause 18: The method of any one of Clauses 1-17, wherein monitoring the first metric or the second metric comprises performing ground truth-free monitoring, the method further comprising analyzing the first metric or the second metric without ground truth data, wherein analyzing the first metric or the second metric includes at least one of: comparing the first metric or the second metric to a threshold, evaluating a statistical property of the first metric or the second metric, or detecting an anomaly in the first metric or the second metric.

[0285] Clause 19: A method for wireless communications by an apparatus comprising: obtaining a first metric associated with a model that is associated with wireless communication; and obtaining, in response to a trigger condition associated with the first metric being satisfied, a second metric associated with the model, the second metric providing a different measure of the model than the first metric.

[0286] Clause 20: The method of Clause 19, wherein the first metric is based on a first set of resources and the second metric is based on a second set of resources, and the second set of resources includes more resources than the first set of resources.

[0287] Clause 21 : The method of any one of Clauses 19-20, further comprising: determining that the trigger condition is satisfied based on the first metric.

[0288] Clause 22: The method of any one of Clauses 19-21, wherein the trigger condition is based on a comparison of the first metric to a threshold value.D&S Ref. No.: QCM2404871WOQualcomm Ref. No.: 2404871WO69

[0289] Clause 23: The method of any one of Clauses 19-22, further comprising obtaining, in response to a second trigger condition associated with the second metric being satisfied, a third metric associated with the model, the third metric providing a different measure of the model than the first metric and the second metric.

[0290] Clause 24: The method of any one of Clauses 19-23, further comprising receiving a configuration from a second apparatus, the configuration including an indication of the first metric.

[0291] Clause 25: The method of Clause 24, wherein the configuration further includes at least one of: an indication of the trigger condition; a threshold value for the first metric; an indication of the second metric; or a time period to monitor the first metric.

[0292] Clause 26: The method of any one of Clauses 19-25, further comprising: sending, to a second apparatus, a request for assistance to monitor the model; and receiving, from the second apparatus, an indication of one or more monitoring parameters in response to the request.

[0293] Clause 27: The method of Clause 26, wherein the one or more monitoring parameters include at least one of: a threshold value for the first metric, a threshold value for the second metric, a time period in which to monitor the first metric, a time period in which to monitor the second metric, a set of resources on which to monitor the first metric, or a set of resources on which to monitor the second metric.

[0294] Clause 28: The method of any one of Clauses 19-27, further comprising: receiving, from a second apparatus, an instruction to revert to monitoring the first metric; and reverting to monitoring the first metric in response to the instruction.

[0295] Clause 29: The method of any one of Clauses 19-28, further comprising obtaining the first metric or information associated with the first metric from a second apparatus different from the first UE.

[0296] Clause 30: The method of any one of Clauses 19-29, wherein the second apparatus that is different from the first UE is at least one of: a second UE, or a network entity.

[0297] Clause 31 : The method of any one of Clauses 19-30, further comprising adjusting a configuration of the model based on the second metric.D&S Ref. No.: QCM2404871WOQualcomm Ref. No.: 2404871WO70

[0298] Clause 32: The of any one of Clauses 19-31, wherein the apparatus is a user equipment (UE), and causing the apparatus to obtain the first metric, comprises causing the apparatus to obtain the first metric at the UE.

[0299] Clause 33: The method of Clause 32, further comprising: sending an indication of the first metric to a network entity, and receiving an indication of the second metric from the network entity.

[0300] Clause 34: The method of any one of Clause 19-33, wherein the apparatus is a network entity, and causing the apparatus to obtain the first metric comprises causing the apparatus to obtain the first metric or information associated with the first metric from a user equipment (UE).

[0301] Clause 35: The method of Clause 34, further comprising: receiving an indication of the first metric from the UE, and sending an indication of the second metric to the UE.

[0302] Clause 36: The method of any one of Clauses 19-34, further comprising adjusting a configuration of the model based on the second metric.

[0303] Clause 37. The method of Clause 36, further comprising: retraining the model based on the second metric; or updating one or more parameters of the model based on the second metric.

[0304] Clause 38: A method for wireless communications by an apparatus comprising: obtaining a first metric associated with a model that is associated with wireless communication; and obtaining, in response to a trigger condition associated with the first metric being satisfied, a second metric associated with the model, the second metric providing a different measure of the model than the first metric.

[0305] Clause 39: A method for wireless communications by an apparatus, comprising: determining that a trigger condition associated with the first metric is satisfied; providing a request for assistance in response to the trigger condition being satisfied; receiving a monitoring configuration based on the request for assistance, the monitoring configuration including an instruction to obtain a second metric associated with the model, the second metric providing a different measure of the model than the first metric; obtaining the second metric based on the monitoring configuration; and performing an action based on the second metric, wherein the action includes at least oneD&S Ref. No.: QCM2404871WOQualcomm Ref. No.: 2404871WO71 of: adapting the model based on the second metric; transmitting the second metric to the base station; receiving an adapted model based on the second metric; or receiving an instruction to adapt the model based on the second metric.

[0306] Clause 40: One or more apparatuses, comprising: one or more memories comprising executable instructions; and one or more processors configured to execute the executable instructions and cause the one or more apparatuses to perform a method in accordance with any one of clauses 1-39.

[0307] Clause 41 : One or more apparatuses, comprising: one or more memories; and one or more processors, coupled to the one or more memories, configured to cause the one or more apparatuses to perform a method in accordance with any one of Clauses 1- 39.

[0308] Clause 42: One or more apparatuses, comprising: one or more memories; and one or more processors, coupled to the one or more memories, configured to perform a method in accordance with any one of Clauses 1-39.

[0309] Clause 43: One or more apparatuses, comprising means for performing a method in accordance with any one of Clauses 1-39.

[0310] Clause 44: One or more non-transitory computer-readable media comprising executable instructions that, when executed by one or more processors of one or more apparatuses, cause the one or more apparatuses to perform a method in accordance with any one of Clauses 1-39.

[0311] Clause 45 : One or more computer program products embodied on one or more computer-readable storage media comprising code for performing a method in accordance with any one of Clauses 1-39.

[0312] Clause 46: A user equipment (UE), comprising: a processing system that includes processor circuitry and memory circuitry that stores code and is coupled with the processor circuitry, the processing system configured to cause the UE to perform a method in accordance with any one of Clauses 1-39.

[0313] Clause 47: A network entity, comprising: a processing system that includes processor circuitry and memory circuitry that stores code and is coupled with the processor circuitry, the processing system configured to cause the network entity to perform a method in accordance with any one of Clauses 1-39.D&S Ref. No.: QCM2404871WOQualcomm Ref. No.: 2404871WO72Additional Considerations

[0314] The preceding description is provided to enable any person skilled in the art to practice the various aspects described herein. The examples discussed herein are not limiting of the scope, applicability, or aspects set forth in the claims. Various modifications to these aspects will be readily apparent to those skilled in the art, and the general principles defined herein may be applied to other aspects. For example, changes may be made in the function and arrangement of elements discussed without departing from the scope of the disclosure. Various examples may omit, substitute, or add various procedures or components as appropriate. For instance, the methods described may be performed in an order different from that described, and various actions may be added, omitted, or combined. Also, features described with respect to some examples may be combined in some other examples. For example, an apparatus may be implemented or a method may be practiced using any number of the aspects set forth herein. In addition, the scope of the disclosure is intended to cover such an apparatus or method that is practiced using other structure, functionality, or structure and functionality in addition to, or other than, the various aspects of the disclosure set forth herein. It should be understood that any aspect of the disclosure disclosed herein may be embodied by one or more elements of a claim.

[0315] The various illustrative logical blocks, modules and circuits described in connection with the present disclosure may be implemented or performed with a general- purpose processor, an Al processor, a digital signal processor (DSP), an ASIC, a field programmable gate array (FPGA) or other programmable logic device (PhD), discrete gate or transistor logic, discrete hardware components, or any combination thereof designed to perform the functions described herein. A general-purpose processor may be a microprocessor, but in the alternative, the processor may be any commercially available processor, controller, microcontroller, or state machine. A processor may also be implemented as a combination of computing devices, e.g., a combination of a DSP and a microprocessor, a plurality of microprocessors, one or more microprocessors in conjunction with a DSP core, a system on a chip (SoC), or any other such configuration.

[0316] As used herein, a phrase referring to “at least one of’ a list of items refers to any combination of those items, including single members. As an example, “at least one of: a, b, or c” is intended to cover a, b, c, a-b, a-c, b-c, and a-b-c, as well as anyD&S Ref. No.: QCM2404871WOQualcomm Ref. No.: 2404871WO73 combination with multiples of the same element (e.g., a-a, a-a-a, a-a-b, a-a-c, a-b-b, a c c, b-b, b-b-b, b-b-c, c-c, and c-c-c or any other ordering of a, b, and c).

[0317] As used herein, the term “determining” encompasses a wide variety of actions. For example, “determining” may include calculating, computing, processing, deriving, investigating, looking up (e.g., looking up in a table, a database or another data structure), ascertaining and the like. Also, “determining” may include receiving (e.g., receiving information), accessing (e.g., accessing data in a memory) and the like. Also, “determining” may include resolving, selecting, choosing, establishing and the like.

[0318] As used herein, “coupled to” and “coupled with” generally encompass direct coupling and indirect coupling (e.g., including intermediary coupled aspects) unless stated otherwise. For example, stating that a processor is coupled to a memory allows for a direct coupling or a coupling via an intermediary aspect, such as a bus.

[0319] The methods disclosed herein comprise one or more actions for achieving the methods. The method actions may be interchanged with one another without departing from the scope of the claims. In other words, unless a specific order of actions is specified, the order and / or use of specific actions may be modified without departing from the scope of the claims. Further, the various operations of methods described above may be performed by any suitable means capable of performing the corresponding functions. The means may include various hardware and / or software component(s) and / or module(s), including, but not limited to a circuit, an application specific integrated circuit (ASIC), or processor.

[0320] The following claims are not intended to be limited to the aspects shown herein, but are to be accorded the full scope consistent with the language of the claims. Reference to an element in the singular is not intended to mean only one unless specifically so stated, but rather “one or more.” The subsequent use of a definite article (e.g., “the” or “said”) with an element (e.g., “the processor”) is not intended to invoke a singular meaning (e.g., “only one”) on the element unless otherwise specifically stated. For example, reference to an element (e.g., “a processor,” “a controller,” “a memory,” “a transceiver,” “an antenna,” “the processor,” “the controller,” “the memory,” “the transceiver,” “the antenna,” etc.), unless otherwise specifically stated, should be understood to refer to one or more elements (e.g., “one or more processors,” “one or more controllers,” “one or more memories,” “one more transceivers,” etc.). The terms “set” andD&S Ref. No.: QCM2404871WOQualcomm Ref. No.: 2404871WO74“group” are intended to include one or more elements, and may be used interchangeably with “one or more.” Where reference is made to one or more elements performing functions (e.g., steps of a method), one element may perform all functions, or more than one element may collectively perform the functions. When more than one element collectively performs the functions, each function need not be performed by each of those elements (e.g., different functions may be performed by different elements) and / or each function need not be performed in whole by only one element (e.g., different elements may perform different sub-functions of a function). Similarly, where reference is made to one or more elements configured to cause another element (e.g., an apparatus) to perform functions, one element may be configured to cause the other element to perform all functions, or more than one element may collectively be configured to cause the other element to perform the functions. Unless specifically stated otherwise, the term “some” refers to one or more. All structural and functional equivalents to the elements of the various aspects described throughout this disclosure that are known or later come to be known to those of ordinary skill in the art are intended to be encompassed by the claims. Moreover, nothing disclosed herein is intended to be dedicated to the public regardless of whether such disclosure is explicitly recited in the claims.D&S Ref. No.: QCM2404871WO

Claims

Qualcomm Ref. No.: 2404871WO75CLAIMS1. An apparatus configured for machine learning model performance monitoring, comprising: one or more memories; and one or more processors coupled to the one or more memories and configured to cause the apparatus to: monitor a first metric associated with a machine learning model using a first level of monitoring granularity, wherein the machine learning model is associated with wireless communication, and wherein the first level of monitoring granularity comprises a first set of one or more parameters associated with a performance of the machine learning model; monitor, in response to a trigger condition associated with the first metric being satisfied, a second metric associated with the machine learning model using a second level of monitoring granularity, wherein the second metric is configured to provide a different measure of the machine learning model than the first metric, wherein the second level of monitoring granularity comprises a second set of one or more parameters associated with the performance of the machine learning model, and wherein the second set of one or more parameters includes at least one parameter that is not included in the first set of one or more parameters; and signal, in response to the second metric satisfying a reporting condition that is not satisfied by the first metric, an indication associated with the performance of the machine learning model.

2. The apparatus of claim 1, wherein the one or more processors are further configured to cause the apparatus to cause an adjustment of one or more operating parameters of at least one of the machine learning model or the apparatus based on the indication.

3. The apparatus of claim 1, wherein the first set of one or more parameters and the second set of one or more parameters are hierarchically related, such that the second set of one or more parameters includes at least one parameter that provides additional information about the performance of the machine learning model beyond information provided by the first set of one or more parameters.

4. The apparatus of claim 1, wherein the trigger condition is configured to indicate a potential performance issue with the machine learning model based on the first metric.D&S Ref. No.: QCM2404871WOQualcomm Ref. No.: 2404871WO765. The apparatus of claim 1 , wherein the apparatus is one of a user equipment (UE) or a network entity in a wireless communications network.

6. The apparatus of claim 1, wherein the first set of one or more parameters and the second set of one or more parameters are based on different sets of resources, and wherein the second set of one or more parameters is based on a larger set of resources than the first set of one or more parameters.

7. The apparatus of claim 1, wherein the trigger condition is based on a comparison of the first metric to a threshold value.

8. The apparatus of claim 1, wherein the one or more processors are further configured to cause the apparatus to monitor, in response to a second trigger condition associated with the second metric being satisfied, a third metric associated with the machine learning model using a third level of monitoring granularity, wherein the third metric is configured to provide a different measure of the machine learning model than the first metric and the second metric.

9. The apparatus of claim 1, wherein the one or more processors are further configured to cause the apparatus to receive a configuration from a second apparatus, the configuration including an indication of at least one of: the first metric, the trigger condition, a threshold value for the first metric, the second metric, or a time period to monitor the first metric.

10. The apparatus of claim 1, wherein the one or more processors are further configured to cause the apparatus to: send, to a second apparatus, a request for assistance to monitor the machine learning model; and receive, from the second apparatus, an indication of one or more monitoring parameters in response to the request, wherein the one or more monitoring parameters include at least one of: a threshold value for the first metric, a threshold value for the second metric,D&S Ref. No.: QCM2404871WOQualcomm Ref. No.: 2404871WO77 a time period in which to monitor the first metric, a time period in which to monitor the second metric, a set of resources on which to monitor the first metric, or a set of resources on which to monitor the second metric.

11. The apparatus of claim 1, wherein the one or more processors are further configured to cause the apparatus to adjust monitoring of the first metric or the second metric based on at least one of the first set of one or more parameters or the second set of one or more parameters.

12. The apparatus of claim 1, wherein the one or more processors are further configured to cause the apparatus to: receive, from a second apparatus, a request for assistance with monitoring the machine learning model; and send, to the second apparatus, an indication of one or more monitoring parameters in response to the request.

13. The apparatus of claim 1, wherein the one or more processors are further configured to cause the apparatus to revert, in response to a third trigger condition being satisfied, to monitor the first metric, wherein the third trigger condition is satisfied when a timer associated with monitoring the second metric expires.

14. The apparatus of claim 13, wherein the one or more processors are further configured to cause the apparatus to send, to a second apparatus, an indication that the apparatus has reverted to monitor the first metric.

15. The apparatus of claim 1, wherein the one or more processors are further configured to cause the apparatus to: receive, from a second apparatus, an instruction to revert to monitoring the first metric; and revert to monitoring the first metric in response to the instruction.

16. The apparatus of claim 1, wherein the one or more processors are further configured to cause the apparatus to perform ground truth based monitoring to monitor the first metric or the second metric, and wherein to cause the apparatus to perform theD&S Ref. No.: QCM2404871WOQualcomm Ref. No.: 2404871WO78 ground truth based monitoring, the one or more processors are further configured to cause the apparatus to: receive, from a second apparatus, a set of ground truth data; and compare the first metric or the second metric to the set of ground truth data, wherein the set of ground truth data includes at least one of: a set of input data and corresponding expected output data for the machine learning model, a set of measurements of a wireless environment, or a set of parameters of a wireless communication in a wireless communications network.

17. The apparatus of claim 1, wherein the one or more processors are further configured to cause the apparatus to perform ground truth- free monitoring to monitor the first metric or the second metric, wherein to cause the apparatus to perform ground truth- free monitoring to monitor the first metric or the second metric, the one or more processors are further configured to cause the apparatus to analyze the first metric or the second metric without ground truth data, and wherein the one or more processors, to analyze the first metric or the second metric without ground truth data, are further configured to cause the apparatus to: compare the first metric or the second metric to a threshold, evaluate a statistical property of the first metric or the second metric, or detect an anomaly in the first metric or the second metric.

18. The apparatus of claim 1, wherein the one or more processors are further configured to cause the apparatus to perform key performance indicator (KPI)-based monitoring to monitor the first metric or the second metric, and wherein to perform KPIbased monitoring to monitor the first metric or the second metric, the one or more processors are further configured to cause the apparatus to: receive, from a second communication device, a set of KPIs; and compare the first metric or the second metric to the set of KPIs, wherein the set of KPIs includes at least one of: a block error rate (BLER), a packet loss rate, a bit error rate (BER),D&S Ref. No.: QCM2404871WOQualcomm Ref. No.: 2404871WO79 a throughput, a spectral efficiency, a latency, or a jitter.

19. A method for wireless communications by an apparatus comprising: monitoring a first metric associated with a machine learning model using a first level of monitoring granularity, wherein the machine learning model is associated with wireless communication, and wherein the first level of monitoring granularity comprises a first set of one or more parameters associated with a performance of the machine learning model; monitoring, in response to a trigger condition associated with the first metric being satisfied, a second metric associated with the machine learning model using a second level of monitoring granularity, wherein the second metric is configured to provide a different measure of the machine learning model than the first metric, wherein the second level of monitoring granularity comprises a second set of one or more parameters associated with the performance of the machine learning model, and wherein the second set of one or more parameters includes at least one parameter that is not included in the first set of one or more parameters; and signaling, in response to the second metric satisfying a reporting condition that is not satisfied by the first metric, an indication associated with the performance of the machine learning model.

20. An apparatus configured for wireless communications, comprising: one or more memories; and one or more processors coupled to the one or more memories and configured to cause the apparatus to: obtain a first metric associated with a model; determine that a trigger condition associated with the first metric is satisfied; provide a request for assistance in response to the trigger condition being satisfied;D&S Ref. No.: QCM2404871WOQualcomm Ref. No.: 2404871WO80 receive a monitoring configuration based on the request for assistance, the monitoring configuration including an instruction to obtain a second metric associated with the model, the second metric providing a different measure of the model than the first metric; obtain the second metric based on the monitoring configuration; and perform an action based on the second metric, wherein the action includes at least one of to: adapt the model based on the second metric; transmit the second metric to a network entity; receive an adapted model based on the second metric; or receive an instruction to adapt the model based on the second metric.D&S Ref. No.: QCM2404871WO

Images