Multi-stage machine learning model monitoring

Abstract

Methods, systems, and devices for wireless communication are described. For example, the described techniques provide for performing a first monitoring process that includes monitoring performance of each respective machine learning model in a set of machine learning models configured at a network entity. The network entity may determine a first subset of the set of machine learning models based on the first monitoring procedure. The network entity may perform a second monitoring process that includes monitoring performance of each respective machine learning model in the first subset of the set of machine learning models.

Description

Background Technology

[0001] The following discussion pertains to wireless communication in relation to monitoring machine learning models. Wireless communication systems are widely deployed to provide various types of communication content, such as voice, video, packet data, message sending and receiving, and broadcasting. These systems can support communication with multiple users by sharing available system resources, such as time, frequency, and power. Examples of such multiple access systems include fourth-generation (4G) systems (such as Long Term Evolution (LTE) systems, LTE-A Advanced (LTE-A) systems, or LTE-A Pro systems) and fifth-generation (5G) systems (which may be referred to as New Radio (NR) systems). These systems may employ technologies such as Code Division Multiple Access (CDMA), Time Division Multiple Access (TDMA), Frequency Division Multiple Access (FDMA), Orthogonal FDMA (OFDMA), or Discrete Fourier Transform Extended Orthogonal Frequency Division Multiplexing (DFT-S-OFDM). A wireless multiple access communication system may include one or more base stations, each supporting wireless communication for communication devices, which may be referred to as User Equipment (UE). Summary of the Invention

[0002] The described technology relates to improved methods, systems, devices, and apparatuses supporting multi-stage machine learning model monitoring. For example, the described technology provides for performing a first monitoring process in a first monitoring phase. The first monitoring process is used to monitor the performance of each corresponding machine learning model in a set of machine learning models configured at a network entity. The network entity may determine a first subset of the set of machine learning models based on the first monitoring process. The first subset may be one or more machine learning models and represents those machine learning models that are most accurate for the conditions in which the network entity is operating. The network entity may then perform a second monitoring process during a second monitoring phase. The second monitoring process may differ from the first monitoring process and may include monitoring the performance of each corresponding machine learning model in the first subset of the set of machine learning models.

[0003] A method performed by a network entity is described. The method may include: performing a first monitoring process, wherein performing the first monitoring process includes monitoring the performance of each corresponding machine learning model in a set of machine learning models configured at the network entity, such that the set of machine learning models can be executed by the network entity; determining a first subset of the set of machine learning models based on the first monitoring process; and performing a second monitoring process, wherein performing the second monitoring process includes monitoring the performance of each corresponding machine learning model in the first subset of the set of machine learning models.

[0004] A network entity is described. The network entity may include: one or more memories storing processor-executable code; and one or more processors coupled to the one or more memories. The one or more processors may be able to operate individually or jointly to execute code such that the network entity: performs a first monitoring process, wherein performing the first monitoring process includes monitoring the performance of each corresponding machine learning model in a set of machine learning models configured at the network entity, such that the set of machine learning models can be executed by the network entity; determines a first subset of the set of machine learning models based on the first monitoring process; and performs a second monitoring process, wherein performing the second monitoring process includes monitoring the performance of each corresponding machine learning model in the first subset of the set of machine learning models.

[0005] Another network entity is described. This network entity may include: components for performing a first monitoring process, wherein performing the first monitoring process includes monitoring the performance of each corresponding machine learning model in a set of machine learning models configured at the network entity, such that the set of machine learning models can be executed by the network entity; components for determining a first subset of the set of machine learning models based on the first monitoring process; and components for performing a second monitoring process, wherein performing the second monitoring process includes monitoring the performance of each corresponding machine learning model in the first subset of the set of machine learning models.

[0006] A non-transitory computer-readable medium having code stored thereon is described. When executed, the code enables a device to: perform a first monitoring process, wherein performing the first monitoring process includes monitoring the performance of each corresponding machine learning model in a set of machine learning models configured at a network entity, such that the set of machine learning models can be executed by the network entity; determine a first subset of the set of machine learning models based on the first monitoring process; and perform a second monitoring process, wherein performing the second monitoring process includes monitoring the performance of each corresponding machine learning model in the first subset of the set of machine learning models.

[0007] In some examples of the methods, network entities, and nontransitory computer-readable media described herein, determining the first subset of the machine learning model set may include operations, features, components, or instructions for comparing the input data distribution information with an operational data distribution corresponding to one or more operational conditions of the processing system, wherein the first subset of the machine learning model set may be determined based on the comparison.

[0008] In some examples of the methods, network entities, and nontransitory computer-readable media described herein, monitoring the performance of each corresponding machine learning model in the first subset of the set of machine learning models may include operations, features, components, or instructions for comparing input-output data distribution information with operational data distributions corresponding to one or more operational conditions of the network entity.

[0009] In some examples of the methods, network entities, and nontransitory computer-readable media described herein, monitoring the performance of each corresponding machine learning model in the first subset of the machine learning model set may include operations, features, components, or instructions for generating the input-output data distribution information based on each corresponding machine learning model in the first subset of the machine learning model set.

[0010] In some examples of the methods, network entities, and nontransitory computer-readable media described herein, monitoring the performance of each corresponding machine learning model in the first subset of the machine learning model set may include operations, features, components, or instructions for: measuring one or more first performance-related parameters; and comparing one or more second performance-related parameters associated with each corresponding machine learning model in the first subset of the machine learning model set with the measured one or more first performance-related parameters.

[0011] In some examples of the methods, network entities, and nontransitory computer-readable media described herein, the second monitoring process may be an inferential performance monitoring process, wherein the one or more second performance-related parameters that can be compared with the one or more first performance-related parameters may be parameters that can be output by each corresponding machine learning model in the first subset of the machine learning model set.

[0012] In some examples of the methods, network entities, and nontransitory computer-readable media described herein, the second monitoring process may be an end-to-end or system performance monitoring process in which one or more performance-related parameters associated with each corresponding machine learning model in the first subset of the set of machine learning models may be compared with threshold performance-related parameters.

[0013] In some examples of the methods, network entities, and nontransitory computer-readable media described herein, determining the first subset of the set of machine learning models may include operations, features, components, or instructions for: determining the distributional similarity between the training dataset associated with each corresponding machine learning model in the set of machine learning models and the data distribution corresponding to one or more operating conditions of the network entity; and determining the first subset of the set of machine learning models based on a comparison of each corresponding distributional similarity with a threshold similarity.

[0014] Some examples of the methods, network entities, and nontransitory computer-readable media described herein may also include operations, features, components, or instructions for: comparing input-output data distribution information with operational data distributions corresponding to one or more operational conditions of the network entity during the second monitoring process, wherein the input-output data distribution information includes a corresponding input-output data distribution associated with each corresponding machine learning model in the set of machine learning models; detecting data drift based on the second monitoring process, wherein the data drift may indicate that one or more corresponding distribution similarities between the input-output distributions are below a threshold distribution similarity relative to the operational data distribution; and, in response to detecting the data drift, re-monitoring the performance of each corresponding machine learning model in the set of machine learning models configured at the network entity via the first monitoring process.

[0015] Some examples of the methods, network entities, and nontransitory computer-readable media described herein may also include operations, features, components, or instructions for receiving instructions for at least one of the first monitoring process or the second monitoring process, wherein at least one of the following exists: performing the first monitoring process includes performing the first monitoring process according to the instruction; and performing the second monitoring process includes performing the second monitoring process according to the instruction.

[0016] Some examples of the methods, network entities, and nontransitory computer-readable media described herein may also include operations, features, components, or instructions for receiving an instruction to switch from the first monitoring process to the second monitoring process, wherein the first monitoring process and the second monitoring process are performed according to the instruction.

[0017] Some examples of the methods, network entities, and nontransitory computer-readable media described herein may also include operations, features, components, or instructions for: determining a second subset of the machine learning model set based on the second monitoring process, wherein the second subset includes one or more machine learning models in the first subset of the machine learning model set; and performing a third monitoring process, wherein performing the third monitoring process includes monitoring the performance of each corresponding machine learning model in the second subset of the machine learning model set.

[0018] The methods, network entities, and some examples of nontransitory computer-readable media described herein may also include operations, features, components, or instructions for sending capability information instructing the network entity to monitor the set of machine learning models via a multi-process monitoring process.

[0019] In some examples of the methods, network entities, and nontransitory computer-readable media described herein, the multi-process monitoring process includes at least a first monitoring process and a second monitoring process.

[0020] Some examples of the methods, network entities, and nontransitory computer-readable media described herein may also include operations, features, components, or instructions for sending instructions on at least one of the first monitoring process or the second monitoring process before applying a multi-process monitoring process to the set of machine learning models, wherein the multi-process monitoring process includes at least the first monitoring process and the second monitoring process.

[0021] Some examples of the methods, network entities, and nontransitory computer-readable media described herein may also include operations, features, components, or instructions for receiving configurations for at least one machine learning model in the set of machine learning models, or for one or more of the first monitoring process or the second monitoring process, via Radio Resource Control (RRC) messages, Media Access Control-Control Element (MAC-CE) messages, Downlink Control Information (DCI) messages, or combinations thereof.

[0022] In some examples of the methods, network entities, and non-transitory computer-readable media described herein, the first monitoring process may have lower processing complexity than the second monitoring process.

[0023] A method performed by a first network entity is described. The method may include: receiving a capability report indicative of the capability of a second network entity to monitor the performance of individual machine learning models in a machine learning model set via a multi-stage monitoring process; and sending control information to the second network entity indicating at least two or more machine learning models in the machine learning model set, wherein the control information further indicates a first monitoring process of a first monitoring process type and a second monitoring process of a second monitoring process type, wherein the first and second monitoring processes are used by the second network entity to monitor the two or more machine learning models in the machine learning model set, and wherein the second monitoring process is based on the first monitoring process, and wherein the first monitoring process type has lower processing complexity than the second monitoring process type.

[0024] A first network entity is described. The first network entity may include: one or more memories storing processor-executable code; and one or more processors coupled to the one or more memories. The one or more processors may be capable of operating individually or jointly to execute code, enabling the first network entity to: receive a capability report indicative of the capability of the second network entity to monitor the performance of individual machine learning models in a machine learning model set via a multi-stage monitoring process; and send control information to the second network entity indicative of at least two or more machine learning models in the machine learning model set, wherein the control information further indicative of a first monitoring process of a first monitoring process type and a second monitoring process of a second monitoring process type, wherein the first monitoring process and the second monitoring process are used by the second network entity to monitor the two or more machine learning models in the machine learning model set, and wherein the second monitoring process is based on the first monitoring process, and wherein the first monitoring process type has lower processing complexity than the second monitoring process type.

[0025] Another first network entity is described. This first network entity may include: components for receiving a capability report indicative of the capability of a second network entity to monitor the performance of individual machine learning models in a machine learning model set via a multi-stage monitoring process; and components for sending control information to the second network entity indicating at least two or more machine learning models in the machine learning model set, wherein the control information further indicates a first monitoring process of a first monitoring process type and a second monitoring process of a second monitoring process type, wherein the first and second monitoring processes are used by the second network entity to monitor the two or more machine learning models in the machine learning model set, and wherein the second monitoring process is based on the first monitoring process, and wherein the first monitoring process type has lower processing complexity than the second monitoring process type.

[0026] A non-transitory computer-readable medium having code stored thereon is described. When executed, the code enables a device to: receive a capability report instructing a second network entity on its ability to monitor the performance of individual machine learning models in a machine learning model set via a multi-stage monitoring process; and send control information to the second network entity instructing at least two or more machine learning models in the machine learning model set, wherein the control information further instructs a first monitoring process of a first monitoring process type and a second monitoring process of a second monitoring process type, wherein the first and second monitoring processes are for monitoring the two or more machine learning models in the machine learning model set by the second network entity, and wherein the second monitoring process is based on the first monitoring process, and wherein the first monitoring process type has lower processing complexity than the second monitoring process type.

[0027] The methods described herein, examples of the first network entity, and some examples of nontransitory computer-readable media may also include operations, features, components, or instructions for sending an instruction to switch the second network entity to use either the first monitoring process type or the second monitoring process type.

[0028] The methods described herein, examples of the first network entity, and some examples of nontransitory computer-readable media may also include operations, features, components, or instructions for: receiving a first instruction for a third monitoring process before applying the multi-process monitoring process to two or more machine learning models in the set of machine learning models; and, in response to the first instruction, sending a second instruction to the second network entity to switch to using the third monitoring process during the multi-stage monitoring process.

[0029] In some examples of the methods described herein, the first network entity, and non-transitory computer-readable media, the control information may be included in RRC messages, MAC-CE messages, DCI messages, or combinations thereof. Attached Figure Description

[0030] Figure 1 and Figure 2 An example of a wireless communication system supporting multi-stage machine learning model monitoring according to one or more aspects of this disclosure is shown.

[0031] Figure 3 and Figure 4 An example of a flowchart supporting multi-stage machine learning model monitoring is shown, according to one or more aspects of this disclosure.

[0032] Figure 5 An example of a process flow supporting multi-stage machine learning model monitoring is shown, according to one or more aspects of this disclosure.

[0033] Figure 6 and Figure 7 A block diagram of an apparatus supporting multi-stage machine learning model monitoring according to one or more aspects of this disclosure is shown.

[0034] Figure 8 A block diagram of a communication manager supporting multi-stage machine learning model monitoring according to one or more aspects of this disclosure is shown.

[0035] Figure 9 A diagram of a system including a device supporting multi-stage machine learning model monitoring, according to one or more aspects of this disclosure, is shown.

[0036] Figures 10 to 12 A flowchart illustrating a method for supporting multi-stage machine learning model monitoring according to one or more aspects of this disclosure is shown. Detailed Implementation

[0037] Wireless devices (such as user equipment (UE) or base stations or components thereof, each of which may be referred to herein as a network entity) may perform machine learning model monitoring. For example, a wireless device may be configured with multiple machine learning models. To verify the accuracy of a given model for the conditions in which the wireless device is operating, the wireless device may monitor each of the machine learning models to determine which model, if any, is suitable for use. As an example, the wireless device may monitor the discrepancies between the data and the machine learning model after the machine learning model has been deployed (e.g., data-concept drift). The machine learning model may deviate from the data based on changes in one or more operating conditions. For example, the machine learning model may be associated with a set of operating conditions (e.g., with the training dataset of the machine learning model), and the model's performance may degrade when operating outside the set of operating conditions. Therefore, the wireless device may monitor the machine learning model to determine whether the set of operating conditions is met, and based on the determination that the set of operating conditions is not met, switch to a different machine learning model or a non-machine learning baseline, or retrain the machine learning model in a different environment, etc. In some cases, the wireless device may be configured with a set of machine learning models and may monitor this set to determine which machine learning model is best suited for the operating conditions. However, monitoring a set of machine learning models may be associated with high complexity and excessive processing requirements.

[0038] To reduce the overall processing associated with monitoring multiple machine learning models, wireless devices can perform machine learning model monitoring through a multi-stage process. The wireless device can monitor a set of machine learning models at least according to a first monitoring process during a first monitoring phase and a second monitoring process during a second monitoring phase. For example, the first monitoring process can be applied to all configured machine learning models at the wireless device and can utilize a monitoring process that is not as processing-intensive as other potential monitoring processes. For example, the first monitoring process can monitor model performance without actually running or executing each model (e.g., by monitoring the input data distribution of each model and comparing the corresponding input data distribution with the current operating conditions). The wireless device can determine a subset of the set of machine learning models based on the first monitoring process. For example, the wireless device can determine a subset of machine learning models whose input data distribution is most similar to the operating conditions. Then, once the total number of machine learning models has been narrowed down through the first monitoring phase, the wireless device can execute a second monitoring process (during the second monitoring phase) to monitor the subset of machine learning models. The second monitoring process may be more processing-intensive than the first monitoring process, which is acceptable because the second monitoring process is only applied to a subset of machine learning models. For example, the second monitoring process may include running each model (e.g., to allow comparison of operating conditions and actual measurements with the model's input-output data distribution, or to compare actual device performance with predicted performance via a performance-based monitoring approach, etc.). The wireless device may switch machine learning models based on monitoring (e.g., autonomously), or may signal instructions for the selected machine learning model based on operating conditions.

[0039] The aspects of this disclosure are initially described in the context of wireless communication systems. These aspects are further described in the context of flowcharts and process flows. The aspects of this disclosure are further illustrated by apparatus diagrams, system diagrams, and flowcharts relating to monitoring of multi-stage machine learning models, and are further described with reference to these diagrams.

[0040] Figure 1 An example of a wireless communication system 100 supporting multi-stage machine learning model monitoring according to one or more aspects of this disclosure is shown. The wireless communication system 100 may include one or more network entities 105, one or more UEs 115, and a core network 130. In some aspects, the wireless communication system 100 may be a Long Term Evolution (LTE) network, an Advanced LTE (LTE-A) network, an LTE-A Pro network, a New Radio (NR) network, or a network operating according to other systems and radio technologies, including future systems and radio technologies not explicitly mentioned herein.

[0041] Network entity 105 may be distributed across a geographical area to form wireless communication system 100 and may include devices in different forms or with different capabilities. In various examples, network entity 105 may be referred to as a network element, mobility element, radio access network (RAN) node, or network equipment, etc. In some aspects, network entity 105 and UE 115 may wirelessly communicate via one or more communication links 125 (e.g., radio frequency (RF) access links). For example, network entity 105 may support coverage area 110 (e.g., a geographical coverage area) within which UE 115 and network entity 105 may establish one or more communication links 125. Coverage area 110 may be an example of a geographical area within which network entity 105 and UE 115 may support the transmission of signals according to one or more radio access technologies (RATs).

[0042] UE 115 can be distributed throughout the coverage area 110 of wireless communication system 100, and each UE 115 can be stationary or mobile, or stationary and mobile at different times. UE 115 can be devices in different forms or with different capabilities. Figure 1 Some example UE 115s are illustrated herein. The UE 115 described herein can be able to support various types of devices, such as... Figure 1 The other UE 115 or network entity 105 shown communicates with it.

[0043] As described herein, a network entity (which may alternatively be referred to as an entity, node, network node, or wireless entity) can be, can be similar to, can include, or can be included in (e.g., can be a component of) the following: base station (e.g., any base station described herein, including a decomposed base station), UE (e.g., any UE described herein), RedCap device, eRedCap device, ambient Internet of Things (IoT) device, energy harvesting (EH) capable device, network controller, apparatus, device, computing system, integrated access and backhaul (IAB) node, distributed unit (DU), central unit (CU), remote / radio unit (RU) (which may also be referred to as a remote radio unit (RRU)), and / or another processing entity configured to perform any of the techniques described herein. For example, a network entity can be a UE. As another example, a network entity can be a base station. As used herein, “network entity” can mean an entity configured to operate in a network (such as network entity 105). For example, “network entity” is not limited to entities currently located in and / or currently operating in a network. Instead, a network entity can be any entity capable of communicating and / or operating within a network.

[0044] The adjectives "first," "second," "third," etc., are used to distinguish between two or more modified nouns in context, and do not imply absolute modifiers applicable only to a specific corresponding entity throughout the document. For example, a network entity may be referred to as "first network entity" in one discussion and as "second network entity" in another, and vice versa. As an example, the first network entity may be configured to communicate with a second network entity or a third network entity. In one aspect of this example, the first network entity may be a UE, the second network entity may be a base station, and the third network entity may be a UE. In another aspect of this example, the first network entity may be a UE, the second network entity may be a base station, and the third network entity may be a base station. In yet other aspects of this example, the first, second, and third network entities may be different from these examples.

[0045] Similarly, references to UE, base station, device, equipment, or computing system may include disclosures of UE, base station, device, equipment, or computing system as network entities. For example, a disclosure of a UE being configured to receive information from a base station also discloses that a first network entity is configured to receive information from a second network entity. Consistent with this disclosure, once a particular example is extended according to this disclosure (e.g., a disclosure of a UE being configured to receive information from a base station also discloses that a first network entity is configured to receive information from a second network entity), a broader example of a narrower example may be interpreted in reverse, but in a broad, open-ended manner. In the above example where a UE is configured to receive information from a base station and a first network entity is configured to receive information from a second network entity, the first network entity may refer to a first UE, a first base station, a first device, a first equipment, a first computing system, a first set of one or more components, or a first processing entity, etc., configured to receive information; and the second network entity may refer to a second UE, a second base station, a second device, a second equipment, a second computing system, a second set of one or more components, or a second processing entity, etc.

[0046] As described herein, different terms may be used in various contexts to describe the transmission of information (e.g., any information, signal, etc.). Disclosure of one communication term includes disclosure of other communication terms. For example, a first network entity may be described as being configured to send information to a second network entity. In this example and consistent with this disclosure, disclosure that a first network entity is configured to send information to a second network entity includes disclosure that the first network entity is configured to provide, transmit, output, communicate, or send information to the second network entity. Similarly, in this example and consistent with this disclosure, disclosure that a first network entity is configured to send information to a second network entity includes disclosure that the second network entity is configured to receive, obtain, or decode information provided, transmitted, output, communicate, or sent by the first network entity.

[0047] As shown in the figure, a network entity (e.g., network entity 105) may include a processing system 106. Similarly, a network entity (e.g., UE 115) may include a processing system 112. A processing system may include one or more components (or sub-components), such as those described herein. For example, a corresponding component among these one or more components may be, similar to, include, or be included in at least one memory, at least one communication interface, or at least one processor. For example, a processing system may include one or more components. In such an example, the one or more components may include a first component, a second component, and a third component. In this example, the first component may be coupled to the second and third components. In this example, the first component may be at least one processor, the second component may be a communication interface, and the third component may be at least one memory. A processing system is generally one or more components of a system capable of performing one or more functions (such as any function or combination of functions described herein). For example, one or more components may receive input information (e.g., any information as input, such as a signal, any digital information, or any other information), one or more components may process the input information to generate output information (e.g., any information as output, such as a signal or any other information), one or more components may perform any function as described herein or any combination thereof. As described herein, “input” and “input information” can be used interchangeably. Similarly, as described herein, “output” and “output information” can be used interchangeably. Any information generated by any component can be provided to one or more other systems or components of network entities such as those described herein. For example, a processing system may include a first component configured to receive or obtain information, a second component configured to process the information to generate output information, and / or a third component configured to provide the output information to other systems or components. In this example, the first component may be a communication interface (e.g., a first communication interface), the second component may be at least one processor (e.g., coupled to the communication interface and / or at least one memory), and the third component may be a communication interface (e.g., a first communication interface or a second communication interface). For example, a processing system may include at least one memory, at least one communication interface, and / or at least one processor, wherein the at least one processor may, for example, be coupled to the at least one memory and the at least one communication interface.

[0048] The processing system of the network entity described herein can interface with one or more other components of the network entity, process information received from one or more other components (such as input information), or output such information to one or more other components. For example, the processing system may include a first component configured to interface with one or more other components of the network entity to receive or obtain information, a second component configured to process the information to generate one or more outputs, and / or a third component configured to output the one or more outputs to one or more other components. In this example, the first component may be a communication interface (e.g., a first communication interface), the second component may be at least one processor (e.g., coupled to the communication interface and / or at least one memory), and the third component may be a communication interface (e.g., the first communication interface or the second communication interface). For example, a chip or modem of the network entity may include the processing system. The processing system may include a first communication interface for receiving or obtaining information, and a second communication interface for outputting, transmitting, or providing information. In some aspects, the first communication interface may be an interface configured to receive input information, and such information may be provided to the processing system. In some aspects, the second system interface may be configured to transmit information output from the chip or modem. The second communication interface can also obtain or receive input information, and the first communication interface can also output, send, or provide information.

[0049] In some aspects, network entity 105 may communicate with core network 130, communicate with each other, or both. For example, network entity 105 may communicate with core network 130 via one or more backhaul communication links 120 (e.g., according to S1, N2, N3, or other interface protocols). In some aspects, network entities 105 may communicate with each other directly (e.g., directly between network entities 105) or indirectly (e.g., via core network 130) via backhaul communication links 120 (e.g., according to X2, Xn, or other interface protocols). In some aspects, network entities 105 may communicate with each other via midhaul communication link 162 (e.g., according to midhaul interface protocol) or fronthaul communication link 168 (e.g., according to fronthaul interface protocol) or any combination thereof. Backhaul communication link 120, midhaul communication link 162, or fronthaul communication link 168 may be or include one or more wired links (e.g., electrical links, fiber optic links), one or more wireless links (e.g., radio links, wireless optical links), etc., or various combinations thereof. UE 115 can communicate with core network 130 via communication link 155.

[0050] One or more network entities in network entity 105 described herein may include or be referred to as base station 140 (e.g., transceiver base station, radio base station, NR base station, access point, radio transceiver, node B, eNodeB (eNB), next-generation node B or gigabit node B (any of which may be referred to as gNB), 5G NB, next-generation eNB (ng-eNB), home node B, home evolution node B, or other suitable terms). In some aspects, network entity 105 (e.g., base station 140) may be implemented in an aggregated (e.g., monolithic, self-contained) base station architecture that may be configured to utilize a protocol stack physically or logically integrated within a single network entity 105 (e.g., a single RAN node, such as base station 140).

[0051] In some aspects, network entity 105 may be implemented in a decomposed architecture (e.g., a decomposed base station architecture, a decomposed RAN architecture) that can be configured to utilize a protocol stack physically or logically distributed between two or more network entities 105 (such as an integrated access backhaul (IAB) network, an open RAN (O-RAN) (e.g., a network configuration sponsored by the O-RAN Alliance), or a virtualized RAN (vRAN) (e.g., a cloud RAN (C-RAN))). For example, network entity 105 may include one or more of the following: a central unit (CU) 160, a distributed unit (DU) 165, a radio unit (RU) 170, a RAN intelligent controller (RIC) 175 (e.g., a near real-time RIC (near RT RIC), a non-real-time RIC (non-RT RIC)), a service management and orchestration (SMO) 180 system, or any combination thereof. RU 170 may also be referred to as a radio headend, an intelligent radio headend, a remote radio headend (RRH), a remote radio unit (RRU), or a transmit-receive point (TRP). One or more components of network entity 105 in a decomposed RAN architecture may be co-located, or one or more components of network entity 105 may be located in distributed locations (e.g., separate physical locations). In some aspects, one or more network entities 105 in a decomposed RAN architecture may be implemented as virtual units (e.g., virtual CU (VCU), virtual DU (VDU), virtual RU (VRU)).

[0052] The functional splitting among CU 160, DU 165, and RU 170 is flexible and can support different functionalities depending on which functions (e.g., network layer functions, protocol layer functions, baseband functions, RF functions, and any combination thereof) are performed at CU 160, DU 165, or RU 170. For example, a protocol stack functional splitting can be used between CU 160 and DU 165, allowing CU 160 to support one or more layers of the protocol stack, and DU 165 to support one or more different layers of the protocol stack. In some respects, CU 160 can host higher protocol layer (e.g., Layer 3 (L3), Layer 2 (L2)) functionalities and signaling (e.g., Radio Resource Control (RRC), Serving Data Adaptation Protocol (SDAP), Packet Data Convergence Protocol (PDCP)). The CU 160 can connect to one or more DU 165s or RU 170s, and the one or more DU 165s or RU 170s can host lower protocol layers, such as Layer 1 (L1) (e.g., Physical (PHY) layer) or L2 (e.g., Radio Link Control (RLC) layer, Medium Access Control (MAC) layer) functionality and signaling, and each can be at least partially controlled by the CU 160. Additionally or alternatively, a protocol stack functional split can be employed between the DU 165 and RU 170, such that the DU 165 can support one or more layers of the protocol stack, and the RU 170 can support one or more different layers of the protocol stack. The DU 165 can (e.g., via one or more RU 170s) support one or more different cells. In some cases, functional decomposition between CU 160 and DU 165, or between DU 165 and RU 170, can be performed within the protocol layer (e.g., some functions of the protocol layer can be performed by one of CU 160, DU 165, or RU 170, while other functions of the protocol layer can be performed by different of CU 160, DU 165, or RU 170). CU 160 can be further functionally decomposed into CU control plane (CU-CP) functions and CU user plane (CU-UP) functions. CU 160 can be connected to one or more DU 165 via midhaul communication link 162 (e.g., F1, F1-c, F1-u), and DU 165 can be connected to one or more RU 170 via fronthaul communication link 168 (e.g., open fronthaul (FH) interface). In some respects, the midhaul communication link 162 or the fronthaul communication link 168 may be implemented based on the interfaces (e.g., channels) between the layers of the protocol stack, each layer of which is supported by the corresponding network entity 105 communicating via such communication links.

[0053] In some wireless communication systems (e.g., wireless communication system 100), the infrastructure and spectrum resources for radio access can support wireless backhaul link capabilities to supplement wired backhaul connections, thereby providing an IAB network architecture (e.g., to core network 130). In some cases, in an IAB network, one or more network entities 105 (e.g., IAB node 104) may be partially controlled by each other. One or more IAB nodes 104 may be referred to as donor entities or IAB donors. One or more DU 165s or one or more Ru 170s may be partially controlled by one or more CU 160s associated with donor network entity 105 (e.g., donor base station 140). One or more donor network entities 105 (e.g., IAB donors) may communicate with one or more additional network entities 105 (e.g., IAB node 104) via supported access and backhaul links (e.g., backhaul communication link 120). IAB node 104 may include an IAB mobile terminal (IAB-MT) controlled (e.g., scheduled) by a DU 165 of a coupled IAB donor. The IAB-MT may include a separate set of antennas for relaying communication with UE 115, or may share the same antennas (e.g., those of RU 170) for access to IAB node 104 via DU 165 of IAB node 104. (e.g., referred to as a virtual IAB-MT (vIAB-MT)). In some aspects, IAB node 104 may include a DU 165 that supports communication links with additional entities (e.g., IAB node 104, UE 115) within a relay chain or configuration (e.g., downstream) of the access network. In such cases, one or more components of the decomposed RAN architecture (e.g., one or more IAB nodes 104 or components of IAB node 104) may be configured to operate according to the techniques described herein.

[0054] When the techniques described herein are applied in the context of a decomposed RAN architecture, one or more components of the decomposed RAN architecture can be configured to support multi-stage machine learning model monitoring as described herein. For example, some operations described as being performed by UE 115 or network entity 105 (e.g., base station 140) may additionally or alternatively be performed by one or more components of the decomposed RAN architecture (e.g., IAB node 104, DU 165, CU 160, RU 170, RIC 175, SMO 180).

[0055] UE 115 may include or be referred to as a mobile device, wireless device, remote device, handheld device, or subscriber device, or any other suitable term, wherein "device" may also be referred to as a unit, station, terminal, or client, etc. UE 115 may also include or be referred to as a personal electronic device, such as a cellular phone, personal digital assistant (PDA), tablet computer, laptop computer, or personal computer. In some aspects, UE 115 may include or be referred to as a wireless local loop (WLL) station, Internet of Things (IoT) device, Internet of Everything (IoE) device, or machine-type communication (MTC) device, etc., which can be implemented in various objects such as electrical appliances or vehicles, instruments, etc.

[0056] The UE 115 described herein can communicate with various types of devices, such as other UEs 115 that sometimes act as relays, network entities 105, and network equipment including macro eNBs or gNBs, small cell eNBs or gNBs, or relay base stations, etc. Figure 1 As shown.

[0057] UE 115 and network entity 105 can wirelessly communicate with each other via one or more communication links 125 (e.g., access links) using resources associated with one or more carriers. The term "carrier" can refer to a set of RF spectrum resources having a defined physical layer structure for supporting communication link 125. For example, a carrier for communication link 125 may include a portion of the RF spectrum band (e.g., a bandwidth portion (BWP)) operating according to one or more physical layer channels for a given radio access technology (e.g., LTE, LTE-A, LTE-A Pro, NR). Each physical layer channel may carry acquisition signaling (e.g., synchronization signals, system information), control signaling coordinating carrier operation, user data, or other signaling. Wireless communication system 100 may support communication with UE 115 using carrier aggregation or multi-carrier operation. Depending on the carrier aggregation configuration, UE 115 may be configured with multiple downlink component carriers and one or more uplink component carriers. Carrier aggregation can be used in conjunction with both frequency division duplex (FDD) component carriers and time division duplex (TDD) component carriers. Communication between network entity 105 and other devices can refer to communication between these devices and any part of network entity 105 (e.g., entity, sub-entity). For example, the terms “send,” “receive,” or “communicate” when referring to network entity 105 can refer to any part of the RAN’s network entity 105 (e.g., base station 140, CU 160, DU 165, RU 170) communicating with another device (e.g., directly or via one or more other network entities 105).

[0058] The signal waveform transmitted via a carrier may include multiple subcarriers (e.g., using multi-carrier modulation (MCM) techniques, such as orthogonal frequency division multiplexing (OFDM) or discrete Fourier transform extended OFDM (DFT-S-OFDM)). In a system employing MCM, a resource element may refer to a resource of one symbol period (e.g., the duration of one modulation symbol) and one subcarrier, in which case the symbol period and subcarrier spacing may be inversely related. The number of bits carried by each resource element may depend on the modulation scheme (e.g., the order of the modulation scheme, the decoding rate of the modulation scheme, or both), such that a relatively high number of resource elements (e.g., in the transmission duration) and a relatively high modulation scheme order correspond to a relatively high communication rate. Wireless communication resources may refer to a combination of RF spectrum resources, temporal resources, and spatial resources (e.g., spatial layers or beams), and the use of multiple spatial resources may increase the data rate or data integrity used for communication with UE 115.

[0059] The time interval for network entity 105 or UE 115 can be expressed as a multiple of a basic time unit, such as the sampling period. seconds, of which It can represent the supported subcarrier spacing, and This can represent the supported Discrete Fourier Transform (DFT) size. The time interval of the communication resources can be organized according to radio frames, each with a specified duration (e.g., 10 milliseconds (ms)). Each radio frame can be identified by a System Frame Number (SFN) (e.g., ranging from 0 to 1023).

[0060] Each frame may include multiple consecutively numbered subframes or time slots, and each subframe or time slot may have the same duration. In some aspects, a frame may (e.g., in the time domain) be divided into subframes, and each subframe may be further divided into a number of time slots. Alternatively, each frame may include a variable number of time slots, and the number of time slots may depend on the subcarrier spacing. Each time slot may include a number of symbol periods (e.g., depending on the length of the cyclic prefix appended to each symbol period). In some wireless communication systems 100, time slots may be further divided into multiple micro-time slots associated with one or more symbols. Excluding the cyclic prefix, each symbol period may be associated with one or more (e.g., The duration of a symbol period is associated with a (number) sampling period. The duration of a symbol period can depend on the subcarrier spacing or the operating frequency band.

[0061] A subframe, time slot, micro-time slot, or symbol can be the smallest scheduling unit of the wireless communication system 100 (e.g., in the time domain) and can be referred to as a Transmission Time Interval (TTI). In some aspects, the duration of the TTI (e.g., the number of symbol periods in the TTI) can be variable. Additionally or alternatively, the smallest scheduling unit of the wireless communication system 100 can be dynamically selected (e.g., in a burst form of a shortened TTI (sTTI)).

[0062] Depending on the technology, carriers can be used to multiplex physical channels for communication. One or more of Time Division Multiplexing (TDM), Frequency Division Multiplexing (FDM), or hybrid TDM-FDM techniques can be used, for example, to multiplex physical control channels and physical data channels for signaling via a downlink carrier. The control region (e.g., control resource set (CORESET)) of the physical control channel can be defined by a set of symbol periods and can extend across the system bandwidth of the carrier or a subset of that bandwidth. One or more control regions (e.g., CORESET) can be configured for a set of UEs 115. For example, one or more UEs in UE 115 can monitor or search for control regions to obtain control information based on one or more search space sets, and each search space set can include one or more control channel candidates in one or more aggregation levels arranged in a concatenated manner. The aggregation level of control channel candidates can refer to the amount of control channel resources (e.g., control channel elements (CCEs)) associated with coded information for a control information format having a given payload size. The search space set may include: a common search space set configured to transmit control information to multiple UEs 115, and a UE-specific search space set used to transmit control information to a specific UE 115.

[0063] In some aspects, network entity 105 (e.g., base station 140, RU 170) may be mobile, and thus provide communication coverage to mobile coverage areas 110. In some aspects, while different coverage areas 110 associated with different technologies may overlap, different coverage areas 110 may be supported by the same network entity 105. In some other examples, overlapping coverage areas 110 associated with different technologies may be supported by different network entities 105. The wireless communication system 100 may include, for example, a heterogeneous network in which different types of network entities 105 provide coverage for various coverage areas 110 using the same or different radio access technologies.

[0064] Some UE 115 devices (such as MTC or IoT devices) can be low-cost or low-complexity devices and can provide automated communication between machines (e.g., via machine-to-machine (M2M) communication). M2M communication or MTC can refer to data communication technologies that allow devices to communicate with each other or with network entity 105 (e.g., base station 140) without human intervention. In some aspects, M2M communication or MTC may include communication from devices with integrated sensors or meters to measure or capture information and relay such information to a central server or application that uses the information or presents it to people interacting with the application. Some UE 115 devices may be designed to collect information or enable automated behavior of machines or other devices. Examples of applications for MTC devices include: smart metering, inventory monitoring, water level monitoring, equipment monitoring, healthcare monitoring, wildlife monitoring, weather and geographic event monitoring, queue management and tracking, remote security sensing, physical access control, and transaction-based commercial charging.

[0065] Wireless communication system 100 may be configured to support ultra-reliable communication or low-latency communication, or various combinations thereof. For example, wireless communication system 100 may be configured to support ultra-reliable low-latency communication (URLLC). UE 115 may be designed to support ultra-reliable, low-latency, or critical functions. Ultra-reliable communication may include private or group communication and may be supported by one or more services, such as push-to-talk, video, or data. Support for ultra-reliable, low-latency functions may include prioritizing services, and such services may be used for public safety or general business applications. The terms “ultra-reliable,” “low-latency,” and “ultra-reliable low-latency” are used interchangeably herein.

[0066] In some aspects, UE 115 may be configured to support direct communication with other UE 115s via device-to-device (D2D) communication link 135 (e.g., according to peer-to-peer (P2P), D2D, or sidelink protocols). In some aspects, one or more UE 115s performing D2D communication in a group may be within the coverage area 110 of network entity 105 (e.g., base station 140, RU 170), which may support aspects of such D2D communication configured (e.g., scheduled by network entity 105). In some aspects, one or more UE 115s in such a group may be outside the coverage area 110 of network entity 105, or may otherwise be unable or not configured to receive transmissions from network entity 105. In some aspects, a group of UE 115s communicating via D2D communication may support a one-to-many (1:M) system, wherein each UE 115 transmits to every other UE 115 in the group. In some respects, network entity 105 can facilitate the scheduling of resources for D2D communication. In some other examples, D2D communication can be performed between UEs 115 without involving network entity 105.

[0067] In some systems, the D2D communication link 135 may be an example of a communication channel (such as a sidelink communication channel) between vehicles (e.g., UE 115). In some aspects, vehicles may communicate using vehicle-to-vehicle (V2X) communication, vehicle-to-vehicle (V2V) communication, or some combination of these. Vehicles may signal information related to traffic conditions, signal control, weather, safety, emergencies, or any other information relevant to the V2X system. In some aspects, vehicles in a V2X system may communicate with roadside infrastructure (such as roadside units), or communicate with the network via one or more network nodes (e.g., network entity 105, base station 140, RU 170) using vehicle-to-network (V2N) communication, or both.

[0068] Core network 130 provides user authentication, access authorization, tracking, Internet Protocol (IP) connectivity, and other access, routing, or mobility functions. Core network 130 can be an evolved packet core (EPC) or a 5G core (5GC), which may include at least one control plane entity (e.g., a mobility management entity (MME), access and mobility management function (AMF)) for managing access and mobility, and at least one user plane entity (e.g., a serving gateway (S-GW), packet data network (PDN) gateway (P-GW), or user plane function (UPF)) for routing packets or interconnecting to external networks. The control plane entity manages non-access stratum (NAS) functions, such as mobility, authentication, and bearer management of UE 115 served by network entity 105 (e.g., base station 140) associated with core network 130. User IP packets can be transferred through user plane entities, which provide IP address allocation and other functions. User plane entities can connect to one or more network operator IP services 150. IP services 150 may include access to the Internet, intranets, IP Multimedia Subsystem (IMS), or packet-switched streaming services.

[0069] Wireless communication system 100 can operate using one or more frequency bands in the range of 300 MHz to 300 GHz. Generally, the area from 300 MHz to 3 GHz is referred to as the Ultra High Frequency (UHF) band or decimeter band because the wavelength range is approximately one decimeter to one meter in length. UHF waves may be blocked or redirected by buildings and environmental features (which may be referred to as clusters), but these waves are sufficient to penetrate structures so that macrocells can provide service to UE 115 located indoors. Compared to communication using smaller frequencies and longer waves in the High Frequency (HF) or Very High Frequency (VHF) portions of the spectrum below 300 MHz, communication using UHF waves can be associated with smaller antennas and shorter ranges (e.g., less than 100 km).

[0070] Wireless communication system 100 may utilize both licensed and unlicensed RF spectrum bands. For example, wireless communication system 100 may use unlicensed frequency bands (such as the 5 GHz Industrial, Scientific, and Medical (ISM) band) to employ Licensed Assisted Access (LAA), LTE Unlicensed (LTE-U) radio access technology, or NR technology. When operating using unlicensed RF spectrum bands, devices such as network entity 105 and UE 115 may employ carrier sensing for collision detection and avoidance. In some aspects, operations using unlicensed frequency bands may be combined with component carriers operating using licensed frequency bands based on carrier aggregation configurations (e.g., LAA). Operations using unlicensed spectrum may include downlink transmission, uplink transmission, P2P transmission, or D2D transmission, etc.

[0071] Network entity 105 (e.g., base station 140, RU 170) or UE 115 may be equipped with multiple antennas that can be used to employ techniques such as transmit diversity, receive diversity, multiple-input multiple-output (MIMO) communication, or beamforming. The antennas of network entity 105 or UE 115 may be located within one or more antenna arrays or antenna panels, which can support MIMO operation or transmit or receive beamforming. For example, one or more base station antennas or antenna arrays may be co-located at an antenna assembly (such as an antenna tower). In some aspects, the antennas or antenna arrays associated with network entity 105 may be located at different geographical locations. Network entity 105 may include an antenna array having a collection of multiple rows and columns of antenna ports that network entity 105 can use to support beamforming for communication with UE 115. Similarly, UE 115 may include one or more antenna arrays that can support various MIMO or beamforming operations. Additionally or alternatively, the antenna panel may support RF beamforming for signals transmitted via the antenna ports.

[0072] Beamforming (also known as spatial filtering, directional transmission, or directional reception) is a signal processing technique that can be used at a transmitting or receiving device (e.g., network entity 105, UE 115) to shape or guide an antenna beam (e.g., a transmit beam, a receive beam) along a spatial path between the transmitting and receiving devices. Beamforming can be achieved by combining signals transmitted via antenna elements of an antenna array such that some signals propagating along a specific orientation relative to the antenna array experience constructive interference, while other signals experience destructive interference. Adjustments to the signals transmitted via the antenna elements may include applying amplitude shifts, phase shifts, or both to the signals carried via the antenna elements associated with the device. The adjustments associated with each of these antenna elements may be defined by a beamforming weight set associated with a specific orientation (e.g., relative to the antenna array of the transmitting or receiving device or relative to some other orientation).

[0073] Network entity 105 or UE 115 may use beam scanning technology as part of beamforming operations. For example, network entity 105 (e.g., base station 140, RU 170) may use multiple antennas or antenna arrays (e.g., antenna panels) to perform beamforming operations for directional communication with UE 115. Some signals (e.g., synchronization signals, reference signals, beam selection signals, or other control signals) may be transmitted multiple times by network entity 105 along different directions. For example, network entity 105 may transmit signals according to different beamforming weight sets associated with different transmission directions. Transmission along different beam directions may be used to identify (e.g., by a transmitting device (such as network entity 105) or by a receiving device (such as UE 115)) the beam direction for later transmission or reception by network entity 105.

[0074] Some signals (such as data signals associated with a specific receiving device) may be transmitted by a transmitting device (e.g., transmitting network entity 105, transmitting UE 115) along a single beam direction (e.g., the direction associated with a receiving device (such as receiving network entity 105 or receiving UE 115). In some aspects, the beam direction associated with transmission along a single beam direction may be determined based on the signals transmitted along one or more beam directions. For example, UE 115 may receive one or more signals transmitted by network entity 105 along different directions and may report to network entity 105 an indication of signals received by UE 115 with the highest signal quality or other acceptable signal quality.

[0075] In some aspects, transmissions performed by a device (e.g., by network entity 105 or UE 115) may be performed using multiple beam directions, and the device may use a combination of digital pre-decoding or beamforming to generate combined beams for transmission (e.g., from network entity 105 to UE 115). UE 115 may report feedback indicating pre-decoding weights for one or more beam directions, and this feedback may correspond to a beam set configured across the system bandwidth or one or more subbands. Network entity 105 may transmit reference signals (e.g., cell-specific reference signals (CRS), channel state information reference signals (CSI-RS)) that may or may not be pre-decoded. UE 115 may provide feedback for beam selection, which may be a pre-decoding matrix indicator (PMI) or codebook-based feedback (e.g., multi-panel codebook, linear combination codebook, port selection codebook). Although these techniques are described with reference to signals transmitted by network entity 105 (e.g., base station 140, RU 170) along one or more directions, UE 115 may use similar techniques to transmit signals multiple times along different directions (e.g., to identify the beam direction used by UE 115 for subsequent transmission or reception), or to transmit signals along a single direction (e.g., to transmit data to a receiving device).

[0076] A receiving device (e.g., UE 115) may perform reception operations according to multiple reception configurations (e.g., directional listening) when receiving various signals (such as synchronization signals, reference signals, beam selection signals, or other control signals) from a transmitting device (e.g., network entity 105). For example, the receiving device may perform reception according to multiple reception directions by: receiving via different antenna subarrays; processing the received signal according to different antenna subarrays; receiving according to different sets of reception beamforming weights (e.g., different directional listening weight sets) applied to signals received at multiple antenna elements of the antenna array; or processing the received signal according to different sets of reception beamforming weights applied to signals received at multiple antenna elements of the antenna array. Any of these operations may be referred to as “listening” according to different reception configurations or reception directions. In some aspects, the receiving device may use a single reception configuration to receive along a single beam direction (e.g., when a data signal is received). A single receiver configuration can be aligned along a beam direction determined based on listening according to different receiver configuration directions (e.g., a beam direction determined to have the highest signal strength, highest signal-to-noise ratio (SNR), or other acceptable signal quality based on listening according to multiple beam directions).

[0077] UE 115 and network entity 105 can support data retransmission to increase the likelihood of successful data reception. Hybrid Automatic Repeat Request (HARQ) feedback is a technique used to increase the likelihood of correctly receiving data via communication links (e.g., communication link 125, D2D communication link 135). HARQ may include a combination of error detection (e.g., using Cyclic Redundancy Check (CRC)), forward error correction (FEC), and retransmission (e.g., Automatic Repeat Request (ARQ)). HARQ can improve throughput at the MAC layer under poor radio conditions (e.g., low signal-to-noise ratio conditions). In some aspects, the device can support same-slot HARQ feedback, in which case the device can provide HARQ feedback in a specific time slot for data received via a previous symbol in that time slot. In some other examples, the device can provide HARQ feedback in subsequent time slots or according to a different time interval.

[0078] A wireless device (such as network entity 105 or UE 115) may perform multi-stage machine learning model monitoring. For example, the wireless device may monitor a set of machine learning models during a first monitoring phase according to at least a first monitoring process, and then monitor a subset of machine learning models during a second monitoring phase according to a second monitoring process. The first monitoring process may involve monitoring the operating conditions of the wireless device or comparing them with a corresponding input data distribution (e.g., training data) associated with each machine learning model in the machine learning models. The wireless device may determine a subset of the set of machine learning models based on the first monitoring process. For example, the wireless device may determine a subset of machine learning models having an input data distribution most similar to the operating conditions at the wireless device. The wireless device may perform a second monitoring process to monitor the subset of machine learning models. For example, the second monitoring process may include running each model (e.g., an input-output data distribution method in which the operating conditions and actual measurements at the wireless device are compared with the input-output data distribution of each model in the subset, a performance-based monitoring method in which the performance of the wireless device is compared with the predicted performance of each model in the subset, etc.). Wireless devices can switch machine learning models based on monitoring (e.g., autonomously), or they can signal instructions for the selected machine learning model based on operating conditions.

[0079] As used in this article, machine learning model monitoring typically refers to monitoring the performance of a machine learning model. However, monitoring model performance does not require the model to actually be run or executed. For example, the performance of a machine learning model can be evaluated based on the distribution of input data associated with a particular machine learning model. This type of monitoring is often less processing-intensive than other monitoring processes where the model is actually executed. Model performance can also be evaluated by running the model and comparing its predicted outputs or related variables to actual measurements from a wireless device. This type of monitoring is typically more processing-intensive than input data distribution processes. Even a combination of input data distribution comparisons and predicted output comparisons can be used to evaluate model performance. Any evaluation of how a machine learning model can or is actually executed, or any evaluation of the conditions under which a machine learning model is intended to be executed, qualifies as an evaluation of the machine learning model's performance.

[0080] Figure 2 An example of a wireless communication system 200 supporting multi-stage machine learning model monitoring according to one or more aspects of this disclosure is shown. The wireless communication system 200 may implement, or be implemented by, various aspects of the wireless communication system 100. For example, the wireless communication system 200 may include network entities 105 and UE 115, which may represent as referenced... Figure 1 Examples of the corresponding devices described.

[0081] UE 115 may use one or more machine learning models 205. In some aspects, UE 115 may perform machine learning model monitoring. That is, UE 115 may monitor one or more machine learning models 205 for data drift, concept drift, or both (e.g., after deploying one or more machine learning models 205). In some aspects, monitoring data drift may be referred to as concept drift detection or learning under concept drift, etc.

[0082] UE 115 can monitor a mismatch (e.g., drift) between one or more data distributions associated with machine learning model 205 and one or more environmental conditions associated with UE 115 at a given time. For example, each machine learning model in one or more machine learning models 205 may be associated with a data distribution, which may be an example of a training dataset (e.g., the conditions under which the machine learning model is trained).

[0083] The data distribution associated with the machine learning model can represent a set of operating conditions under which the machine learning model can be effectively used (e.g., peak performance). In some aspects, the machine learning model can be associated with degraded performance when one or more environmental conditions at UE 115 deviate from the set of operating conditions associated with the machine learning model (e.g., exceed a threshold).

[0084] UE 115 can monitor one or more machine learning models 205 to identify data drift and thus avoid performance degradation of one or more machine learning models 205 associated with differences between the corresponding set of operating conditions of one or more machine learning models 205 and one or more environmental conditions at UE 115.

[0085] After identifying (e.g., detecting) data drift, UE 115 may switch one or more machine learning models in use 205, fall back to a non-machine learning model, train a global machine learning model (e.g., a general machine learning model associated with a wide range of operating conditions), perform online retraining or calibration of machine learning model 205, or a combination thereof.

[0086] For example, UE 115 may use one or more machine learning models 205 to select beams in beam set 220. For instance, UE 115 may monitor one or more machine learning models 205, select a machine learning model from one or more machine learning models 205, and select beams in beam set 220 based on the selected machine learning model.

[0087] UE 115 can monitor the performance of one or more machine learning models 205 according to intermediate performance monitoring methods, end-to-end performance monitoring methods, input data distribution similarity methods, input-output data distribution similarity methods, or any combination thereof.

[0088] Intermediate performance monitoring methods may include identifying one or more intermediate metrics associated with the performance of a machine learning model. For example, UE 115 may evaluate the performance of each machine learning model by comparing the output (e.g., a metric of prediction) of each machine learning model in one or more machine learning models 205 with the output of a metric of measurement.

[0089] For example, in the beam selection example, the intermediate performance monitoring method may include: UE 115 determining the difference between a measured reference signal received power (RSRP) value of beam set 220 and a predicted RSRP value of beam set 220 (e.g., based on one or more machine learning models 205). UE 115 may determine one or more predicted beams (e.g., the best predicted beams of one or more machine learning models 205) based on the predicted RSRP value, and determine one or more measured beams (e.g., the best measured beams) based on the measured RSRP value. UE 115 may compare one or more predicted beams with one or more measured beams (e.g., during a prediction instance) to determine the performance level of each machine learning model in one or more machine learning models 205.

[0090] Additionally or alternatively, in the example of interference prediction, intermediate performance methods may include: UE 115 determining one or more mean squared error (MSE) values ​​between the predicted interference level (e.g., based on one or more machine learning models 205) and the measured (e.g., actual) interference level. UE 115 may determine (e.g., identify) one or more machine learning models (e.g., the most accurate machine learning model) among one or more machine learning models 205 for one or more interference categories based on the determined one or more MSE values.

[0091] End-to-end performance methods may include identifying one or more end-to-end performance metrics associated with the performance of the machine learning model. For example, UE 115 may monitor one or more system performance metrics (e.g., throughput, user-perceived throughput (UPT), latency, etc.) to evaluate the performance of machine learning model 250.

[0092] In an example of beam selection, UE 115 may use one or more machine learning models from machine learning models 205 to select a beam from beam set 220. UE 115 may use the selected beam to communicate with network entity 105 and measure throughput (e.g., during a predicted instance). UE 115 may evaluate the performance level of the machine learning model based on the measured throughput. In some respects, one or more factors unrelated to the machine learning model (e.g., deep fading, low link quality, etc.) may affect the measured throughput. In such examples, the end-to-end performance approach may be associated with relatively inaccurate monitoring of one or more machine learning models 205.

[0093] Input data distribution similarity methods, input-output data distribution similarity methods, or both may include determining (e.g., measuring) the distribution similarity between a measured input data distribution and a machine learning model input data distribution, or between a measured input-output data distribution and a machine learning model input-output data distribution.

[0094] For example, in an example of predicting interference via machine learning model 205, UE 115 may compare one or more input data distributions (e.g., normalized input data distributions) of RSRP measurements associated with the training dataset for each of the one or more machine learning models 205. That is, UE 115 may compare one or more input training data distributions with interference data distributions (e.g., actual RSRP measurements from UE 115) to determine which of the one or more machine learning models 205 (e.g., if any) has the highest similarity (e.g., the highest overlap with the interference data distribution). For example, UE 115 may select the machine learning model from the one or more machine learning models 205 associated with the training RSRP dataset that is most similar to the actual RSRP measurement at UE 115.

[0095] Additionally or alternatively, UE 115 may compare one or more joint input-output data distributions associated with one or more machine learning models 205 with one or more joint input-output data distributions associated with measurements at UE 115. For example, one or more joint input-output distributions may represent a combination of input and output distributions (e.g., a statistical combination).

[0096] In some respects, one or more joint input-output data distributions may be associated with a threshold similarity (e.g., statistical similarity requirement) between the training conditions of one or more machine learning models 205 and the environmental (e.g., operational) conditions at the UE 115. For example, the threshold similarity may be based on the performance degradation of one or more machine learning models 205 that occurs based on the similarity between the training conditions and the environmental conditions.

[0097] To generate one or more joint input-output distributions, UE 115 may use (e.g., run) one or more machine learning models 205 to generate corresponding output distributions using the corresponding input distributions and to combine the input and output distributions. In the example of interference prediction, UE 115 may input the corresponding RSRP training distribution into one or more machine learning models 205 to generate an output distribution predicting throughput.

[0098] In some respects, UE 115 can determine (e.g., calculate) the similarity of the corresponding distributions based on Kullback-Leibler divergence, Kolmogorov-Smirnov test, or bulldozer distance.

[0099] In some respects, the performance of one or more machine learning models 205 may be affected by changes in the signal-to-interference-plus-noise ratio (SINR) of the input reference signal used to train the machine learning model, the scheduling mode at network entity 105 (e.g., single-user (SU) multiple-input multiple-output (MIMO), multi-user (MU) MIMO, etc.), the type of reference signal, operating conditions (e.g., bandwidth, frequency band, beam characteristics, etc.), energy per resource element (EPRE), number of ports, number of panels, number of antenna elements, environmental changes (e.g., rural, urban, high Doppler, low Doppler, high interference, low interference, etc.) or any combination thereof.

[0100] One or more machine learning models 205 at UE 115 may include machine learning models associated with various training datasets to address various properties that may affect one or more machine learning models 205 described above. In such cases, the performance of monitoring one or more machine learning models 205 at UE 115 (e.g., a relatively large number of machine learning models) may be associated with a high level of complexity (e.g., relative to monitoring a smaller number of machine learning models).

[0101] Network entity 105 may configure UE 115 to perform machine learning model monitoring based on one or more machine learning modeling methods. For example, UE 115 may send a capability message 210 to network entity 105, which indicates the capability of UE 115 to perform multi-stage machine learning model monitoring. Based on the received capability message 210, network entity 105 may send a configuration message 215 to UE 115, which indicates one or more machine learning modeling methods, one or more monitoring stages, or both.

[0102] For example, network entity 105 may (e.g., via configuration message 215) configure UE 115 to monitor machine learning model 250 according to a first monitoring type (e.g., a first monitoring method) during a first monitoring phase, monitor the machine learning model according to a second monitoring type during a second monitoring phase, and so on. In some aspects, network entity 105 may instruct the configuration via RRC messages (e.g., statically), via MAC-CE messages (e.g., semi-statically), or via DCI messages (e.g., dynamically).

[0103] In some respects, network entity 105 can configure UE 115 to switch monitoring methods. For example, network entity 105 can configure the UE to switch the monitoring type from a second monitoring type to a first monitoring type during the second monitoring phase.

[0104] UE 115 may report recommended monitoring methods to network entity 105. For example, UE 115 may report recommended monitoring methods before or after receiving configuration message 215. In some aspects, the reported recommended monitoring methods may include switching one or more monitoring methods for one or more monitoring phases. For example, UE 115 may recommend switching the monitoring method from a first monitoring type to a second monitoring type during a first monitoring phase.

[0105] Figure 3 An example of a flowchart 300 supporting multi-stage machine learning model monitoring according to one or more aspects of this disclosure is shown. Flowchart 300 may implement, or be implemented by, various aspects of wireless communication system 100, wireless communication system 200, or both. For example, flowchart 300 may be implemented by a wireless device such as network entity 105 or UE 115, which may represent as referenced... Figure 1 and Figure 2 Examples of the corresponding devices described.

[0106] Wireless devices can perform machine learning model monitoring on one or more machine learning models. For example, a wireless device can perform machine learning model monitoring to detect data drift, select (e.g., switch) a machine learning model, calibrate (e.g., retrain, fine-tune, etc.) one or more machine learning models, or a combination thereof. In some aspects, machine learning model monitoring may include two or more stages (e.g., multi-stage monitoring), where each of the two or more stages may be associated with a corresponding monitoring type.

[0107] For example, at 310, the wireless device may perform a first monitoring phase on a set of machine learning models 305 (e.g., all machine learning models configured at the wireless device). In some aspects, the wireless device may perform the first monitoring phase according to a first monitoring type.

[0108] At point 315, the wireless device may select a subset 325 of machine learning models. For example, the wireless device may select a subset 325 of machine learning models from a set 305 of machine learning models for monitoring during the second monitoring phase.

[0109] In some respects, the wireless device may select a subset 325 of machine learning models to satisfy threshold distribution similarity. For example, the wireless device may select machine learning models from a set 305 of machine learning models that are associated with corresponding training datasets that have at least threshold distribution similarity to the actual data. In the example of interference prediction, the wireless device may select for the subset 325 of machine learning models a machine learning model associated with an RSRP training data distribution that has threshold distribution similarity (e.g., 90% similarity) compared to the RSRP distribution measured at the wireless device.

[0110] At 320, the wireless device may perform a second phase of monitoring on a subset 325 of the machine learning models (e.g., a subset 325 of the machine learning models selected by the wireless device at 315). In some aspects, the wireless device may perform a first monitoring phase according to a first monitoring type.

[0111] In some respects, the second-phase monitoring can trigger the first-phase monitoring. For example, a wireless device can revert to the first-phase monitoring based on the detection of data drift during the second-phase monitoring. The second-phase monitoring can produce performance results at the wireless device that are below a configured (e.g., pre-configured) threshold (e.g., similarity between the training distribution and the data distribution). For example, the threshold could be a performance threshold associated with the second-phase monitoring.

[0112] In some respects, the performance threshold can be similar to the threshold distribution used by the wireless device at 315 to select a subset 325 of machine learning models. That is, the wireless device can detect during the second phase of monitoring that one or more machine learning models in the subset 325 do not meet the performance threshold at a second prediction time associated with performing the second phase of monitoring at 320, which occurs after the first prediction time associated with performing the first phase of monitoring at 310. For example, one or more machine learning models may meet the performance threshold at the first prediction time but not at the second prediction time (e.g., due to changes in one or more operating conditions, one or more environmental conditions, or both at the wireless device).

[0113] In some aspects, the wireless device may revert to first-stage monitoring to identify one or more machine learning models in the machine learning model set 305. For example, the machine learning model set 305 may include one or more machine learning models that do not meet a performance threshold at a first prediction time but do meet the performance threshold at a second prediction time. Reverting to first-stage monitoring may allow the wireless device to identify one or more machine learning models that have a training data distribution most similar to, or both of, the current operating conditions, current environmental conditions, or both at the wireless device.

[0114] Figure 4 An example of a flowchart 400 supporting multi-stage machine learning model monitoring according to one or more aspects of this disclosure is shown. Flowchart 400 may implement various aspects of wireless communication system 100, wireless communication system 200, or both, or be implemented by these aspects. For example, flowchart 400 may be implemented by a wireless device such as network entity 105 or UE 115, which may represent as referenced... Figure 1 and Figure 2 Examples of the corresponding devices described.

[0115] Wireless devices can perform machine learning model monitoring on a collection of machine learning models. For example, a wireless device can perform machine learning model monitoring to detect data drift, select (e.g., switch) machine learning models, calibrate (e.g., retrain, fine-tune, etc.) the collection of machine learning models, or a combination thereof. In some aspects, machine learning model monitoring may include two or more stages (e.g., multi-stage monitoring), where each of the two or more stages may be associated with a corresponding monitoring type.

[0116] For example, at 405, the wireless device can perform a measurement. For example, in the case of beam prediction, the wireless device can measure the RSRP. The measured RSRP can represent the operating conditions at the wireless device over a time period (e.g., within one or more prediction instances).

[0117] At point 410, the wireless device may determine one or more data distributions associated with one or more machine learning models. For example, the wireless device may determine one or more input data distributions associated with a set of machine learning models (e.g., all machine learning models at the wireless device) during a first monitoring phase. In some aspects, determining one or more input data distributions may not be associated with running a set of machine learning models.

[0118] Additionally or alternatively, in the second monitoring phase, the wireless device may determine one or more input-output data distributions associated with a subset of the set of machine learning models. That is, the wireless device may run a subset of machine learning models to determine one or more output data distributions and combine the corresponding one or more output data distributions with one or more input data distributions to produce one or more input-output data distributions.

[0119] In some respects, during the second monitoring phase, the wireless device may determine one or more intermediate performance metrics (e.g., throughput, UPE, latency, etc.). For example, the wireless device may run a subset of machine learning models to determine one or more performance metrics associated with each machine learning model in the subset of machine learning models.

[0120] In some respects, wireless devices can perform the first monitoring phase with a lower level of complexity than the second monitoring phase. That is, determining the input data distribution during the first monitoring phase can be associated with lower processing complexity compared to determining the input-output data distribution or performance metric during the second monitoring phase (e.g., based on a subset of running a machine learning model to produce the output data distribution, performance metric, or both). Generally, wireless devices can perform multi-stage monitoring, where processing complexity gradually increases at each stage.

[0121] At 415, the wireless device can plot one or more data distributions associated with one or more machine learning models. In some aspects, the wireless device can determine input measurements at multiple prediction instances to plot one or more input data distributions. Alternatively, the wireless device can run one or more machine learning models to determine output measurements at multiple prediction instances, thereby plotting one or more input-output data distributions.

[0122] At 420, the wireless device can compare one or more data distributions with a measurement (e.g., the measurement performed at 405). That is, the wireless device can compare one or more data distributions with a measurement to determine the corresponding similarity between the training set, performance, or combinations thereof associated with each machine learning model and the operating conditions at the wireless device.

[0123] For example, in the first monitoring phase, the wireless device may determine (e.g., by computation, calculation, etc.) the similarity of the input data distribution to a first set 425-a of data distributions. The first set 425-a of data distributions may represent the data distribution of the set of machine learning models (e.g., all machine learning models) at the wireless device.

[0124] In the case of beam selection, the wireless device can compare the measured RSRP value on the predicted instance with the input RSRP distribution of each machine learning model in the set of machine learning models. For example, the wireless device can compare the measurement (e.g., representing one or more operating conditions at the wireless device) with a first model 415-a, a second model 415-b, and a third model 415-c.

[0125] Additionally or alternatively, during the second monitoring phase, the wireless device may determine (e.g., by computation, calculation, etc.) the similarity of input-output data distributions for a second set 425-b of data distributions. The second set 425-b of data distributions may represent a subset of the first set 425-a of data distributions. For example, the wireless device may select a subset of machine learning models associated with the second set 425-b of data distributions (e.g., at 430) from the set of machine learning models associated with the first set 425-a of data distributions based on a comparison of the data distributions at 420 during the first monitoring phase.

[0126] In the case of beam selection, the wireless device can compare the joint distribution of the measured RSRP value and the predicted beam with one or more joint distributions of the input RSRP value and the predicted beam associated with each machine learning model in the subset of machine learning models to determine the corresponding similarity associated with each machine learning model.

[0127] In some aspects, during the second monitoring phase, the wireless device may determine (e.g., by calculation, computation, etc.) the accuracy level of one or more predicted metrics associated with each machine learning model in a subset of machine learning models. For example, the wireless device may determine the difference between the RSRP value associated with a beam measured by the wireless device and the RSRP value associated with a beam predicted by each machine learning model. Additionally or alternatively, the wireless device may compare one or more system performance metrics (e.g., throughput, UPT, latency) for each machine learning model. That is, the wireless device may avoid performing measurements at 405 and compare the corresponding system performance metrics associated with each machine learning model at 420.

[0128] At 430, the wireless device may select one or more machine learning models based on the comparison at 420. For example, the wireless device may select one or more machine learning models that have the highest similarity to the measurement at 405. In the first monitoring phase, similarity can be between the measurement and one or more input data distributions (e.g., a first set of data distributions 425-a), while in the second monitoring phase, similarity can be between the joint distribution of the measurement and one or more predicted metrics and the joint distribution of the input data (e.g., training data associated with each machine learning model) and one or more metrics predicted via the machine learning models (e.g., a second set of data distributions 425-b). Additionally or alternatively, the wireless device may select one or more machine learning models that have the highest level of accuracy, the best performance metric, or both.

[0129] At 435, the wireless device may utilize one or more machine learning models selected at 430 to perform an additional monitoring phase. For example, the wireless device may proceed to a second monitoring phase after selecting a subset of machine learning models during the first monitoring phase. Additionally or alternatively, the wireless device may perform one or more monitoring phases after the second monitoring phase.

[0130] In some respects, a wireless device can recover from a higher monitoring phase (e.g., a second monitoring phase) to a lower monitoring phase (e.g., a first monitoring phase) based on a comparison of the data distribution at 420 locations. For example, a wireless device can recover to a lower monitoring phase based on the detection of data drift in one or more selected models in a given monitoring phase.

[0131] Figure 5 An example of a process flow 500 supporting multi-stage machine learning model monitoring according to one or more aspects of this disclosure is shown. In some aspects, process flow 500 may implement as referenced Figure 1 – Figure 4The described wireless communication system 100, wireless communication system 200, and aspects of flowcharts 300 and 400, or aspects implemented by these aspects. For example, process flow 500 may include network entity 105-a and network entity 105-b, which may be as described in reference... Figure 1 and Figure 2 Examples of the corresponding devices described.

[0132] The following alternative examples may be implemented, in which some steps are performed in a different order than described or not at all. In some cases, steps may include additional features not mentioned below, or additional steps may be added. Although network entities 105-a and 105-b are shown as performing the operations of process flow 500, some aspects of some operations may also be performed by one or more other wireless devices.

[0133] Network entity 105-a can execute multi-stage monitoring processes. For example, network entity 105-a can execute two or more monitoring processes to monitor the set of machine learning models at network entity 105-a. Network entity 105-a can select machine learning models from the set of machine learning models based on the multi-stage monitoring processes.

[0134] At point 505, network entity 105-a may send a capability report to network entity 105-b. For example, network entity 105-a may send capability information instructing network entity 105-a to monitor the capabilities of a set of machine learning models via a multi-stage monitoring process. The multi-stage monitoring process may include at least a first monitoring process and a second monitoring process.

[0135] At point 510, network entity 105-b may send instructions to network entity 105-a regarding machine learning models and monitoring processes. For example, network entity 105-a may receive configurations for at least one machine learning model in a set of machine learning models, or configurations for one or more of the first or second monitoring processes.

[0136] In some respects, network entity 105-b may indicate configuration statically, semi-statically, or dynamically. For example, network entity 105-b may indicate configuration via RRC messages, MAC-CE messages, DCI messages, or combinations thereof.

[0137] At point 515, network entity 105-a may send instructions to network entity 105-b regarding the monitoring process. For example, before applying a multi-stage process monitoring process to a set of machine learning models, network entity 105-a may send instructions for at least one of the first or second monitoring processes. In other words, network entity 105-a may send instructions for recommended monitoring processes for one or more stages of the multi-stage monitoring process.

[0138] At point 520, network entity 105-b can send instructions for switching monitoring procedures. For example, network entity 105-a can receive instructions for switching from a first monitoring procedure to a second monitoring procedure.

[0139] At point 525, network entity 105-a may perform a first monitoring process. For example, network entity 105-a may monitor the performance of each corresponding machine learning model in a set of machine learning models configured at network entity 105-a (e.g., via the configuration received at point 510). The set of machine learning models may be executable by the processing system of network entity 105-a. For example, network entity 105-a may generate the output of each machine learning model. In some aspects, network entity 105-a may avoid generating outputs according to the first monitoring process. For example, the first monitoring process may include monitoring based on the similarity of the input data distribution of each corresponding machine learning model.

[0140] At 530, network entity 105-a can compare the input data distribution. Network entity 105-a can monitor input data distribution information (e.g., during a first monitoring process at 525). The input data distribution information may include the corresponding input data distribution associated with each corresponding machine learning model in the set of machine learning models. In some aspects, the input data distribution information may be associated with the dataset used to train each machine learning model. For example, network entity 105-a can compare the training data associated with each machine learning model and the operational data distribution associated with network entity 105-a. The operational data distribution may be associated with one or more operating conditions of network entity 105-a (e.g., or the processing system of network entity 105-a).

[0141] In some respects, network entity 105-a can determine the distributional similarity between the training dataset associated with each corresponding machine learning model in the set of machine learning models and the operational data distribution of network entity 105-a (e.g., or the processing system of network entity 105-a).

[0142] At point 535, network entity 105-a may determine a subset of machine learning models. For example, network entity 105-a may determine a subset of machine learning models based on the first monitoring process at point 525. In some aspects, network entity 105-a may determine a subset of machine learning models based on a comparison of the input data distribution at point 530. For example, network entity 105-a may select a subset of machine learning models that have the greatest similarity to the input data distribution of network entity 105-a's operating data distribution. Additionally or alternatively, network entity 105-a may determine a subset of machine learning models based on a comparison of each corresponding distribution similarity with a threshold similarity. For example, network entity 105-a may select machine learning models in the set of machine learning models that satisfy the threshold similarity in the subset.

[0143] At 540, network entity 105-a may perform a second monitoring process. For example, network entity 105-a may monitor the performance of each corresponding machine learning model in a subset of the machine learning model set according to (e.g., as configured at 510) the second monitoring process.

[0144] In some respects, the second monitoring process can be an end-to-end or system performance monitoring process. For example, network entity 105-a can compare one or more performance-related parameters associated with each corresponding machine learning model in a subset of the machine learning model set with a threshold performance-related parameter. The performance-related parameter could be throughput or UPT, etc. Network entity 105-a can continue using one or more machine learning models based on the performance-related parameter meeting the threshold. Alternatively, network entity 105-a can avoid using one or more machine learning models based on the performance-related parameter being below the threshold.

[0145] In some aspects, the second monitoring process may include monitoring input-output data distribution information. For example, input-output data distribution information may include the corresponding input-output data distribution associated with each corresponding machine learning model in the set of machine learning models.

[0146] At point 545, network entity 105-a can generate input-output data distributions. In some aspects, to monitor input-output data distribution information, network entity 105-a can generate input-output data distribution information. For example, network entity 105-a can generate input-output distribution information based on each corresponding machine learning model in a first subset of machine learning models. Generating input-output distribution information can involve running each machine learning model in the first subset of machine learning models to produce an output distribution. For example, network entity 105-a can generate input-output distributions by combining the corresponding input distribution (e.g., training data) with the resulting output distribution. In other words, input-output distribution information can be associated with the aggregation of input distribution information and output distribution information.

[0147] At 550, network entity 105-a can compare input-output data distributions. For example, network entity 105-a can compare input-output data distribution information with the operational data distribution of network entity 105-a (e.g., or the processing system of network entity 105-a).

[0148] At position 555, network entity 105-a can measure performance-related parameters. For example, the second monitoring process can be a performance-based monitoring process. In some aspects, network entity 105-a can measure performance-related parameters such as inferred performance (e.g., predicted beam accuracy) or system performance (e.g., throughput, UPT, latency, etc.).

[0149] At position 560, network entity 105-a can compare performance-related parameters. For example, network entity 105-a can compare the performance-related parameters for each machine learning model with the performance-related parameters measured by network entity 105-a (e.g., at position 555).

[0150] In some aspects, network entity 105-a may detect data drift during a first monitoring process, a second monitoring process, or both. For example, network entity 105-a may detect data drift based on a comparison of the input data distribution at 530, a comparison of the input-output data distribution at 545, or both. In some aspects, network entity 105-a may resume the first monitoring process based on the detection of data drift during the second monitoring process. For example, network entity 105-a may detect data drift at 550 and, in response to this detection, re-monitor the machine learning model ensemble via the first monitoring process.

[0151] Network entity 105-a may perform one or more additional monitoring processes. For example, network entity 105-a may determine a second subset of machine learning models based on a second monitoring process (e.g., from a subset of machine learning models). Network entity 105-a may perform a third monitoring process, which includes monitoring the performance of each corresponding machine learning model in the second subset of the machine learning model set.

[0152] In some respects, network entity 105-a can perform multi-stage monitoring processes, in which the processing complexity gradually increases. For example, the first monitoring process may have lower processing complexity than the second monitoring process, the second monitoring process may have lower processing complexity than the third monitoring process, and so on.

[0153] Figure 6 A block diagram 600 of a device 605 supporting multi-stage machine learning model monitoring according to one or more aspects of this disclosure is shown. Device 605 may be an example of aspects of network entity 105 as described herein. Device 605 may include a receiver 610, a transmitter 615, and a communication manager 620. Device 605, or one or more components of device 605 (e.g., receiver 610, transmitter 615, and communication manager 620), may include at least one processor that may be coupled to at least one memory to individually or jointly support or implement the described techniques. Each of these components may communicate with each other (e.g., via one or more buses).

[0154] Receiver 610 may provide components for acquiring (e.g., receiving, determining, identifying) information (such as user data, control information, or any combination thereof (e.g., I / Q samples, symbols, packets, protocol data units, service data units)) associated with various channels (e.g., control channels, data channels, information channels, channels associated with a protocol stack). The information may be passed to other components of device 605. In some aspects, receiver 610 may support acquiring information by receiving signals via one or more antennas. Additionally or alternatively, receiver 610 may support acquiring information by receiving signals via one or more wired (e.g., electrical, fiber optic) interfaces, wireless interfaces, or any combination thereof.

[0155] Transmitter 615 may provide components for outputting (e.g., transmitting, providing, conveying, transmitting) information generated by other components of device 605. For example, transmitter 615 may output information associated with various channels (e.g., control channels, data channels, information channels, channels associated with a protocol stack), such as user data, control information, or any combination thereof (e.g., I / Q samples, symbols, packets, protocol data units, service data units). In some aspects, transmitter 615 may support outputting information by transmitting signals via one or more antennas. Additionally or alternatively, transmitter 615 may support outputting information by transmitting signals via one or more wired (e.g., electrical, fiber optic) interfaces, wireless interfaces, or any combination thereof. In some aspects, transmitter 615 and receiver 610 may be co-located in a transceiver, which may include or be coupled to a modem.

[0156] The communication manager 620, receiver 610, transmitter 615, or various combinations thereof, or various components thereof, may be examples of components used to perform various aspects of multi-stage machine learning model monitoring as described herein. For example, the communication manager 620, receiver 610, transmitter 615, or various combinations thereof, or components thereof, may be able to perform one or more of the functions described herein.

[0157] In some aspects, the communication manager 620, receiver 610, transmitter 615, or various combinations or components thereof may be implemented in hardware (e.g., in communication management circuitry). The hardware may include at least one of a processor, DSP, CPU, ASIC, FPGA, or other programmable logic device, microcontroller, discrete gate or transistor logic device, discrete hardware component, or any combination thereof, configured as or otherwise individually or collectively to support components for performing the functions described herein. In some aspects, at least one processor and at least one memory coupled to said at least one processor may be configured to perform one or more of the functions described herein (e.g., instructions stored in at least one memory are executed individually or collectively by one or more processors).

[0158] Additionally or alternatively, the communication manager 620, receiver 610, transmitter 615, or various combinations or components thereof may be implemented in code executed by at least one processor (e.g., as communication management software or firmware). If implemented in code executed by at least one processor, the functionality of the communication manager 620, receiver 610, transmitter 615, or various combinations or components thereof may be performed by any combination of a general-purpose processor, DSP, CPU, ASIC, FPGA, microcontroller, or these or other programmable logic devices (e.g., configured as or otherwise individually or collectively to support components for performing the functions described in this disclosure).

[0159] In some respects, the communication manager 620 may be configured to use or otherwise cooperate with the receiver 610, the transmitter 615, or both to perform various operations (e.g., receiving, acquiring, monitoring, outputting, transmitting). For example, the communication manager 620 may receive information from the receiver 610, transmit information to the transmitter 615, or be integrated with the receiver 610, the transmitter 615, or both to acquire information, output information, or perform various other operations as described herein.

[0160] For example, the communication manager 620 is capable of, configured to, or operable to support components for performing a first monitoring process, wherein performing the first monitoring process includes monitoring the performance of each corresponding machine learning model in a set of machine learning models configured at the network entity, such that the set of machine learning models can be executed by the network entity. The communication manager 620 is capable of, configured to, or operable to support components for determining a first subset of the set of machine learning models based on the first monitoring process. The communication manager 620 is capable of, configured to, or operable to support components for performing a second monitoring process, wherein performing the second monitoring process includes monitoring the performance of each corresponding machine learning model in the first subset of the set of machine learning models.

[0161] For example, the communication manager 620 can be configured or operated to support components for receiving capability reports indicating the ability of a second network entity to monitor the performance of individual machine learning models in a machine learning model set via a multi-stage monitoring process. The communication manager 620 can be configured or operated to support components for sending control information to the second network entity indicating at least two or more machine learning models in the machine learning model set, wherein the control information also indicates a first monitoring process of a first monitoring process type and a second monitoring process of a second monitoring process type, wherein the first and second monitoring processes are used by the second network entity to monitor two or more machine learning models in the machine learning model set, and wherein the second monitoring process is based on the first monitoring process, and wherein the first monitoring process type has lower processing complexity than the second monitoring process type.

[0162] By including or configuring a communication manager 620 according to an example as described herein, device 605 (e.g., controlling receiver 610, transmitter 615, communication manager 620 or a combination thereof or at least one processor otherwise coupled to them) can support techniques for reducing processing, lowering power consumption, and utilizing communication resources more efficiently.

[0163] Figure 7 A block diagram 700 of a device 705 supporting multi-stage machine learning model monitoring according to one or more aspects of this disclosure is shown. Device 705 may be an example of aspects of device 605 or network entity 105 as described herein. Device 705 may include a receiver 710, a transmitter 715, and a communication manager 720. Device 705, or one or more components of device 705 (e.g., receiver 710, transmitter 715, and communication manager 720), may include at least one processor that can be coupled to at least one memory to support the described techniques. Each of these components may communicate with each other (e.g., via one or more buses).

[0164] Receiver 710 may provide components for acquiring (e.g., receiving, determining, identifying) information (such as user data, control information, or any combination thereof (e.g., I / Q samples, symbols, packets, protocol data units, service data units)) associated with various channels (e.g., control channels, data channels, information channels, channels associated with a protocol stack). The information may be passed to other components of device 705. In some aspects, receiver 710 may support acquiring information by receiving signals via one or more antennas. Additionally or alternatively, receiver 710 may support acquiring information by receiving signals via one or more wired (e.g., electrical, fiber optic) interfaces, wireless interfaces, or any combination thereof.

[0165] Transmitter 715 may provide components for outputting (e.g., transmitting, providing, conveying, transmitting) information generated by other components of device 705. For example, transmitter 715 may output information associated with various channels (e.g., control channels, data channels, information channels, channels associated with a protocol stack), such as user data, control information, or any combination thereof (e.g., I / Q samples, symbols, packets, protocol data units, service data units). In some aspects, transmitter 715 may support outputting information by transmitting signals via one or more antennas. Additionally or alternatively, transmitter 715 may support outputting information by transmitting signals via one or more wired (e.g., electrical, fiber optic) interfaces, wireless interfaces, or any combination thereof. In some aspects, transmitter 715 and receiver 710 may be co-located in a transceiver, which may include or be coupled to a modem.

[0166] Device 705 or its various components may be examples of parts for performing various aspects of multi-stage machine learning model monitoring as described herein. For example, communication manager 720 may include a first monitoring component 725, a first subset component 730, a second monitoring component 735, a capability report receiver 740, a control information transmitter 745, or any combination thereof. Communication manager 720 may be examples of aspects of communication manager 620 as described herein. In some aspects, communication manager 720 or its various components may be configured to use or otherwise cooperate with receiver 710, transmitter 715, or both to perform various operations (e.g., receiving, acquiring, monitoring, outputting, transmitting). For example, communication manager 720 may receive information from receiver 710, transmit information to transmitter 715, or be integrated in combination with receiver 710, transmitter 715, or both to acquire information, output information, or perform various other operations as described herein.

[0167] The first monitoring component 725 is capable of, configured to, or operable to support components for performing a first monitoring process, wherein performing the first monitoring process includes monitoring the performance of each corresponding machine learning model in a set of machine learning models configured at the network entity, such that the set of machine learning models can be executed by the network entity. The first subset component 730 is capable of, configured to, or operable to support components for determining a first subset of the set of machine learning models based on the first monitoring process. The second monitoring component 735 is capable of, configured to, or operable to support components for performing a second monitoring process, wherein performing the second monitoring process includes monitoring the performance of each corresponding machine learning model in the first subset of the set of machine learning models.

[0168] Capability report receiver 740 is capable of, configured to, or operable to support components for receiving capability reports indicating the ability of a second network entity to monitor the performance of individual machine learning models in a machine learning model set via a multi-stage monitoring process. Control information transmitter 745 is capable of, configured to, or operable to support components for sending control information to the second network entity indicating at least two or more machine learning models in the machine learning model set, wherein the control information also indicates a first monitoring process of a first monitoring process type and a second monitoring process of a second monitoring process type, wherein the first and second monitoring processes are used by the second network entity to monitor two or more machine learning models in the machine learning model set, and wherein the second monitoring process is based on the first monitoring process, and wherein the first monitoring process type has lower processing complexity than the second monitoring process type.

[0169] Figure 8A block diagram 800 of a communication manager 820 supporting multi-stage machine learning model monitoring according to one or more aspects of this disclosure is shown. The communication manager 820 may be an example of aspects of the communication manager 620, communication manager 720, or both as described herein. The communication manager 820 or its various components may be examples of components for performing various aspects of multi-stage machine learning model monitoring as described herein. For example, the communication manager 820 may include a first monitoring component 825, a first subset component 830, a second monitoring component 835, a capability report receiver 840, a control information transmitter 845, a measurement component 850, a distribution similarity component 855, a data drift component 860, a process indication receiver 865, a second subset component 870, a third monitoring component 875, a capability information transmitter 880, a process indication transmitter 885, a control information receiver 890, or any combination thereof. These components, or each of their components or sub-components (e.g., one or more processors, one or more memories), may communicate with each other directly or indirectly (e.g., via one or more buses), and such communication may include communication within the protocol layers of the protocol stack, communication associated with logical channels of the protocol stack (e.g., between protocol layers of the protocol stack, within devices, components, or virtualization components associated with network entity 105, between devices, components, or virtualization components associated with network entity 105), or any combination thereof.

[0170] The first monitoring component 825 is capable of, configured to, or operable to support components for performing a first monitoring process, wherein performing the first monitoring process includes monitoring the performance of each corresponding machine learning model in a set of machine learning models configured at the network entity, such that the set of machine learning models can be executed by the network entity. The first subset component 830 is capable of, configured to, or operable to support components for determining a first subset of the set of machine learning models based on the first monitoring process. The second monitoring component 835 is capable of, configured to, or operable to support components for performing a second monitoring process, wherein performing the second monitoring process includes monitoring the performance of each corresponding machine learning model in the first subset of the set of machine learning models.

[0171] In some respects, in order to support the determination of a first subset of a set of machine learning models, the first subset component 830 is capable of, configured to, or operable to support components for comparing input data distribution information with operational data distributions corresponding to one or more operational conditions of the processing system, wherein the first subset of the set of machine learning models is determined based on the comparison.

[0172] In some respects, in order to support monitoring the performance of each corresponding machine learning model in a first subset of the machine learning model set, the second monitoring component 835 is capable of, configured to, or operable to support a component for comparing input-output data distribution information with the operational data distribution corresponding to one or more operational conditions of the network entity.

[0173] In some respects, in order to support monitoring the performance of each corresponding machine learning model in the first subset of the machine learning model set, the second monitoring component 835 is capable of, configured to, or operable to support components for generating input-output data distribution information based on each corresponding machine learning model in the first subset of the machine learning model set.

[0174] In some aspects, to support monitoring the performance of each corresponding machine learning model in a first subset of the machine learning model set, the measurement component 850 is capable of, configured to, or operable to support components for measuring one or more first performance-related parameters. In some aspects, to support monitoring the performance of each corresponding machine learning model in the first subset of the machine learning model set, the second monitoring component 835 is capable of, configured to, or operable to support components for comparing one or more second performance-related parameters associated with each corresponding machine learning model in the first subset of the machine learning model set with the measured one or more first performance-related parameters.

[0175] In some respects, the second monitoring process is an inferential performance monitoring process, wherein one or more second performance-related parameters, compared with one or more first performance-related parameters, are parameters output by each corresponding machine learning model in a first subset of the set of machine learning models.

[0176] In some respects, the second monitoring process is an end-to-end or system performance monitoring process in which one or more performance-related parameters associated with each corresponding machine learning model in a first subset of the machine learning model set are compared with threshold performance-related parameters.

[0177] In some aspects, to support the determination of a first subset of the machine learning model set, the distribution similarity component 855 is capable of, configured to, or operable to support components for determining the distribution similarity between the training dataset associated with each corresponding machine learning model in the machine learning model set and the data distribution corresponding to one or more operating conditions of the network entity. In some aspects, to support the determination of a first subset of the machine learning model set, the first subset component 830 is capable of, configured to, or operable to support components for determining a first subset of the machine learning model set based on a comparison of each corresponding distribution similarity with a threshold similarity.

[0178] In some aspects, the second monitoring component 835 is capable of, configured to, or operable to support components for comparing input-output data distribution information with operational data distributions corresponding to one or more operational conditions of the network entity during the second monitoring process, wherein the input-output data distribution information includes a corresponding input-output data distribution associated with each corresponding machine learning model in the machine learning model set. In some aspects, the data drift component 860 is capable of, configured to, or operable to support components for detecting data drift based on the second monitoring process, wherein data drift indicates that one or more corresponding distribution similarities between input-output distributions are below a threshold distribution similarity relative to the operational data distribution. In some aspects, the first monitoring component 825 is capable of, configured to, or operable to support components for re-monitoring the performance of each corresponding machine learning model in the machine learning model set configured at the network entity in response to the detection of data drift and via the first monitoring process.

[0179] In some aspects, the process indication receiver 865 is capable of, configured to, or operable to support components for receiving indications to at least one of a first monitoring process or a second monitoring process, wherein at least one of the following conditions exists: In some aspects, the first monitoring component 825 is capable of, configured to, or operable to support components for performing a first monitoring process, performing the first monitoring process including performing the first monitoring process according to the indication. In some aspects, the second monitoring component 835 is capable of, configured to, or operable to support components for performing a second monitoring process, performing the second monitoring process including performing the second monitoring process according to the indication.

[0180] In some respects, the process indication receiver 865 is capable, configured, or operable to support components for receiving an indication to switch from a first monitoring process to a second monitoring process, wherein the first and second monitoring processes are performed according to the indication.

[0181] In some aspects, the second subset component 870 is capable of, configured to, or operable to support components for determining a second subset of the set of machine learning models based on a second monitoring process, wherein the second subset includes one or more machine learning models from a first subset of the set of machine learning models. In some aspects, the third monitoring component 875 is capable of, configured to, or operable to support components for performing a third monitoring process, wherein performing the third monitoring process includes monitoring the performance of each corresponding machine learning model in the second subset of the set of machine learning models.

[0182] In some respects, the capability information transmitter 880 is capable of, configured to, or able to operate to support components for transmitting capability information that indicates the ability of a network entity to monitor a set of machine learning models through a multi-stage monitoring process.

[0183] In some respects, a multi-stage monitoring process includes at least a first monitoring process and a second monitoring process.

[0184] In some respects, the process indication transmitter 885 is capable of, configured to, or able to operate to support components for sending indications to at least one of a first monitoring process or a second monitoring process before applying a multi-stage monitoring process to a set of machine learning models, wherein the multi-stage monitoring process includes at least a first monitoring process and a second monitoring process.

[0185] In some respects, the control information receiver 890 is capable of, configured to, or operable to support components for receiving configurations for at least one machine learning model in a set of machine learning models, or for one or more of the first or second monitoring processes, via RRC messages, Media Access Control-Control Element (MAC-CE) messages, DCI messages, or combinations thereof.

[0186] In some respects, the first monitoring process has lower processing complexity than the second monitoring process.

[0187] Capability report receiver 840 is capable of, configured to, or operable to support components for receiving capability reports indicating the ability of a second network entity to monitor the performance of individual machine learning models in a machine learning model set via a multi-stage monitoring process. Control information transmitter 845 is capable of, configured to, or operable to support components for sending control information to the second network entity indicating at least two or more machine learning models in the machine learning model set, wherein the control information also indicates a first monitoring process of a first monitoring process type and a second monitoring process of a second monitoring process type, wherein the first and second monitoring processes are used by the second network entity to monitor two or more machine learning models in the machine learning model set, and wherein the second monitoring process is based on the first monitoring process, and wherein the first monitoring process type has lower processing complexity than the second monitoring process type.

[0188] In some respects, the process indication transmitter 885 is capable of, configured to, or able to operate to support components for sending indications to a second network entity to switch to using either a first monitoring process type or a second monitoring process type.

[0189] In some aspects, the process indication receiver 865 is capable of, configured to, or operable to support components for receiving a first indication to a third monitoring process before applying a multi-stage monitoring process to two or more machine learning models in a set of machine learning models. In some aspects, the third monitoring component 875 is capable of, configured to, or operable to support components for sending a second indication to a second network entity in response to the first indication, indicating that the second entity is switching to use the third monitoring process during the multi-stage monitoring process.

[0190] In some respects, control information is included in RRC messages, Media Access Control-Control Element (MAC-CE) messages, DCI messages, or combinations thereof.

[0191] Figure 9 A diagram of a system 900 including device 905 supporting multi-stage machine learning model monitoring, according to one or more aspects of this disclosure, is shown. Device 905 may be an example of device 605, device 705, or network entity 105 as described herein, or may include components thereof. Device 905 may communicate with one or more network entities 105, one or more UEs 115, or any combination thereof, and such communication may include communication via one or more wired interfaces, one or more wireless interfaces, or any combination thereof. Device 905 may include components supporting output and obtaining communication, such as a communication manager 920, a transceiver 910, an antenna 915, at least one memory 925, code 930, and at least one processor 935. These components may communicate electronically or otherwise coupled (e.g., operational ground, communication ground, functional ground, electronic ground, electrical ground) via one or more buses (e.g., bus 940).

[0192] Transceiver 910 may support bidirectional communication via a wired link, a wireless link, or both as described herein. In some aspects, transceiver 910 may include a wired transceiver and be capable of bidirectional communication with another wired transceiver. Additionally or alternatively, in some aspects, transceiver 910 may include a wireless transceiver and be capable of bidirectional communication with another wireless transceiver. In some aspects, device 905 may include one or more antennas 915 that are capable of (e.g., concurrently) transmitting or receiving wireless transmissions. Transceiver 910 may also include a modem for: modulating a signal; providing the modulated signal for transmission (e.g., via one or more antennas 915, via a wired transmitter); receiving the modulated signal (e.g., from one or more antennas 915, from a wired receiver); and demodulating the signal. In some embodiments, transceiver 910 may include one or more interfaces, such as one or more interfaces coupled to one or more antennas 915 configured to support various receive or acquire operations, or one or more interfaces coupled to one or more antennas 915 configured to support various transmit or output operations, or combinations thereof. In some embodiments, transceiver 910 may include one or more processors or one or more memory components or be configured to couple to said one or more processors or one or more memory components, said one or more processors or one or more memory components being operable to perform or support operations based on received or acquired information or signals, or generate information or other signals for transmission or other output, or any combination thereof. In some embodiments, transceiver 910, or transceiver 910 and one or more antennas 915, or transceiver 910 and one or more antennas 915 and one or more processors or one or more memory components (e.g., at least one processor 935, at least one memory 925, or both) may be included in a chip or chip assembly mounted in device 905. In some respects, transceiver 910 may be able to operate to support communication via one or more communication links (e.g., communication link 125, backhaul communication link 120, midhaul communication link 162, fronthaul communication link 168).

[0193] At least one memory 925 may include RAM, ROM, or any combination thereof. At least one memory 925 may store computer-readable, computer-executable code 930 including instructions that, when executed by one or more processors of at least one processor 935, cause device 905 to perform the various functions described herein. Code 930 may be stored in a non-transitory computer-readable medium such as system memory or another type of memory. In some cases, code 930 may not be directly executable by a processor of at least one processor 935, but may enable a computer (e.g., when compiled and executed) to perform the functions described herein. In some cases, at least one memory 925 may also include a BIOS, among other things, that controls basic hardware or software operation, such as interaction with peripheral components or devices. In some aspects, at least one processor 935 may include multiple processors, and at least one memory 925 may include multiple memories. One or more of the multiple processors may be coupled to one or more of the multiple memories, which may be configured individually or collectively to perform the various functions described herein (e.g., as part of a processing system).

[0194] At least one processor 935 may include intelligent hardware devices (e.g., general-purpose processors, DSPs, ASICs, CPUs, FPGAs, microcontrollers, programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, or any combination thereof). In some cases, at least one processor 935 may be configured to operate a memory array using a memory controller. In some other cases, the memory controller may be integrated into one or more processors in at least one processor 935. At least one processor 935 may be configured to execute computer-readable instructions stored in memory (e.g., one or more memories in at least one memory 925) to cause device 905 to perform various functions (e.g., functions or tasks supporting multi-stage machine learning model monitoring). For example, device 905 or components of device 905 may include at least one processor 935 and at least one memory 925 coupled to one or more processors in at least one processor 935, wherein at least one processor 935 and at least one memory 925 are configured to perform the various functions described herein. At least one processor 935 may be an example of a cloud computing platform (e.g., one or more physical nodes and supporting software such as an operating system, virtual machine, or container instance) that can (e.g., by executing code 930) host functions for performing the functions of device 905. At least one processor 935 may be any one or more suitable processors capable of executing scripts or instructions of one or more software programs stored in device 905 (such as within one or more memories in at least one memory 925). In some aspects, at least one processor 935 may include multiple processors, and at least one memory 925 may include multiple memories. One or more of the multiple processors may be coupled to one or more of the multiple memories, which may be configured individually or collectively to perform the various functions described herein. In some aspects, at least one processor 935 may be a component of a processing system, which may refer to a system of machines (such as a series of machines), circuitry (including, for example, one or both of processor circuitry (which may include at least one processor 935) and memory circuitry (which may include at least one memory 925)) or components that receive or receive input and process the input to produce, generate, or obtain a set of outputs. The processing system may be configured to perform one or more of the functions described herein. Therefore, at least one processor 935 or a processing system including at least one processor 935 may be configured, configured to, or operated to cause device 905 to perform one or more of the functions described herein. Additionally, as described herein, “configured to,” “configurable to,” and “operable to” are used interchangeably and may be associated with the ability to perform one or more of the functions described herein when executing code stored in at least one memory 925 or otherwise.

[0195] In some aspects, bus 940 may support communication at the protocol layer of the protocol stack (e.g., within a protocol layer). In some aspects, bus 940 may support communication associated with logical channels of the protocol stack (e.g., between protocol layers of the protocol stack), which may include communication performed within components of device 905, or communication performed between different components of device 905 that are co-addressable or may be located in different locations (e.g., where device 905 may refer to a system in which one or more of communication manager 920, transceiver 910, at least one memory 925, code 930 and at least one processor 935 may be located in one component of different components or partitioned between different components).

[0196] In some aspects, the communication manager 920 can manage (e.g., via one or more wired or wireless backhaul links) various aspects of communication with the core network 130. For example, the communication manager 920 can manage the transfer of data communication between client devices (such as one or more UEs 115). In some aspects, the communication manager 920 can manage communication with other network entities 105 and may include a controller or scheduler for coordinating with other network entities 105 to control communication with UE 115. In some aspects, the communication manager 920 may support an X2 interface within LTE / LTE-A wireless communication network technology to provide communication between network entities 105.

[0197] For example, the communication manager 920 is capable of, configured to, or operable to support components for performing a first monitoring process, wherein performing the first monitoring process includes monitoring the performance of each corresponding machine learning model in a set of machine learning models configured at the network entity, such that the set of machine learning models can be executed by the network entity. The communication manager 920 is capable of, configured to, or operable to support components for determining a first subset of the set of machine learning models based on the first monitoring process. The communication manager 920 is capable of, configured to, or operable to support components for performing a second monitoring process, wherein performing the second monitoring process includes monitoring the performance of each corresponding machine learning model in the first subset of the set of machine learning models.

[0198] For example, the communication manager 920 is capable of, configured to, or operable to support components for receiving capability reports indicating the ability of a second network entity to monitor the performance of individual machine learning models in a machine learning model set via a multi-stage monitoring process. The communication manager 920 is capable of, configured to, or operable to support components for sending control information to the second network entity indicating at least two or more machine learning models in the machine learning model set, wherein the control information also indicates a first monitoring process of a first monitoring process type and a second monitoring process of a second monitoring process type, wherein the first and second monitoring processes are used by the second network entity to monitor two or more machine learning models in the machine learning model set, and wherein the second monitoring process is based on the first monitoring process, and wherein the first monitoring process type has lower processing complexity than the second monitoring process type.

[0199] By including or configuring a communication manager 920 according to an example as described herein, device 905 can support techniques for improving and reducing user experience related to processing, reducing power consumption, utilizing communication resources more efficiently, and improving the utilization of processing power.

[0200] In some aspects, the communication manager 920 may be configured to use or otherwise coordinate with the transceiver 910, one or more antennas 915 (e.g., where applicable), or any combination thereof to perform various operations (e.g., receiving, acquiring, monitoring, outputting, transmitting). Although the communication manager 920 is illustrated as a separate component, in some aspects, one or more functions described with reference to the communication manager 920 may be supported or performed by the transceiver 910, one or more processors in at least one processor 935, one or more memories in at least one memory 925, code 930, or any combination thereof (e.g., by a processing system including at least a portion of at least one processor 935, at least one memory 925, code 930, or any combination thereof). For example, code 930 may include instructions that can be executed by one or more processors in at least one processor 935 to cause the device 905 to perform various aspects of multi-stage machine learning model monitoring as described herein, or at least one processor 935 and at least one memory 925 may be otherwise configured to perform or support such operations individually or jointly.

[0201] Figure 10 A flowchart illustrating a method 1000 for supporting multi-stage machine learning model monitoring according to various aspects of this disclosure is shown. The operation of method 1000 can be implemented by a network entity or its components as described herein. For example, the operation of method 1000 can be implemented by, as referenced... Figures 1 to 9The network entity described performs the functions described. In some aspects, the network entity may execute a set of instructions to control the functional elements of the network entity to perform the functions described. Additionally or alternatively, the network entity may use dedicated hardware to perform aspects of the functions described.

[0202] At 1005, the method may include performing a first monitoring process, wherein performing the first monitoring process includes monitoring the performance of each corresponding machine learning model in a set of machine learning models configured at the network entity, such that the set of machine learning models can be executed by the network entity. The operation of box 1005 may be performed according to examples as disclosed herein. In some aspects, aspects of the operation of 1005 may be derived from references... Figure 8 The first monitoring component 825 described is executed.

[0203] At 1010, the method may include determining a first subset of the set of machine learning models based on a first monitoring process. The operation of box 1010 may be performed according to examples disclosed herein. In some aspects, aspects of the operation of 1010 may be derived from references... Figure 8 The first subset component 830 described is executed.

[0204] At 1015, the method may include performing a second monitoring process, wherein performing the second monitoring process includes monitoring the performance of each corresponding machine learning model in a first subset of the machine learning model set. The operation of box 1015 may be performed according to examples as disclosed herein. In some aspects, aspects of the operation of 1015 may be derived from references... Figure 8 The second monitoring component 835 described is executed.

[0205] Figure 11 A flowchart illustrating a method 1100 for supporting multi-stage machine learning model monitoring according to various aspects of this disclosure is shown. The operation of method 1100 can be implemented by a network entity or its components as described herein. For example, the operation of method 1100 can be implemented by, as referenced... Figures 1 to 9 The network entity described performs the functions described. In some aspects, the network entity may execute a set of instructions to control the functional elements of the network entity to perform the functions described. Additionally or alternatively, the network entity may use dedicated hardware to perform aspects of the functions described.

[0206] At 1105, the method may include performing a first monitoring process, wherein performing the first monitoring process includes monitoring the performance of each corresponding machine learning model in a set of machine learning models configured at the network entity, such that the set of machine learning models can be executed by the network entity. The operation of box 1105 may be performed according to examples as disclosed herein. In some aspects, aspects of the operation of 1105 may be derived from references... Figure 8 The first monitoring component 825 described is executed.

[0207] At 1110, the method may include determining the distributional similarity between the training dataset associated with each corresponding machine learning model in the set of machine learning models and the data distribution corresponding to one or more operating conditions of the network entity. The operation of box 1110 may be performed according to examples as disclosed herein. In some aspects, aspects of the operation of 1110 may be derived from references... Figure 8 The described distribution similarity component 855 is executed.

[0208] At 1115, the method may include determining a first subset of the machine learning model set based on a comparison of each corresponding distribution similarity with a threshold similarity. The operation of box 1115 can be performed according to examples as disclosed herein. In some aspects, aspects of the operation of 1115 can be derived from references... Figure 8 The first subset component 830 described is executed.

[0209] At 1120, the method may include determining a first subset of the set of machine learning models based on a first monitoring process. The operation of box 1120 may be performed according to examples as disclosed herein. In some aspects, aspects of the operation of 1120 may be derived from references... Figure 8 The first subset component 830 described is executed.

[0210] At 1125, the method may include performing a second monitoring process, wherein performing the second monitoring process includes monitoring the performance of each corresponding machine learning model in a first subset of the machine learning model set. The operation of box 1125 may be performed according to examples as disclosed herein. In some aspects, aspects of the operation of 1125 may be derived from references... Figure 8 The second monitoring component 835 described is executed.

[0211] Figure 12 A flowchart illustrating a method 1200 for supporting multi-stage machine learning model monitoring according to various aspects of this disclosure is shown. The operation of method 1200 can be implemented by a network entity or its components as described herein. For example, the operation of method 1200 can be implemented by, as referenced... Figures 1 to 9 The network entity described performs the functions described. In some aspects, the network entity may execute a set of instructions to control the functional elements of the network entity to perform the functions described. Additionally or alternatively, the network entity may use dedicated hardware to perform aspects of the functions described.

[0212] At 1205, the method may include receiving a capability report indicative of the ability of a second network entity to monitor the performance of individual machine learning models in a ensemble of machine learning models via a multi-stage monitoring process. The operation of box 1205 may be performed according to examples as disclosed herein. In some aspects, aspects of the operation of 1205 may be derived from references... Figure 8The described capability report is executed by receiver 840.

[0213] At 1210, the method may include sending control information to a second network entity indicating at least two or more machine learning models in a set of machine learning models, wherein the control information further indicates a first monitoring process of a first monitoring process type and a second monitoring process of a second monitoring process type, wherein the first and second monitoring processes are used by the second network entity to monitor two or more machine learning models in the set of machine learning models, and wherein the second monitoring process is based on the first monitoring process, and wherein the first monitoring process type has lower processing complexity than the second monitoring process type. The operation of block 1210 may be performed according to examples as disclosed herein. In some aspects, aspects of the operation of 1210 may be derived from references... Figure 8 The control information transmitter 845 described is executed.

[0214] The following provides an overview of the various aspects of this disclosure:

[0215] Aspect 1: A method for wireless communication by a network entity, the method comprising: performing a first monitoring process, wherein performing the first monitoring process includes monitoring the performance of each corresponding machine learning model in a set of machine learning models configured at the network entity, such that the set of machine learning models can be executed by the network entity; determining a first subset of the set of machine learning models based on the first monitoring process; and performing a second monitoring process, wherein performing the second monitoring process includes monitoring the performance of each corresponding machine learning model in the first subset of the set of machine learning models.

[0216] Aspect 2: According to the method of aspect 1, wherein performing the first monitoring process includes monitoring input data distribution information, wherein the input data distribution information includes a corresponding input data distribution associated with each corresponding machine learning model in the machine learning model set, and wherein determining the first subset of the machine learning model set further includes: comparing the input data distribution information with an operational data distribution corresponding to one or more operational conditions of the processing system, wherein the first subset of the machine learning model set is determined based on the comparison.

[0217] Aspect 3: The method according to any one of Aspects 1 to 2, wherein performing the second monitoring process includes monitoring input-output data distribution information, wherein the input-output data distribution information includes a corresponding input-output data distribution associated with each corresponding machine learning model in the set of machine learning models, and wherein monitoring the performance of each corresponding machine learning model in the first subset of the set of machine learning models further includes comparing the input-output data distribution information with an operational data distribution corresponding to one or more operational conditions of the network entity.

[0218] Aspect 4: The method according to any one of Aspects 1 to 3, wherein performing the second monitoring process includes monitoring input-output data distribution information, wherein the input-output data distribution information includes a corresponding input-output data distribution associated with each corresponding machine learning model in the set of machine learning models, and wherein monitoring the performance of each corresponding machine learning model in the first subset of the set of machine learning models further includes generating the input-output data distribution information based on each corresponding machine learning model in the first subset of the set of machine learning models.

[0219] Aspect 5: The method according to any one of Aspects 1 to 4, wherein monitoring the performance of each corresponding machine learning model in the first subset of the machine learning model set further comprises: measuring one or more first performance-related parameters; and comparing one or more second performance-related parameters associated with each corresponding machine learning model in the first subset of the machine learning model set with the measured one or more first performance-related parameters.

[0220] Aspect 6: According to the method of aspect 5, wherein the second monitoring process is an inferential performance monitoring process, wherein the one or more second performance-related parameters compared with the one or more first performance-related parameters are parameters output by each corresponding machine learning model in the first subset of the machine learning model set.

[0221] Aspect 7: The method according to any one of Aspects 1 to 6, wherein the second monitoring process is an end-to-end or system performance monitoring process, wherein one or more performance-related parameters associated with each corresponding machine learning model in the first subset of the machine learning model set are compared with threshold performance-related parameters.

[0222] Aspect 8: The method according to any one of Aspects 1 to 7, wherein determining the first subset of the set of machine learning models further comprises: determining the distribution similarity between the training dataset associated with each corresponding machine learning model in the set of machine learning models and the data distribution corresponding to one or more operating conditions of the network entity; and determining the first subset of the set of machine learning models based on a comparison of each corresponding distribution similarity with a threshold similarity.

[0223] Aspect 9: The method according to any one of Aspects 1 to 8, the method further comprising: comparing input-output data distribution information with operational data distributions corresponding to one or more operating conditions of the network entity during the second monitoring process, wherein the input-output data distribution information includes a corresponding input-output data distribution associated with each corresponding machine learning model in the set of machine learning models; detecting data drift based on the second monitoring process, wherein the data drift indicates that one or more corresponding distribution similarities between the input-output distributions are below a threshold distribution similarity relative to the operational data distribution; and in response to detecting the data drift, re-monitoring the performance of each corresponding machine learning model in the set of machine learning models configured at the network entity via the first monitoring process.

[0224] Aspect 10: The method according to any one of aspects 1 to 9, the method further comprising: receiving an instruction for at least one of the first monitoring process or the second monitoring process, wherein at least one of the following exists: performing the first monitoring process includes performing the first monitoring process according to the instruction; or performing the second monitoring process includes performing the second monitoring process according to the instruction.

[0225] Aspect 11: The method according to any one of aspects 1 to 10, the method further comprising: receiving an instruction to switch from the first monitoring process to the second monitoring process, wherein the first monitoring process and the second monitoring process are performed according to the instruction.

[0226] Aspect 12: The method according to any one of Aspects 1 to 11, the method further comprising: determining a second subset of the set of machine learning models based on the second monitoring process, wherein the second subset includes one or more machine learning models in the first subset of the set of machine learning models; and performing a third monitoring process, wherein performing the third monitoring process includes monitoring the performance of each corresponding machine learning model in the second subset of the set of machine learning models.

[0227] Aspect 13: The method according to any one of Aspects 1 to 12, the method further comprising: sending capability information instructing the network entity to monitor the set of machine learning models via a multi-process monitoring process.

[0228] Aspect 14: According to the method of aspect 13, the multi-process monitoring process includes at least the first monitoring process and the second monitoring process.

[0229] Aspect 15: The method according to any one of Aspects 1 to 14, the method further comprising: sending an instruction to at least one of the first monitoring process or the second monitoring process before applying a multi-process monitoring process to the set of machine learning models, wherein the multi-process monitoring process includes at least the first monitoring process and the second monitoring process.

[0230] Aspect 16: The method according to any one of Aspects 1 to 15, the method further comprising: receiving configuration for at least one machine learning model in the set of machine learning models or configuration for one or more of the first monitoring process or the second monitoring process via an RRC message, a MAC-CE message, a DCI message or a combination thereof.

[0231] Aspect 17: The method according to any one of Aspects 1 to 16, wherein the first monitoring process has lower processing complexity than the second monitoring process.

[0232] Aspect 18: A method for wireless communication at a first network entity, the method comprising: receiving a capability report indicating the capability of a second network entity to monitor the performance of individual machine learning models in a set of machine learning models via a multi-stage monitoring process; and sending control information to the second network entity indicating at least two or more machine learning models in the set of machine learning models, wherein the control information further indicates a first monitoring process of a first monitoring process type and a second monitoring process of a second monitoring process type, wherein the first monitoring process and the second monitoring process are for monitoring the two or more machine learning models in the set of machine learning models by the second network entity, and wherein the second monitoring process is based on the first monitoring process, and wherein the first monitoring process type has lower processing complexity than the second monitoring process type.

[0233] Aspect 19: The method according to aspect 18 further includes: sending an instruction to the second network entity to switch to using either the first monitoring process type or the second monitoring process type.

[0234] Aspect 20: The method according to any one of aspects 18 to 19, the method further comprising: receiving a first instruction for a third monitoring process before applying the multi-process monitoring process to the two or more machine learning models in the set of machine learning models; and sending a second instruction to the second network entity to switch to using the third monitoring process during the multi-stage monitoring process in response to the first instruction.

[0235] Aspect 21: The method according to any one of Aspects 18 to 20, wherein the control information is included in an RRC message, a MAC-CE message, a DCI message, or a combination thereof.

[0236] Aspect 22: A network entity comprising: one or more memories storing processor-executable code; and one or more processors coupled to the one or more memories and capable of operating individually or jointly to execute the code to cause the network entity to perform a method according to any one of Aspects 1 to 17.

[0237] Aspect 23: A network entity comprising at least one component for performing the method according to any one of aspects 1 to 17. Code is stored thereon that, when executed, causes the means.

[0238] Aspect 24: A non-transitory computer-readable medium having code stored thereon, said code, when executed, causing a device to perform the method according to any one of aspects 1 to 17.

[0239] Aspect 25: A first network entity comprising: one or more memories storing processor-executable code; and one or more processors coupled to the one or more memories and capable of operating individually or jointly to execute the code to cause the first network entity to perform a method according to any one of Aspects 18 to 21.

[0240] Aspect 26: A first network entity comprising at least one component for performing the method according to any one of aspects 18 to 21.

[0241] Aspect 27: A non-transitory computer-readable medium having code stored thereon, said code, when executed, causing a device to perform the method according to any one of aspects 18 to 21.

[0242] The methods described herein outline possible specific implementations, and the operations and steps can be rearranged or otherwise modified, and other specific implementations are also possible. Furthermore, aspects from two or more of these methods can be combined.

[0243] While aspects of LTE, LTE-A, LTE-A Pro, or NR systems may be described for illustrative purposes, and the terms LTE, LTE-A, LTE-A Pro, or NR may be used in most of the description, the techniques described herein are also applicable to networks outside of LTE, LTE-A, LTE-A Pro, or NR networks. For example, the techniques described can be applied to a variety of other wireless communication systems, such as Ultra Mobile Broadband (UMB), IEEE 802.11 (Wi-Fi), IEEE 802.16 (WiMAX), IEEE 802.20, Flash-OFDM, and other systems and radio technologies not explicitly mentioned herein.

[0244] The information and signals described herein can be represented using any of a variety of different techniques and skills. For example, data, instructions, commands, information, signals, bits, symbols, and chips mentioned throughout the description can be represented by voltage, current, electromagnetic waves, magnetic fields or magnetic particles, light fields or optical particles, or any combination thereof.

[0245] The various exemplary blocks and components described herein can be implemented or performed using a general-purpose processor, DSP, ASIC, CPU, FPGA or other programmable logic device, discrete gate or transistor logic device, discrete hardware component, or any combination thereof, designed to perform the functions described herein. A general-purpose processor may be a microprocessor, but in alternative embodiments, a processor may be any processor, controller, microcontroller, or state machine. A processor may also be implemented as a combination of computing devices (e.g., a combination of a DSP and a microprocessor, multiple microprocessors, one or more microprocessors cooperating with a DSP core, or any other such configuration). Any function or operation described herein that can be performed by a processor may be performed by multiple processors capable of performing the described functions or operations individually or jointly.

[0246] The functions described herein can be implemented using hardware, software executed by a processor, firmware, or any combination thereof. When implemented using software executed by a processor, the functions can be stored as one or more instructions or code on a computer-readable medium or transmitted using one or more instructions or code on a computer-readable medium. Other examples and specific implementations are within the scope of this disclosure and the claims. For example, due to the nature of software, the functions described herein can be implemented using software executed by a processor, hardware, firmware, hardwiring, or any combination of these. Features implementing the functions can also be physically located in various locations, including various portions distributed such that the functions are implemented in different physical locations.

[0247] Computer-readable media includes both non-transitory computer storage media and communication media, encompassing any medium that facilitates the transfer of a computer program from one location to another. Non-transitory storage media can be any available medium accessible by a general-purpose or special-purpose computer. By way of example, and not limitation, non-transitory computer-readable media may include RAM, ROM, electrically erasable programmable ROM (EEPROM), flash memory, compressed optical disc (CD) ROM or other optical disc storage devices, magnetic disk storage devices or other magnetic storage devices, or any other non-transitory medium that can be used to carry or store desired program code components in the form of instructions or data structures, and accessible by a general-purpose or special-purpose computer or a general-purpose or special-purpose processor. Furthermore, any connection is appropriately referred to as computer-readable media. For example, if software is transmitted from a website, server, or other remote source using coaxial cable, fiber optic cable, twisted pair, digital subscriber line (DSL), or wireless technologies such as infrared, radio, and microwave, then coaxial cable, fiber optic cable, twisted pair, DSL, or wireless technologies such as infrared, radio, and microwave are included within the definition of computer-readable media. As used herein, disks and optical discs include CDs, laser discs, optical discs, digital multifunction discs (DVDs), floppy disks, and Blu-ray discs. Disks can magnetically reproduce data, and optical discs can optically reproduce data using lasers. Combinations of the above are also included within the scope of computer-readable media. Any function or operation described herein that can be performed by memory can be performed by multiple memories capable of performing the described function or operation individually or jointly.

[0248] As used herein, the term "or" is inclusive unless restrictive language is used relative to the listed alternatives. For example, a reference to "X is based on A or B" should be interpreted as including, within its scope, X is based on A, X is based on B, and X is based on both A and B. In this respect, a reference to "X is based on A or B" means "at least one of A or B" or "one or more of A or B," because "or" is inclusive. Similarly, a reference to "X is based on A, B, or C" should be interpreted as including, within its scope, X is based on A, X is based on B, X is based on C, X is based on both A and B, X is based on both A and C, X is based on both B and C, and X is based on both A, B, and C. In this respect, a reference to "X is based on A, B, or C" means "at least one of A, B, or C" or "one or more of A, B, or C," because "or" is inclusive. As an example of restrictive language, the reference to "X is based on either A or B" should be interpreted as including, within its scope, both X based on A and X based on B, but excluding X based on both A and B. Furthermore, as used herein, the phrase "based on" should not be interpreted as a reference to a closed set of information, one or more conditions, one or more factors, etc. In other words, the phrase "based on A" (where "A" can be information, conditions, factors, etc.) should be interpreted as "based on at least A," unless specifically stated differently. Moreover, as used herein, the phrase "set" should be understood to include the possibility of a set having one member. That is, the phrase "set" should be interpreted in the same way as "one or more" or "at least one."

[0249] As used herein, including in claims, the article “a” preceding a noun is open-ended and is understood to refer to “at least one” or “one or more” of those nouns. Therefore, the terms “a,” “at least one,” “one or more,” and “at least one of one or more” are interchangeable. For example, where a claim enumerates “components” performing one or more functions, each of the individual functions may be performed by a single component or by any combination of multiple components. Thus, the term “component” having a characteristic or performing a function may refer to “at least one of one or more components” having a particular characteristic or performing a particular function. Subsequent references to a component introduced with the article “a” using the terms “the” or “the” may refer to any or all of the one or more components. For example, a component introduced with the article “a” may be understood to mean “one or more components,” and subsequent reference to “the component” in a claim may be understood to be equivalent to referring to “at least one of the one or more components.” Similarly, subsequent references to a component introduced with the terms “the” or “the” as “one or more components” may refer to any or all of the one or more components. For example, reference to "the one or more components" in the subsequent claims can be understood as equivalent to reference to "at least one of the one or more components".

[0250] The term "determine" encompasses a variety of actions, and therefore, "determine" can include calculation, computation, processing, derivation, investigation, lookup (such as by searching in a table, database, or other data structure), identification, and similar actions. Furthermore, "determine" can include receiving (e.g., receiving information), accessing (e.g., accessing data stored in memory), etc. Moreover, "determine" can include parsing, obtaining, selecting, choosing, building, and other similar actions.

[0251] In the accompanying drawings, similar components or features may have the same reference numerals. Furthermore, various components of the same type can be distinguished by adding a dash after the reference numeral and a second reference numeral to differentiate between similar components. If only the first reference numeral is used in the description, the description applies to any of the similar components having the same first reference numeral, regardless of the second reference numeral or other subsequent reference numerals.

[0252] The description herein, illustrated with reference to the accompanying drawings, describes an example configuration and does not represent all examples that can be implemented or that are within the scope of the claims. The term "example" as used herein means "used as an example, instance, or illustration," and not "preferred" or "advantageous over other examples." The detailed description includes specific details used to provide an understanding of the described techniques. However, these techniques can be practiced without these specific details. In some instances, structures and devices are shown in block diagram form to avoid obscuring the concept of the described examples.

[0253] The description herein is provided to enable those skilled in the art to implement or use this disclosure. Various modifications to this disclosure will be apparent to those skilled in the art, and the general principles defined herein may be applied to other variations without departing from the scope of this disclosure. Therefore, this disclosure is not limited to the examples and designs described herein, but should be granted the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. A network entity for wireless communication, the network entity comprising: Processing system, the processing system being configured to: A first monitoring process is performed, wherein, in order to perform the first monitoring process, the processing system is configured to monitor the performance of each corresponding machine learning model in a set of machine learning models configured at the network entity, such that the set of machine learning models can be executed by the processing system; Based on the first monitoring process, a first subset of the machine learning model set is determined; as well as A second monitoring process is performed, wherein, in order to perform the second monitoring process, the processing system is configured to monitor the performance of each corresponding machine learning model in the first subset of the machine learning model set.

2. The network entity of claim 1, wherein, To perform the first monitoring process, the processing system is configured to monitor input data distribution information, wherein the input data distribution information includes a corresponding input data distribution associated with each corresponding machine learning model in the machine learning model set, and wherein, to determine the first subset of the machine learning model set, the processing system is configured to: The input data distribution information is compared with the operational data distribution corresponding to one or more operational conditions of the processing system, wherein the first subset of the machine learning model set is determined based on the comparison.

3. The network entity of claim 1, wherein, To perform the second monitoring process, the processing system is configured to monitor input-output data distribution information, wherein the input-output data distribution information includes a corresponding input-output data distribution associated with each corresponding machine learning model in the machine learning model set, and wherein, to monitor the performance of each corresponding machine learning model in the first subset of the machine learning model set, the processing system is configured to: The input-output data distribution information is compared with the operation data distribution corresponding to one or more operating conditions of the processing system.

4. The network entity of claim 1, wherein, To perform the second monitoring process, the processing system is configured to monitor input-output data distribution information, wherein the input-output data distribution information includes a corresponding input-output data distribution associated with each corresponding machine learning model in the machine learning model set, and wherein, to monitor the performance of each corresponding machine learning model in the first subset of the machine learning model set, the processing system is configured to: The input-output data distribution information is generated based on each corresponding machine learning model in the first subset of the machine learning model set.

5. The network entity of claim 1, wherein, To monitor the performance of each corresponding machine learning model in the first subset of the machine learning model set, the processing system is configured to: Measure one or more primary performance-related parameters; as well as One or more second performance-related parameters associated with each corresponding machine learning model in the first subset of the machine learning model set are compared with one or more first performance-related parameters that are measured.

6. The network entity of claim 5, wherein the second monitoring process is an inferential performance monitoring process, wherein the one or more second performance-related parameters compared with the one or more first performance-related parameters are parameters output by each corresponding machine learning model in the first subset of the machine learning model set.

7. The network entity of claim 1, wherein the second monitoring process is an end-to-end or system performance monitoring process, wherein one or more performance-related parameters associated with each corresponding machine learning model in the first subset of the machine learning model set are compared with threshold performance-related parameters.

8. The network entity of claim 1, wherein, To determine the first subset of the set of machine learning models, the processing system is configured to: Determine the distributional similarity between the training dataset associated with each corresponding machine learning model in the set of machine learning models and the data distribution corresponding to one or more operating conditions of the processing system; as well as The first subset of the set of machine learning models is determined based on a comparison of the similarity of each corresponding distribution with the threshold similarity.

9. The network entity according to claim 1, wherein the processing system is configured to: During the second monitoring process, the input-output data distribution information is compared with the operation data distribution corresponding to one or more operating conditions of the processing system, wherein the input-output data distribution information includes the corresponding input-output data distribution associated with each corresponding machine learning model in the set of machine learning models; Data drift is detected based on the second monitoring process, wherein the data drift indication includes one or more corresponding distribution similarities between the input-output distributions in the input-output data distribution information relative to the operational data distribution being below a threshold distribution similarity; and In response to the detection of the data drift, the performance of each corresponding machine learning model in the set of machine learning models configured at the processing system is re-monitored via the first monitoring process.

10. The network entity of claim 1, wherein the processing system is configured to: Receive an instruction for at least one of the first monitoring process or the second monitoring process, wherein at least one of the following conditions exists: In order to perform the first monitoring process, the processing system is configured to perform the first monitoring process according to the instruction; or In order to perform the second monitoring process, the processing system is configured to perform the second monitoring process according to the instructions.

11. The network entity of claim 1, wherein the processing system is configured to: Receive an instruction to switch from the first monitoring process to the second monitoring process, wherein the processing system is configured to execute the first monitoring process and the second monitoring process according to the instruction.

12. The network entity of claim 1, wherein the processing system is configured to: A second subset of the machine learning model set is determined based on the second monitoring process, wherein the second subset includes one or more machine learning models from the first subset of the machine learning model set; and A third monitoring process is performed, wherein, in order to perform the third monitoring process, the processing system is configured to monitor the performance of each corresponding machine learning model in a second subset of the machine learning model set.

13. The network entity of claim 1, wherein the processing system is configured to: Send capability information instructing the network entity to monitor the set of machine learning models through a multi-stage monitoring process.

14. The network entity of claim 13, wherein the multi-stage monitoring process includes at least the first monitoring process and the second monitoring process.

15. The network entity of claim 1, wherein the processing system is configured to: Before applying the multi-stage monitoring process to the set of machine learning models, an instruction is sent to at least one of the first monitoring process or the second monitoring process, wherein the multi-stage monitoring process includes at least the first monitoring process and the second monitoring process.

16. The network entity of claim 1, wherein the processing system is configured to: The configuration for at least one machine learning model in the set of machine learning models, or for one or more of the first monitoring process or the second monitoring process, is received via Radio Resource Control (RRC) messages, Medium Access Control-Control Element (MAC-CE) messages, Downlink Control Information (DCI) messages, or combinations thereof.

17. The network entity of claim 1, wherein the first monitoring process has lower processing complexity than the second monitoring process.

18. A method for wireless communication by a network entity, the method comprising: Perform a first monitoring process, wherein performing the first monitoring process includes monitoring the performance of each corresponding machine learning model in the set of machine learning models configured at the network entity, such that the set of machine learning models can be executed by the network entity; Based on the first monitoring process, a first subset of the machine learning model set is determined; as well as Perform a second monitoring process, wherein performing the second monitoring process includes monitoring the performance of each corresponding machine learning model in the first subset of the machine learning model set.

19. The method of claim 18, wherein, Performing the first monitoring process includes monitoring input data distribution information, wherein the input data distribution information includes a corresponding input data distribution associated with each corresponding machine learning model in the machine learning model set, and wherein determining the first subset of the machine learning model set further includes: The input data distribution information is compared with the operational data distribution corresponding to one or more operational conditions of the network entity, wherein the first subset of the machine learning model set is determined based on the comparison.

20. The method of claim 18, wherein performing the second monitoring procedure comprises monitoring input-output data distribution information, wherein the input-output data distribution information comprises a respective input-output data distribution associated with each respective machine learning model in the set of machine learning models, and wherein, Monitoring the performance of each corresponding machine learning model in the first subset of the machine learning model set further includes: The input-output data distribution information is compared with the operation data distribution corresponding to one or more operation conditions of the network entity.

21. The method of claim 18, wherein performing the second monitoring process includes monitoring input-output data distribution information, wherein the input-output data distribution information includes a corresponding input-output data distribution associated with each corresponding machine learning model in the set of machine learning models, and wherein, Monitoring the performance of each corresponding machine learning model in the first subset of the machine learning model set further includes: The input-output data distribution information is generated based on each corresponding machine learning model in the first subset of the machine learning model set.

22. The method of claim 18, wherein monitoring the performance of each corresponding machine learning model in the first subset of the machine learning model set further comprises: Measure one or more primary performance-related parameters; as well as One or more second performance-related parameters associated with each corresponding machine learning model in the first subset of the machine learning model set are compared with one or more first performance-related parameters that are measured.

23. The method of claim 22, wherein the second monitoring process is an inferential performance monitoring process, wherein the one or more second performance-related parameters compared with the one or more first performance-related parameters are parameters output by each corresponding machine learning model in the first subset of the machine learning model set.

24. The method of claim 18, wherein the second monitoring process is an end-to-end or system performance monitoring process, wherein one or more performance-related parameters associated with each corresponding machine learning model in the first subset of the machine learning model set are compared with threshold performance-related parameters.

25. The method of claim 18, wherein determining the first subset of the set of machine learning models further comprises: Determine the distributional similarity between the training dataset associated with each corresponding machine learning model in the set of machine learning models and the data distribution corresponding to one or more operating conditions of the network entity; as well as The first subset of the set of machine learning models is determined based on a comparison of the similarity of each corresponding distribution with the threshold similarity.

26. The method according to claim 18, further comprising: During the second monitoring process, the input-output data distribution information is compared with the operation data distribution corresponding to one or more operation conditions of the network entity, wherein the input-output data distribution information includes the corresponding input-output data distribution associated with each corresponding machine learning model in the set of machine learning models; Data drift is detected based on the second monitoring process, wherein the data drift indication includes one or more corresponding distribution similarities between the input-output distributions in the input-output data distribution information relative to the operational data distribution being below a threshold distribution similarity; as well as In response to the detection of the data drift, the performance of each corresponding machine learning model in the set of machine learning models configured at the network entity is re-monitored via the first monitoring process.

27. A network entity, the network entity comprising: Components for performing a first monitoring process, wherein performing the first monitoring process includes monitoring the performance of each corresponding machine learning model in a set of machine learning models configured at the network entity, such that the set of machine learning models can be executed by the network entity; Components used to determine a first subset of the set of machine learning models based on the first monitoring process; and Components for performing a second monitoring process, wherein performing the second monitoring process includes monitoring the performance of each corresponding machine learning model in the first subset of the machine learning model set.

28. The network entity according to claim 27, further comprising: A component for receiving an indication of at least one of the first monitoring process or the second monitoring process, wherein at least one of the following conditions exists: A component for performing the first monitoring process, wherein performing the first monitoring process includes performing the first monitoring process according to the instruction; or A component for performing the second monitoring process, which includes performing the second monitoring process according to the instructions.

29. A non-transitory computer-readable medium having code stored thereon, the code causing a device to: The first monitoring process is performed, in which, In order to perform the first monitoring process, the device is configured to monitor the performance of each corresponding machine learning model in a set of machine learning models configured at the network entity, such that the set of machine learning models can be executed by the network entity; Based on the first monitoring process, a first subset of the machine learning model set is determined; as well as Perform a second monitoring process, wherein performing the second monitoring process includes monitoring the performance of each corresponding machine learning model in the first subset of the machine learning model set.

30. The non-transitory computer-readable medium of claim 29, wherein the code, when executed, causes the device to: Send capability information instructing the network entity to monitor the set of machine learning models through a multi-stage monitoring process.

Images