Machine learning model monitoring management

Abstract

A method includes measuring, by an apparatus, a first characteristic of a wireless communication system. The apparatus predicts success of a first operation of the wireless communication system based on an inference in a first model. Based on an inference in the first model and a configuration parameter indicating that a second characteristic is to be measured, the apparatus measures the second characteristic of the wireless communication, and updates the first model based on measurement of the second characteristic of the wireless communication system being performed.

Description

TECHNICAL FIELD

[0001] Various exemplary embodiments relate generally to wireless networks, and more particularly, to methods for machine learning model monitoring management. BACKGROUND

[0002] In supervised learning models, machine learning (ML) model monitoring is used to detect data or concept drift. To detect concept drift, the ML model monitoring needs to know the ground truth. However, obtaining the ground truth often has an associated cost. This cost can also include a performance reduction of the ML operation.

[0003] In an example case of ML-based inter-frequency layer measurement reduction, concept drift can occur, for example, if one of the capacity layer cells fails due to technical problems, or the coverage of two layers changes for any other reason. This can make the inter-frequency measurement prediction model inaccurate, and if the change is more persistent, the model can need to be retrained. SUMMARY

[0004] In one aspect of the disclosure, a method includes measuring, by an apparatus, a first characteristic of a wireless communication system. The apparatus predicts success of a first operation of the wireless communication system based on an inference in a first model. Based on the inference in the first model and a configuration parameter indicating that a second characteristic is to be measured, the apparatus measures the second characteristic of the wireless communication, and updates the first model based on the measurement of the second characteristic of the wireless communication system being performed.

[0005] In one aspect of the method, the first characteristic is a characteristic of a first communication layer of the wireless communication system.

[0006] In one aspect of the method, the first operation is a transmission from the first communication layer of the wireless communication system to a second communication layer of the wireless communication system.

[0007] In one aspect of the method, the second characteristic is a characteristic of the second communication layer of the wireless communication system.

[0008] In one aspect of the method, measuring the second characteristic of the second layer of the wireless communication is positive based on the inference that the transmission from the first layer of the wireless communication system to the second layer of the wireless communication system is successful.

[0009] In one aspect of the method, the configuration parameter is a value between 0 and 1.

[0010] In one aspect of the method, for a configuration value of 0, the measurement of the second characteristic of the second layer of the wireless communication is not performed.

[0011] In one aspect of the method, for a configuration value of 1, the measurement of the second characteristic of the second layer of the wireless communication is performed every time.

[0012] In one aspect of the method, the second characteristic of the second layer of the wireless communication system is a characteristic related to a capability of the second layer to provide coverage.

[0013] In one aspect of the method, the second characteristic of the second layer of the wireless communication system is a characteristic related to a capability of the second layer to provide coverage.

[0014] In one aspect of the method, the configuration parameter is a model monitoring window.

[0015] In one aspect of the method, the configuration parameter is based on an amount of detected drift or degradation.

[0016] In one aspect of the method, the updating of the first model is based on a decision to measure the second characteristic of the second layer of the wireless communication system.

[0017] In one aspect of the method, the configuration parameter comprises a first configuration parameter that is a value between 0 and 1 and a second configuration parameter that is a model monitoring window.

[0018] In one aspect of the method, the second characteristic is not measured based on the inference in the first model and the configuration parameter indicating not to measure the second characteristic.

[0019] In one aspect of the method, the first model is not updated.

[0020] In one aspect of the method, the apparatus is a user equipment (UE).

[0021] In one aspect of the method, the apparatus is a network node.

[0022] In one aspect of the disclosure, a user equipment (UE) includes at least one processor and at least one memory storing instructions that, when executed by the at least one processor, cause the user equipment to perform at least any of the preceding methods.

[0023] In one aspect of the disclosure, an apparatus includes at least one processor and at least one memory storing instructions that, when executed by the at least one processor, cause the apparatus to perform at least any of the preceding methods.

[0024] In one aspect of the disclosure, a processor-readable medium stores instructions that, when executed by at least one processor of an apparatus, cause the apparatus to perform at least any of the preceding methods.

[0025] According to some aspects, the subject matter of the independent claims is provided. Some other aspects are defined in the dependent claims. BRIEF DESCRIPTION OF DRAWINGS

[0026] Some example embodiments will now be described with reference to the accompanying drawings.

[0027] Figure 1 is a diagram of example embodiments of wireless networking between a network system and a user equipment (UE) according to one illustrated aspect of the present disclosure; Figure 2 is a diagram of example components of a network system according to one illustrated aspect of the present disclosure; Figure 3 is a flow diagram of an example method for machine learning model monitoring management according to one illustrated aspect of the present disclosure; Figure 4 is a diagram of an example machine learning model monitoring window according to one illustrated aspect of the present disclosure; Figure 5 is a flow diagram of an example method for inter-frequency layer measurement machine learning model monitoring management according to one illustrated aspect of the present disclosure; and Figure 6 is a diagram of example embodiments of components of a UE or network apparatus according to one illustrated aspect of the present disclosure. DETAILED DESCRIPTION

[0028] In the following description, certain specific details are set forth in order to provide a thorough understanding of the disclosed aspects. However, one skilled in the relevant art will recognize that various aspects can be practiced without one or more of the specific details, or with other methods, components, materials, etc. In other instances, well-known structures associated with transmitters, receivers, or transceivers are not shown or described in detail to avoid obscuring aspects described herein.

[0029] Reference throughout this specification to "one aspect" or "an aspect" means that a particular feature, structure, or characteristic described in connection with the aspect is included in at least one aspect. Thus, the appearances of the phrase "in one aspect" or "in an aspect" in various places throughout this specification are not necessarily all referring to the same aspect. Furthermore, the particular features, structures, or characteristics can be combined in any suitable manner in one or more aspects.

[0030] The embodiments described in this disclosure can be implemented in wireless network devices, such as, but not limited to, devices utilizing wireless network systems such as Worldwide Interoperability for Microwave Access (WiMAX), Global System for Mobile Communications (GSM, 2G), GSM EDGE Radio Access Network (GERAN), General Packet Radio Service (GRPS), Universal Mobile Telecommunication System (UMTS, 3G) based on Basic Wideband Code Division Multiple Access (W-CDMA), High Speed Packet Access (HSPA), Long Term Evolution (LTE), LTE-Advanced, Enhanced LTE (eLTE), 5G New Radio (5G NR), 5G Advance, 6G (and beyond), and 802.11ax (Wi-Fi 6). The term "eLTE" denotes here LTE evolution connected to a 5G core. LTE is also referred to as Evolved UMTS Terrestrial Radio Access (EUTRA) or Evolved UMTS Terrestrial Radio Access Network (EUTRAN).

[0031] The present disclosure can use the term "serving network device" to refer to a network node or network device (or a part thereof) that provides service for a UE. As used herein, the terms "send to," "receive from," and "cooperate with" (and variations thereof) include communication that can or can not involve communication through one or more intermediary devices or nodes. The term "acquire" (and variations thereof) includes acquiring for the first time or reacquiring after the first time. The term "connect" can refer to a physical connection or a logical connection.

[0032] The present disclosure uses 5G NR as an example of a wireless network, and can use a smartphone and / or an extended reality headset as an example of a UE. It should be understood that these examples are merely illustrative, and the present disclosure is applicable to other wireless networks and user equipment.

[0033] Figure 1 is a schematic diagram depicting an example of a wireless network between a network system 100 and a user equipment (UE) 150. The network system 100 can include one or more network nodes 120, one or more servers 110, and / or one or more network devices 130 (e.g., test devices). The network nodes 120 will be described in more detail below. As used herein, the term "network device" can refer to any component of the network system 100, such as a server 110, a network node 120, a network device 130, any component of the foregoing, and / or any other component of the network system 100. Examples of network devices include, but are not limited to, devices implementing various aspects of 5G NR, and the like. The present disclosure describes embodiments related to 5G NR and embodiments involving aspects defined by the Third Generation Partnership Project (3GPP). However, embodiments related to other wireless network technologies are contemplated to be within the scope of the present disclosure.

[0034] The following description provides further details of examples of network nodes. In a 5G NR network, a gNodeB (also referred to as gNB) can comprise, for example, a node that provides New Radio (NR) user plane and control plane protocol terminations towards the UE, and interfaces with a 5G Core (5GC) through an NG interface, e.g., in accordance with 3GPP TS 38.300 V16.6.0 (2021-06) section 3.2, which is hereby incorporated by reference herein.

[0035] The gNB supports various protocol layers, e.g., Layer 1 (LI) - physical layer, Layer 2 (L2), and Layer 3 (L3).

[0036] Layer 2 (L2) of NR is split into the following sublayers: Medium Access Control (MAC), Radio Link Control (RLC), Packet Data Convergence Protocol (PDCP), and Service Data Adaptation Protocol (SDAP), e.g., in which: o the physical layer provides transport channels to the MAC sublayer; o the MAC sublayer provides logical channels to the RLC sublayer; o the RLC sublayer provides RLC channels to the PDCP sublayer; o the PDCP sublayer provides radio bearers to the SDAP sublayer; o the SDAP sublayer provides Quality of Service (QoS) flows to the 5GC; o control channels include Broadcast Control Channel (BCCH) and Physical Control Channel (PCCH).

[0037] Layer 3 (L3) includes, e.g., Radio Resource Control (RRC), e.g., in accordance with 3GPP TS 38.300 V16.6.0 (2021-06) section 6, which is hereby incorporated by reference herein.

[0038] A gNB central unit (gNB-CU) includes, e.g., a logical node that hosts, e.g., radio resource control (RRC), service data adaptation protocol (SDAP), and packet data convergence protocol (PDCP) protocols of a gNB or RRC and PDCP protocols of an en-gNB, which controls the operation of one or more gNB distributed units (gNB-DUs). The gNB-CU terminates the Fl interface connected with the gNB-DU. The gNB-CU can also be referred to herein as a CU, central unit, centralized unit, or control unit.

[0039] A gNB-distributed unit (gNB-DU) comprises, for example, logical nodes hosting, for example, radio link control (RLC), medium access control (MAC), and physical (PHY) layers of a gNB or en-gNB, and whose operation is partly controlled by a gNB-CU. One gNB-DU supports one or more cells. One cell is supported by only one gNB-DU. A gNB-DU terminates the F1 interface connected with a gNB-CU. A gNB-DU can also be referred to herein as a DU or distributed unit.

[0040] As used herein, the term "network node" can refer to any one of a gNB, a gNB-CU, or a gNB-DU, or any combination thereof. A RAN (Radio Access Network) node or network node, e.g., a gNB, a gNB-CU, or a gNB-DU, or parts thereof, can be implemented using, for example, an apparatus having at least one processor and / or at least one memory having processor-readable instructions ("programs") configured to support and / or provide and / or handle CU and / or DU related functions and / or features, and / or at least one protocol (sub)layer of a RAN (Radio Access Network), e.g., layer 2 and / or layer 3. There can be different splits of functions between a central unit and a distributed unit. Examples of such apparatuses and components will be described below in connection with Figure 6

[0041] A gNB-CU and a gNB-DU part can be, for example, co-located or physically separated. A gNB-DU can even be further split, for example, into two parts, for example, one comprising processing devices and one comprising antennas. A central unit (CU) can also be referred to as a baseband unit / radio equipment controller / cloud RAN / virtual RAN (BBU / REC / C-RAN / V-RAN), open RAN (O-RAN), or parts thereof. A distributed unit (DU) can also be referred to as a remote radio head / remote radio unit / radio equipment / radio unit (RRH / RRU / RE / RU), or parts thereof. Hereinafter, in various example embodiments of the present disclosure, a network node supporting central unit functions or at least one of layer 3 protocols of a radio access network can be, for example, a gNB-CU. Similarly, a network node supporting distributed unit functions or at least one of layer 2 protocols of a radio access network can be, for example, a gNB-DU.

[0042] A gNB-CU can support one or more gNB-DUs. A gNB-DU can support one or more cells and thus can support a serving cell of a user equipment (UE) or a candidate cell for procedures such as handover, dual connectivity, and / or carrier aggregation.

[0043] ​A user equipment (UE) 150 can be or include a wireless or mobile device, a device with a wireless interface that interacts with a RAN (Radio Access Network), a smartphone, a vehicle mounted device, an IoT device or M2M device, or the like type of user equipment. Such a UE 150 can include at least one processor and at least one memory including program code, where the at least one memory and the computer program code are configured to, with the at least one processor, cause the apparatus at least to perform certain operations, such as RRC connection with a RAN. Examples of components of a UE will be described in Figure 6 In embodiments, the UE 150 can be configured to generate a message (e.g., including a cell ID) to be transmitted over the air to the RAN (e.g., to reach and communicate with a serving cell). In embodiments, the UE 150 can generate and transmit and receive RRC messages containing one or more RRC PDUs (Packet Data Units). Those skilled in the art will understand the RRC protocol and other procedures that a UE can perform.

[0044] With continued reference to Figure 1 In an example of a 5G NR network, the network system 100 provides one or more cells that define coverage areas of the network system 100. As described above, the network system 100 can include a gNB of a 5G NR network or can include any other apparatus configured to control wireless communications and manage wireless resources within a cell. As used herein, the term "resource" can refer to a wireless resource, such as a resource block (RB), a physical resource block (PRB), a radio frame, a subframe, a time slot, a sub-band, a frequency region, a subcarrier, a beam, or the like. In embodiments, the network node 120 can be referred to as a base station.

[0045] Figure 1 One example is provided, merely illustrating the network system 100 and the UE 150. Those skilled in the art will understand that the network system 100 includes Figure 1 components not shown in

[0046] Figure 2 is Figure 1 a block diagram of example components of the network system 100. A 5G NR network can be described as an example of the network system 100, and it is contemplated that various aspects of the following description will apply to other types of network systems as well. The network system can operate according to the signals and connections shown in Figure 1 , such that the UE 150 communicates with the network system 100 through a radio access network 225. Further, the network system can be divided into user plane components and functions and control plane components and functions, as shown and described herein. Unless otherwise noted, the terms "component," "function," and "service" can be used interchangeably herein, and they can refer to and be implemented by instructions executed by one or more processors.

[0047] Example functions of the components are described below. The example functions are merely illustrative, and it is understood that the components described herein can perform other operations and functions. Moreover, the connections between the components can be virtual connections over service-based interfaces, such that any component can communicate with any other component. In this way, any component can act as a service "producer" for any other component as a service "consumer."

[0048] For example, a core network 210 is described in the control plane of the network system. The core network 210 can include an authentication server function (AUSF) 211, an access and mobility function (AMF) 212, and a session management function (SMF) 213. The core network 210 can also include a network slice selection function (NSSF) 214, a network exposure function (NEF) 215, a network repository function (NRF) 216, and a unified data management function (UDM) 217, which can include a unified data repository (UDR) 224.

[0049] Additional components and functions of the core network 210 can include an application function 218, a policy control function (PCF) 219, a network data analytics function (NWDAF) 220, an analytics data repository function (ADRF) 221, a management data analytics function (MDAF) 222, and an operations and management function (OAM) 223.

[0050] The user plane includes the UE 150, a radio access network (RAN) 225, a user plane function (UPF) 226, and a data network (DN) 227. The RAN 225 can include one or more components described in connection with Figure 1 The RAN 225 can not be limited to such components. The UPF 226 provides connectivity for data transmitted through the RAN 225. The DN 226 identifies services from service providers, Internet access, and third-party services, for example.

[0051] The AMF 212 handles connection and mobility tasks. The AUSF 211 receives an authentication request from the AMF 212 and interacts with the UDM 217 to authenticate and verify network responses to determine authentication success. The SMF 213 performs packet data unit (PDU) session management and manages session context with the UPF 226.

[0052] The NSSF 214 can select a network slice instance (NSI) and determine allowed network slice selection assistance information (NSSAI). The selection and determination are used to set the AMF 212 to serve the UE 150. The NEF 215 securely exposes services offered by the network functions to third parties, enabling third parties to create specialized services by using the network services. The NRF 216 serves as a repository of information of network functions, allowing them to register and discover each other.

[0053] The UDM 217 generates authentication vectors for use by the AUSF 211 and AMF 212, and provides user identification handling. The UDM 217 can be connected to the UDR 224, a repository storing data related to authentication, applications, and the like. The AF 218 provides application services (e.g., streaming services, etc.) to users. The PCF 219 provides policy control functions. For example, the PCF 219 can assist with network slicing and mobility management, as well as provide quality of service (QoS) and charging functions.

[0054] The NWDAF 220 collects data (e.g., from the UE 150 and network system) to perform network analytics, and provides insights to functions that leverage analytics in providing services. The ADRF 221 allows consumers to store, retrieve, and delete data and analytics. The MDAF 222 provides additional data analytics services for network functions. The OAM 223 provides configuration and management handling functions to manage elements in the network or connected to the network (e.g., UE 150, network nodes, etc.).

[0055] Figure 2 This is merely one example of a network system components, and variations are contemplated to be within the scope of the present disclosure. In embodiments, the network system can include other components not shown in Figure 2 In embodiments, the network system can not include each component shown in Figure 2 In embodiments, the components and connections can be implemented with different connections than those shown in Figure 2 Such and other embodiments are contemplated to be within the scope of the present disclosure.

[0056] While more details will be provided below, in supervised learning models, a machine learning (ML) model monitors is used to detect data or concept drift. To detect concept drift, the ML model monitoring needs to know the true values. However, obtaining the true values often has an associated cost. This cost can also include a decrease in performance of the ML operation.

[0057] In the example case of ML-based inter-frequency layer measurement reduction, concept drift can occur, for example, if one of the capacity layer cells is closed due to technical problems, or the coverage of the two layers is changed for any other reason. This can make the inter-frequency measurement prediction model inaccurate, and if the changes are more permanent, the model can need to be retrained.

[0058] For example, in the case of a positive prediction (e.g. when the model predicts that the UE can connect to a cell in the capacity tier), the capacity tier can be measured to ensure that the handover is possible, a ground truth can be established, and it can be determined whether the prediction was correct. However, in the case of a negative prediction, the ground truth is still established by measuring the capacity tier, which can not result in a reduction of inter-frequency measurements and introduce a cost in terms of measurement savings.

[0059] Therefore, when monitoring involves a cost, an operator of the ML solution can wish to configure and optimize the monitoring to optimize the ability to monitor the ML performance and detect any degradation therein with a trade-off with the monitoring cost.

[0060] Although more details will be provided below, in brief, described herein is a technique of performing measurements in which a measurement (e.g. a capacity measurement) is performed on a negative prediction inferred from the ML model, the frequency is reduced or performed with a given probability. Therefore, concept drift in both directions, false positives as well as false negatives, can be detected, e.g. if there is a capacity cell failure or an increase in the coverage area of the capacity tier. If a false negative is measured, appropriate decisions can be made to handover the UE to the capacity tier, which can improve the network quality of service (QoS).

[0061] Data drift can refer to a situation in which the distribution of the input data to the ML model changes from the distribution in the training data, which can result in the model being inaccurate. Concept drift can refer to a change in the function between the input samples to the ML model and the corresponding labels compared to the training dataset used to train the ML model, which can also result in inaccuracy.

[0062] ML model monitoring can be used to monitor the performance of the ML solution, in particular to detect data or concept drift in supervised learning ML models. If a performance degradation is detected, corrective measures can be taken. In the case of data or concept drift, the ML model can be retrained using a new training dataset in which the drift has been taken into account. In various embodiments, the monitoring can be performed as described in UK Application No. 2307062.6, which is incorporated by reference in its entirety.

[0063] Since monitoring has an associated cost, if the capacity tier is always measured when the model predicts negatively, no measurements can be saved. If the capacity tier is not measured for negative predictions, any concept drift that tends to more false negatives cannot be detected, nor can any decisions based thereon be corrected.

[0064] By adjusting the probability or frequency of measuring the negative prediction, measurement savings can be achieved without sacrificing concept drift detection. To illustrate, using a zero probability implementation, no measurements are performed, and using a 1.0 probability, measurements are always performed. By selecting a value between 0 and 100, the monitoring level can be controlled between these two extremes. Measuring the true value too frequently can be too costly, but measuring too infrequently can make drift detection too slow or too unreliable because there are not enough true value measurements to obtain statistical correlation.

[0065] As used herein, communication with a radio access network (RAN) can refer to and mean communication with a portion of the RAN, e.g., with a network node (e.g., a DU and / or CU) or another portion of the RAN. As used herein, communication with a core network can refer to and mean communication with one or more services / applications of the core network, e.g., an AMF or another service of the core network.

[0066] As used herein, the terms "first" and "second" and the like can refer to a first or second instance of a message sent / received by a component (e.g., a UE, an apparatus, etc.) or a first or second component in a sequence of described components. Thus, these terms are used in a non-limiting manner and can refer to any message, operation, device, component, etc.

[0067] According to the above brief description, Figure 3 is a flowchart of an example method 300 of machine learning model monitoring management according to one illustrated aspect of the disclosure. The following paragraphs will describe various operations. Although the various operations are presented in a certain order, it should be understood that the order of the operations is not limiting, the operations can be performed in any order, and certain operations can or can not be performed.

[0068] As Figure 3 As shown, in block 310, success of a first operation is predicted based on an inference in a first model (e.g., a ML model). For example, in various embodiments, in a wireless communication system, it can be predicted whether a UE can be switched from a first frequency to a second frequency (e.g., tier B to tier A) based on measurements (e.g., capacity, etc.) of these tiers. In various embodiments, the first tier (e.g., tier A) can be a macro cell tier that provides full coverage for UE communications, while the second tier (e.g., tier B) can be a capacity tier that provides better network QoS (e.g., better throughput than the first tier). In various embodiments, the second tier can have a different frequency than the first tier and can also implement a different radio access technology (RAT).

[0069] In block 320, the determination to measure the second characteristic of the wireless communication is based on the inference in block 310 and a configuration parameter. As described above, if the measurement of the second characteristic is always performed, then measurement savings can not be realized, and if the measurement of the second characteristic is not performed, then the ML model used to predict the success of the first operation (e.g., a handover operation from tier B to tier A) can become inaccurate. Thus, in various embodiments, a configuration parameter can be introduced to configure whether the measurement of the second characteristic is performed.

[0070] In some embodiments, the configuration parameter can include a probability parameter set between 0-100% (0.0-1.0) as described above. For example, a configuration parameter of 0.5 can indicate that the measurement of the second characteristic is performed 50% of the time.

[0071] In some embodiments, the configuration parameter can include configuring the length of the model monitoring window to compute the ML performance metrics. For example, the trade-off in model monitoring is between how quickly concept or data drift can be detected and how noisy the drift measurements are. This can be controlled by the length of the filter applied to compute the model accuracy. For example, a simple moving average window can be used to compute the false positive or negative rate, precision, recall, or f1 score of the model. The longer the window, the less noisy the monitoring is, but the slower it can be to detect any changes.

[0072] According to the above, Figure 4 is a diagram of an example machine learning model monitoring window 400 according to one illustrative aspect of the disclosure. As Figure 4 indicated, model accuracy metrics can be computed based on its past predictions according to the ML monitoring window length. When a new prediction is made and the true value is determined, the window 410a with the window monitoring window length can move forward and the metrics are computed again, resulting in window 410b.

[0073] In some embodiments, additional accuracy metrics of the ML model where the effective impact of the model monitoring is converted into ML model accuracy metrics based on how the ML predictions can be used and applied.

[0074] For example, there can be challenges in selecting and configuring the right model monitoring. In addition, the cost and to some extent the benefit of different model monitoring configurations can be monitored. If true negatives are measured, unnecessary measurements can have been made when the capacity tier cell was not available, so it effectively becomes a false positive behavior. Similarly, when model monitoring is applied, if false negatives are measured (e.g., the model predicted that the capacity tier cell was not available when it actually was), an action can be performed based on the measurement and a handover to the capacity tier is triggered, which can partially compensate for the cost of measuring true negatives.

[0075] Thus, two confusion matrices can be presented: one that does not take into account the impact of model monitoring, and one that does. Similarly, individual precision, recall, and f1 scores can be computed. Thus, one can understand the trade-off between the measures of savings under different amounts of model monitoring and connecting to the capacity layer when possible. Model monitoring allows the ability to detect concept drift and trigger retraining when necessary.

[0076] In some embodiments, automation can be utilized to configure the ML model monitoring level. For example, the monitoring amount / frequency can be configured lower when the model accuracy is high each time it is monitored, but can be automatically increased when the accuracy is below a specified threshold.

[0077] Referring again to Figure 3 , in step 330, if the determination in step 320 is positive, a second characteristic of the wireless communication system is measured. Thus, in step 330, a measurement can be made regarding capacity, etc. (e.g., on tier A).

[0078] In step 340, the first model is updated based on the measured second characteristic. From here, the method 300 can end or return to step 310 using the updated model learned through machine learning.

[0079] Figure 3 The operations / steps of are merely illustrative and variations are contemplated within the scope of the disclosure. In embodiments, the operations can include Figure 3 other operations not shown in Figure 3 In embodiments, the operations can not include every operation shown in Figure 3 In embodiments, the operations can be implemented in a different order than shown in

[0080] The operations are described below from the perspective of an apparatus. The apparatus can include a UE, a network apparatus, a network node, or another device described above. In various embodiments, all of the operations can be performed by the apparatus. In various embodiments, some of the operations are performed by the apparatus. From such a perspective, the method can include measuring a first characteristic of a wireless communication system, predicting success of a first operation of the wireless communication system based on an inference in a first model, measuring a second characteristic of the wireless communication by the apparatus based on the inference in the first model and a configuration parameter indicating that the second characteristic is to be measured, and updating the first model based on performing the measurement of the second characteristic of the wireless communication system.

[0081] Figure 5 is a flow diagram of an example method 500 for machine learning model monitoring management for inter-frequency tier measurements, according to one illustrated aspect of the present disclosure. AsFigure 5 As shown, the macro cell (coverage) can provide full coverage for the UE and can be referred to as tier A. The micro cell (capacity) can provide partial coverage but include more throughput for the UE and can be referred to as tier B. The UE can be connected to either the micro cell or the macro cell and have the ability to measure their respective cell reference signal received power (RSRP). The model learns to predict the probability of connecting to the micro cell given the RSRP of the macro cell to which the UE is currently connected.

[0082] At block 505, the UE measures tier B and at block 510, the tier B measurements are provided to the ML model. At block 515, the ML model predicts whether a handover (HO) to tier A is likely.

[0083] If the prediction is positive at block 515, at block 525, measurements of tier A are performed according to the configuration parameters as described above. In various embodiments, the configuration parameters can include a probability parameter, where the measurements are performed based on the value of the probability parameter being between 0.0-1.0, as described above. In various embodiments, the configuration parameters can include additional parameters as described above.

[0084] If the prediction is negative at block 515, then model monitoring can be performed at block 520 according to the configuration parameters as described above.

[0085] If after measuring tier A at block 525, it is determined at block 530 that a HO to tier A is not likely, then adjustments (e.g., probabilities) are made according to the ground truth used to update the ML model.

[0086] If after measuring tier A at block 525, it is determined at block 530 that a HO to tier A is likely, then the UE can be handed over to tier A at block 535. At block 540, the UE continues to measure tier A to determine whether the UE can remain in coverage with tier A. For example, if at block 545, it is possible to lose coverage in tier A, then in various embodiments, the UE can handover to tier B at block 550.

[0087] Reference is now made to Figure 6, showing a block diagram of example components of a UE or network device (e.g., of a RAN or core network). The device includes electronic storage 610, processor 620, network interface 640, and memory 650. The various components can be communicatively coupled to each other. Processor 620 can be and can include any type of processor, such as a single core central processing unit (CPU), a multi core CPU, a microprocessor, a digital signal processor (DSP), a system on chip (SoC), or any other type of processor. Memory 650 can be a volatile type of memory, such as RAM, or a non-volatile type of memory, such as NAND flash. Memory 650 includes processor-readable instructions that are executable by processor 620 to cause the device to perform various operations, including the operations mentioned herein, such as Figures 3-4

[0088] Electronic storage 610 can be and include any type of electronic storage, such as a hard disk drive, solid state drive, optical disk, and / or other non-transitory computer readable medium, among other types of electronic storage. Electronic storage 610 stores processor-readable instructions for causing or configuring the device to perform its operations and also stores data related to such operations, such as storing data related to the 5G NR standard, among other data. Network interface 640 can implement wireless network technologies, such as 5G NR and / or other wireless network technologies.

[0089] Figure 6 The components shown in FIG. 6 are merely examples, as one of skill in the art will understand that the device includes other components not shown and can include multiple instances of any of the components shown. These and other embodiments are contemplated within the scope of this disclosure. For example, a transmitter and receiver can be included as components for transmitting and receiving signals.

[0090] Further embodiments of the present disclosure include the following examples. Example 1.1. A device comprising: means for measuring, by the device, a first characteristic of a wireless communication system; means for predicting, by the device, success of a first operation of the wireless communication system based on an inference in a first model; means for measuring, by the device, a second characteristic of the wireless communication based on the inference in the first model and a configuration parameter indicating that the second characteristic is to be measured; and means for updating, by the device, the first model based on performing the measurement of the second characteristic of the wireless communication system.

[0091] Example 1.2. The device of Example 1.1, wherein the first characteristic is a characteristic of a first communication layer of the wireless communication system.

[0092] ​Example 1.3. The apparatus of Example 1.2, wherein the first operation is a transmission from the first communication layer of the wireless communication system to the second communication layer of the wireless communication system.

[0093] Example 1.4. The apparatus of Example 1.3, wherein the second characteristic is a characteristic of the second communication layer of the wireless communication system.

[0094] Example 1.5. The apparatus of Example 1.1, wherein measuring the second characteristic of the second layer of the wireless communication is positive based on an inference of success of the transmission from the first layer of the wireless communication system to the second layer of the wireless communication system.

[0095] Example 1.6. The apparatus of Example 1.5, wherein the configuration parameter is a value between 0 and 1.

[0096] Example 1.7. The apparatus of Example 1.6, wherein for a configuration value of 0, the measurement of the second characteristic of the second layer of the wireless communication is not performed.

[0097] Example 1.8. The apparatus of Example 1.6, wherein for a configuration value of 1, the measurement of the second characteristic of the second layer of the wireless communication is performed each time.

[0098] Example 1.9. The apparatus of Example 1.6, wherein for a first value of the configuration value, the measurement of the second characteristic of the second layer of the wireless communication is performed a greater number of times than for a second value of the configuration value, wherein the second value of the configuration value is less than the first value of the configuration value.

[0099] Example 1.10. The apparatus of Example 1.1, wherein the second characteristic of the second layer of the wireless communication system is a characteristic related to a capability of the second layer to provide coverage.

[0100] Example 1.11. The apparatus of Example 1.1, wherein the configuration parameter is a model monitoring window.

[0101] Example 1.12. The apparatus of Example 1.1, wherein the configuration parameter is based on an amount of detected drift or degradation.

[0102] Example 1.13. The apparatus of Example 1.1, wherein the updating of the first model is based on a determination to measure the second characteristic of the second layer of the wireless communication system.

[0103] Example 1.14. The apparatus of Example 1.1, wherein the configuration parameter includes a first configuration parameter that is a value between 0 and 1 and a second configuration parameter that is a model monitoring window.

[0104] Example 1.15. The apparatus of Example 1.1, wherein the second characteristic is not measured based on an inference in the first model and a configuration parameter indicating not to measure the second characteristic.

[0105] Example 1.16. The apparatus of Example 1.15, wherein the first model is not updated.

[0106] Example 1.17. The apparatus of Example 1.1, wherein the apparatus is a user equipment (UE).

[0107] Example 1.18. The apparatus of Example 1.1, wherein the apparatus is a network node.

[0108] Example 2.1. An apparatus comprising: at least one processor; and at least one memory storing instructions that, when executed by the at least one processor, cause the apparatus at least to: measure a first characteristic of a wireless communication system; predict, based on an inference in a first model, a success of a first operation of the wireless communication system; measure a second characteristic of the wireless communication based on the inference in the first model and a configuration parameter indicating that the second characteristic is to be measured; and update the first model based on performing the measurement of the second characteristic of the wireless communication system.

[0109] Example 2.2. The apparatus of Example 2.1, wherein the apparatus is a user equipment (UE).

[0110] Example 2.3. The apparatus of Example 2.1, wherein the apparatus is a network node.

[0111] Embodiments and aspects disclosed herein are examples of the present disclosure and can be implemented in various forms. For example, although certain embodiments herein are described as separate embodiments, each embodiment herein can be combined with one or more other embodiments herein. The specific structural and functional details disclosed herein are not to be interpreted as limiting but are to be construed as a basis for the claims and as a representative basis for teaching one skilled in the art to variously employ the present disclosure in virtually any appropriately detailed structure. Like reference numerals can refer to similar or identical elements throughout the figures.

[0112] The phrases "in one aspect," "in various aspects," "in some aspects" or "in other aspects" can each refer to one or more than one of the same or different aspects. The phrase "plurality" can refer to two or more.

[0113] In various embodiments, the terms "first message" and "second message" and any subsequent messages can refer to any message sent or received in a certain order and are not necessarily limited to any particular message.

[0114] The phrases "in one embodiment", "in various embodiments", "in some embodiments", "in other embodiments", "in one or more embodiments", or "in at least one embodiment" can each refer to the same or different embodiments. The phrases "A or B" means "(A), (B) or (A and B)". The phrase "at least one of A, B and C" means "(A), (B), (C), (A and B), (A and C), (B and C) or (A, B and C)".

[0115] Any method, procedure, algorithm, or code described herein can be converted to or expressed as a programming language or computer program. As used herein, the terms "programming language" and "computer program" each include any language used to specify instructions to a computer, and include (but are not limited to) the following languages and their derivatives: assembly language, Basic, batch files, BCPL, C, C+, C++, Delphi, Fortran, Java, JavaScript, machine code, operating system command languages, Pascal, Perl, PL1, Python, scripting languages, Visual Basic, meta-languages that specify programs per se, and all first, second, third, fourth, fifth, or higher generation computer languages. Database and other data schemas are also included, as are any other meta-languages. No distinction is made between languages that are interpreted, compiled, or use both compilation and interpretation. No distinction is made between compiled and source versions of a program. Thus, a reference to a program, where a programming language can exist in multiple states (such as source, compiled, object, or linked), is a reference to the program in any and all such states. A reference to a program can encompass the actual instructions and / or the intent of those instructions.

[0116] While aspects of the disclosure have been illustrated and described in the drawings and foregoing description, the disclosure is not intended to be limited to the aspects illustrated and described, as such will be within the scope of the claims, and other modifications, changes, and substitutions are intended in the foregoing disclosure.

Claims

1. A method comprising: measuring, by an apparatus, a first characteristic of a wireless communication system; predicting, by the apparatus, success of a first operation of the wireless communication system based on an inference in a first model; measuring, by the apparatus, a second characteristic of the wireless communication system based on the inference in the first model and based on a configuration parameter indicating that the second characteristic is to be measured; and updating, by the apparatus, the first model based on the measurement of the second characteristic of the wireless communication system being performed.

2. The method of claim 1, wherein the first characteristic is a characteristic of a first communication layer of the wireless communication system.

3. The method of claim 2, wherein the first operation is a transmission from the first communication layer of the wireless communication system to a second communication layer of the wireless communication system.

4. The method of claim 3, wherein the second characteristic is a characteristic of the second communication layer of the wireless communication system.

5. The method of claim 1, wherein measuring the second characteristic of the second layer of the wireless communication is positive based on an inference that a transmission from the first layer of the wireless communication system to a second layer of the wireless communication system is successful.

6. The method of claim 5, wherein the configuration parameter is a value between 0 and 1.

7. The method of claim 6, wherein the measurement of the second characteristic of the second layer of the wireless communication is not performed for a configuration value of 0.

8. The method of claim 6, wherein the measurement of the second characteristic of the second layer of the wireless communication is performed every time for a configuration value of 1.

9. The method of claim 6, wherein the measurement of the second characteristic of the second layer of the wireless communication is performed a greater number of times for a first value of the configuration value than for a second value of the configuration value, wherein the second value of the configuration value is less than the first value of the configuration value.

10. The method of claim 1, wherein the second characteristic of the second layer of the wireless communication system is a characteristic related to a capability of the second layer to provide coverage.

11. The method of claim 1, wherein the configuration parameter is a model monitoring window.

12. The method of claim 1, wherein the configuration parameter is based on an amount of detected drift or degradation.

13. The method of claim 1, wherein the updating of the first model is based on the determination that the second characteristic of the second layer of the wireless communication system is to be measured.

14. The method of claim 1, wherein the configuration parameter comprises a first configuration parameter that is a value between 0 and 1 and a second configuration parameter that is a model monitoring window.

15. The method of claim 1, wherein the second characteristic is not measured based on the inference in the first model and the configuration parameter indicating that the second characteristic is not to be measured.

16. The method of claim 15, wherein the first model is not updated.

17. The method of claim 1, wherein the apparatus is a user equipment (UE). ​ 18. The method of claim 1, wherein the apparatus is a network node.

19. A user equipment (UE), comprising: at least one processor; and at least one memory storing instructions that, when executed by the at least one processor, cause the UE to perform at least the method of any one of claims 1-18.

20. An apparatus, comprising: at least one processor; and at least one memory storing instructions that, when executed by the at least one processor, cause the apparatus to perform at least the method of any one of claims 1-18.

21. A processor-readable medium storing instructions that, when executed by at least one processor of an apparatus, cause the apparatus to perform at least the method of any one of claims 1-18.

Images