Machine learning model monitoring management

Abstract

A method includes measuring, by an apparatus, a first characteristic of a wireless communication system. The apparatus predicts based on an inference in a first model, a success of a first operation of the wireless communication system. Based on the inference in the first model and on a configuration parameter indicating a second characteristic is to be measured, the apparatus measures the second characteristic of the wireless communication, and updates the first model based on the measuring of the second characteristic of the wireless communication system being performed.

Description

MACHINE LEARNING MODEL MONITORING MANAGEMENTFIELD

[0001] Various example embodiments relate generally to wireless networks and, more particularly, to a method for machine learning model monitoring management.BACKGROUND

[0002] In supervised learning models, machine learning (ML) model monitoring is utilized to detect data or concept drift. To detect concept drift, the ML model monitoring requires knowing the ground truth. However, obtaining the ground truth often has an associated cost. This cost may also include reduced performance of the ML operation.

[0003] In an example case of ML-based inter- frequency layer measurement reduction, concept drift may occur, for example, if one of the capacity layer cells goes down because of a technical issue, or the coverage of the two layers changes for any other reason. This may render the interfrequency measurement prediction model inaccurate and if the change is more permanent, may require re-training of the model.SUMMARY

[0004] In an aspect of the present disclosure, a method includes measuring, by an apparatus, a first characteristic of a wireless communication system. The apparatus predicts based on an inference in a first model, a success of a first operation of the wireless communication system. Based on the inference in the first model and on a configuration parameter indicating a second characteristic is to be measured, the apparatus measures the second characteristic of the wireless communication, and updates the first model based on the measuring of the second characteristic of the wireless communication system being performed.

[0005] In an aspect of the method, the first characteristic is a characteristic of a first communication layer of the wireless communication system.

[0006] In an aspect of the method, the first operation is a transfer from the first communication layer of the wireless communication system to a second communication layer of the wireless communication system.

[0007] In an aspect of the method, the second characteristic is a characteristic of the second communication layer of the wireless communication system.

[0008] In an aspect of the method, measuring the second characteristic of the second layer of the wireless communication is based on the inference of transfer success from the first layer of the wireless communication system to a second layer of the wireless communication system being positive.

[0009] In an aspect of the method, the configuration parameter is a value between 0 and 1.

[0010] In an aspect of the method, no measuring of the second characteristic of the second layer of the wireless communication is performed for a configuration value of 0.

[0011] In an aspect of the method, measuring of the second characteristic of the second layer of the wireless communication is performed every time for a configuration value of 1.

[0012] In an aspect of the method, measuring of the second characteristic of the second layer of the wireless communication is performed a greater number of times for a first value of the configuration value than for a second value of the configuration value, wherein the second value of the configuration value is less than the first value of the configuration value.

[0013] In an aspect of the method, the second characteristic of the second layer of the wireless communication system is a characteristic related to an ability of the second layer to provide coverage.

[0014] In an aspect of the method, the configuration parameter is a model monitoring window.

[0015] In an aspect of the method, the configuration parameter is based on an amount of detected drift or degradation.

[0016] In an aspect of the method, the updating of the first model is based on the determination to measure the second characteristic of the second layer of the wireless communication system.

[0017] In an aspect of the method, the configuration parameter includes a first configuration parameter that is a value between 0 and 1, and a second configuration parameter that is a model monitoring window.

[0018] In an aspect of the method, the second characteristic is not measured based upon the inference in the first model and the configuration parameter indicating a second characteristic is not to be measured.

[0019] In an aspect of the method, the first model is not updated.

[0020] In an aspect of the method, the apparatus is a user equipment (UE).

[0021] In an aspect of the method, the apparatus is a network node.

[0022] In an aspect of the present disclosure, a user equipment (UE) includes at least one processor, and at least one memory storing instructions which, when executed by the at least one processor, cause the user equipment at least to perform any of the foregoing methods.

[0023] In an aspect of the present disclosure, an apparatus includes at least one processor, and at least one memory storing instructions which, when executed by the at least one processor, cause the apparatus at least to perform any of the foregoing methods.

[0024] In an aspect of the present disclosure, a processor-readable medium storing instructions which, when executed by at least one processor of an apparatus, cause the apparatus at least to perform any of the foregoing methods.

[0025] According to some aspects, there is provided the subject matter of the independent claims. Some further aspects are defined in the dependent claims.BRIEF DESCRIPTION OF THE DRAWINGS

[0026] Some example embodiments will now be described with reference to the accompanying drawings.

[0027] FIG. 1 is a diagram of an example embodiment of wireless networking between a network system and a user equipment (UE), according to one illustrated aspect of the disclosure;

[0028] FIG. 2 is a diagram of example components of a network system, according to one illustrated aspect of the disclosure;

[0029] FIG. 3 is a flow diagram of an example method for machine learning model monitoring management, according to one illustrated aspect of the disclosure;

[0030] FIG. 4 is a diagram of an example machine learning model monitoring window, according to one illustrated aspect of the disclosure;

[0031] FIG. 5 is a flow diagram of an example method for machine learning model monitoring management for inter-frequency layer measurements, according to one illustrated aspect of the disclosure; and

[0032] FIG. 6 is a diagram of an example embodiment of components of a UE or of a network apparatus, according to one illustrated aspect of the present disclosure.DETAILED DESCRIPTION

[0033] In the following description, certain specific details are set forth in order to provide a thorough understanding of disclosed aspects. However, one skilled in the relevant art will recognize that aspects may be practiced without one or more of these specific details or with other methods, components, materials, etc. In other instances, well-known structures associated with transmitters, receivers, or transceivers have not been shown or described in detail to avoid unnecessarily obscuring descriptions of the aspects.

[0034] Reference throughout this specification to “one aspect” or “an aspect” means that a particular feature, structure, or characteristic described in connection with the aspect is included in at least one aspect. Thus, the appearances of the phrases “in one aspect” or “in an aspect” in various places throughout this specification are not necessarily all referring to the same aspect. Furthermore, the particular features, structures, or characteristics may be combined in any suitable manner in one or more aspects.

[0035] Embodiments described in the present disclosure may be implemented in wireless networking apparatuses, such as, without limitation, apparatuses utilizing Worldwide Interoperability for Microwave Access (WiMAX), Global System for Mobile communications (GSM, 2G), GSM EDGE radio access Network (GERAN), General Packet Radio Service (GRPS), Universal Mobile Telecommunication System (UMTS, 3G) based on basic wideband-code division multiple access (W-CDMA), high-speed packet access (HSPA), Long Term Evolution (LTE), LTE- Advanced, enhanced LTE (eLTE), 5G New Radio (5G NR), 5G Advance, 6G (and beyond) and 802.1 lax (Wi-Fi 6), among other wireless networking systems. The term ‘eLTE’ here denotes the LTE evolution that connects to a 5G core. LTE is also known as evolved UMTS terrestrial radio access (EUTRA) or as evolved UMTS terrestrial radio access network (EUTRAN).

[0036] The present disclosure may use the term “serving network device” to refer to a network node or network device (or a portion thereof) that services a UE. As used herein, the terms “transmit to,” “receive from,” and “cooperate with,” (and their variations) include communications that may or may not involve communications through one or more intermediate devices or nodes. The term “acquire” (and its variations) includes acquiring in the first instance or reacquiring after the first instance. The term “connection” may mean a physical connection or a logical connection.

[0037] The present disclosure uses 5G NR as an example of a wireless network and may use smartphones and / or extended reality headsets as an example of UEs. It is intended and shall be understood that such examples are merely illustrative, and the present disclosure is applicable to other wireless networks and user equipment.

[0038] FIG. 1 is a diagram depicting an example of wireless networking between a network system 100 and a user equipment (UE) 150. The network system 100 may include one or more network nodes 120, one or more servers 110, and / or one or more network equipment 130 (e.g., test equipment). The network nodes 120 will be described in more detail below. As used herein, the term “network apparatus” may refer to any component of the network system 100, such as the server 110, the network node 120, the network equipment 130, any component(s) of the foregoing, and / or any other component(s) of the network system 100. Examples of network apparatuses include, without limitation, apparatuses implementing aspects of 5G NR, among others. The present disclosure describes embodiments related to 5GNR and embodiments that involve aspects defined by 3rd Generation Partnership Project (3GPP). However, it is contemplated that embodiments relating to other wireless networking technologies are encompassed within the scope of the present disclosure.

[0039] The following description provides further details of examples of network nodes. In a 5G NR network, a gNodeB (also known as gNB) may include, e.g., a node that provides new radio (NR) user plane and control plane protocol terminations towards the UE and that is connected via a NG interface to the 5G core (5GC), e.g., according to 3GPP TS 38.300 V16.6.0 (2021-06) section 3.2, which is hereby incorporated by reference herein.

[0040] A gNB supports various protocol layers, e.g., Layer 1 (LI) - physical layer, Layer 2 (L2), and Layer 3 (L3).

[0041] The layer 2 (L2) of NR is split into the following sublayers: Medium Access Control (MAC), Radio Link Control (RLC), Packet Data Convergence Protocol (PDCP) and Service Data Adaptation Protocol (SDAP), where, e.g.: o The physical layer offers to the MAC sublayer transport channels; o The MAC sublayer offers to the RLC sublayer logical channels; o The RLC sublayer offers to the PDCP sublayer RLC channels; o The PDCP sublayer offers to the SDAP sublayer radio bearers; o The SDAP sublayer offers to 5GC quality of service (QoS) flows;o Control channels include broadcast control channel (BCCH) and physical control channel (PCCH).

[0042] Layer 3 (L3) includes, e.g., radio resource control (RRC), e.g., according to 3GPP TS 38.300 V16.6.0 (2021-06) section 6, which is hereby incorporated by reference herein.

[0043] A gNB central unit (gNB-CU) includes, e.g., a logical node hosting, e.g., radio resource control (RRC), service data adaptation protocol (SDAP), and packet data convergence protocol (PDCP) protocols of the gNB or RRC and PDCP protocols of the en-gNB, that controls the operation of one or more gNB distributed units (gNB-DUs). The gNB-CU terminates the Fl interface connected with the gNB-DU. A gNB-CU may also be referred to herein as a CU, a central unit, a centralized unit, or a control unit.

[0044] A gNB Distributed Unit (gNB-DU) includes, e.g., a logical node hosting, e.g., radio link control (RLC), media access control (MAC), and physical (PHY) layers of the gNB or en- gNB, and its operation is partly controlled by the gNB-CU. One gNB-DU supports one or multiple cells. One cell is supported by only one gNB-DU. The gNB-DU terminates the Fl interface connected with the gNB-CU. A gNB-DU may also be referred to herein as DU or a distributed unit.

[0045] As used herein, the term “network node” may refer to any of a gNB, a gNB-CU, or a gNB-DU, or any combination of them. A RAN (radio access network) node or network node such as, e.g., a gNB, gNB-CU, or gNB-DU, or parts thereof, may be implemented using, e.g., an apparatus with at least one processor and / or at least one memory with processor-readable instructions (“program”) configured to support and / or provision and / or process CU and / or DU related functionality and / or features, and / or at least one protocol (sub-)layer of a RAN (radio access network), e.g., layer 2 and / or layer 3. Different functional splits between the central and distributed unit are possible. An example of such an apparatus and components will be described in connection with FIG. 6 below.

[0046] The gNB-CU and gNB-DU parts may, e.g., be co-located or physically separated. The gNB-DU may even be split further, e.g., into two parts, e.g., one including processing equipment and one including an antenna. A central unit (CU) may also be called baseband unit / radio equipment controller / cloud-RAN / virtual-RAN (BBU / REC / C-RAN / V-RAN), open-RAN (O- RAN), or part thereof. A distributed unit (DU) may also be called remote radio head / remote radio unit / radio equipment / radio unit (RRH / RRU / RE / RU), or part thereof. Hereinafter, in variousexample embodiments of the present disclosure, a network node, which supports at least one of central unit functionality or a layer 3 protocol of a radio access network, may be, e.g., a gNB-CU. Similarly, a network node, which supports at least one of distributed unit functionality or a layer 2 protocol of the radio access network, may be, e.g., a gNB-DU.

[0047] A gNB-CU may support one or multiple gNB-DUs. A gNB-DU may support one or multiple cells and, thus, could support a serving cell for a user equipment (UE) or support a candidate cell for handover, dual connectivity, and / or carrier aggregation, among other procedures.

[0048] The user equipment (UE) 150 may be or include a wireless or mobile device, an apparatus with a radio interface to interact with a RAN (radio access network), a smartphone, an in-vehicle apparatus, an loT device, or a M2M device, among other types of user equipment. Such UE 150 may include: at least one processor; and at least one memory including program code; where the at least one memory and the computer program code are configured to, with the at least one processor, cause the apparatus at least to perform certain operations, such as, e.g., RRC connection to the RAN. An example of components of a UE will be described in connection with FIG. 6. In embodiments, the UE 150 may be configured to generate a message (e.g., including a cell ID) to be transmitted via radio towards a RAN (e.g., to reach and communicate with a serving cell). In embodiments, the UE 150 may generate and transmit and receive RRC messages containing one or more RRC PDUs (packet data units). Persons skilled in the art will understand RRC protocol as well as other procedures a UE may perform.

[0049] With continuing reference to FIG. 1, in the example of a 5G NR network, the network system 100 provides one or more cells, which define a coverage area of the network system 100. As described above, the network system 100 may include a gNB of a 5G NR network or may include any other apparatus configured to control radio communication and manage radio resources within a cell. As used herein, the term “resource” may refer to radio resources, such as a resource block (RB), a physical resource block (PRB), a radio frame, a subframe, a time slot, a sub-band, a frequency region, a sub-carrier, a beam, etc. In embodiments, the network node 120 may be called a base station.

[0050] FIG. 1 provides an example and is merely illustrative of a network system 100 and a UE 150. Persons skilled in the art will understand that the network system 100 includes components not illustrated in FIG. 1 and will understand that other user equipment may be in communication with the network system 100.

[0051] FIG. 2 is a block diagram of example components of the network system 100 of FIG. 1. A 5G NR network may be described as an example of the network system 100, and it is intended that aspects of the following description shall be applicable to other types of network systems, as well. The network system may operate in accordance with the signals and connections shown in FIG. 1 such that the UE 150 is in communication with the network system 100 through the radio access network 225. Additionally, the network system may be divided into user plane components and functions and control plane components and functions, as shown and described herein. Unless indicated otherwise, the terms “component”, “function”, and “service” may be used interchangeably herein, and they may refer to and be implemented by instructions executed by one or more processors.

[0052] Example functions of the components are described below. The example functions are merely illustrative, and it shall be understood that additional operations and functions may be performed by the components described herein. Additionally, the connections between components may be virtual connections over service-based interfaces such that any component may communicate with any other component. In this manner, any component may act as a service “producer,” for any other component that is a service “consumer,” to provide services for network functions.

[0053] For example, a core network 210 is described in the control plane of the network system. The core network 210 may include an authentication server function (AUSF) 211, an access and mobility function (AMF) 212, and a session management function (SMF) 213. The core network 210 may also include a network slice selection function (NSSF) 214, a network exposure function (NEF) 215, a network repository function (NRF) 216, and a unified data management function (UDM) 217, which may include a uniform data repository (UDR) 224.

[0054] Additional components and functions of the core network 210 may include an application function 218, policy control function (PCF) 219, network data analytics function (NWDAF) 220, analytics data repository function (ADRF) 221, management data analytics function (MDAF) 222, and operations and management function (0AM) 223.

[0055] The user plane includes the UE 150, a radio access network (RAN) 225, a user plane function (UPF) 226, and a data network (DN) 227. The RAN 225 may include one or more components described in connection with FIG. 1, such as one or more network nodes. However, the RAN 225 may not be limited to such components. The UPF 226 provides connection for databeing transmitted over the RAN 225. The DN 226 identifies services from service providers, Internet access, and third party services, for example.

[0056] The AMF 212 processes connection and mobility tasks. The AUSF 211 receives authentication requests from the AMF 212 and interacts with UDM 217 to authenticate and validate network responses for determination of successful authentication. The SMF 213 conducts packet data unit (PDU) session management, as well as manages session context with the UPF 226.

[0057] The NSSF 214 may select a network slicing instance (NSI) and determine the allowed network slice selection assistance information (NSSAI). This selection and determination is utilized to set the AMF 212 to provide service to the UE 150. The NEF 215 secures access to network services for third parties to create specialized network services. The NRF 216 acts as a repository to store network functions to allow the functions to register with and discover each other.

[0058] The UDM 217 generates authentication vectors for use by the AUSF 211 and ADM 212 and provides user identification handling. The UDM 217 may be connected to the UDR 224 which stores data associated with authentication, applications, or the like. The AF 218 provides application services to a user (e.g., streaming services, etc.). The PCF 219 provides policy control functionality. For example, the PCF 219 may assist in network slicing and mobility management, as well as provide quality of service (QoS) and charging functionality.

[0059] The NWDAF 220 collects data (e.g., from the UE 150 and the network system) to perform network analytics and provide insight to functions that utilize the analytics in the providing of services. The ADRF 221 allows the storage, retrieval, and removal of data and analytics by consumers. The MDAF 222 provides additional data analytics services for network functions. The 0AM 223 provides provisioning and management processing functions to manage elements in or connected to the network (e.g., UE 150, network nodes, etc.).

[0060] FIG. 2 is merely an example of components of a network system, and variations are contemplated to be within the scope of the present disclosure. In embodiments, the network system may include other components not illustrated in FIG. 2. In embodiments, the network system may not include every component illustrated in FIG. 2. In embodiments, the components and connections may be implemented with different connections than those illustrated in FIG. 2. Such and other embodiments are contemplated to be within the scope of the present disclosure.

[0061] Although further detail will be provided below, in supervised learning models, machine learning (ML) model monitoring is utilized to detect data or concept drift. To detect concept drift, the ML model monitoring requires knowing the ground truth. However, obtaining the ground truth often has an associated cost. This cost may also include reduced performance of the ML operation.

[0062] In an example case of ML-based inter- frequency layer measurement reduction, concept drift may occur, for example, if one of the capacity layer cells goes down because of a technical issue, or the coverage of the two layers changes for any other reason. This may render the interfrequency measurement prediction model inaccurate and if the change is more permanent, may require re-training of the model.

[0063] For example, in the case of positive predictions (e.g., when the model predicts that a UE can connect to a cell in the capacity layer), the capacity layer may be measured to be sure the handover is possible, the ground truth may be established, and it may be determined if the prediction was correct or not. However, in case that the model predicts negative, the ground truth is still established by measuring the capacity layer, which may not result in a reduction of interfrequency measurements and introduce a cost in terms of less saved measurements.

[0064] Accordingly, when the monitoring involves a cost, the operator of the ML solution may desire to configure and optimize the monitoring to optimize the tradeoff between being able to monitor the ML performance and detect any degradations in it and the cost of the monitoring.

[0065] Although further detail will be provided below, briefly, described herein is a technique for performing measurements where measurements are performed (e.g., capacity measurements) for negative prediction inferred from an ML model with a reduced frequency or with a given probability. Accordingly, concept drift in both directions, false positives as well as false negatives may be detected, for example if there is a capacity cell failure or the coverage area of the capacity layer is increased. If a false negative is measured, a proper decision can be made to hand a UE over to the capacity layer, which may improve network quality of service (QoS).

[0066] Data drift may refer to a situation where the distribution of the ML input data changes from what it was in the training data, which can lead to model inaccuracy. Concept drift may refer to the function between input samples to an ML model and corresponding labels changing compared to the training data set used to train the ML model, which can lead to inaccuracy as well.

[0067] ML model monitoring may be used to monitor the performance of an ML solution and especially to detect data or concept drift in supervised learning ML models. If a performancedegradation is detected, corrective actions can be taken. In case of data or concept drift, the ML model may be re-trained with a new training dataset, where the drift has been considered. In various embodiments, monitoring may be performed as described in UK Application No. 2307062.6, incorporated herein by reference as if fully set forth.

[0068] Since monitoring has an associated cost, if the capacity layer is always measured when the model predicts negative, there may be no saving of any measurements. If the capacity layer is not measured for negative predictions, there can be no detection of any concept drift towards more false negatives no correction any decisions based on them.

[0069] By adjusting the probability or frequency of measuring negative predictions, measurement savings may be attained without sacrificing detection of concept drift. For purposes of example, with a zero probability implementation, no measurements are performed and with a 1.0 probability, measurements are always made. By selecting a value between 0 and 100, the level of monitoring may be controlled between these two extremes. Measuring the ground truth too often may be too costly but measuring it too infrequently may make the detection of drift too slow or too unreliable because of not having enough ground truth measurements for statistical relevance.

[0070] As used herein, a communication with a radio access network (RAN) may refer to and mean a communication with a portion of a RAN, such as with a network node (e.g., a DU and / or a CU), or another portion of a RAN. As used herein, a communication with a core network may refer to and mean a communication with one or more services / applications of the core network, such as AMF or another service of a core network.

[0071] As used herein, the terms “first” and “second”, or the like, may refer to a first or second instance of a message being transmitted / received by a component (e.g., UE, apparatus, etc.), or a first or second component in a sequence of described components. As such, the terms are used in a non-limiting manner, and can refer to any message, operation, device, component, or the like.

[0072] In accordance with the brief description, FIG. 3 is a flow diagram of an example method 300 for machine learning model monitoring management, according to one illustrated aspect of the disclosure. The following paragraphs will describe various operations. Although the various operations are presented in an order, it will be understood that the order of the operations is not limiting, the operations may be performed in any order, and some operations may or may not be performed.

[0073] As shown in FIG. 3, in block 310, a success of a first operation is predicted based on an inference in a first model (e.g., ML model). For example, in various embodiments, in a wireless communication system, a prediction may be made whether a UE may be handed over from a first frequency to a second frequency (e.g., layer B to layer A) based upon measurements (such as capacity, etc.) for those layers. In various embodiments, a first layer (e.g., layer A) may be a macro cell layer offering full coverage for a UE communication, whereas a second layer (e.g., layer B) may be a capacity layer, which offers better network QoS, (e.g., better throughput than the first layer). In various embodiments, the second layer may have a different frequency than the first layer and may also implement a different Radio Access Technology (RAT).

[0074] In block 320, a determination to measure a second characteristic of the wireless communication may be based on the inference in block 310 as well as a configuration parameter. As described above, if measurements are always performed of the second characteristic, no measurement savings may be realized, whereas if no measurements are of the second characteristic are performed, the ML model for predicting success of the first operation (e.g., a handover operation from layer B to layer A) may become inaccurate. Accordingly, in various embodiments, a configuration parameter may be introduced that configures whether or not to perform the measurement of the second characteristic.

[0075] In some embodiments, the configuration parameter may include a probability parameter set between 0-100% (0.0-1.0) as described above. For example, a configuration parameter of 0.5 may indicate to perform the measuring of the second characteristic 50% of the time.

[0076] In some embodiments, the configuration parameter may include configuring a length of a model monitoring window to calculate ML performance metrics. For example, a tradeoff in model monitoring is between how fast concept or data drift can be detected and how noisy the measurement of the drift is. This can be controlled with the length of the filter that is applied to calculate the model accuracy. For example, a simple moving average window may be used to calculate the model false positive or negative rate, precision, recall or fl -score. The longer the window, the less noisy the monitoring will be, but also the slower it may detect any changes.

[0077] In accordance with the above, FIG. 4 is a diagram of an example machine learning model monitoring window 400, according to one illustrated aspect of the disclosure. As shown in FIG. 4, model accuracy metrics may be calculated based on its past predictions according to the ML monitoring window length. As a new prediction is made and ground truth is determined, a window 410a having a window monitoring window length may be moved forward and the metrics calculated again, resulting in a window 410b.

[0078] In some embodiments, additional accuracy metrics for the ML model, where the effective impact of the model monitoring is translated into the ML model accuracy metrics based on how the ML predictions may be used and applied.

[0079] For example, a challenge may be faced in choosing and configuring the right amount of model monitoring. Also the cost, and to some extent the benefit, of different model monitoring amount configurations can be monitored. If a true negative is measured, an unnecessary measurement may have been made, when a capacity layer cell was not available, so in effect it becomes false positive behavior. Similarly, when applying the model monitoring, if a false negative is measured (e.g., the model predicts that a capacity layer cell is not available when one really is), then an act may be performed based on the measurement and trigger a handover to the capacity layer, which may partially compensate for the cost of measuring true negatives.

[0080] Accordingly, two confusion matrixes may be presented: one where the impact of model monitoring is not considered and one where it is. Similarly, separate precision, recall and fl -scores may be calculated. Accordingly, the tradeoff between the measurements saved and connecting to the capacity layer, when possible, with different amounts of model monitoring can be understood. Model monitoring allows the ability to detect the concept drift and trigger re-training, if necessary.

[0081] In some embodiments, automation may be utilized for configuring the ML model monitoring level. For example, when the model accuracy, per monitoring, is high, the monitoring amount / frequency can be configured lower, but may be automatically increased, when the accuracy falls below a specified threshold.

[0082] Referring again to FIG. 3, in step 330, the second characteristic of the wireless communication system is measured if the determination in step 320 is positive. Accordingly, in step 330, measurements may be made (e.g., to layer A) regarding capacity, etc.

[0083] In step 340, the first model is updated based upon the measured second characteristic. From here, the method 300 may end or revert to step 310 using the updated model learned via machine learning.

[0084] The operations / steps of FIG. 3 are merely illustrative, and variations are contemplated to be within the scope of the present disclosure. In embodiments, the operations may include other operations not illustrated in FIG. 3. In embodiments, the operations may not include every operation illustrated in FIG. 3. In embodiments, the operations may be implemented in a different order than that illustrated in FIG. 3. Such and other embodiments are contemplated to be within the scope of the present disclosure. Persons of skill in the art will appreciate that, although various example components are described as perform various functions, other components may perform those functions described in the method 300.

[0085] The following describes operations from the perspective of an apparatus. The apparatus may include a UE, a network apparatus, a network node, or another device described above. In various embodiments, all operations may be performed by the apparatus. In various embodiments, some operations are performed by the apparatus. From such a perspective, a method may include measuring, a first characteristic of a wireless communication system, predicting, based on an inference in a first model, a success of a first operation of the wireless communication system, based on the inference in the first model and on a configuration parameter indicating a second characteristic is to be measured, measuring, by the apparatus, the second characteristic of the wireless communication, and updating the first model based on the measuring of the second characteristic of the wireless communication system being performed.

[0086] FIG. 5 is a flow diagram of an example method 500 for machine learning model monitoring management for inter-frequency layer measurements, according to one illustrated aspect of the disclosure. As shown in FIG. 5, a macro cell (coverage) may provide full coverage for a UE and may be referred to as layer A. Micro cells (capacity) may provide partial coverage for UEs but include more throughput, and may be referred to as layer B. UEs can connect to either the micro or macro cells with the ability to measure their respective cell reference signal received power (RSRP). The model learns to predict the probability of connecting to a micro cell given the RSRPs of the macro cell that the UE is currently connected to.

[0087] At block 505, the UE measures layer B and at block 510, the layer B measurements are provided to the ML model. At block 515, the ML model predicts if a handover (HO) to layer A is possible.

[0088] If the prediction is positive at block 515, at block 525, a measurement of layer A is performed in accordance with the configuration parameter as described above. In various embodiments, the configuration parameter may include a probability parameter, where the measurement is performed based upon the value of the probability parameter being between 0.0- 1.0 as described above. In various embodiments, the configuration parameter may include additional parameters as described above.

[0089] If the predication is negative at block 515, then model monitoring may be performed at block 520 in accordance with the configuration parameter(s) as described above.

[0090] If after measuring layer A at block 525, it is determined that a HO to layer A is not possible at block 530, then an adjustment is made (e.g., probability) according to the ground truth for updating the ML model.

[0091] If after measuring layer A at block 525, it is determined that a HO to layer A is possible at block 530, then a handover of the UE to layer A may be effected at block 535. At block 540, the UE continues to measure layer A to determine if the UE may remain in coverage with layer A. If, for example, coverage may be lost in layer A at block 545, then the UE, in various embodiments may handover to layer B at block 550.

[0092] Referring now to FIG. 6, there is shown a block diagram of example components of a UE or a network apparatus (e.g., of a RAN or a core network). The apparatus includes an electronic storage 610, a processor 620, a network interface 640, and a memory 650. The various components may be communicatively coupled with each other. The processor 620 may be and may include any type of processor, such as a single-core central processing unit (CPU), a multi-core CPU, a microprocessor, a digital signal processor (DSP), a System- on- Chip (SoC), or any other type of processor. The memory 650 may be a volatile type of memory, e.g., RAM, or a non-volatile type of memory, e.g., NAND flash memory. The memory 650 includes processor-readable instructions that are executable by the processor 620 to cause the apparatus to perform various operations, including those mentioned herein, such as the operations of FIGS. 3-4.

[0093] The electronic storage 610 may be and include any type of electronic storage used for storing data, such as hard disk drive, solid state drive, optical disc, and / or other non-transitorycomputer-readable mediums, among other types of electronic storage. The electronic storage 610 stores processor-readable instructions for causing or configured for causing the apparatus to perform its operations and also stores data associated with such operations, such as storing data relating to 5 G NR standards, among other data. The network interface 640 may implement wireless networking technologies such as 5G NR and / or other wireless networking technologies.

[0094] The components shown in FIG. 6 are merely examples, and persons skilled in the art will understand that an apparatus includes other components not illustrated and may include multiples of any of the illustrated components. Such and other embodiments are contemplated to be within the scope of the present disclosure. For example, a transmitter and a receiver may be included as components for transmitting and receiving signals.

[0095] Further embodiments of the present disclosure include the following examples.

[0096] Example 1.1. An apparatus, comprising: means for measuring, by an apparatus, a first characteristic of a wireless communication system; means for predicting, by the apparatus, based on an inference in a first model, a success of a first operation of the wireless communication system; means for, based on the inference in the first model and on a configuration parameter indicating a second characteristic is to be measured, measuring, by the apparatus, the second characteristic of the wireless communication; and means for updating, by the apparatus, the first model based on the measuring of the second characteristic of the wireless communication system being performed.

[0097] Example 1.2. The apparatus of example 1.1, wherein the first characteristic is a characteristic of a first communication layer of the wireless communication system.

[0098] Example 1.3. The apparatus of example 1.2, wherein the first operation is a transfer from the first communication layer of the wireless communication system to a second communication layer of the wireless communication system.

[0099] Example 1.4. The apparatus of example 1.3, wherein the second characteristic is a characteristic of the second communication layer of the wireless communication system.

[0100] Example 1.5. The apparatus of example 1.1, wherein measuring the second characteristic of the second layer of the wireless communication is based on the inference of transfer success from the first layer of the wireless communication system to a second layer ofthe wireless communication system being positive.

[0101] Example 1.6. The apparatus of example 1.5, wherein the configuration parameter is a value between 0 and 1.

[0102] Example 1.7. The apparatus of example 1.6, wherein no measuring of the second characteristic of the second layer of the wireless communication is performed for a configuration value of 0.

[0103] Example 1.8. The apparatus of example 1.6, wherein measuring of the second characteristic of the second layer of the wireless communication is performed every time for a configuration value of 1.

[0104] Example 1.9. The apparatus of example 1.6, wherein measuring of the second characteristic of the second layer of the wireless communication is performed a greater number of times for a first value of the configuration value than for a second value of the configuration value, wherein the second value of the configuration value is less than the first value of the configuration value.

[0105] Example 1.10. The apparatus of example 1.1, wherein the second characteristic of the second layer of the wireless communication system is a characteristic related to an ability of the second layer to provide coverage.

[0106] Example l. i l. The apparatus of example 1.1, wherein the configuration parameter is a model monitoring window.

[0107] Example 1.12. The apparatus of example 1.1, wherein the configuration parameter is based on an amount of detected drift or degradation.

[0108] Example 1.13. The apparatus of example 1.1, wherein the updating of the first model is based on the determination to measure the second characteristic of the second layer of the wireless communication system.

[0109] Example 1.14. The apparatus of example 1.1, wherein the configuration parameter includes a first configuration parameter that is a value between 0 and 1, and a second configuration parameter that is a model monitoring window.

[0110] Example 1.15. The apparatus of example 1.1, wherein the second characteristic is not measured based upon the inference in the first model and the configuration parameter indicating a second characteristic is not to be measured.

[0111] Example 1.16. The apparatus of example 1.15, wherein the first model is notupdated.

[0112] Example 1.17. The apparatus of example 1.1, wherein the apparatus is a user equipment (UE).

[0113] Example 1.18. The apparatus of example 1.1, wherein the apparatus is a network node.

[0114] Example 2.1. An apparatus, comprising: at least one processor; and at least one memory storing instructions which, when executed by the at least one processor, cause the apparatus at least to: measure, a first characteristic of a wireless communication system; predict, based on an inference in a first model, a success of a first operation of the wireless communication system; based on the inference in the first model and on a configuration parameter indicating a second characteristic is to be measured, measure the second characteristic of the wireless communication; and update the first model based on the measuring of the second characteristic of the wireless communication system being performed.

[0115] Example 2.2. The apparatus of example 2.1, wherein the apparatus is a user equipment (UE).

[0116] Example 2.3. The apparatus of example 2.1, wherein the apparatus is a network node.

[0117] The embodiments and aspects disclosed herein are examples of the present disclosure and may be embodied in various forms. For instance, although certain embodiments herein are described as separate embodiments, each of the embodiments herein may be combined with one or more of the other embodiments herein. Specific structural and functional details disclosed herein are not to be interpreted as limiting, but as a basis for the claims and as a representative basis for teaching one skilled in the art to variously employ the present disclosure in virtually any appropriately detailed structure. Like reference numerals may refer to similar or identical elements throughout the description of the figures.

[0118] The phrases “in an aspect,” “in aspects,” “in various aspects,” “in some aspects,” or “in other aspects” may each refer to one or more of the same or different aspects in accordance with this present disclosure. The phrase “a plurality of’ may refer to two or more.

[0119] In various embodiments, the terms “first message” and “second message”, as well as any subsequent messages may refer to any messages that are transmitted or received in an order and are not necessarily limited to any particular message.

[0120] The phrases “in an embodiment,” “in embodiments,” “in various embodiments,” “in some embodiments,” or “in other embodiments” may each refer to one or more of the same or different embodiments in accordance with the present disclosure. A phrase in the form “A or B” means “(A), (B), or (A and B).” A phrase in the form “at least one of A, B, or C” means “(A); (B); (C); (A and B); (A and C); (B and C); or (A, B, and C) ”

[0121] Any of the herein described methods, programs, algorithms or codes may be converted to, or expressed in, a programming language or computer program. The terms “programming language” and “computer program,” as used herein, each include any language used to specify instructions to a computer, and include (but is not limited to) the following languages and their derivatives: Assembler, Basic, Batch files, BCPL, C, C+, C++, Delphi, Fortran, Java, JavaScript, machine code, operating system command languages, Pascal, Perl, PL1, Python, scripting languages, Visual Basic, metalanguages which themselves specify programs, and all first, second, third, fourth, fifth, or further generation computer languages. Also included are database and other data schemas, and any other meta- languages. No distinction is made between languages which are interpreted, compiled, or use both compiled and interpreted approaches. No distinction is made between compiled and source versions of a program. Thus, reference to a program, where the programming language could exist in more than one state (such as source, compiled, object, or linked) is a reference to any and all such states. Reference to a program may encompass the actual instructions and / or the intent of those instructions.

[0122] While aspects of the present disclosure have been shown in the drawings, it is not intended that the present disclosure be limited thereto, as it is intended that the present disclosure be as broad in scope as the art will allow and that the specification be read likewise. Therefore, the above description should not be construed as limiting, but merely as exemplifications of particular aspects. Those skilled in the art will envision other modifications within the scope and spirit of the claims appended hereto.

Claims

WHAT IS CLAIMED IS:

1. A method, comprising: measuring, by an apparatus, a first characteristic of a wireless communication system; predicting, by the apparatus, based on an inference in a first model, a success of a first operation of the wireless communication system; based on the inference in the first model and on a configuration parameter indicating a second characteristic is to be measured, measuring, by the apparatus, the second characteristic of the wireless communication; and updating, by the apparatus, the first model based on the measuring of the second characteristic of the wireless communication system being performed.

2. The method of claim 1, wherein the first characteristic is a characteristic of a first communication layer of the wireless communication system.

3. The method of claim 2, wherein the first operation is a transfer from the first communication layer of the wireless communication system to a second communication layer of the wireless communication system.

4. The method of claim 3, wherein the second characteristic is a characteristic of the second communication layer of the wireless communication system.

5. The method of claim 1, wherein measuring the second characteristic of the second layer of the wireless communication is based on the inference of transfer success from the first layer of the wireless communication system to a second layer of the wireless communication system being positive.

6. The method of claim 5, wherein the configuration parameter is a value between 0 and 1.

7. The method of claim 6, wherein no measuring of the second characteristic of the second layer of the wireless communication is performed for a configuration value of 0.

8. The method of claim 6, wherein measuring of the second characteristic of the second layer of the wireless communication is performed every time for a configuration value of 1.

9. The method of claim 6, wherein measuring of the second characteristic of the second layer of the wireless communication is performed a greater number of times for a first value of the configuration value than for a second value of the configuration value, wherein the second value of the configuration value is less than the first value of the configuration value.

10. The method of claim 1 , wherein the second characteristic of the second layer of the wireless communication system is a characteristic related to an ability of the second layer to provide coverage.

11. The method of claim 1, wherein the configuration parameter is a model monitoring window.

12. The method of claim 1, wherein the configuration parameter is based on an amount of detected drift or degradation.

13. The method of claim 1, wherein the updating of the first model is based on the determination to measure the second characteristic of the second layer of the wireless communication system.

14. The method of claim 1, wherein the configuration parameter includes a first configuration parameter that is a value between 0 and 1, and a second configuration parameter that is a model monitoring window.

15. The method of claim 1, wherein the second characteristic is not measured based upon the inference in the first model and the configuration parameter indicating a second characteristic is not to be measured.

16. The method of claim 15, wherein the first model is not updated.

17. The method of claim 1, wherein the apparatus is a user equipment (UE).

18. The method of claim 1 , wherein the apparatus is a network node.

19. A user equipment (UE), comprising: at least one processor; and at least one memory storing instructions which, when executed by the at least one processor, cause the UE at least to perform a method as in any of claims 1-18.

20. An apparatus, comprising: at least one processor; and at least one memory storing instructions which, when executed by the at least one processor, cause the apparatus at least to perform a method as in any of claims 1-18.

21. A processor-readable medium storing instructions which, when executed by at least one processor of an apparatus, cause the apparatus at least to perform a method as in any one of claims 1-18.

Images