Machine learning based beam management monitoring with support of minimization of drive test data in wireless communication networks

Abstract

Various example embodiments relate to devices, methods, apparatuses and computer readable mediums for machine learning based beam management monitoring with support of minimization of drive test data in wireless communication networks A network device comprises means for receiving beam prediction information of at least one beam predicted by a machine learning model, monitoring performance of the machine learning model to obtain a monitored performance, and determining an action based on the monitored performance of the machine learning model. The monitoring performance of the machine learning model comprises comparing the beam prediction information to a database comprising ground truth data of beam prediction, and the database is created based on minimization of drive test, MDT, measurement data collected from a plurality of user equipments.

Description

MACHINE LEARNING BASED BEAM MANAGEMENT MONITORING WITHSUPPORT OF MINIMIZATION OF DRIVE TEST DATA IN WIRELESSCOMMUNICATION NETWORKSTECHNICAL FIELD

[0001] Various example embodiments described herein generally relate to communication technologies, and more particularly, to devices, methods, apparatuses and computer readable mediums for machine learning (ML) based beam management monitoring with support of minimization of drive test (MDT) data in wireless communication networks.BACKGROUND

[0002] Certain abbreviations that may be found in the description and / or in the figures are herewith defined as follows:3 GPP Third Generation Partnership ProjectAl Artificial IntelligenceBM Beam ManagementBS Base StationCN Core NetworkLCM Life Cycle ManagementMDT Minimization of Drive TestNWDAF Network Data Analytics Function0AM Operation, Administration and MaintenanceRAN Radio Access NetworkUAI UE Assistance InformationUE User Equipment

[0003] Communications at high frequencies e.g. millimeter wave (mmWave) utilize beamforming to compensate for severe propagation loss. The base station and user equipments (UEs) align their beams through a set of operations known as beam management (BM) to ensure sufficient beamforming gain. Conventional BM requires a large number of beam measurements and reports to identify and update an optimal transmit-receive beam pair over time, which leads to significant overhead, increased communication delay, and degraded spectral efficiency. To address these issues, AI / ML (Artificial Intelligence / Machine Learning) based solutions have been proposed where AI / ML models / algorithms (e.g. neural network models) are trained for beam prediction in time and / or spatial domain.SUMMARY

[0004] A brief summary of exemplary embodiments is provided below to provide basic understanding of some aspects of various embodiments. It should be noted that this summary is not intended to identify key features of essential elements or define scopes of the embodiments, and its sole purpose is to introduce some concepts in a simplified form as a preamble for a more detailed description provided below.

[0005] In a first aspect, an example embodiment of a network device is provided. The network device may comprise means for receiving beam prediction information of at least one beam predicted by a machine learning model, means for monitoring performance of the machine learning model to obtain a monitored performance, and means for determining an action based on the monitored performance of the machine learning model. The monitoring performance of the machine learning model may comprise comparing the beam prediction information to a database comprising ground truth data of beam prediction. The database may be created based on minimization of drive test, MDT, measurement data collected from a plurality of user equipments.

[0006] In a second aspect, an example embodiment of a user equipment is provided. The user equipment may comprise means for transmitting beam prediction information of at least one beam predicted by a machine learning model to a network device, and means for receiving instructions. The received instructions may comprise at least one of instructions to switch the machine learning model to another machine learning model for beam prediction, instructions to retrain the machine learning model, instructions to perform beam selection without using the machine learning model, or instructions to collect MDT measurement data and transmit the MDT measurement data to the network device.

[0007] Example embodiments of methods, apparatuses and computer readable mediums for machine learning based beam management monitoring are also provided. Such example embodiments generally correspond to the above example embodiments of the network device and the user equipment, and a repetitive description thereof is omitted here for convenience.

[0008] Other features and advantages of the example embodiments of the present disclosure will also be apparent from the following description of specific embodiments when read in conjunction with the accompanying drawings, which illustrate, by way ofexample, the principles of example embodiments of the present disclosure.BRIEF DESCRIPTION OF THE DRAWINGS

[0009] Some example embodiments will now be described, by way of non-limiting examples, with reference to the accompanying drawings.

[0010] Fig 1 is a schematic diagram illustrating a wireless communication network in which example embodiments of the present disclosure may be practiced.

[0011] Fig 2 is a high-level flowchart illustrating a process according to an example embodiment of the present disclosure.

[0012] Fig 3 is a signaling diagram illustrating a process according to an example embodiment of the present disclosure.

[0013] Fig 4 is a signaling diagram illustrating a process according to an example embodiment of the present disclosure.

[0014] Fig 5 is a signaling diagram illustrating a process according to an example embodiment of the present disclosure.

[0015] Fig 6 is a schematic flowchart illustrating a method according to an example embodiment of the present disclosure.

[0016] Fig 7 is a schematic flowchart illustrating a method according to an example embodiment of the present disclosure.

[0017] Fig 8 is a schematic block diagram illustrating devices in a communication system according to an example embodiment of the present disclosure.

[0018] Throughout the drawings, same or similar reference numbers indicate same or similar elements. A repetitive description on the same elements would be omitted.DETAILED DESCRIPTION

[0019] Herein below, some example embodiments are described in detail with reference to the accompanying drawings. The following description includes specific details for the purpose of providing a thorough understanding of various concepts. However, it will be apparent to those skilled in the art that these concepts may be practiced without these specific details. In some instances, well known circuits, techniques and components are shown in block diagram form to avoid obscuring the described concepts and features.

[0020] Fig 1 illustrates, by way of example, a schematic diagram of a wireless communication network 100 in which example embodiments of the present disclosure may be practiced. The wireless communication network 100 may be for example a cellularcommunication network, which includes one or more base stations 120 that form a so-called radio access network (RAN) to provide access to the network for a plurality of user equipments (UEs), such as UEs 110a, 110b. One base station 120 and two UEs 110a, 110b in communication with the base station 120 are shown as an example in Fig. 1. The base station 120 may connect to a core network (CN) 130. The core network 130 can provide secure and reliable connectivity to a data network (not shown) and access to networking services. The core network 130 may include numerous network functions (NFs) (not shown) to support functionalities such as subscriber data management, authorization, authentication policy management, session management, mobility management, etc. In an example embodiment, the core network 130 may be deployed on cloud, e.g. a public cloud, a private cloud or a hybrid cloud.

[0021] The base station 120 may include one or more antenna panels, which each may include a number of dual-polarized antenna elements arrayed in columns and rows. The base station 120 can control amplitudes and phase shifts of transmit signals provided to the antenna elements such that the signals radiated from the antenna elements coherently combine together for a particular transmit direction and destructively cancel each other out for other directions, forming an aggregated signal beam with more power propagating in the particular direction. Similarly, the UE 110 may include a plurality of antenna elements to form a receive signal beam. The base station 120 and the UE 110 may need to align their beams during both initial access and data transmissions to ensure an acceptable communication quality. Toward this goal, 3GPP has introduced a beam management (BM) procedure which includes a set of operations such as beam sweeping, beam measurement, beam selection and beam reporting to achieve precise alignment of the transmitter and receiver beams. The operations are periodically repeated to update the optimal transmitreceive beam pair over time. However, because a large number of beam measurements and reports are required to identify the optimal beam pair, the conventional BM procedure usually suffers from significant overhead, increased communication delay, and degraded spectral efficiency.

[0022] In recent years, AI / ML techniques have attracted great interests, especially for radio resource optimization and “cognitive” network management. 3 GPP has confirmed in Release 18 that BM is one of the AI / ML use cases. AI / ML models / algorithms can automatically learn beam characteristics and distribution from training data through neural networks (NNs), thus achieving high beam prediction accuracy. It can also reduce overhead and latency, especially when the base station and the UE have a larger number of beams,because the AI / ML models / algorithms can make the beam prediction (in time and / or spatial domain) without measuring every transmit beam using every receive beam. The performance of the radio access network (RAN) can be improved with AI / ML usage as long as the ML model is running as expected with the required prediction accuracy.

[0023] On the other hand, the degradation of the ML model would have undesirable effects of wasting radio resources, as well as causing other radio network performance deterioration. For instance, environment change or network configuration adjustment / update may have an impact on the beam distribution and thus lead to the degradation of the ML model performance or accuracy. Therefore, it is desirable to perform ML model performance monitoring to continuously check the ML model performance and early detect any possible important changes / updates / adjustments as part of life cycle management (LCM) of the ML functionality.

[0024] Example embodiments of the present disclosure herein describe techniques for monitoring ML based beam management performance. As a part of beam management, ability of ML model to learn to predict the best beam(s) for a UE may be associated with a propagation path from the base station to the UE. Accordingly, UE position information may be of great help to support the ML model in learning and predicting the best beam(s). In addition, the UE position information will also help the network to perform ML model performance monitoring by comparing current UE position and ML based beam prediction with history information. The history information may comprise, for example, UE positions and selected serving beam (e.g. beam identity (ID)) corresponding to the UE position. The selected beam may be the best beam or an optimal beam for a given UE position. For example, the history information may comprise entries of “UE position, beam ID, cell ID”. In some example embodiments, minimization of drive test (MDT) measurement reports, either logged or immediate, may be used to collect the history information of UE position and selected serving beam. The MDT measurement report can include periodic reporting of UE position and radio measurements of the UE at the position. A mechanism to trigger the MDT data collection for the ML based beam management performance monitoring is also described in some example embodiments.

[0025] Fig 2 is a high-level flowchart illustrating a process 200 for monitoring ML based beam management performance according to an example embodiment of the present disclosure. In general, the process 200 may include a data preparation phase to collect MDT measurements as historic data into a database, and a monitoring phase to evaluate the ML based beam management performance by comparing beam prediction of the ML model withthe database of the historic data. The database may include processed MDT measurements.

[0026] As shown in Fig. 2, the data preparation phase may include a step 210 of collecting MDT measurement data corresponding to a plurality of UEs 110 in a region of interest. The MDT measurement data may include periodic reporting of geographical position of a corresponding UE, which may be acquired from a global positioning system (GPS) receiver implemented at the UE 110, or from a network positioning service received at the UE 110. The MDT measurement data may also include periodic reporting of radio measurements corresponding to the UE position. For instance, the UE 110 may measure reference signal received power (RSRP) of downlink (DL) reference signals and report one or more best beams, one or more cells providing the one or more best beams, and corresponding RSRP measurement values of the one or more best beams. Alternatively or additionally, reference signal received quality (RSRQ), signal to interference plus noise ratio (SINR) or other metrics of the DL reference signals may be measured at the UE 110 to determine the one or more best beams. The step 210 may be performed until sufficient data samples, e.g. enough representative position - beam / cell pairs, are collected for the region of interest. Each sample may include at least UE position coordinates, an identity of one or more best beams associated with the UE position, and a cell identity of each best beam. In an example embodiment, each data sample may further include an RSRP measurement value of each best beam, and a timestamp of the sample. In case of time domain beam prediction monitoring, it would be beneficial to collect sufficient data samples during a predefined time interval. The best beam may refer to a beam that has been selected according to selection criteria, such as RSRP, RSRQ, SINR or other metrics.

[0027] At 212, the collected MDT measurement data may be stored in a database. It would be appreciated that the steps 210, 212 may be performed in parallel with each other, until sufficient data samples are collected and stored in the database. In an example embodiment, the collected MDT measurement data may be pre-processed before being stored in the database. For instance, the collected MDT measurement data may be filtered to ensure that only MDT measurement data which meets requirements for the ML based BM performance monitoring would be stored in the database. An example requirement may include that a link quality metric such as throughput or RSRP is higher than or equal to a predetermined threshold. When sufficient data samples are collected and stored in the database, the process 200 may proceed to the monitoring phase where the database 212 prepared with the MDT measurement data is exploited for the monitoring purpose.

[0028] In the monitoring phase, decisions or beam predictions of a ML model may becollected at 214 and compared with the database to evaluate performance of the ML model at 216. The ML model decision may include a prediction of one or more best beams and corresponding cell identity inferred by the ML model at least based on available beam measurements. The beam measurements may be in spatial or temporal domain. . For instance, for the spatial domain beam prediction, the ML model may receive RSRP values of a measured set of beams as input, and output the identity of the selected / predicted beam. It would be appreciated that the measured set of beams may be a part (not all) of the transmit beams, and the selected / predicted beam may not be included in the measured set of beams (but included in the transmit beams). For the time domain beam prediction, the ML model may further receive time as input. The MDT measurement data stored in the database are used as ground truth data of the beam prediction to evaluate the performance of the ML model. It allows to evaluate if the ML model is performing as expected with the required prediction accuracy. In an example embodiment, some metrics may be defined to evaluate the ML model performance, which will be discussed in detail later. For example, the comparison may be performed, for a given UE position, between predicted beam ID and a beam ID in the database. As another example, the comparison may be performed, in case of temporal beam prediction, using a vector of consecutive selected beams without a need to use UE position.

[0029] Fig 3 illustrates a process 300 according to an example embodiment of the present disclosure. The process 300 includes operations performed at the UEs 110, the base station 120, an operation, administration and management (0AM) function 132 and a network data analytics function (NWDAF) 134 as described below, and the operations may be performed in the data preparation phase of the process 200 shown in Fig. 2. The UEs 110, the base station 120, the 0AM 132 and the NWDAF 134 each may include a plurality of components, circuitries, modules, elements or means to perform the operations, which may be implemented in various manners including but not limited to for example software, hardware, firmware or any combination thereof.

[0030] Referring to Fig. 3, at 310, the 0AM 132 may subscribe to an ML based BM monitoring service with the NWDAF 134. NWDAF is a function standardized in 3GPP SA2 (Service and System Aspect Work Group 2) responsible to collect data from other network functions (NFs) as well as from 0AM and provide analytics to the NFs as well as the 0AM. It may be embodied as a separate NF in the core network 130, or it may be collocated with another NF. For instance, the NWDAF 134 may be implemented at an RAN node or a core network entity. In an example embodiment, the 0 AM 132 may use aNnwdaf AnalyticsSubscription Subscribe signaling to subscribe to the ML based BM monitoring service, and the NWDAF 134 may assign a unique service identity (ID) to the subscribed ML based BM monitoring service.

[0031] At 312, the 0AM 132 or the NWDAF 134 may transmit to one or more base stations 120 an indication to trigger the ML based BM monitoring service. The one or more base stations 120 may serve a region of interest where ML based BM is enabled. In an example embodiment, the indication may also include the service ID of the ML based BM monitoring service.

[0032] In response to the indication to trigger the ML based BM monitoring service, the base station 120 may request UEs 110 connected to the base station 120 to report UE capabilities available for the ML based BM monitoring service at 314, and the UEs 110 may transmit a capability report indicating the UE capabilities available for the ML based BM monitoring service to the base station 120 at 316. The UE capability report may indicate for example whether the UE is able to support the ML based BM monitoring service. If a UE runs an ML based beam management procedure, it would be able to support the ML based BM monitoring service. Then the UE capability report may further indicate UE capabilities that are currently relevant to the ML based BM monitoring service, for example beam management specificities (time and / or spatial domain) supported at the UE, the number of best beams that the UE can measure and report to the network, beam quality metrics e.g. RSRP, RSRQ, or SINR measured at the UE, measurement period, positioning capability and accuracy, and the like.

[0033] In an example embodiment, the base station 120 may aggregate the UE capabilities received from different UEs 110 at 318. For instance, the base station 120 may divide the UEs 110 into groups based on UE capabilities. The UEs which supports the ML based BM monitoring service may be classified into one or more groups, and each group may include one or more UEs that have similar capabilities relevant to the ML based BM monitoring service. The UEs which do not support the ML based BM monitoring service may be omitted or classified into a separate group. Then the base station 120 may transmit the UE capabilities to the 0AM 132 or the NWDAF 134 at 320.

[0034] The 0AM 132 or the NWDAF 134 may analyze the received UE capabilities and determine a minimization of drive test (MDT) configuration for the ML based BM monitoring service at 322. In an example embodiment, the 0AM 132 or the NWDAF 134 may determine the MDT configuration per UE group with different UE capabilities (e.g. with different measurement duration). The MDT configuration may specify MDTmeasurements for the ML based BM monitoring service. For instance, for the spatial domain ML based BM procedure, the MDT may collect not only RSRP (or other beam quality metrics) of the best beam, but also one or more (or a predetermined number of) neighboring / candidate beams with RSRP higher than a predetermined threshold. For the time domain ML based BM procedure, the MDT may collect measurements during a predefined interval. The goal is to ensure collection of consecutive RSRP measurements at the UE where the best beam may potentially change over time. The MDT configuration may also specify the beam quality metrics to be measured, the measurement period, the positioning accuracy requirement, etc. In an example embodiment, the OAM 132 or the NWDAF 134 may determine the MDT configuration for both time domain and spatial domain monitoring. If the MDT configuration is determined at the OAM 132, the OAM 132 may transmit the determined MDT configuration to the NWDAF 134. If the MDT configuration is determined at the NWDAF 134 the NWDAF 134 may transmit the determined MDT configuration to the OAM 132.

[0035] When the MDT configuration is determined, MDT measurements may be collected following the MDT configuration. Here two options for the MDT measurement collection are provided. In Option A, the MDT measurements may be collected with assistance of the OAM 132, while in Option B, the MDT measurements may be collected without assistance of the OAM 132.

[0036] With continued reference to Fig. 3, in Option A, the OAM 132 may transmit an MDT measurement request to the one or more base stations 120 serving the region of interest at 324. The MDT measurement request may include the MDT configuration for the ML based BM monitoring service. The base station 120 may forward the MDT measurement request to the UEs 110 connected to the base station 120 at 326. Based on the MDT measurement request (i.e. the MDT configuration), the UEs 110 may perform MDT measurements and transmit the MDT measurement data to the base station 120 at 328. The MDT measurement data may include those specified in the MDT configuration. For instance, the MDT measurement data may include at least UE position coordinates, one or more best beam identities and corresponding cell identity(ies). In an example, the MDT measurement data may further include quality metrics e.g. RSRP, RSRQ, SINR of each beam and a timestamp indicating the time when the MDT measurement data is acquired. The base station 120 may send the collected MDT measurement data to the OAM 132 at 330.

[0037] Upon receiving the MDT measurement data, the OAM 132 may pre-process / filter the MDT measurement data to ensure that the MDT measurement data is compliant with theMDT configuration for the ML based BM monitoring service at 332. The OAM 132 may discard the MDT measurement data incompliant with the MDT configuration, and transmit the MDT measurement data compliant with the MDT configuration to the NWDAF 134 at 334. In an example embodiment, the base station 120 may pre-process / filter the received MDT measurement data before transmitting the MDT measurement data to the OAM 132, and the step 332 may be omitted.

[0038] In Option B, the NWDAF 134 may send the MDT measurement request to one or more base stations 120 at 336, and then the base station 120 may send the MDT measurement request to the UEs 110 connected to the base station 120 at 338. In response to the MDT measurement request, the UEs 110 may perform MDT measurements and transmit the MDT measurement data to the base station 120 at 340. Then the base station 120 may send the collected MDT measurement data to the NWDAF 134 at 342. The NWDAF 134 may pre- process / filter the MDT measurement data to ensure that the MDT measurement data is compliant with the MDT configuration for the ML based BM monitoring service at 344. In an example embodiment, the base station 120 may pre-process / filter the received MDT measurement data before transmitting the MDT measurement data to the NWDAF 134, and the step 344 may be omitted.

[0039] When the NWDAF 134 receives the MDT measurement data of the UEs 110 from the OAM 132 or the one or more base stations 120 and optionally pre-processes / filters the MDT measurement data, the NWDAF 134 may store the MDT measurement data in a database. The MDT measurement data stored in the database may be used as ground truth data to evaluate performance of an ML model for beam prediction, which will be described below.

[0040] Fig 4 illustrates a process 400 according to an example embodiment of the present disclosure. The process 400 includes operations performed at the UEs 110, the base station 120, the OAM 132 and the NWDAF 134 as described below, and the operations may be performed in the monitoring phase of the process 200 shown in Fig. 2. The UEs 110, the base station 120, the OAM 132 and the NWDAF 134 each may include a plurality of components, circuitries, modules, elements or means to perform the operations, which may be implemented in various manners including but not limited to for example software, hardware, firmware or any combination thereof.

[0041] Referring to Fig. 4, at 410, the UE 110 may transmit beam prediction information of one or more beams predicted by an ML model to the NWDAF 134. The ML model may be deployed at the UE 110 to predict one or more best beams for the UE 110 based on e.g.position or other measurements measured at the UE 110. The beam prediction information may include e.g. UE position, identities of the predicted beam(s) / cell(s) associated with the UE position, a timestamp, and optionally additional information. In another example embodiment, the ML model may be deployed at a network entity e.g. the base station 120, and it makes the beam prediction for the UE 110 based on UE measurements received from the UE 110. The UE 110 may receive the beam prediction information from the network entity and then transmit the beam prediction information to the NWDAF 134. In an example embodiment, the NWDAF 134 may receive the beam prediction information from the network entity running the ML model. In case the ML model is deployed at the same network entity as the NWDAF 134, the NWDAF 134 may receive the beam prediction information from the memory / storage of the network entity.

[0042] With the received beam prediction information, the NWDAF 134 may monitor performance of the ML model at 412. For instance, the NWDAF 134 may compare the beam prediction information with the database comprising the MDT measurement data to determine performance of the ML model. The MDT measurement data stored in the database are used as ground truth data of the beam prediction to evaluate the performance of the ML model. For instance, the predicted beam ID in the beam prediction information may be compared with the beam ID in the database associated with identical or similar UE position and / or timestamp, e.g. within ± threshold meters / milliseconds. Some metrics may be defined to quantize the comparison result. For example, the comparison result may be percentage of correct predictions, portion of time the predictions are correct, or some other quantities suitable for the purpose of the ML model performance evaluation. At 414, the NWDAF 134 may transmit the monitored performance of the ML model to the 0AM 132 if needed.

[0043] In an example embodiment, the monitoring step 412 may be performed at the 0AM 132 instead. For instance, the 0AM 132 may receive the beam prediction information from the UEs 110 (e.g. via one or more base stations 120) and retrieve the database from the NWDAF 134 since the 0AM 132 can already have direct access to the MDT measurements. Then the 0AM 132 may compare the beam prediction information with the database to evaluate the performance of the ML model. The 0AM 132 may also transmit the monitored performance of the ML model to the NWDAF 134 if needed. It would be appreciated that the monitoring step 412 may also be perform at another network entity e.g. the base station 120 or other network functions in the core network, as long as the network entity can create or have access to the above-mentioned database and receive the beam prediction information.

[0044] At 416, the 0AM 132 or the NWDAF 134 may determine an action based on themonitored performance of the ML model. If the monitored performance of the ML model is higher than or equal to a predetermined threshold, the OAM 132 or the NWDAF 134 may determine that the ML model is running as expected with the required prediction accuracy, and no action is needed. If the monitored performance of the ML model is lower than the threshold, the OAM 132 or the NWDAF 134 may decide to retrain the ML model or switch the ML model to another ML model for beam prediction. The OAM 132 or the NWDAF 134 may instruct to retrain the ML model or instruct to switch the ML model to another ML model, if the ML model is running in another entity, such as in the UE 110. Alternatively, the OAM 132 or the NWDAF 134 may instruct the UE 110 and optionally the base station 120 to perform legacy beam selection without using the ML model. If the UE position and / or timestamp in the beam prediction information received from a threshold number of UEs 110 does not have a match in the MDT measurement data stored in the database, the OAM 132 or the NWDAF 134 may decide to collect additional MDT measurement data to enrich the database. For example, the OAM 132 or the NWDAF 134 may instruct the UE 110 to collect MDT measurement data and transmit the collected data back to the network entity (OAM or NWDAF) for storing into the database. The additional MDT measurement data may be collected according to the process 300 discussed above with respect to Fig. 3, and a detailed description thereof is omitted here for convenience.

[0045] It would be appreciated that the ML based BM performance monitoring is dependent on environment conditions. If the conditions change, the beam distribution may also change, and the database prepared before the condition change may not be applicable for the ML model performance monitoring any longer. To address this issue, a process 500 illustrated in Fig. 5 may be performed to detect the condition change. The process 500 includes operations performed at the UEs 110, the base station 120, the OAM 132 and the NWDAF 134 as described below. The UEs 110, the base station 120, the OAM 132 and the NWDAF 134 each may include a plurality of components, circuitries, modules, elements or means to perform the operations, which may be implemented in various manners including but not limited to for example software, hardware, firmware or any combination thereof.

[0046] Referring to Fig. 5, the process 500 may be initiated at the network side or at the UEs 110. In case that the process 500 is initiated at the network side, the OAM 132 or the NWDAF 134 may identify a condition change relevant for monitoring the performance of the ML model at 510. For instance, obstacles in the propagation path between the UE 110 and the base station 120 may decrease the RSRP measured at the UE 110 and thus have an impact on the beam selection at the UE 110. Hence the OAM 132 or the NWDAF 134 mayidentify the RSRP change relevant for the ML model performance monitoring. Here the RSRP change is described as an example, but other condition changes such as RSRQ change, SINR change may also be identified as being relevant for the ML model performance monitoring. It would be appreciated that at the time the OAM 132 or the NWDAF 134 identifies the condition change relevant for the ML model performance monitoring at 510, the identified condition change may not have happened. In other words, the condition is in a normal range, but the OAM 132 or the NWDAF 134 may identify that change of the condition is relevant for the ML model performance monitoring.

[0047] The OAM 132 or the NWDAF 134 may transmit information of the identified condition change to one or more base stations 120 at 512, and each base station 120 may forward the information of the condition change to the UEs 110 connected to the base station 120 at 514. Based on the received information, the UEs 110 can verify whether the condition change has happened or not. For instance, the information of the identified condition change may comprise a predetermined RSRP value range. If the RSRP value measured at the UE 110 falls within the predetermined range, the UE 110 may determine that the condition change has not yet happened. If the RSRP value measured at the UE 110 falls outside the predetermined range, the UE 110 may determine that the condition change has happened. Then the UE 110 may transmit a verification indicating whether the condition change has happened to the base station 120 at 516. For instance, the UE 110 may transmit the verification in UE assistance information (UAI) to the base station 120. In another example, the verification may comprise an acknowledgement (ACK) / non-acknowledgement (NACK) type indication transmitted on physical uplink control channel (PUCCH) or physical uplink shared channel (PUSCH). The base station 120 may send the verifications received from the UEs 110 to the OAM 132 or the NWDAF 134 at 518.

[0048] In another example embodiment where the process 500 is initiated at UE, the UEs 110 may detect a condition change based on its own implementation at 520. For instance, if the UE 110 identifies a condition change possibly relevant for the ML model performance monitoring and the condition value measured at the UE goes out of a normal range preconfigured or predetermined at the UE 110, the UE 110 may determine that the condition change is detected. Then the UE 110 may send an indication of the detected condition change to the base station 120 at 522, and the base station 120 may forward the indication of the condition change to the OAM 132 or the NWDAF 134 at 524. The indication of the condition change may be transmitted in UE assistance information (UAI) and it may include the condition value measured at the UE 110. At 526, the OAM 132 or the NWDAF 134 mayevaluate whether the condition change is relevant to the monitoring of the ML model performance.

[0049] If it is verified or evaluated that the condition change is relevant for the monitoring of the ML model performance, the 0AM 132 or the NWDAF 134 may take actions in response to the condition change at 528. For instance, the 0AM 132 or the NWDAF 134 may stop monitoring with the current database and initiate a process to update the database or establish a new database with up-to-date MDT measurement data. If the condition change goes beyond the environment where the ML model is trained for beam prediction, the 0AM 132 or the NWDAF 134 may decide to retrain the ML model, switch the ML model to another ML model, or fallback to the legacy beam selection without using any ML model. If it is verified or evaluated that the condition change is irrelevant for the monitoring of the ML model performance, the 0AM 132 or the NWDAF 134 may take no action but keep monitoring the ML model performance as before.

[0050] In the process 500, the network or the UEs can timely identify and detect condition changes that are relevant for the monitoring of the ML model performance and take necessary actions if needed. The process 500 may be performed in parallel with the process 200 shown in Fig. 2 to ensure a robust and resilient scheme for the ML model performance monitoring.

[0051] Fig 6 illustrate a method 600 performed at a network device 140 according to an example embodiment of the present disclosure. The network device 140 may be implemented as a RAN node or a CN network function. For instance, the network device 140 may be implemented as the base station 120, the 0AM 132, or the NWDAF 134. The network device 140 may include a plurality of components, circuitries, modules, elements or means to perform respective steps of the method 600. The components, circuitries, modules, elements or means each may be implemented in various manners including but not limited to for example software, hardware, firmware or any combination thereof. Since the operations of the base station 120, the 0AM 132 and the NWDAF 134 have been described above with reference to Figs. 1-5, the method 600 will be described briefly here and details thereof may refer to the above description.

[0052] Referring to Fig. 6, the method 600 may include a step 610 of receiving beam prediction information of one or more beams predicted by a ML model, a step 620 of monitoring performance of the ML model, and a step 630 of determining an action based on the monitored performance of the ML model.

[0053] The ML model for beam prediction may be deployed at the UE 110, or at a networkentity associated with the UE 110. For instance, the ML model may be deployed at a RAN node e.g. the base station 120 serving the UE 110 or a CN network function associated with the UE 110. The network device 140 may receive the beam prediction information from the UE 110 or from the network entity. If the ML model is deployed at the network device 140, the network device 140 may receive the beam prediction information from its own memory.

[0054] The network device 140 may monitor the performance of the ML model by comparing the received beam prediction information to a database which comprises ground truth data of beam prediction. The database may be created based on MDT measurement data collected from the UEs 110, and the MDT measurement data may indicate one or more optimal beams measured at the UEs 110. The network device 140 may compare the predicted optimal beam(s) with the measured optimal beam(s) with position and / or time approximation to monitor the performance of the ML model. In another example embodiment, the comparison may be performed without using the position and / or time information. For instance, in case of temporal beam prediction, the comparison may be performed using a vector of consecutive selected beams.

[0055] Based on the monitored performance of the ML model, the network device 140 may take necessary actions. For instance, if the monitored performance of the ML model is below a threshold, the network device 140 may take one or more actions of switching or instructing a device running the ML model to switch the ML model to another ML model for beam prediction, triggering retraining of the ML model, performing or instructing a device performing ML based beam prediction / sel ection to perform beam selection without using the ML model, or collecting additional MDT measurement data to enrich the database.

[0056] In an example embodiment, the method 600 may further include a step (not shown) of identifying a condition change in the network of the network device 140, which is relevant for monitoring the performance of the ML model, and a step (not shown) of indicating the identified condition change to one or more UEs 110 for the UEs 110 to verify the condition change. Then a verification of the condition change may be received from the UEs 110.

[0057] In an example embodiment, the method 600 may further include a step (not shown) of receiving information on a condition change from one or more of UEs 110, and a step (not shown) of evaluating relevance of the condition change to the monitoring of the ML model performance.

[0058] The condition change may be based on at least one of RSRP value change, RSRQ value change or SINR value change. If the condition change is verified or evaluated as being relevant to the monitoring of the ML model performance, the network device 140 may takenecessary actions e.g. stopping monitoring with the current database and triggering a process to update the database or establish a new database with up-to-date MDT measurement data.

[0059] The method 600 may further include steps performed at the base station 120, the 0AM 132 ortheNWDAF 134 as discussed above with reference to Figs. 1-5, and a repetitive description thereof is omitted here for convenience.

[0060] Fig. 7 illustrate a method 700 performed at the UE 110 according to an example embodiment of the present disclosure. The UE 110 may include a plurality of components, circuitries, modules, elements or means to perform respective steps of the method 700. The components, circuitries, modules, elements or means each may be implemented in various manners including but not limited to for example software, hardware, firmware or any combination thereof. Since the operations of the UE 110 have been described above with reference to Figs. 1-5, the method 700 will be described briefly here and details thereof may refer to the above description.

[0061] Referring to Fig. 7, the method 700 may include a step 710 of transmitting beam prediction information of at least one beam predicted by a ML model to a network device, and a step 720 of receiving instructions.

[0062] The UE 110 may transmit the beam prediction information to the network device which operates to monitor performance of the ML model that makes the beam prediction for the UE 110. The instructions received at the UE 110 may be transmitted from the network device which determines the instructions based on the monitored performance of the ML model, or from another network entity which receives the monitored performance from the network device and determines the instructions based on the monitored performance. The instructions received at the UE 110 may include at least one of instructions to switch the ML model to another ML model for beam prediction, instruction to retrain the ML model, instructions to perform beam selection without using the ML model, or instructions to collect MDT measurement data and transmit the MDT measurement data to the network device.

[0063] In an example embodiment, the method 700 may further include a step (not shown) of detecting a condition change in the network of the UE 110 and the network device, and a step (not shown) of transmitting information of the condition change to the network device. The condition change may be based on at least one of RSRP value change, RSRQ value change, SINR value change. The network device may identify whether the condition change is relevant to the monitoring of the ML model performance.

[0064] In an example embodiment, the method 700 may further include a step (not shown) of receiving from the network device an indication of a condition change relevant to themonitoring of the ML model performance, and a step (not shown) of transmitting a verification of the condition change to the network device. The verification may indicate whether the condition change has happened.

[0065] The method 700 may further include steps performed at the UEs 110 as discussed above with reference to Figs. 1-5, and a repetitive description thereof is omitted here for convenience.

[0066] Fig. 8 is a schematic block diagram illustrating devices in a communication system 800 according to an example embodiment of the present disclosure. As shown in Fig. 8, the communication system 800 may include one or more terminal devices 810, one or more radio access network (RAN) devices 820, and one or more core network (CN) devices 830. In an example embodiment, the terminal device 810 may be implemented as the UE 110 discussed above, the RAN device 820 may be implemented as the base station 120 discussed above, and the CN device 830 may be implemented as the 0AM 132 and / or the NWDAF 134 discussed above. In another example embodiment, the NWDAF 134 may be embodied at the RAN device 820.

[0067] Referring to Fig. 8, the terminal device 810 may comprise one or more processors 811, one or more memories 812 and one or more transceivers 813 interconnected through one or more buses 814. The one or more buses 814 may be address, data, or control buses, and may include any interconnection mechanism such as series of lines on a motherboard or integrated circuit, fiber, optics or other optical communication equipment, and the like. Each of the one or more transceivers 813 may comprise a receiver and a transmitter, which are connected to one or more antennas 816. The terminal device 810 may wirelessly communicate with the RAN device 820 through the one or more antennas 816. The one or more memories 812 may include instructions 815 which, when executed by the one or more processors 811, may cause the terminal device 810 to perform operations and procedures relating to the UE 110 as described above.

[0068] The RAN device 820 may comprise one or more processors 821, one or more memories 822, one or more transceivers 823 and one or more network interfaces 827 interconnected through one or more buses 824. The one or more buses 824 may be address, data, or control buses, and may include any interconnection mechanism such as a series of lines on a motherboard or integrated circuit, fiber, optics or other optical communication equipment, and the like. Each of the one or more transceivers 823 may comprise a receiver and a transmitter, which are connected to one or more antennas 826. The RAN device 820 may operate as a base station for the terminal device 810 and wirelessly communicate withterminal device 810 through the one or more antennas 826. The one or more network interfaces 827 may provide wired or wireless communication links through which the RAN device 820 may communicate with other network devices, entities, elements or functions. For example, the RAN device 820 may communicate with the CN device 830 via backhaul connections 828. The one or more memories 822 may include instructions 825 which, when executed by the one or more processors 821, may cause the RAN device 820 to perform operations and procedures relating to the base station 120 and / or the NWDAF 134 as described above.

[0069] The CN device 830 may comprise one or more processors 831, one or more memories 832, and one or more network interfaces 837 interconnected through one or more buses 834. The one or more buses 834 may be address, data, or control buses, and may include any interconnection mechanism such as a series of lines on a motherboard or integrated circuit, fiber, optics or other optical communication equipment, and the like. The CN device 830 may operate as a core network function node and wired or wirelessly communicate with the RAN device 820 through one or more links. The one or more network interfaces 837 may provide wired or wireless communication links through which the CN device 830 may communicate with other network devices, entities, elements or functions. The one or more memories 832 may include instructions 835 which, when executed by the one or more processors 831, may cause the CN device 830 to perform operations and procedures relating to the 0AM 132 and / or the NWDAF 134 as described above.

[0070] The one or more processors 811, 821 and 831 discussed above may be of any appropriate type that is suitable for the local technical network, and may include one or more of general purpose processors, special purpose processor, microprocessors, a digital signal processor (DSP), one or more processors in a processor based multi-core processor architecture, as well as dedicated processors such as those developed based on Field Programmable Gate Array (FPGA) and Application Specific Integrated Circuit (ASIC). The one or more processors 811, 821 and 831 may be configured to control other elements of the terminal / RAN / core network devices and operate in cooperation with them to implement the procedures discussed above.

[0071] The one or more memories 812, 822 and 832 may include at least one storage medium in various forms, such as a transitory memory and / or a non-transitory memory. The transitory memory may include, but not limited to, for example, a random access memory (RAM) or a cache. The non-transitory memory may include, but not limited to, for example, a read only memory (ROM), a hard disk, a flash memory, and the like. The term “non-transitory,” as used herein, is a limitation of the medium itself (i.e., tangible, not a signal) as opposed to a limitation on data storage persistency (e.g., RAM vs. ROM). Further, the one or more memories 812, 822 and 832 may include but not limited to an electric, a magnetic, an optical, an electromagnetic, an infrared, or a semiconductor system, apparatus, or device or any combination of the above.

[0072] It would be understood that blocks in the drawings may be implemented in various manners, including software, hardware, firmware, or any combination thereof. In some embodiments, one or more blocks may be implemented using software and / or firmware, for example, machine-executable instructions stored in the storage medium. In addition to or instead of machine-executable instructions, parts or all of the blocks in the drawings may be implemented, at least in part, by one or more hardware logic components. For example, and without limitation, illustrative types of hardware logic components that can be used include Field-Programmable Gate Arrays (FPGAs), Application-Specific Integrated Circuits (ASICs), Application-Specific Standard Products (ASSPs), System-on-Chip systems (SOCs), Complex Programmable Logic Devices (CPLDs), etc.

[0073] Some exemplary embodiments further provide program instruction or instructions which, when executed by one or more processors, may cause a device or apparatus to perform the procedures described above. The program instruction for carrying out procedures of the exemplary embodiments may be written in any combination of one or more programming languages. The program instruction may be provided to one or more processors or controllers of a general purpose computer, special purpose computer, or other programmable data processing apparatus, such that the program instruction, when executed by the processor or controller, cause the functions / operations specified in the flowcharts and / or block diagrams to be implemented. The program instruction may execute entirely on a machine, partly on the machine, as a stand-alone software package, partly on the machine and partly on a remote machine or entirely on the remote machine or server.

[0074] Some exemplary embodiments further provide a computer program product or a computer readable medium having the program instruction or instructions stored therein. The computer readable medium may be any tangible medium that may contain, or store a program for use by or in connection with an instruction execution system, apparatus, or device. The machine readable medium may be a machine readable signal medium or a machine readable storage medium. A machine readable medium may include but is not limited to an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any suitable combination of the foregoing. More specificexamples of the machine readable storage medium would include an electrical connection having one or more wires, a portable computer diskette, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or Flash memory), an optical fiber, a portable compact disc read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

[0075] The following are additional examples.

[0076] Example 1. A method, comprising: receiving beam prediction information of at least one beam predicted by a machine learning model; monitoring performance of the machine learning model to obtain a monitored performance, wherein the monitoring performance of the machine learning model comprises comparing the beam prediction information to a database comprising ground truth data of beam prediction, wherein the database is created based on minimization of drive test, MDT, measurement data collected from a plurality of user equipments; and determining an action based on the monitored performance of the machine learning model.

[0077] Example 2. The method of example 1, wherein, in a case where the monitored performance is below a threshold, the action comprises at least one of: switching or instructing to switch the machine learning model to another machine learning model for beam prediction; retraining or instructing to retrain the machine learning model for beam prediction; performing or providing instructions to perform beam selection without using the machine learning model; or collecting and / or instructing to collect additional MDT measurement data to enrich the database comprising ground truth data.

[0078] Example 3. The method of example 1 or 2, further comprising receiving information on a condition change from a user equipment.

[0079] Example 4. The method of example 3, further comprising evaluating relevance of the condition change to the monitoring of performance of the machine learning model.

[0080] Example 5. The method of example 1 or 2, further comprising: identifying a condition change in a network of a network device performing the method, wherein the condition change is relevant for monitoring performance of the machine learning model; and indicating the identified condition change to a user equipment for the user equipment toverify the condition change.

[0081] Example 6. The method of example 5, further comprising: receiving a verification of the condition change from the user equipment.

[0082] Example 7. The method of any of examples 3 to 6, wherein the condition change is based on at least one of: reference signal received power, RSRP, value change, signal to interference and noise ratio, SINK value change, or reference signal received quality, RSRQ, value change.

[0083] Example 8. A method, comprising: transmitting, to a network device, beam prediction information of at least one beam predicted by a machine learning model; and receiving instructions comprising at least one of: instructions to switch the machine learning model to another machine learning model for beam prediction; instructions to retrain the machine learning model; instructions to perform beam selection without using the machine learning model; or instructions to collect MDT measurement data and transmit the MDT measurement data to the network device.

[0084] Example 9. The method of example 8, further comprising: detecting a condition change in a network of the network device; and transmitting, to the network device, information on the condition change.

[0085] Example 10. The method of example 8, further comprising: receiving, from the network device, an indication of a condition change; and transmitting, to the network device, a verification of the condition change.

[0086] Example 11. The method of example 9 or 10, wherein the condition change is based on at least one of: reference signal received power, RSRP, value change, signal to interference and noise ratio, SINR value change, or reference signal received quality, RSRQ, value change.

[0087] Example 12. An apparatus, comprising: at least one processor; and at least one memory storing instructions that, when executed by the at least one processor, cause the apparatus at least to perform: receiving beam prediction information of at least one beam predicted by a machinelearning model; monitoring performance of the machine learning model to obtain a monitored performance, wherein the monitoring performance of the machine learning model comprises comparing the beam prediction information to a database comprising ground truth data of beam prediction, wherein the database is created based on minimization of drive test, MDT, measurement data collected from a plurality of user equipments; and determining an action based on the monitored performance of the machine learning model.

[0088] Example 13. The apparatus of example 12, wherein, in a case where the monitored performance is below a threshold, the action comprises at least one of: switching or instructing to switch the machine learning model to another machine learning model for beam prediction; retraining or instructing to retrain the machine learning model for beam prediction; performing or providing instructions to perform beam selection without using the machine learning model; or collecting and / or instructing to collect additional MDT measurement data to enrich the database comprising ground truth data.

[0089] Example 14. The apparatus of example 12 or 13, wherein the at least one memory stores instructions that, when executed by the at least one processor, cause the apparatus at least to perform: receiving information on a condition change from a user equipment.

[0090] Example 15. The apparatus of example 14, wherein the at least one memory stores instructions that, when executed by the at least one processor, cause the apparatus at least to perform: evaluating relevance of the condition change to the monitoring of performance of the machine learning model.

[0091] Example 16. The apparatus of example 12 or 13, wherein the at least one memory stores instructions that, when executed by the at least one processor, cause the apparatus at least to perform: identifying a condition change in a network of the apparatus, wherein the condition change is relevant for monitoring performance of the machine learning model; and indicating the identified condition change to a user equipment for the user equipment to verify the condition change.

[0092] Example 17. The apparatus of example 16, wherein the at least one memory stores instructions that, when executed by the at least one processor, cause the apparatus at least to perform: receiving a verification of the condition change from the user equipment.

[0093] Example 18. The apparatus of any of examples 14 to 17, wherein the condition change is based on at least one of: reference signal received power, RSRP, value change, signal to interference and noise ratio, SINK value change, or reference signal received quality, RSRQ, value change.

[0094] Example 19. An apparatus, comprising: at least one processor; and at least one memory storing instructions that, when executed by the at least one processor, cause the apparatus at least to perform: transmitting, to a network device, beam prediction information of at least one beam predicted by a machine learning model; and receiving instructions comprising at least one of: instructions to switch the machine learning model to another machine learning model for beam prediction; instructions to retrain the machine learning model; instructions to perform beam selection without using the machine learning model; or instructions to collect MDT measurement data and transmit the MDT measurement data to the network device.

[0095] Example 20. The apparatus of example 19, wherein the at least one memory stores instructions that, when executed by the at least one processor, cause the apparatus at least to perform: detecting a condition change in a network of the network device; and transmitting, to the network device, information on the condition change.

[0096] Example 21. The apparatus of example 19, wherein the at least one memory stores instructions that, when executed by the at least one processor, cause the apparatus at least to perform: receiving, from the network device, an indication of a condition change; and transmitting, to the network device, a verification of the condition change.

[0097] Example 22. The apparatus of example 20 or 21, wherein the condition change isbased on at least one of: reference signal received power, RSRP, value change, signal to interference and noise ratio, SINK value change, or reference signal received quality, RSRQ, value change.

[0098] Example 23. A computer readable medium comprising program instructions that, when executed by an apparatus, cause the apparatus to at least perform: receiving beam prediction information of at least one beam predicted by a machine learning model; monitoring performance of the machine learning model to obtain a monitored performance, wherein the monitoring performance of the machine learning model comprises comparing the beam prediction information to a database comprising ground truth data of beam prediction, wherein the database is created based on minimization of drive test, MDT, measurement data collected from a plurality of user equipments; and determining an action based on the monitored performance of the machine learning model.

[0099] Example 24. The computer readable medium of example 23, wherein, in a case where the monitored performance is below a threshold, the action comprises at least one of: switching or instructing to switch the machine learning model to another machine learning model for beam prediction; retraining or instructing to retrain the machine learning model for beam prediction; performing or providing instructions to perform beam selection without using the machine learning model; or collecting and / or instructing to collect additional MDT measurement data to enrich the database comprising ground truth data.

[0100] Example 25. The computer readable medium of example 23 or 24, further comprising program instructions that, when executed by an apparatus, cause the apparatus to at least perform: receiving information on a condition change from a user equipment.

[0101] Example 26. The computer readable medium of example 25, further comprising program instructions that, when executed by an apparatus, cause the apparatus to at least perform: evaluating relevance of the condition change to the monitoring of performance of the machine learning model.

[0102] Example 27. The computer readable medium of example 23 or 24, further comprising program instructions that, when executed by an apparatus, cause the apparatusto at least perform: identifying a condition change in a network of the apparatus, wherein the condition change is relevant for monitoring performance of the machine learning model; and indicating the identified condition change to a user equipment for the user equipment to verify the condition change.

[0103] Example 28. The computer readable medium of example 27, further comprising program instructions that, when executed by an apparatus, cause the apparatus to at least perform: receiving a verification of the condition change from the user equipment.

[0104] Example 29. The computer readable medium of any of examples 25 to 28, wherein the condition change is based on at least one of: reference signal received power, RSRP, value change, signal to interference and noise ratio, SINK value change, or reference signal received quality, RSRQ, value change.

[0105] Example 30. A computer readable medium comprising program instructions that, when executed by an apparatus, cause the apparatus to at least perform: transmitting, to a network device, beam prediction information of at least one beam predicted by a machine learning model; and receiving instructions comprising at least one of: instructions to switch the machine learning model to another machine learning model for beam prediction; instructions to retrain the machine learning model; instructions to perform beam selection without using the machine learning model; or instructions to collect MDT measurement data and transmit the MDT measurement data to the network device.

[0106] Example 31. The computer readable medium of example 30, further comprising program instructions that, when executed by an apparatus, cause the apparatus to at least perform: detecting a condition change in a network of the network device; and transmitting, to the network device, information on the condition change.

[0107] Example 32. The computer readable medium of example 30, further comprising program instructions that, when executed by an apparatus, cause the apparatus to at least perform: receiving, from the network device, an indication of a condition change; andtransmitting, to the network device, a verification of the condition change.

[0108] Example 33. The computer readable medium of example 31 or 32, wherein the condition change is based on at least one of reference signal received power, RSRP, value change, signal to interference and noise ratio, SINK value change, or reference signal received quality, RSRQ, value change.

[0109] As used in the application, the term “circuitry” may refer to one or more or all of the following:(a) hardware-only circuit implementations (such as implementations in only analog and / or digital circuitry);(b) combinations of hardware circuits and software, such as (as applicable): (i) a combination of analog and / or digital hardware circuit(s) with software / firmware, and (ii) any portions of hardware processor(s) with software (including digital signal processor(s)), software, and memory(ies) that work together to cause an apparatus, such as a mobile phone or server, to perform various functions; and(c) hardware circuit(s) and / or processor(s), such as a microprocessor s) or a portion of a microprocessor(s), that requires software (e.g., firmware) for operation, but the software may not be present when it is not needed for operation.The definition of circuitry applies to all uses of this term in the application, including in any claims. As a further example, as used in this application, the term circuitry also covers an implementation of merely a hardware circuit or processor (or multiple processors) or portion of a hardware circuit or processor and its (or their) accompanying software and / or firmware. The term circuitry also covers, for example and if applicable to the particular claim element, a baseband integrated circuit or processor integrated circuit for a mobile device or a similar integrated circuit in server, a cellular network device, or other computing or network device.

[0110] As used herein, “at least one of the following: ” and “at least one of ” and similar wording, where the list of two or more elements are joined by “and” or “or”, mean at least any one of the elements, or at least any two or more of the elements, or at least all the elements.

[0111] Further, while operations are depicted in a particular order, this should not be understood as requiring that such operations be performed in the particular order shown or in sequential order, or that all illustrated operations be performed, to achieve desirable results. In certain circumstances, multitasking and parallel processing may be advantageous.Likewise, while several specific implementation details are contained in the above discussions, these should not be construed as limitations on the scope of the present disclosure, but rather as descriptions of features that may be specific to particular embodiments. Certain features that are described in the context of separate embodiments may also be implemented in combination in a single embodiment. Conversely, various features that are described in the context of a single embodiment may also be implemented in multiple embodiments separately or in any suitable sub-combination.

[0112] Although the subject matter has been described in a language that is specific to structural features and / or method actions, it is to be understood the subject matter defined in the appended claims is not limited to the specific features or actions described above. On the contrary, the above-described specific features and actions are disclosed as an example of implementing the claims.

Claims

CLAIMS:

1. A network device, comprising means for performing: receiving beam prediction information of at least one beam predicted by a machine learning model; monitoring performance of the machine learning model to obtain a monitored performance, wherein the monitoring performance of the machine learning model comprises comparing the beam prediction information to a database comprising ground truth data of beam prediction, wherein the database is created based on minimization of drive test, MDT, measurement data collected from a plurality of user equipments; and determining an action based on the monitored performance of the machine learning model.

2. The network device of claim 1, wherein, in a case where the monitored performance is below a threshold, the action comprises at least one of: switching or instructing to switch the machine learning model to another machine learning model for beam prediction; retraining or instructing to retrain the machine learning model for beam prediction; performing or providing instructions to perform beam selection without using the machine learning model; or collecting and / or instructing to collect additional MDT measurement data to enrich the database comprising ground truth data.

3. The network device of claim 1 or 2, further comprising means for performing: receiving, from a user equipment, information on a condition change.

4. The network device of claim 3, further comprising means for performing: evaluating relevance of the condition change to the monitoring of performance ofthe machine learning model.

5. The network device of claim 1 or 2, further comprising means for performing: identifying a condition change in a network of the network device, wherein the condition change is relevant for monitoring performance of the machine learning model; and indicating the identified condition change to a user equipment for the user equipment to verify the condition change.

6. The network device of claim 5, further comprising means for performing: receiving, from the user equipment, a verification of the condition change.

7. The network device of any of claims 3 to 6, wherein the condition change is based on at least one of: reference signal received power, RSRP, value change, signal to interference and noise ratio, SINK value change, or reference signal received quality, RSRQ, value change.

8. A user equipment, comprising means for performing: transmitting, to a network device, beam prediction information of at least one beam predicted by a machine learning model; and receiving instructions comprising at least one of: o instructions to switch the machine learning model to another machine learning model for beam prediction; o instructions to retrain the machine learning model for beam prediction; o instructions to perform beam selection without using the machine learning model; or o instructions to collect minimization of drive test, MDT, measurement data and transmit the MDT measurement data to the network device.

9. The user equipment of claim 8, further comprising means for performing: detecting a condition change in a network of the user equipment and the network device; and transmitting, to the network device, information on the condition change.

10. The user equipment of claim 8, further comprising means for performing: receiving, from the network device, an indication of a condition change; and transmitting, to the network device, a verification of the condition change.

11. The user equipment of claim 9 or 10, wherein the condition change is based on at least one of: reference signal received power, RSRP, value change, signal to interference and noise ratio, SINK value change, or reference signal received quality, RSRQ, value change.

12. A method, comprising: receiving beam prediction information of at least one beam predicted by a machine learning model; monitoring performance of the machine learning model to obtain a monitored performance, wherein the monitoring performance of the machine learning model comprises comparing the beam prediction information to a database comprising ground truth data of beam prediction, wherein the database is created based on minimization of drive test, MDT, measurement data collected from a plurality of user equipments; and determining an action based on the monitored performance of the machine learning model.

13. A method, comprising: transmitting, to a network device, beam prediction information of at least one beam predicted by a machine learning model; andreceiving instructions comprising at least one of: o instructions to switch the machine learning model to another machine learning model for beam prediction; o instructions to retrain the machine learning model for beam prediction; o instructions to perform beam selection without using the machine learning model; or o instructions to collect MDT measurement data and transmit the MDT measurement data to the network device.

14. A computer readable medium comprising program instructions that, when executed by an apparatus, cause the apparatus to at least perform: receiving beam prediction information of at least one beam predicted by a machine learning model; monitoring performance of the machine learning model to obtain a monitored performance, wherein the monitoring performance of the machine learning model comprises comparing the beam prediction information to a database comprising ground truth data of beam prediction, wherein the database is created based on minimization of drive test, MDT, measurement data collected from a plurality of user equipments; and determining an action based on the monitored performance of the machine learning model.

15. A computer readable medium comprising program instructions that, when executed by an apparatus, cause the apparatus to at least perform: transmitting, to a network device, beam prediction information of at least one beam predicted by a machine learning model; and receiving instructions comprising at least one of: o instructions to switch the machine learning model to another machine learning model for beam prediction; o instructions to retrain the machine learning model for beam prediction;instructions to perform beam selection without using the machine learning model; or instructions to collect MDT measurement data and transmit the MDT measurement data to the network device.

Images