Monitoring method and wireless communication device

Abstract

The present disclosure provides a monitoring method for monitoring a machine learning (ML) model. At least one wireless communication device performs a cellular communication task using a first ML model as a monitored ML model, and monitors the first ML model using a second ML model as a monitoring ML model. The at least one wireless communication device evaluates performance of the monitored ML model based on the monitoring.

Description

TECHNICAL FIELD

[0001] The present application relates to the field of communication systems, and in particular to a monitoring method and a wireless communication device. BACKGROUND

[0002] Wireless communication systems, such as the Third Generation (3G) mobile phone standards and technologies, are well known. These 3G standards and technologies are developed by the Third Generation Partnership Project (3GPP). The third generation of wireless communication was mainly developed to support macro cellular mobile phone communications. The communication systems and networks have evolved into broadband and mobile systems. In a cellular wireless communication system, a User Equipment (UE) is connected to a Radio Access Network (RAN) over a wireless link. The RAN comprises a set of Base Stations (BS) providing wireless links to a plurality of UEs located within the cells of the base stations, and an interface to a Core Network (CN) providing overall network control. As known, the RAN and the CN each perform respective functions related to the overall network. The Third Generation Partnership Project developed a so-called Long Term Evolution (LTE) system, i.e. an Evolved Universal Mobile Telecommunication System Territorial Radio Access Network (E-UTRAN), for mobile access networks, in which one or more macro cells are supported by base stations called evolved NodeBs or eNBs. More recently, LTE has further evolved towards a so-called 5G or New Radio (NR) system, in which one or more cells are supported by base stations called gNBs.

[0003] TECHNICAL PROBLEM

[0004] In 3GPP Rel-18, a Study Item (SI) on "Artificial Intelligence (AI) / Machine Learning (ML) New Radio Air Interface" will start to develop. The AI / ML is applied to the 3GPP telecommunication system, and several use cases are studied, including Channel State Information (CSI) feedback compression, beam management, and positioning.

[0005] Generally, beam selection is based on the measurement of Channel State Information-Reference Signal (CSI-RS) / Synchronization Signal Block (SSB). This procedure consumes a lot of reference signals and produces delay. Therefore, predictive beam switching is proposed to reduce the delay. Applying ML to beam management is under research.

[0006] After deploying an AI / ML model, the AI / ML model should be monitored to ensure normal operation. For example, the deployed AI / ML model is monitored to determine whether the beam predicted by the AI / ML model is accurate for beam management, whether the positioning performed by the AI / ML model is still accurate, and / or whether the reported CSI can be fully recovered.

[0007] ML model monitoring is crucial for ML model deployment. How to monitor AI / ML models for telecommunications has not been fully discussed.

[0008] Therefore, there is a need for a monitoring method for machine learning models of wireless communication devices in telecommunications. SUMMARY

[0009] An object of the present invention is to propose a wireless communication device, such as a User Equipment (UE) or a base station, and a monitoring method.

[0010] In a first aspect, one embodiment of the present invention provides a monitoring method for monitoring a Machine Learning (ML) model, which can be executed in at least one wireless communication device, comprising:

[0011] using a first ML model as a monitored ML model to perform a cellular communication task;

[0012] using a second ML model as a monitoring ML model to monitor the first ML model; and

[0013] evaluating the performance of the monitored ML model based on the monitoring.

[0014] In a second aspect, one embodiment of the present invention provides a wireless communication device comprising a processor configured to invoke and run a computer program stored in a memory to cause the device installed with the processor to perform the disclosed method.

[0015] The disclosed method can be implemented in a chip. The chip can include a processor configured to invoke and run a computer program stored in a memory to cause a device installed with the chip to perform the disclosed method.

[0016] The disclosed method can be programmed as computer executable instructions stored in a non-transitory computer readable medium. When the non-transitory computer readable medium is loaded into a computer, the processor of the computer is directed to perform the disclosed method.

[0017] The non-transitory computer readable medium can include at least one of the group consisting of a hard disk, a compact disc read only memory (CD-ROM), an optical storage device, a magnetic storage device, a read only memory (ROM), a programmable read only memory (PROM), an erasable programmable read only memory (EPROM), an electrically erasable programmable read only memory (EEPROM), and a flash memory.

[0018] The disclosed method can be programmed as a computer program product that causes a computer to perform the disclosed method.

[0019] The disclosed method can be programmed as a computer program that causes a computer to perform the disclosed method.

[0020] Advantageous effects

[0021] The present invention provides embodiments to address the issues in the AI / ML model monitoring. In some embodiments of the present invention, the monitoring of AI / ML models is no longer limited by the time required to collect real data, and the AI / ML models can be monitored in time. In some embodiments, the monitoring of ML models is limited to collecting the real data at a predetermined physical location. The proposed method breaks these limitations, enabling the ML model monitoring to be performed at any time and any place.

[0022] Embodiments of the present invention can be applied to evaluate model generalization, including monitoring and evaluating the generalization AI / ML models.

[0023] In some embodiments of the present invention, synthetic real data is synthesized and used as an alternative to the real data to assist active model monitoring or model switching or AI / ML model pre-training. Thus, the system performance can be improved. BRIEF DESCRIPTION OF DRAWINGS

[0024] In order to more clearly illustrate the embodiments of the present application or the related art, the following drawings will briefly be introduced in the embodiments. Obviously, the drawings are only some of the embodiments of the present application. Those skilled in the art can obtain other drawings from these drawings without paying the prior art.

[0025] Figure 1 A schematic diagram of an example wireless communication system is shown, including a user equipment (UE), a base station, and a network entity.

[0026] Figure 2 A schematic diagram is shown, showing the interaction of the data collection, model training, and model inference and feedback, where the ML models are monitored during model inference and updated according to them.

[0027] Figure 3 A schematic diagram is shown, showing one embodiment of the disclosed method.

[0028] Figure 4 A schematic diagram is shown, showing an example where both the monitoring model and the monitored model are deployed on the UE.

[0029] Figure 5 A schematic diagram is shown, showing an example where the monitored model is deployed on the UE and the monitoring model is deployed on the gNB.

[0030] Figure 6 A schematic diagram is shown, showing an example where the monitored model is deployed on the gNB and the monitoring model is deployed on the UE.

[0031] Figure 7 A schematic diagram is shown, showing an example where both the monitoring model and the monitored model are deployed on the gNB.

[0032] Figure 8 A schematic diagram is shown, showing an example of a bilateral model.

[0033] Figure 9 A schematic diagram is shown, showing an example of a bilateral model.

[0034] Figure 10 A schematic diagram is shown, showing an example of a bilateral model.

[0035] Figure 11 A schematic diagram is shown, showing an example of two autoencoder models for CSI feedback compression.

[0036] Figure 12 A schematic diagram is shown, showing a wireless communication system according to embodiments of the present application. DETAILED DESCRIPTION

[0037] The technical matters, structural features, implementation objectives and effects of the embodiments of the present application are described in detail with reference to the drawings. Specifically, the terms in the embodiments of the present application are only used for the purpose of describing the specific embodiments, and are not intended to limit the present application.

[0038] Embodiments of the present application relate to Artificial Intelligence (AI) and Machine Learning (ML) for New Radio (NR) air interface, and solve the problems of data collection, model monitoring and model generalization.

[0039] In some embodiments, the performance of the AI / ML model is monitored as part of the AI / ML model life cycle management. If the performance of the AI / ML model decreases, the AI / ML model can be retrained or switched to another AL / ML model. In some embodiments of the present application, one AI / ML model is used to monitor the performance of another AI / ML model.

[0040] After deploying an AI / ML model, the AI / ML model should be monitored to ensure normal operation. For example, an AI / ML model monitors the deployed AI / ML model to determine whether the predicted beam of the AI / ML model is accurate for beam management, whether the positioning performed by the AI / ML model is still accurate, and / or whether the reported CSI can be fully recovered.

[0041] In one scheme of AI / ML model monitoring, to continue monitoring the AI / ML model, real data is collected, reported and compared with the AI / ML model output.

[0042] For example, to determine whether the predicted beam is accurate, all the beams are measured at the prediction time (or prediction time window). The beam index (or ID) of the beam with the strongest Reference Signal Received Power (RSRP) and this strongest RSRP are collected and compared with the AI / ML model output.

[0043] For example, for an AI / ML model trained to provide Channel State Information (CSI) feedback, if the original CSI (original CSI refers to the original CSI-RS value) is selected as the input of the AI / ML model, and the recovered CSI is the output of the decoder of the autoencoder AI / ML model. The original CSI itself must be reported to the gNB as real data and compared with the decoder output. If the difference between the two exceeds a level, the AI / ML model is not working properly, should be retrained, or switched to another model, or trigger fallback to traditional non-AI methods such as codebook type I and codebook type II.

[0044] In another scheme of AI / ML model monitoring, the confidence is output as a measure of the AI / ML model output. For example, for an AI / ML model trained to provide positioning, the confidence is calculated for the positioning result output by the AI / ML model.

[0045] For AI / ML model monitoring, real data needs to be collected, and sometimes it is difficult to obtain or cumbersome to collect.

[0046] For example, for the AI / ML model trained to provide UE positioning, the real data is the real position of the UE. However, it is difficult to obtain the real position of the UE. The real position in the training data can be collected from some tags of positioning reference units (PRUs) or third parties. Collecting real data from PRUs or tags is not suitable for timely monitoring of the AI / ML model on site during normal operation of the UE.

[0047] For example, for the AI / ML model trained to provide beam management cases, beam prediction in the time domain aims to reduce latency. The collection of real data requires the transmission and measurement of reference signals, which is both time-consuming and resource-consuming.

[0048] Data collection for offline model training is not a big problem. However, data collection for AI / ML model monitoring sometimes encounters challenges due to the limited amount of real data collected.

[0049] On the other hand, if the confidence of the monitored AI / ML model is selected as the monitoring key performance indicator (KPI), when the monitored AI / ML model becomes unreliable, this confidence also becomes unreliable. Therefore, the calculation of the confidence requires a more complex AI / ML model or a complex method.

[0050] A generalizing AI / ML model is an AI / ML model that is trained to handle all unseen datasets. In machine learning, generalization is a definition used to show how a trained model classifies or predicts on unseen data. The training capability of a generalizing AI / ML model can be referred to as generalization capability. A proper way to evaluate the training capability of a generalizing AI / ML model is to compare the performance of the generalizing AI / ML model with the performance of a non-generalizing AI / ML model for a specific scenario.

[0051] In some embodiments of the present application, one AI / ML model is used to monitor another AI / ML model. For simplicity, AI / ML model, AI / ML model, and model are used interchangeably in the description. An AI / ML model used to monitor one or more AI / ML models is referred to as a monitoring model. An AI / ML model monitored by a monitoring AI / ML model is referred to as a monitored model.

[0052] The monitoring model can be deployed on different nodes, including a UE, a base station (e.g., gNB), or a third node. Each of the monitoring model and the monitored model can be a one-sided model or a two-sided model.

[0053] In the description of embodiments of the present application, model switching includes turning off or deactivating one model and turning on or activating another model.

[0054] The third node can include an application server, a gNB, or a UE.

[0055] Reference Figure 1 A telecommunications system including a UE 10a, a base station 20a, a base station 20b, and a network entity device 30 performs the disclosed methods according to embodiments of the present application. Figure 1For illustrative purposes, the system can include more UEs, BSs, and CN entities without limitation. Connections between devices and components within devices are shown with lines and arrows in the figures. The UE 10a can include a processor 11a, a memory 12a, and a transceiver 13a. The base station 20a can include a processor 21a, a memory 22a, and a transceiver 23a. The base station 20b can include a processor 21b, a memory 22b, and a transceiver 23b. The network entity device 30 can include a processor 31, a memory 32, and a transceiver 33. Each of the processors 11a, 21a, 21b, and 31 can be configured to implement the proposed functions, procedures, and / or methods described in the present specification. Layers of radio interface protocols can be implemented in the processors 11a, 21a, 21b, and 31. Each of the memories 12a, 22a, 22b, and 32 can be operable to store various programs and information for operating the connected processors. Each of the transceivers 13a, 23a, 23b, and 33 can be operable to transmit and / or receive radio signals with the connected processors. Each of the base stations 20a and 20b can be one of an eNB, a gNB, or other wireless nodes.

[0056] Each of the processors 11a, 21a, 21b, and 31 can include a general-purpose central processing unit (CPU), an application-specific integrated circuit (ASIC), other chip sets, logic circuits, and / or data processing devices. Each of the memories 12a, 22a, 22b, and 32 can include read-only memory (ROM), random-access memory (RAM), flash memory, memory cards, storage media and / or other storage devices. Each of the transceivers 13a, 23a, 23b, and 33 can include baseband circuitry and radio frequency (RF) circuitry to process radio frequencies signals. When the embodiments are implemented in software, the techniques described herein can be implemented using a suitably-programmed processor, such as the processors 11a, 21a, 21b, and 31, and a memory storing program code, such as the memories 12a, 22a, 22b, and 32. The program code stored in the memory can include one or more modules, programs, functions, entities, etc. that, when executed by the processor, implement the techniques described herein. The memory can be implemented within or external to the processor, and can be communicatively coupled to the processor by various means known in the art.

[0057] The network entity device 30 can be a node in the CN. The CN can include a LTE CN or a 5GC, which can include a User Plane Function (UPF), a Session Management Function (SMF), a Mobility Management Function (AMF), a Unified Data Management (UDM), a Policy Control Function (PCF), a Control Plane (CP) / User Plane (UP) split (CUPS), an Authentication Server (AUSF), a Network Slice Selection Function (NSSF), and the network exposure function (NEF).

[0058] Referring to Figure 2 The system 100 for the machine learning general aspects in NR or NR air interface includes a data collection unit 101, a model training unit 102, an executor 103, and a model inference 104. Note that, Figure 2 The monitoring method is not necessarily limited to the present example. The monitoring method is applicable to any machine learning-based design. The general steps include data collection and / or model training and / or model inference and / or executor.

[0059] The data collection unit 101 is a function that provides input data for the model training unit 102 and the model inference unit 104. AI / ML algorithm-specific data preparation (e.g., data pre-processing and cleaning, formatting, and conversion) is not performed in the data collection unit 101.

[0060] Examples of input data can include measurements from multiple UEs or different network entities, feedback from the executor 103, and output from the AI / ML model.

[0061] Training data is the data required as input to the AI / ML model training unit 102.

[0062] Inference data is the data required as input to the AI / ML model inference unit 104.

[0063] The model training unit 102 is the function that performs the ML model training, validation, and testing. If needed, the model training unit 102 is also responsible for data preparation (e.g., data pre-processing and cleaning, formatting, and transformation) based on the training data passed by the data collection unit 101.

[0064] Model deployment / update between units 102 and 104 involves deploying or updating an AI / ML model (e.g., a trained machine learning model 105a or 105b) to the model inference unit 104. The model training unit 102 trains a machine learning model 105a using the data unit as training data and generates a trained machine learning model 105b from the machine learning model 105a.

[0065] The model inference unit 104 is the function that provides AI / ML model inference output (e.g., a prediction or a decision). The AI / ML model inference output is the output of the machine learning model 105b. If needed, the model inference unit 104 is also responsible for data preparation (e.g., data pre-processing and cleaning, formatting, and transformation) based on the inference data passed by the data collection unit 101.

[0066] The output shown between units 103 and 104 is the inference output of the AI / ML model produced by the model inference unit 104.

[0067] The actuator 103 is the function that receives the output from the model inference unit 104 and triggers or performs a corresponding action. The actuator 103 can trigger an action for other entities or itself.

[0068] Feedback between units 103 and 101 is information that can be needed to derive training or inference data or performance feedback.

[0069] General aspects of AI / ML model monitoring

[0070] General aspects of embodiments of the method are detailed below. Some embodiments use at least one AI / ML model to assist or monitor at least another AI / ML model. The model monitoring can be a pattern, which can be activated or deactivated by at least one or any combination of {RRC, MAC-CE, or DCI}. The ML model monitoring is typically done within a time window. Within this time window, if the monitored model is continuously declared as failed by the UE or gNB or third node, the monitored model is determined as failed by the gNB or third node or UE. In response, a model switch or fallback to a non-AI method can be triggered. As an alternative, if the monitored model is declared as failed at least once, the ML model is determined as failed. As another alternative, if the monitored model is declared as failed more than a threshold number of times, the ML model is determined as failed. The threshold is configured by the gNB or third node for the UE / gNB.

[0071] Referring to Figures 3 to 7 The examples of the UE 10 in the description can include one of the UE 10a. The examples of the gNB 20 in the description can include the base station 20a or 20b. Note that while the gNB is described as one example of a base station in the following, the disclosed method can be implemented in any other type of base station, like an eNB or a beyond-5G base station. The uplink (UL) transmission of control signals or data can be a transmission operation from a UE to a base station. The downlink (DL) transmission of control signals or data can be a transmission operation from a base station to a UE. The disclosed method is detailed below. The UE 10 and the base station, like the gNB 20, perform the monitoring method based on machine learning.

[0072] Figure 3 Embodiments of the disclosed method are shown. At least one wireless communication device performs a monitoring method based on machine learning. In one embodiment, the at least one wireless communication device can include a user equipment (UE). In another embodiment, the at least one wireless communication device can include a base station. In yet another embodiment, the at least one wireless communication device can include a combination of multiple UEs and base stations.

[0073] Referring to Figure 3 At least one wireless communication device performs a monitoring method for monitoring a machine learning (ML) model. The at least one wireless communication device can include a combination of a user equipment (UE), a base station, or a third node.

[0074] The at least one wireless communication device performs a cellular communication task using the first ML model as a monitored ML model (S010). For example, the cellular communication task includes one or more of channel state information (CSI) reporting, time domain beam prediction, spatial domain beam prediction, and user equipment (UE) positioning.

[0075] The at least one wireless communication device monitors the first ML model using a second ML model as a monitoring ML model (S012).

[0076] The at least one wireless communication device evaluates the performance of the monitored ML model based on the monitoring (S014).

[0077] In one embodiment, a monitored model deployment device that deploys and activates the monitored ML model is different from a monitored model training device that trains the monitored ML model. The monitored ML model is downloaded from the monitored model training device to the monitored model deployment device. The monitored ML model is deployed and activated at one of a user equipment (UE), a base station, or a third node. The monitored model deployment device can be one of the UE, the base station, or the third node. The monitoring ML model is trained at one of the UE, the base station, or the third node. The monitored model training device can be one of the UE, the base station, or the third node. The monitored ML model can be activated by a downlink control information (DCI) signal, a radio resource control (RRC) signal, or a medium access control (MAC) control element (CE).

[0078] In one embodiment, a monitored model deployment device that deploys and activates the monitored ML model is different from a model evaluation device that performs the evaluation. The result of the monitoring is reported from the monitored model deployment device to the model evaluation device. The monitored ML model is deployed and activated at one of a user equipment (UE), a base station, or a third node. The monitored model deployment device can be one of the UE, the base station, or the third node. The evaluation is performed at one of the UE, the base station, or the third node. The model evaluation device can be one of the UE, the base station, or the third node. The result of the monitoring includes at least one combination of inputs and outputs of the monitored ML model.

[0079] In one embodiment, a monitoring model deployment device that deploys and activates the monitoring ML model is different from a model evaluation device that performs the evaluation. The results of the monitoring are reported from the monitoring model deployment device to the model evaluation device. The monitoring ML model is deployed and activated on one of a user equipment (UE), a base station, or a third node. The monitoring model deployment device can be one of the UE, the base station, or the third node. The evaluation is performed on one of the UE, the base station, or the third node. The model evaluation device can be one of the UE, the base station, or the third node. The results of the monitoring include at least one combination of inputs and outputs of the monitoring ML model.

[0080] In one embodiment, a monitoring model deployment device that deploys and activates the monitoring ML model is different from a model evaluation device that performs the evaluation. The results of the monitoring are reported from the monitoring model deployment device to the model evaluation device. The monitoring ML model is deployed and activated on one of a user equipment (UE), a base station, or a third node. The monitoring model deployment device can be one of the UE, the base station, or the third node. The evaluation is performed on one of the UE, the base station, or the third node. The model evaluation device can be one of the UE, the base station, or the third node. The results of the monitoring include at least one combination of inputs and outputs of the monitoring ML model.

[0081] Both the monitored model and the monitoring model are deployed and activated on a UE:

[0082] Referring to Figure 4 Both the monitored model and the monitoring model are deployed and activated on a UE 10. The monitored model and / or the monitoring model are trained on a gNB 20 or a third node.

[0083] The procedures of the embodiments are detailed as follows:

[0084] The gNB 20 and / or the third node trains the monitoring model and the monitored model.

[0085] Both the monitoring model and the monitored model are downloaded to the UE.

[0086] Both the monitoring model and the monitored model are deployed on the UE.

[0087] The deployment of the monitoring model and the monitored model is confirmed by the UE, e.g., by sending a message or a Hybrid Automatic Repeat Request (HARQ) Acknowledgment (ACK) or Negative Acknowledgment (NACK) to indicate whether the deployment is successful.

[0088] When the deployment is successful, both the monitoring model and the monitored model are activated on the UE.

[0089] After a period of time, the result of the monitoring, referred to as the monitoring result, is reported to the gNB 20. The monitoring result can be the difference between the output of the monitored model and the output of the monitoring model.

[0090] Additionally, the report can be a measure of the performance of the monitored model, referred to as model performance, e.g., the confidence or a probability indicating the prediction accuracy of the monitored model.

[0091] Additionally, if the measure of the monitoring is continuously qualified within a time window, the report is a status confirmation of the monitored model and does not trigger retraining or model switching. Otherwise, the report is a request for model retraining or model switching.

[0092] An example of signaling between the gNB 20 and the UE 10 is detailed as follows. The gNB 20 configures the UE 10 by providing a configuration to the UE 10. In response to the configuration, the UE 10 monitors the monitored model using the monitoring model. The gNB 20 can send the configuration to the UE 10 through a Downlink Control Information (DCI) signal, a Radio Resource Control (RRC) signal, or a Medium Access Control (MAC) Control Element (CE).

[0093] The configuration can include a monitoring time window or a monitoring period.

[0094] The configuration can include a monitoring mode. The monitoring mode indicates whether the monitoring is performed by the monitoring ML model or by collecting real data to compare with the output of the monitored ML model. In one embodiment, the monitoring mode indicates that the UE 10 is monitored by another AI / ML model (i.e., monitored by the monitoring model). In some embodiments, the monitoring mode indicates that the UE 10 is monitored by collecting real data without the help of the monitoring model.

[0095] Further, when the monitoring model is provided by the gNB 20 or the third node, the default setting of the monitoring mode is to monitor using the configured monitoring model.

[0096] Further, when the monitoring model is not provided by the gNB 20 and / or the third node, the default setting of the monitoring mode is to monitor by collecting the real data without the help of the monitoring model.

[0097] The monitoring mode, the monitored model and / or the monitoring model can be activated / deactivated jointly or separately. The activation signal (e.g., activating the monitoring mode, the monitored model and / or the monitoring model) or the deactivation signal (e.g., deactivating the monitoring mode, the monitored model and / or the monitoring model) for model activation can be carried in a DCI signal, a RRC signal or a MAC-CE.

[0098] In one embodiment, activating the monitoring model and activating the monitoring mode are two different operations. Once the monitoring model is activated, the monitored model is in the monitoring mode.

[0099] The post-processing of AI / ML models, such as the monitored model and / or the monitoring model, can include fine-tuning, model quantization, model distillation, model pruning, etc. The post-processing, for example, can further include adjusting or reducing AI / ML model complexity or customizing the AI / ML model.

[0100] In one embodiment, the monitored model and the monitoring model are integrated into one model and downloaded from the gNB 20 to the UE 10 over the air interface. The gNB configures the UE for post-processing by at least one of RRC signaling, MAC-CE or DCI. Note that at least one of {the monitored model, the monitoring model} can be obtained by post-processing of the ML model. By Figures 4 to 11 The configuration signaling of the {monitored model and / or the monitoring model} is at least one of {RRC, MAC-CE or DCI signaling} by post-processing of the one-sided model and the two-sided model in the examples in

[0101] In one embodiment, the UE 10 post-processes the monitored model. That is, only the monitored model is obtained by post-processing on the UE 10.

[0102] In one embodiment, the UE 10 post-processes the monitoring model. That is, only the monitoring model is obtained by post-processing on the UE 10.

[0103] In one embodiment, the UE 10 post-processes the monitoring model and the monitored model. That is, both the monitoring model and the monitored model are obtained on the UE 10 by post-processing.

[0104] Additionally, the gNB 20 performs the post-processing.

[0105] Additionally, the third node performs the post-processing.

[0106] Additionally, both the monitored model and the monitoring model are trained at the third node.

[0107] The monitored model is deployed and activated on the UE, and the monitoring model is deployed and activated on the gNB:

[0108] Referring to Figure 5 , the monitored model is deployed and activated on the UE 10, and the monitoring model is deployed and activated on the gNB 20.

[0109] The monitored model can be trained or deployed on the gNB 20 or the third node. The procedure of the embodiments is detailed as follows:

[0110] The monitored model is downloaded to the UE 10.

[0111] The monitored model is deployed on the UE 10.

[0112] The deployment is confirmed by the UE 10, for example, by sending a message or a hybrid automatic repeat request (HARQ) acknowledgement (ACK) or negative acknowledgement (NACK).

[0113] The monitored model is activated by the UE 10 or the gNB 20.

[0114] After a period of time, the output of the monitored model is reported to the gNB 20. For example, the output of the monitored model can include CSI, position information (coordinates), and / or beam information (beam index, RSRP).

[0115] The monitoring model and / or the monitoring model is activated by the gNB 20.

[0116] The UE 10 reports the input and output of the monitored model to the gNB 20. The input and output of the monitored model are further processed in the gNB 20 and input to the monitoring model.

[0117] In one embodiment, the report can include at least one of {real data, input of the monitored model, output of the monitored model, auxiliary information of the monitoring model}.

[0118] An example of signaling between the gNB 20 and the UE 10 is detailed as follows.

[0119] The monitoring mode, the monitored model and / or the monitoring model can be activated / deactivated jointly or separately. An activation signal (e.g., activating the monitoring mode, the monitored model and / or the monitoring model) or a deactivation signal (e.g., deactivating the monitoring mode, the monitored model and / or the monitoring model) for model activation / deactivation can be carried in a DCI signal, a RRC signal or a MAC-CE.

[0120] The assistance information from the UE 10 to the monitoring model of the gNB 20 can include at least one of the following: the current channel condition, L1-RSRP, L1-SINR, etc. RSRP stands for Reference Signal Received Power (RSRP), and SINR stands for Signal-to-Interference plus Noise Ratio (SINR).

[0121] The monitored model is activated at the gNB and the monitoring model is activated at the UE:

[0122] Referring to Figure 6 The monitored model is activated at the gNB 20 and the monitoring model is activated at the UE 10. The monitored model can be trained or deployed on the gNB 20 or a third node.

[0123] The procedure of the embodiments is detailed as follows.

[0124] The monitoring model is downloaded to the UE 10.

[0125] The monitoring model is deployed on the UE 10.

[0126] The deployment is confirmed by the UE 10, e.g., reporting whether the deployment is successful by sending a message or a Hybrid Automatic Repeat Request (HARQ) acknowledgement (ACK) or negative acknowledgement (NACK).

[0127] When the deployment is successful, the monitoring mode and / or the monitoring model is activated by the gNB 20.

[0128] If the deployment is not successful, the gNB 20 can re-deploy the monitoring model or deploy another monitoring model.

[0129] After a period of time, the output of the monitoring model is reported to the gNB 20, e.g., location information (coordinates), beam information (beam index, RSRP) and confidence.

[0130] Signaling examples between the gNB 20 and the UE 10 are detailed as follows.

[0131] The monitoring mode, the monitored model, and / or the monitoring model can be activated / deactivated jointly or separately. An activation signal (e.g., activating the monitoring mode, the monitored model, and / or the monitoring model) or a deactivation signal (e.g., deactivating the monitoring mode, the monitored model, and / or the monitoring model) for model activation can be carried in a DCI signal, a RRC signal, or a MAC-CE.

[0132] The reporting during the monitoring mode.

[0133] The monitored model is activated at gNB and the monitoring model is activated at gNB:

[0134] Referring to Figure 7 The deployed monitored model and monitoring model are both deployed at the gNB 20.

[0135] The monitored model can be trained or deployed at the gNB 20 or a third node.

[0136] The procedures of the embodiments are detailed as follows:

[0137] The gNB 20 informs the UE 10 that the monitored model has been activated.

[0138] The UE 10 reports the input of the monitored model to the gNB 20.

[0139] The gNB 20 informs the UE 10 that the monitoring model has been activated.

[0140] The UE 10 reports the input of the monitoring model to the gNB 20.

[0141] In some examples, the input of the monitored model and the input of the monitoring model can be the same.

[0142] In some examples, the input of the monitored model and the input of the monitoring model can be different. For example, the input of the monitored model and the input of the monitoring model can have different number of input beam indices and / or L1-RSRP.

[0143] Signaling examples between the gNB 20 and the UE 10 are detailed as follows.

[0144] The monitoring mode, the monitored model and / or the monitoring model can be activated / deactivated jointly or separately. An activation signal (e.g., activating the monitoring mode, the monitored model and / or the monitoring model) or a deactivation signal (e.g., deactivating the monitoring mode, the monitored model and / or the monitoring model) for model activation / deactivation can be carried in a DCI signal, a RRC signal or a MAC-CE.

[0145] As an example, the report of the monitored model during the monitoring mode can be assistance information, including at least one of {Time of Arrival (TOA), Time Difference of Arrival (TDOA), channel condition, Doppler, L1-SINR}.

[0146] In addition, the AI / ML model can run on the UE 10 / gNB 20 and be monitored by the third node.

[0147] At least one of the monitoring model and the monitored model is trained at the third node.

[0148] The monitoring model and the monitored model are delivered to the UE 10 / gNB 20 respectively.

[0149] The UE 10 requests the monitoring model and / or the monitored model from the third node.

[0150] The third node can send the AI / ML model (monitoring model and / or the monitored model) to the UE 10, and the UE 10 sends the confirmation to the third node.

[0151] The gNB 20 requests the monitoring model and / or the monitored model from the third node. The third node can send the AI / ML model (monitoring model and / or the monitored model) to the gNB 20, and the gNB 20 sends the confirmation to the third node.

[0152] In addition, if the monitored model is located at the third node, the UE 10 can send the report to the third node instead of the gNB 20.

[0153] In addition, if the monitored model is located at the third node, the UE 10 can send the report to the gNB 20.

[0154] The input and output of the monitoring AI / ML model:

[0155] As an example, the input and output of the monitoring model have the same quantity / metric as the monitored model. For example, both the monitoring model and the monitored model output the prediction of the best beam from the same input time or space domain. For example, both the monitoring model and the monitored model output the position of the UE 10 from the same input information (e.g., reference signal, time difference of arrival (TDOA), angle of arrival (AOA), channel impulse response (CIR)).

[0156] The difference between the monitoring model and the monitored model is the difference in the level of complexity. For example, the monitoring ML model has a first level of complexity, the monitored ML model has a second level of complexity, and the first level of complexity is greater than the second level of complexity.

[0157] As an example, the monitored model is a low-complexity model, e.g., post-processed. In general, the monitoring model with high complexity has high-level performance. The UE 10 can execute AI / ML models of different levels of complexity according to the UE capability of the UE 10. The UE 10 can frequently run low-complexity AI / ML models, while less frequently run high-complexity models only for model monitoring.

[0158] In addition, the UE 10 can deploy a low-complexity model as a monitored model. A high-complexity model is run on the gNB 20 as a monitoring model.

[0159] Another example is model generalization:

[0160] A generalization model is referred to as an AI / ML model that can handle all unseen subsets of data. In general, a generalization AI / ML model has qualified performance in a range of settings and scenarios. An AI / ML model for a specific scenario has better performance in that scenario. To measure the performance of a generalization AI / ML model in a specific setting, an AI / ML model for a specific scenario can be selected as the monitoring model or the baseline model in these settings.

[0161] The generalization AI / ML model can be deployed on the UE 10 or the gNB 20 for interference. And the AI / ML model for a specific scenario can be deployed on the UE 10 or the gNB 20 to measure the generalization capability of the generalization AI / ML model in these scenarios. If the output difference of the two models is too large, e.g., greater than a threshold, the generalization AI / ML model is considered infeasible.

[0162] Monitoring AI / ML model output: the synthetic data used to replace real data

[0163] The monitoring mode indicates whether the monitoring is performed by the monitoring ML model or by collecting real data to compare with the output of the monitored ML model. The monitoring ML model generates synthetic data from auxiliary information, and in the evaluation, the synthetic data replaces the real data to compare with the output of the monitored ML model. The auxiliary information includes at least one of channel condition, beam index, reference signal received power (RSRP), signal to interference and noise ratio (SINR), and channel impulse response (CIR).

[0164] The real data can be difficult to obtain (e.g., the real location of the UE 10) or cumbersome to acquire (e.g., the best beam). Model monitoring usually works in a responsive manner. That is, after the performance degradation occurs, the model becomes unreliable.

[0165] The ground truth data can be replaced by synthetic data generated by an artificial intelligence / machine learning (AI / ML) model. The input of the AI / ML model can be the auxiliary information (e.g., channel condition, beam index, L1-RSRP, L1-SINR, CIR), data / noise reference signal reported by a user equipment (UE). The output of the AI / ML model is the synthetic data.

[0166] The input of the AI / ML model can be historical data, such as the historical measurements or monitoring results. The output of the AI / ML model can be the performance of the AI / ML model at the current time.

[0167] Monitoring AI / ML model output: confidence.

[0168] The input of the AI / ML model can be the auxiliary information (e.g., channel condition, beam index, L1-RSRP, L1-SINR, CIR), data / noise reference signal reported by a UE. The output of the AI / ML model is the synthetic data. The output of the monitoring model is confidence.

[0169] In one example, the historical data can be the confidence. The AI / ML model predicts confidence, indicating whether the monitored model is likely to work normally within at least the subsequent time unit.

[0170] Bilateral model:

[0171] In one embodiment, the monitoring model includes a bilateral model and has a first monitoring sub-model working in a first model deployment device and a second monitoring sub-model working in a second model deployment device.

[0172] The monitored model comprises another bilateral model and has a first monitored sub-model working in the first model deployment device and a second monitored sub-model working in the second model deployment device.

[0173] The first model deployment device comprises a UE and the second model deployment device comprises a base station.

[0174] In one embodiment, the monitoring is performed to monitor an output of the first monitoring sub-model and an output of the first monitored sub-model, and the evaluating is performed to evaluate the difference between the output of the first monitoring sub-model and the output of the first monitored sub-model.

[0175] In one embodiment, the monitoring is performed to monitor an output of the second monitoring sub-model and an output of the second monitored sub-model, and the evaluating is performed to evaluate the difference between the output of the second monitoring sub-model and the output of the second monitored sub-model.

[0176] The difference between the output of the first monitoring sub-model and the output of the first monitored sub-model or the difference between the output of the second monitoring sub-model and the output of the second monitored sub-model is calculated using at least one of mean square error (MSE), normalized mean square error (NMSE), cosine similarity, confidence, and accuracy.

[0177] Bilateral model for positioning:

[0178] For the positioning case, the bilateral model works as follows, the first part (the first monitored sub-model NN1_1 or first monitoring sub-model NN2_1) extracts features (time difference, angle of arrival (AOA), time of arrival (TOA), time difference of arrival (TDOA), etc.) from the input (channel impulse response / reference signal). The features are sent to the gNB 20 for further processing to obtain the output (UE 10 position information / coordinates).

[0179] The first monitored sub-model NN1_1 is coupled with a second monitored sub-model NN1_2. The first monitoring sub-model NN2_1 is coupled with a second monitoring sub-model NN2_2.

[0180] The first monitored sub-model NN1 1 and the second monitored sub-model NN1 2 have better performance, e.g. have more complex model structure and / or higher complexity, compared to the first monitoring sub-model NN2 1 and the second monitoring sub-model NN2 2.

[0181] For the double-sided model, the model monitoring can be performed separately.

[0182] Figure 8 An example of double-sided model for positioning is shown. A second double-sided model (first monitoring sub-model NN2 1 and second monitoring sub-model NN2 2) deployed as the monitoring model monitors the first double-sided model (first monitored sub-model NN1 1 and the second monitored sub-model NN1 2) deployed as the monitored model.

[0183] The monitoring can be performed using at least one of the following ways.

[0184] The monitoring can be performed at the outputs of the first monitored sub-model NN1 1 and the first monitoring sub-model NN2 1. The measured key performance indicator (KPI) can be at least one of the following: MSE, NMSE, cosine similarity, confidence and accuracy.

[0185] The monitoring can be performed at the outputs of the first monitoring sub-model NN2 1 and the second monitoring sub-model NN2 2. The measured KPI can be at least one of the following: MSE, NMSE, cosine similarity, confidence and accuracy.

[0186] Alternatively, at least one of the first monitoring sub-model NN2 1 and the second monitoring sub-model NN2 2 is deployed at a third node.

[0187] In another example, for the double-sided model, the model monitoring can be performed jointly at the UE 10 side.

[0188] In one embodiment, the second monitoring sub-model and the second monitored sub-model are integrated into one sub-model.

[0189] Figure 9 A double-sided model for positioning is shown. A double-sided model (first monitoring sub-model NN2 1 and second sub-model NN12) is deployed to monitor another double-sided model (first monitored sub-model NN1 1 and the second sub-model NN12). At the gNB side, the second sub-model NN12 can process the outputs of the first monitored sub-model NN1 1 and the second monitored sub-model NN1 2.

[0190] The monitoring can be performed using at least one of the following ways.

[0191] The monitoring can be performed with respect to a difference between the output of the first monitored sub-model NN1 1 and the output of the second monitoring sub-model NN2 2. The KPI for measuring the difference between the output of the first monitored sub-model NN1 1 and the output of the second monitoring sub-model NN2 2 can be at least one of: MSE, NMSE, cosine similarity, confidence, and accuracy.

[0192] The monitoring can be performed with respect to a difference between the output of the second sub-model NN12 obtained from the output of the first monitored sub-model NN1 1 and the output of the second sub-model NN12 obtained from the second monitoring sub-model NN2 2. The KPI for measuring the difference can be at least one of: MSE, NMSE, cosine similarity, confidence, and accuracy.

[0193] Alternatively, at least one of the second sub-models NN12 is deployed at a third node.

[0194] In another example, for the double-sided model, the monitoring can be performed jointly at the gNB side.

[0195] In one embodiment, the first monitoring sub-model and the first monitored sub-model can be integrated into one sub-model.

[0196] Figure 10 A double-sided model for positioning is shown. A double-sided model (first sub-model NN3 1 and the second monitoring sub-model NN3 2') is deployed to monitor another double-sided model (first sub-model NN3 1 and the second monitored sub-model NN3 2). At the gNB side, the second monitoring sub-model NN3 2 receives the output of the first sub-model NN3 1 as input to the second monitoring sub-model NN3 2, and the second monitored sub-model NN3 2' receives the output of the first sub-model NN3 1 as input to the second monitored sub-model NN3 2'. The second monitored sub-model NN3 2' is used as a monitoring model to monitor the performance of the second monitoring sub-model NN3 2.

[0197] The procedure of the embodiments is detailed as follows:

[0198] The gNB 20 informs the UE 10 about the deployment of the second monitored sub-model NN3 2' as a monitoring model. Monitoring of the first sub-model NN3 1 and the second monitoring sub-model NN3 2 is performed at the gNB 20.

[0199] In another example, the second monitored sub-model NN3_2' is deployed at a third node. The output of the first sub-model NN3_1 is reported to both the gNB 20 and the third node.

[0200] Channel State Information (CSI) feedback compression:

[0201] In one embodiment, the monitoring model comprises a first autoencoder for reporting CSI, the first monitoring sub-model is used as a first encoder of the first autoencoder for compressing CSI, and the second monitoring sub-model is used as a first decoder of the first autoencoder for decompressing the compressed CSI from the first encoder.

[0202] In one embodiment, the monitored model comprises a second autoencoder for reporting CSI, the first monitored sub-model is used as a second encoder of the second autoencoder for compressing CSI, and the second monitored sub-model is used as a second decoder of the second autoencoder for decompressing the compressed CSI from the second encoder.

[0203] In one embodiment, the first autoencoder and the second autoencoder report CSI according to a configured monitoring period. In one embodiment, the monitoring period is reported by the UE. In one embodiment, the base station determines the monitoring period according to UE capability.

[0204] For the double-sided model, e.g., the autoencoder, a second model is deployed to measure the performance of the first model.

[0205] In one embodiment, the first model and the second model can have different complexities.

[0206] Alternatively, the first model and the second model can have different encoder output quantization levels. For example, the monitoring ML model has a first quantization level, the monitored ML model has a second quantization level, and the first quantization level is greater than the second quantization level.

[0207] Figure 11 Two autoencoder models for CSI feedback compression and CSI feedback are shown. One autoencoder (referred to as the first autoencoder) can include an encoder A10 and a decoder A11. The other autoencoder (referred to as the second autoencoder) can include an encoder A20 and a decoder A21.

[0208] The first autoencoder can be a more complex model, while the second autoencoder is a simplified model. The first autoencoder can be a touchstone or benchmark for the performance of the second autoencoder.

[0209] Typically, the raw CSI and / or Eigen vectors need to be reported by the UE with high precision as ground truth for model monitoring and / or data collection. In this way, two autoencoder models are deployed and the ground truth is no longer needed in model monitoring. Thus, the feedback overhead of reporting raw CSI and / or Eigen vectors as the ground truth for model monitoring and / or data collection is reduced.

[0210] The principle of determining the AI / ML model as the touchstone (monitoring model) can be a more stringent performance metric, such as a more stringent cosine similarity, minimum mean square error (MMSE) / throughput, and / or prediction accuracy (e.g., 95% prediction accuracy).

[0211] As another example, the second model or both models can be obtained, e.g., by post-processing the same AI / ML model. Thus, initially, one model is downloaded from the gNB 20 to the UE 10, if the autoencoder model is trained at the gNB 20.

[0212] The compressed vectors are reported by the UE 10. The procedure of configuring the UE 10 to run two AI / ML models is detailed as follows. The gNB 20 can configure the UE 10 and the UE 10 reports the two compressed vectors.

[0213] The UE 10 reports the compressed vectors once. When the radio resources are limited, the UE 10 reports the compressed vectors according to a priority rule.

[0214] In one embodiment, when the multiple compressed vectors have been reported and overlap, the compressed vector with the larger size is discarded. When the compressed vectors have the same size, the compressed vector with the larger age is discarded. Alternatively, the UE 10 randomly discards one vector.

[0215] Alternatively, the compressed vector a with the smaller size is discarded to ensure performance. Since the smaller the compression ratio, the better the performance. When the compressed vectors have the same size, the compressed vector with the larger age is discarded. Alternatively, the UE 10 randomly discards one vector.

[0216] The monitoring period is a UE capability of the UE 10. The monitoring period can be reported by the UE 10. Alternatively, the gNB 20 can explicitly determine the monitoring period according to the UE capability (storage, Central Processing Unit (CPU), memory, Floating-point Operations Per Second (FLOPS),...) of the UE 10. After receiving the UE capability from the UE 10, the gNB 20 configures the monitoring period.

[0217] Alternatively, the gNB 20 configures the monitoring period regardless of the UE capability of the UE 10.

[0218] In one embodiment, the UE 10 reports the compressed vectors with different periods.

[0219] The monitoring period is a UE capability of the UE 10. The monitoring period can be reported by the UE 10. Alternatively, the gNB 20 can explicitly determine the monitoring period according to the UE capability (storage, CPU, memory, FLOPS,...) of the UE 10. After receiving the UE capability from the UE 10, the gNB 20 configures the monitoring period.

[0220] Alternatively, the gNB 20 configures the monitoring period regardless of the UE capability of the UE 10.

[0221] Note that the method in this embodiment is not limited to the model monitoring. When multiple compressed vectors have to be reported with limited resources, the UE 10 reports the compressed vectors according to a priority rule.

[0222] In one embodiment, the compressed vector with a larger age is discarded. When the compressed vectors have the same age, the compressed vector with a larger size is discarded. Alternatively, the UE 10 randomly discards one vector.

[0223] Note that the deployment of the monitoring model and the model under monitoring for the CSI feedback compression can follow a similar way as the bilateral positioning model, e.g., jointly performed or separately performed at the gNB 20 / UE 10 side, or even performed in a third node. The related signaling can be reported to the third node or reported to the gNB 20 and forwarded to the third node.

[0224] ML-assisted non-AI (applicable to all use cases):

[0225] In another example, the monitoring model outputs the synthetic data to replace the ground truth. The synthetic data is generated for further processing. One use of the synthetic data is model monitoring.

[0226] The synthetic data is generated according to the assistance information, e.g., current channel condition, indoor environment, outdoor environment, moving speed of the UE 10, Doppler shift, etc.

[0227] The assistance information and synthetic data can be collected and sent to a node as hyperparameters of an AI / ML model. In one example, the AI / ML model is the monitoring model that can generate the synthetic data.

[0228] When the monitoring model is deployed at the UE 10, the UE 10 can obtain the assistance information for the monitoring model, e.g., current channel condition.

[0229] When the monitoring model is deployed at the gNB 20 / third node, the UE 10 can report the assistance information to the gNB 20 / third node.

[0230] In the example of training the monitored model for providing positioning, it is difficult to obtain the ground truth of the real position. The UE 10 can approach the positioning reference unit (PRU) or some tag to obtain its real position at one time. The reference is PRU. The UE 10 can only refer to some tags or PRUs for monitoring. This limits the model monitoring at the UE 10. An AI / ML model for generating data (referred to as data generation neural network) can be deployed to generate the input data (e.g., the CIR, TOA, reference signal) and output data. The input data and the output data of the AI / ML model for generating data (referred to as data generation neural network) are used by the monitoring model as the corresponding synthetic data of the replacement ground truth to monitor whether the monitored model works normally using the synthetic data.

[0231] For example, the input data in the synthetic data can include one or more of CIR, reference signal (RS), TDOA, AOA, angle of departure (AOD), and channel condition. For example, the output data in the synthetic data can include position information. The synthetic data includes input data for the monitoring model, which is associated with the output data of the monitoring model. For example, the format of the synthetic data includes:

[0232] {input [CIR / RS / TDOA / AOA / AOD / channel condition], output [position information]}.

[0233] In the example of training the monitored model for generating beam management data, it is difficult to collect the ground truth data both in terms of resources and time.

[0234] For example, the input data in the synthetic data can comprise one or more subsets of all beams. For example, the output data in the synthetic data can comprise one or more predicted beams.

[0235] The synthetic data comprises input data associated with the output data, for example in the following format:

[0236] {input [subset of all beams], output [predicted beams]} or {input [measurements of subset of all beams], output [measurements of predicted beams in time or space]}. {input [measurements of subset of all beams and beam IDs of these subset beams], output [measurements of predicted beams in time or space and beam IDs of these predicted beams]}.

[0237] Monitoring schemes:

[0238] The gNB 20 configures the UE 10 by providing the UE 10 with a configuration on one or more monitoring schemes. The configuration is at least one of {Radio Resource Control (RRC) signaling, Medium Access Control (MAC)-Control Element (CE), or Downlink Control Information (DCI) signaling}. In response to the configuration, the UE 10 uses one or more monitoring schemes. The one or more monitoring schemes can be selected from one or more of:

[0239] • collecting and / or reporting the ground truth;

[0240] • collecting or generating and / or reporting synthetic data for replacing ground truth; and

[0241] • using the same type of input and / or output of the monitored ML model and the monitoring model.

[0242] More specifically, the monitoring ML model monitors the first ML model according to an activated monitoring scheme among a plurality of monitoring schemes, and the plurality of monitoring schemes comprises:

[0243] • comparing the output of the monitored ML model with the ground truth for the monitored ML model;

[0244] • comparing the output of the monitored ML model with synthetic data for replacing the ground truth; and

[0245] • comparing the output of the monitored ML model with the output of the monitoring model based on the same input of the monitored ML model and the monitoring model.

[0246] One of the multiple monitoring schemes is activated as the activated monitoring scheme according to a configuration. The configuration can be carried in a DCI signal, RRC signal or MAC CE.

[0247] The configuration is configured by RRC signal or MAC-CE and activated by DCI signaling.

[0248] The configuration is configured by RRC signal or MAC-CE and deactivated by DCI signaling. For multiple monitoring models, the specific location of the monitoring model and the monitored model can be Figures 4 to 11 Any combination of some of all the examples in the above.

[0249] Multiple models:

[0250] In one embodiment, the monitoring model comprises multiple component monitoring models instead of a single model, similar to random forest in ML. The outputs of the multiple component monitoring models are mathematically processed, e.g. by averaging, maximum, minimum, etc., as a benchmark of the component monitoring models. The benchmark is considered as the output of the monitoring model or the result of the monitoring. The result of the monitoring model is obtained by averaging or voting. Voting means majority wins, when the majority of monitoring models (or the majority of monitoring sub-models) indicate that the monitored model fails, the monitored model is judged to fail. Or, if at least one scheme indicates that the monitored model fails, the monitored model is judged to fail. Or, if all schemes indicate that the monitored model fails, the monitored model is judged to fail.

[0251] The monitoring model can be configured to comprise only one component monitoring model or multiple component monitoring models according to a configuration, as configured by the gNB 20 or the third node in the configuration. The benchmark of the monitoring model can be carried in a DCI signal, RRC signal or MAC CE. The configuration can be carried in a DCI signal, RRC signal or MAC CE to inform the UE 10.

[0252] By default, the configuration is a single AI / ML model as one monitoring model.

[0253] Figure 12 is a block diagram of an example system 700 for wireless communication according to embodiments of the present disclosure. Embodiments described herein can implement into the system using any suitably configured hardware and / or software. Figure 12The system 700 is shown to include radio frequency (RF) circuitry 710, baseband circuitry 720, processing elements 730, memory / storage 740, a display 750, a camera 760, a sensor 770, and an input / output (I / O) interface 780, which are each coupled together by one or more busses 790.

[0254] The processing elements 730 can include circuitry such as, but not limited to, one or more single-core or multi-core processors. The processor(s) can include any combination of general-purpose processors and dedicated processors such as graphics processors and application processors. The processors can be coupled with the memory / storage and configured to execute instructions stored in the memory / storage to provide the functionality of various applications and / or an operating system running on the system.

[0255] The radio control functions can include, but are not limited to, signal modulation, encoding, decoding, radio frequency shifting, etc. In some embodiments, the baseband circuitry can provide communication compatible with one or more radio technologies. For example, in some embodiments, the baseband circuitry can support communication with a 5G NR, Long Term Evolution (LTE), Evolved Universal Terrestrial Radio Access Network (EUTRAN), and / or other wireless metropolitan area networks (WMAN), wireless local area networks (WLANs), and / or wireless personal area networks (WPANs). Embodiments in which the baseband circuitry is configured to support radio communications of more than one wireless protocol can be referred to as multi-mode baseband circuitry. In various embodiments, the baseband circuitry 720 can include circuitry to operate with signals that are not strictly considered a part of a baseband frequency. For example, in some embodiments the baseband circuitry can include circuitry to operate with signals having intermediate frequencies.

[0256] In various embodiments, the system 700 can be a mobile computing device such as, but not limited to, a laptop computing device, a tablet computing device, a netbook, an ultrabook, a smartphone, etc. In various embodiments, the system can have more or fewer components, and / or different architectures. Where appropriate, the methods described herein can be implemented as computer programs. The computer program can be stored on storage media, such as non-transitory storage media.

[0257] The embodiments of the present disclosure are a combination of techniques / processes that can be adopted in 3GPP specifications for creating a final product.

[0258] If the software functional unit is realized and used and sold as a product, it can be stored in a readable storage medium in a computer. Based on this understanding, the technical solution proposed by the present disclosure can be substantially or partially realized in the form of a software product. Alternatively, part of the technical solution beneficial to the conventional technology can be realized in the form of a software product. The software product in the computer is stored in a storage medium, including a plurality of commands for a computing device (such as a personal computer, a server, or a network device) to run all or some of the steps disclosed in the embodiments of the present disclosure. The storage medium includes a USB flash disk, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a floppy disk, or other types of media capable of storing program codes.

[0259] The present disclosure provides a monitoring method for monitoring an ML model. The invention provides embodiments for solving problems in monitoring of AI / ML models. In some embodiments of the present disclosure, the monitoring of AI / ML models is no longer limited to the time required to collect true value data, and the AI / ML model can be monitored in time.

[0260] The embodiments of the present disclosure can be applied to evaluate model generalization, including monitoring and evaluating the generalized AI / ML model.

[0261] In some embodiments of the present disclosure, synthetic true value data is synthesized and regarded as true value data to assist in actively switching or pre-training AI / ML models. The system performance can be improved thereby.

[0262] Although the present disclosure has been described in conjunction with the embodiments that are considered to be the most practical and preferred, it should be understood that the present disclosure is not limited to the disclosed embodiments, but is intended to cover various arrangements made without departing from the scope of the appended claims, which are the broadest interpretation allowed under the law.

Claims

1. A monitoring method for monitoring a machine learning (ML) model, executable in at least one wireless communication device, characterized in that, comprises: performing a cellular communication task using a first ML model as a monitored ML model; monitoring the first ML model using a second ML model as a monitoring ML model; and evaluating performance of the monitored ML model based on the monitoring; wherein the monitoring model comprises a bilateral model and has a first monitoring sub-model working in a first model deployment device and a second monitoring sub-model working in a second model deployment device; and the monitored model comprises another bilateral model and has a first monitored sub-model working in the first model deployment device and a second monitored sub-model working in the second model deployment device wherein the monitoring ML model monitors the first ML model according to an active monitoring scheme among a plurality of monitoring schemes, and the plurality of monitoring schemes comprises: comparing an output of the monitored ML model with a ground truth for the monitored ML model; comparing the output of the monitored ML model with synthetic data for replacing the ground truth; and comparing the output of the monitored ML model with an output of the monitoring model based on a same input of the monitored ML model and the monitoring model.

2. The method of claim 1, wherein, the cellular communication task comprises one or more of channel state information, CSI, reporting, beam prediction in time domain, beam prediction in spatial domain, and positioning of a user equipment, UE.

3. The method of claim 1, wherein, a monitored model deployment device that deploys and activates the monitored ML model is different from a monitored model training device that trains the monitored ML model; and the monitored ML model is downloaded from the monitored model training device to the monitored model deployment device.

4. The method of claim 3, wherein, the monitored ML model is deployed and activated at one of a user equipment, UE, a base station, or a third node; and the monitoring ML model is trained at one of the UE, the base station, or the third node.

5. The method of claim 4, wherein, the monitored ML model is activated by a downlink control information, DCI, signal, a radio resource control, RRC, signal, or a medium access control, MAC, control element, CE.

6. The method of claim 1, wherein, a monitored model deployment device that deploys and activates the monitored ML model is different from a model evaluation device that performs the evaluation; and a result of the monitoring is reported from the monitored model deployment device to the model evaluation device.

7. The method of claim 6, wherein, the monitored ML model is deployed and activated at one of a user equipment, UE, a base station, or a third node; and the evaluation is performed at one of the UE, the base station, or the third node.

8. The method of claim 6, wherein, the result of the monitoring comprises at least one combination of an input and an output of the monitored ML model.

9. The method of claim 1, wherein, a monitoring model deployment device that deploys and activates the monitoring ML model is different from a monitoring model training device that trains the monitoring ML model; and the monitoring ML model is downloaded from the monitoring model training device to the monitoring model deployment device.

10. The method of claim 9, wherein, the monitoring ML model is deployed and activated at one of a user equipment, UE, a base station, or a third node; and the monitoring ML model is trained at one of the UE, the base station, or the third node.

11. The method of claim 10, wherein, The monitoring ML model is activated by a DCI signal, a RRC signal, or a MAC CE.

12. The method of claim 1, wherein, A monitoring model deployment device that deploys and activates the monitoring ML model is different from a model evaluation device that performs the evaluation; and Results of the monitoring are reported from the monitoring model deployment device to the model evaluation device.

13. The method of claim 12, wherein, The monitoring ML model is deployed and activated at one of a user equipment, UE, a base station, or a third node; and The evaluation is performed at one of the user equipment, the base station, or the third node.

14. The method of claim 12, wherein, The results of the monitoring include at least one combination of inputs and outputs of the monitoring ML model.

15. The method of claim 1, wherein, A monitoring mode indicates whether the monitoring is performed by the monitoring ML model or by collecting ground truth data to compare with outputs of the monitored ML model.

16. The method of claim 15, wherein, The monitoring ML model generates synthetic data from auxiliary information, and the synthetic data replaces the ground truth data in the evaluation to compare with outputs of the monitored ML model.

17. The method of claim 16, wherein, The auxiliary information includes at least one of channel conditions, beam indices, reference signal received power, RSRP, signal to interference plus noise ratio, SINR, channel impulse response, CIR, indoor environment, outdoor environment, UE’s moving speed, and Doppler shift.

18. The method of claim 16, wherein, The synthetic data includes input data for the monitoring model, which is associated with output data of the monitoring model.

19. The method of claim 1, wherein, The monitoring ML model has a first complexity level, the monitored ML model has a second complexity level, and the first complexity level is greater than the second complexity level.

20. The method of claim 1, wherein, The monitoring model includes a scenario-specific ML model, and the monitored model includes a general-purpose ML model.

21. The method of claim 1, wherein, Both the monitoring model and the monitored model output from the same input a best beam in time or space domain.

22. The method of claim 1, wherein, Both the monitoring model and the monitored model output from the same input a UE location.

23. The method of claim 1, wherein, The first model deployment device includes a user equipment, UE, and the second model deployment device includes a base station.

24. The method of claim 1, wherein, The monitoring is performed to monitor outputs of the first monitoring sub-model and the first monitored sub-model, and the evaluation is performed to evaluate a difference between the outputs of the first monitoring sub-model and the first monitored sub-model; Or The monitoring is performed to monitor outputs of the second monitoring sub-model and the second monitored sub-model, and the evaluation is performed to evaluate a difference between the outputs of the second monitoring sub-model and the second monitored sub-model.

25. The method of claim 24, wherein, The first monitoring sub-model and the first monitored sub-model are integrated into one sub-model; or The second monitoring sub-model and the second monitored sub-model are integrated into one sub-model.

26. The method of claim 24, wherein, The difference between the output of the first monitoring sub-model and the output of the first monitored sub-model or the difference between the output of the second monitoring sub-model and the output of the second monitored sub-model is calculated using at least one of mean square error (MSE), normalized mean square error (NMSE), cosine similarity, confidence, and accuracy.

27. The method of claim 1, wherein, The monitoring model comprises a first autoencoder for reporting CSI, the first monitoring sub-model is used as a first encoder of the first autoencoder for compressing CSI, and the second monitoring sub-model is used as a first decoder of the first autoencoder for decompressing the compressed CSI from the first encoder; and The monitored model comprises a second autoencoder for reporting CSI, the first monitored sub-model is used as a second encoder of the second autoencoder for compressing CSI, and the second monitored sub-model is used as a second decoder of the second autoencoder for decompressing the compressed CSI from the second encoder. The monitoring ML model has a first quantization level, the monitored ML model has a second quantization level, and the first quantization level is greater than the second quantization level.

28. The method of claim 27, wherein, The first autoencoder and the second autoencoder report CSI according to a configured monitoring period.

29. The method of claim 27, wherein, The monitoring period is reported by a UE; or 30. The method of claim 29, wherein, The monitoring period is determined by a base station according to a UE capability. One of the plurality of monitoring schemes is activated as the activated monitoring scheme according to a configuration.

31. The method of claim 1, wherein, The configuration is carried in a DCI signal, an RRC signal, or a MAC CE.

32. The method of claim 31, wherein, The configuration is configured by an RRC signal or a MAC-CE and activated by DCI signaling.

33. The method of claim 31, wherein, The configuration is configured by an RRC signal or a MAC-CE and deactivated by DCI signaling.

34. The method of claim 31, wherein, The monitoring model comprises a plurality of component monitoring models, and the outputs of the plurality of component monitoring models are mathematically processed as a benchmark of the component monitoring model.

35. The method of claim 1, wherein, The monitoring model can be configured to include only one component monitoring model or a plurality of component monitoring models according to a configuration.

36. The method of claim 1, wherein, The configuration is carried in a DCI signal, an RRC signal, or a MAC CE.

37. The method of claim 36, wherein, A processor configured to invoke and run a computer program stored in a memory to enable a device installed with the processor to perform the method of any one of claims 1 to 37.

38. A wireless communication device, comprising: A processor configured to invoke and run a computer program stored in a memory to enable a device installed with the chip to perform the method of any one of claims 1 to 37.

39. A chip comprising:

40. A computer-readable storage medium, wherein a computer program is stored, wherein the computer program enables a computer to perform the method of any one of claims 1 to 37.

41. A computer program product comprising a computer program, wherein the computer program enables a computer to perform the method of any one of claims 1 to 37. ​

Images