Monitoring method and wireless communication equipment

Abstract

The present disclosure provides a monitoring method for monitoring a machine learning (ML) model, and a method for monitoring a machine learning (ML) model. At least one wireless communication device performs cellular communication tasks using a first ML model as a monitored ML model, and monitors the first ML model using a second ML model as a monitoring ML model. The at least one wireless communication device evaluates performance of the monitored ML model based on the monitoring.

Description

Technical Field

[0001] The present invention relates to the field of communication systems, and in particular to a monitoring method and wireless communication equipment. Background Art

[0002] Wireless communication systems, such as the third generation (3G) mobile phone standards and technologies are well known. These 3G standards and technologies were developed by the Third Generation Partnership Project (3GPP). The third generation of wireless communication is mainly to support macrocellular mobile phone communications. Communication systems and networks have evolved into broadband and mobile systems. In a cellular wireless communication system, a user equipment (UE) is connected to a radio access network (RAN) via a wireless link. The RAN includes a group of base stations (BS) that provide wireless links for multiple UEs located in a cell covered by the base station, and an interface connected to a core network (CN) that provides overall network control. As is known, the RAN and the CN each perform their respective functions related to the entire network. The 3rd Generation Partnership Project developed the so-called Long Term Evolution (LTE) system, the Evolved Universal Mobile Telecommunication System Territorial Radio Access Network (E-UTRAN), for mobile access networks, where one or more macro cells are supported by base stations called evolved Node Bs or eNBs. More recently, LTE has further evolved into the so-called 5G or New Radio (NR) system, where one or more cells are supported by base stations called gNBs. Technical issues

[0003] In 3GPP Rel-18, the Study Item (SI) of "Artificial Intelligence (AI) / Machine Learning (ML) New Air Interface" will start development. The AI / ML is applied to the 3GPP telecommunication system and several use cases are studied, including Channel State Information (CSI) feedback compression, beam management and positioning.

[0004] Typically, beam selection is based on the measurement of the Channel State Information-Reference Signal (CSI-RS) / Synchronization Signal Block (SSB). This process consumes a lot of reference signals and generates delays. Therefore, predictive beam switching is proposed to reduce the delay. The application of ML to beam management is under study.

[0005] After the AI / ML model is deployed, the AI / ML model should be monitored to ensure proper operation. For example, the deployed AI / ML model is monitored to determine whether the beam predicted by the AI / ML model is accurate for beam management, whether the positioning performed by the AI / ML model is still accurate, and / or whether the reported CSI can be fully recovered.

[0006] ML model monitoring is critical for ML model deployment. How to monitor AI / ML models for telecommunications is still not fully discussed.

[0007] Therefore, there is a need for a monitoring method for machine learning models of wireless communication devices in telecommunications. Summary of the invention

[0008] An object of the present invention is to provide a wireless communication device, such as a user equipment (UE) or a base station, and a monitoring method.

[0009] In a first aspect, an embodiment of the invention provides a monitoring method for monitoring a machine learning (ML) model, which can be performed in at least one wireless communication device, comprising: using the first ML model as a monitored ML model to perform a cellular communication task; Using a second ML model as a monitoring ML model to monitor the first ML model; and Evaluate performance of the monitored ML model based on the monitoring.

[0010] In a second aspect, an embodiment of the invention provides a wireless communication device, comprising a processor configured to call and run a computer program stored in a memory, so that the device on which the processor is installed performs the disclosed method.

[0011] The disclosed method may be implemented in a chip, which may include a processor configured to call and run a computer program stored in a memory, so that a device equipped with the chip executes the disclosed method.

[0012] The disclosed method can be programmed as computer-executable instructions stored in a non-transitory computer-readable medium. When the non-transitory computer-readable medium is loaded into a computer, the processor of the computer is instructed to execute the disclosed method.

[0013] The non-transitory computer-readable medium may include at least one of the following group consisting of: a hard disk, a read-only compact disk (CD-ROM), an optical storage device, a magnetic storage device, a read-only memory (Read Only Memory), a programmable read-only memory (Programmable Read Only Memory), an erasable programmable read-only memory (ErasableProgrammable Read Only Memory), an EPROM, an electrically erasable programmable read-only memory (ElectricallyErasableProgrammable Read Only Memory) and a flash memory.

[0014] The disclosed method may be programmed as a computer program product, which enables a computer to execute the disclosed method.

[0015] The disclosed method may be programmed as a computer program, which causes a computer to execute the disclosed method. Beneficial effects

[0016] The invention provides embodiments to solve the problems in the AI / ML model monitoring. In some embodiments of the invention, the monitoring of AI / ML models is no longer limited by the time required to collect real data, and the AI / ML models can be monitored in a timely manner. In some embodiments, the monitoring of ML models is limited to collecting the real data at a predetermined physical location. The proposed method breaks these limitations and enables the ML model monitoring to be performed at any time and any place.

[0017] The embodiments of the present invention may be applied to evaluate model generalization, including monitoring and evaluating the generalized AI / ML model.

[0018] In some embodiments of the present invention, synthetic real data is synthesized and used as a substitute for the real data to assist in active model monitoring or model switching or AI / ML model pre-training, thereby improving system performance. BRIEF DESCRIPTION OF THE DRAWINGS

[0019] In order to more clearly illustrate the embodiments of the present invention or related technologies, the following diagrams will be briefly introduced in the embodiments. Obviously, the drawings are only some embodiments of the present invention. A person skilled in the art can obtain other diagrams based on these diagrams without paying the above premises.

[0020] Figure 1 A schematic diagram of an exemplary wireless communication system is shown, which includes a user equipment (UE), a base station and a network entity.

[0021] Figure 2 A schematic diagram is shown showing the interaction of data collection, model training, and model inference and feedback, where the ML model is monitored during model inference and updated based on them.

[0022] Figure 3 A schematic diagram is shown, illustrating one embodiment of the disclosed method.

[0023] Figure 4 A schematic diagram is shown, showing an example in which both the monitoring model and the monitored model are deployed on the UE.

[0024] Figure 5 A schematic diagram is shown, showing an example where a monitored model is deployed on a UE and a monitoring model is deployed on a gNB.

[0025] Figure 6 A schematic diagram is shown, showing an example where a monitored model is deployed on a gNB and a monitoring model is deployed on a UE.

[0026] Figure 7 A schematic diagram is shown, showing an example where both the monitoring model and the monitored model are deployed on the gNB.

[0027] Figure 8 A schematic diagram is shown, showing an example of a bilateral model.

[0028] Fig. 9 A schematic diagram is shown, showing an example of a bilateral model.

[0029] Fig.10 A schematic diagram is shown, showing an example of a bilateral model.

[0030] Fig.11 A schematic diagram is shown showing examples of two autoencoder models for CSI feedback compression.

[0031] Fig.12 A schematic diagram is shown, illustrating a wireless communication system according to an embodiment of the present invention. DETAILED DESCRIPTION

[0032] With reference to the drawings, the technical matters, structural features, implementation objectives and effects of the embodiments of the present invention are described in detail. Specifically, the terms in the embodiments of the present invention are only used to describe the purpose of the specific embodiments, rather than to limit the present invention.

[0033] Embodiments of the present invention relate to artificial intelligence (AI) and machine learning (ML) for New Radio (NR) air interface and solve the problems of data collection, model monitoring and model generalization.

[0034] In some embodiments, the performance of the AI / ML model is monitored as part of the AI / ML model lifecycle management. If the performance of the AI / ML model degrades, the AI / ML model can be retrained or switched to another AI / ML model. In some embodiments of the present invention, one AI / ML model is used to monitor the performance of another AI / ML model.

[0035] After the AI / ML model is deployed, the AI / ML model should be monitored to ensure proper operation. For example, an AI / ML model monitors the deployed AI / ML model to determine whether the beam predicted by the AI / ML model is accurate for beam management, whether the positioning performed by the AI / ML model is still accurate, and / or whether the reported CSI can be fully recovered.

[0036] In one scenario of AI / ML model monitoring, in order to continue to monitor the AI / ML model, real data is collected, reported and compared with the AI / ML model output.

[0037] For example, to determine whether the predicted beam is accurate, all the beams are measured at the prediction time (or prediction time window). The beam index (or ID) of the beam with the strongest reference signal received power (RSRP) and the strongest RSRP are collected and compared with the AI / ML model output.

[0038] For example, for an AI / ML model trained to provide Channel State Information (CSI) feedback, if the original CSI (the original CSI refers to the original CSI-RS value) is selected as the input of the AI / ML model, and the recovered CSI is the output of the decoder of the autoencoder AI / ML model. The original CSI itself must be reported to the gNB as real data and compared with the decoder output. If the difference between the two exceeds a level, the AI / ML model is not working properly and should be retrained, or switched to another model, or trigger a fallback to traditional non-AI methods such as codebook type I and codebook type II.

[0039] In another embodiment of AI / ML model monitoring, the confidence level is output as a measure of the output of the AI / ML model. For example, for an AI / ML model trained to provide positioning, the confidence level is calculated for the positioning result output by the AI / ML model.

[0040] For AI / ML model monitoring, real data needs to be collected, which is sometimes difficult to obtain or cumbersome to collect.

[0041] For example, for the AI / ML model trained to provide UE positioning, the real data is the real location of the UE. However, it is difficult to obtain the real location of the UE. The real location in the training data can be collected from a positioning reference unit (PRU) or some tags of a third party. Collecting the real data from PRU or tags is not suitable for timely monitoring of the AI / ML model in the field during normal operation of the UE.

[0042] For example, for the AI / ML model trained to provide beam management cases, beam prediction in the time domain aims to reduce latency. The collection of real data requires the transmission and measurement of reference signals, which is both time-consuming and resource-intensive.

[0043] Data collection for offline model training is not a big problem. However, data collection for AI / ML model monitoring can sometimes be challenging due to the limited amount of real data collected.

[0044] On the other hand, if the confidence of the monitored AI / ML model is selected as the monitoring key performance indicator (KPI), when the monitored AI / ML model becomes unreliable, the confidence also becomes unreliable. Therefore, the calculation of the confidence requires a more complex AI / ML model or a complex method.

[0045] A generalized AI / ML model is an AI / ML model that is trained to handle all unseen data sets. In machine learning, generalization is a definition that shows how a trained model can classify or predict unseen data. The training ability of a generalized AI / ML model can be called generalization ability. An appropriate way to evaluate the training ability of a generalized AI / ML model is to compare the performance of the generalized AI / ML model with the performance of a non-generalized AI / ML model for a specific scenario.

[0046] In some embodiments of the present invention, one AI / ML model is used to monitor another AI / ML model. For simplicity, AI / ML model, AI / ML model, and model are used interchangeably in the description. An AI / ML model used to monitor one or more AI / ML models is referred to as a monitoring model. The AI / ML model monitored by the monitoring AI / ML model is referred to as a monitored model.

[0047] The monitoring model can be deployed on different nodes, including UE, base station (e.g., gNB) or a third node. Each of the monitoring model and the monitored model can be a unilateral model or a bilateral model.

[0048] In the description of the embodiment of the present invention, model switching includes closing or deactivating one model and opening or activating another model.

[0049] The third node may include an application server, a gNB, or a UE.

[0050] Reference Figure 1 The telecommunication system including UE 10a, base station 20a, base station 20b and network entity device 30 executes the disclosed method according to the embodiment of the present invention. Figure 1 For illustrative purposes only, not limiting, the system may include more UE, BS and CN entities. The connections between devices and between device components are shown in the figure with lines and arrows. The UE 10a may include a processor 11a, a memory 12a and a transceiver 13a. The base station 20a may include a processor 21a, a memory 22a and a transceiver 23a. The base station 20b may include a processor 21b, a memory 22b and a transceiver 23b. The network entity device 30 may include a processor 31, a memory 32 and a transceiver 33. Each of the processors 11a, 21a, 21b and 31 may be configured to implement the proposed functions, procedures and / or methods described in this description. The layers of the wireless interface protocol may be implemented in the processors 11a, 21a, 21b and 31. Each of the memories 12a, 22a, 22b and 32 may be operable to store various programs and information to operate the connected processors. Each of the transceivers 13a, 23a, 23b and 33 is operably coupled to a connected processor to send and / or receive wireless signals. Each of the base stations 20a and 20b can be one of an eNB, a gNB or other wireless nodes.

[0051] Each of the processors 11a, 21a, 21b and 31 may include a general-purpose central processing unit (CPU), an application-specific integrated circuit (ASIC), other chipsets, logic circuits and / or data processing devices. Each of the memories 12a, 22a, 22b and 32 may include a read-only memory (ROM), a random access memory (RAM), a flash memory, a memory card, a storage medium and / or other storage devices. Each of the transceivers 13a, 23a, 23b and 33 may include a baseband circuit and a radio frequency (RF) circuit to process radio frequency signals. When the embodiments are implemented in software, the techniques described herein may be implemented by modules, programs, functions, entities, etc. that perform the functions. The modules may be stored in a memory and executed by the processor. The memory may be implemented inside the processor or outside the processor, where these may be communicatively coupled to the processor in various ways known in the art.

[0052] The network entity device 30 may be a node in CN. CN may include LTE CN or 5GC, which may include a user plane function (UPF), a session management function (SMF), a mobility management function (AMF), a unified data management (UDM), a policy control function (PCF), a control plane (CP) / user plane (UP) separation (CUPS), an authentication server (AUSF), a network slice selection function (NSSF) and the network exposure function (NEF).

[0053] Reference Figure 2 , the system 100 for the general aspects of machine learning in NR or NR air interface includes a data collection unit 101, a model training unit 102, an actuator 103 and a model inference 104. Note that, Figure 2The monitoring method is not necessarily limited to this example. The monitoring method is applicable to any design based on machine learning. The general steps include data collection and / or model training and / or model reasoning and / or actuator.

[0054] The data collection unit 101 is a function that provides input data to the model training unit 102 and the model reasoning unit 104. AI / ML algorithm-specific data preparation (e.g., data preprocessing and cleaning, formatting, and conversion) is not performed in the data collection unit 101.

[0055] Examples of input data may include measurements from multiple UEs or different network entities, feedback from actuator 103, and output from AI / ML models.

[0056] The training data is the data required as input to the AI / ML model training unit 102 .

[0057] The inference data is data required as input to the AI / ML model inference unit 104 .

[0058] The model training unit 102 is a function for performing the ML model training, verification and testing. If necessary, the model training unit 102 is also responsible for data preparation (e.g., data preprocessing and cleaning, formatting and conversion) based on the training data delivered by the data collection unit 101.

[0059] Model deployment / update between units 102 and 104 involves deploying or updating an AI / ML model (e.g., training a machine learning model 105a or 105b) to the model reasoning unit 104. The model training unit 102 uses the data unit as training data to train the machine learning model 105a, and generates a trained machine learning model 105b from the machine learning model 105a.

[0060] The model inference unit 104 is a function of providing AI / ML model inference output (e.g., prediction or decision). The AI / ML model inference output is the output of the machine learning model 105b. If necessary, the model inference unit 104 is also responsible for data preparation (e.g., data preprocessing and cleaning, formatting and conversion) based on the inference data delivered by the data collection unit 101.

[0061] The output shown between unit 103 and unit 104 is the inference output of the AI / ML model generated by the model inference unit 104.

[0062] The actuator 103 is a function that receives the output from the model reasoning unit 104 and triggers or executes corresponding actions. The actuator 103 can trigger actions for other entities or itself.

[0063] Feedback between unit 103 and unit 101 is information that may be needed to derive training or inference data or performance feedback. General Aspects of AI / ML Model Monitoring

[0064] General aspects of embodiments of the method are detailed below. Some embodiments use at least one AI / ML model to assist or monitor at least one other AI / ML model. The model monitoring can be a mode that can be activated or deactivated by at least one or any combination of {RRC, MAC-CE or DCI}. The ML model monitoring is typically performed within a time window. Within this time window, if the monitored model is continuously declared as faulty by the UE or gNB or a third node, the monitored model is determined to be faulty by the gNB or the third node or the UE. In response, a model switch or fallback to a non-AI method can be triggered. As an alternative, the ML model is determined to be faulty if the monitored model is declared as faulty at least once. As another alternative, the ML model is determined to be faulty if the number of times the monitored model is declared as faulty is greater than a threshold. The threshold is configured by the gNB or the third node for the UE / gNB.

[0065] Reference Figures 3 to 7 , the example of UE 10 described may include one of the UE 10a. The example of gNB 20 described may include the base station 20a or 20b. Note that although the gNB is described as an example of a base station in the following content, the disclosed method can be implemented in any other type of base station, such as an eNB or a base station beyond 5G. Uplink (Uplink, UL) transmission of control signals or data may be a transmission operation from the UE to the base station. Downlink (Downlink, DL) transmission of control signals or data may be a transmission operation from the base station to the UE. The disclosed method is described in detail as follows. The UE 10 and the base station (such as gNB 20) perform the monitoring method based on machine learning.

[0066] Figure 3 Embodiments of the disclosed method are shown. At least one wireless communication device performs a machine learning-based monitoring method. In one embodiment, the at least one wireless communication device may include a user equipment (UE). In another embodiment, the at least one wireless communication device may include a base station. In yet another embodiment, the at least one wireless communication device may include a combination of multiple UEs and base stations.

[0067] Reference Figure 3 , at least one wireless communication device performs a monitoring method for monitoring a machine learning (ML) model. The at least one wireless communication device may include a user equipment (UE), a base station, or a combination of a third node.

[0068] The at least one wireless communication device uses the first ML model as a monitored ML model to perform a cellular communication task (S010). For example, the cellular communication task includes one or more of channel state information (CSI) reporting, time domain beam prediction, spatial domain beam prediction, and user equipment (UE) positioning.

[0069] The at least one wireless communication device monitors the first ML model using a second ML model as a monitoring ML model ( S012 ).

[0070] The at least one wireless communication device evaluates performance of the monitored ML model based on the monitoring (S014).

[0071] In one embodiment, the monitored model deployment device that deploys and activates the monitored ML model is different from the monitored model training device that trains the monitored ML model. The monitored ML model is downloaded from the monitored model training device to the monitored model deployment device. The monitored ML model is deployed and activated in one of a user equipment (UE), a base station, or a third node. The monitored model deployment device may be one of the user equipment, the base station, or the third node. The monitoring ML model is trained in one of the user equipment, the base station, or the third node. The monitored model training device may be one of the user equipment, the base station, or the third node. The monitored ML model may be activated by a downlink control information (DCI) signal, a radio resource control (RRC) signal, or a medium access control (MAC) control element (CE).

[0072] In one embodiment, the monitored model deployment device that deploys and activates the monitored ML model is different from the model evaluation device that performs the evaluation. The result of the monitoring is reported from the monitored model deployment device to the model evaluation device. The monitored ML model is deployed and activated at one of a user equipment (UE), a base station, or a third node. The monitored model deployment device may be one of the user equipment, the base station, or the third node. The evaluation is performed at one of the user equipment, the base station, or the third node. The model evaluation device may be one of the user equipment, the base station, or the third node. The result of the monitoring includes at least one combination of an input and an output of the monitored ML model.

[0073] In one embodiment, the monitoring model deployment device that deploys and activates the monitoring ML model is different from the monitoring model training device that trains the monitoring ML model. The monitoring ML model is downloaded from the monitoring model training device to the monitoring model deployment device. The monitoring ML model is deployed and activated in one of a user equipment (UE), a base station, or a third node. The monitoring model deployment device may be one of the user equipment, the base station, or the third node. The monitoring ML model is trained in one of the user equipment, the base station, or the third node. The monitoring model training device may be one of the user equipment, the base station, or the third node. The monitoring ML model may be activated by a downlink control information (DCI) signal, a radio resource control (RRC) signal, or a media access control (MAC) control element (CE).

[0074] In one embodiment, the monitoring model deployment device that deploys and activates the monitoring ML model is different from the model evaluation device that performs the evaluation. The result of the monitoring is reported from the monitoring model deployment device to the model evaluation device. The monitoring ML model is deployed and activated at one of a user equipment (UE), a base station, or a third node. The monitoring model deployment device may be one of the user equipment, the base station, or the third node. The evaluation is performed at one of the user equipment, the base station, or the third node. The model evaluation device may be one of the user equipment, the base station, or the third node. The result of the monitoring includes at least one combination of the input and output of the monitoring ML model. The monitored model and the monitoring model are both deployed and activated on the UE:

[0075] Reference Figure 4 , the monitored model and the monitoring model are both deployed and activated on the UE 10. The monitored model and / or the monitoring model are trained on the gNB 20 or the third node.

[0076] The procedures of the embodiment are described in detail as follows:

[0077] The gNB 20 and / or the third node trains the monitoring model and the monitored model.

[0078] The monitoring model and the monitored model are both downloaded to the UE.

[0079] The monitoring model and the monitored model are both deployed on the UE.

[0080] The deployment of the monitoring model and the monitored model is confirmed by the UE, for example, by sending a message or a Hybrid Automatic Repeat Request (HARQ) Acknowledgment (ACK) or Negative Acknowledgment (NACK) to indicate whether the deployment is successful.

[0081] When the deployment is successful, both the monitoring model and the monitored model are activated on the UE.

[0082] After a period of time, the monitoring result (referred to as the monitoring result) is reported to the gNB 20. The monitoring result may be the difference between the output of the monitored model and the output of the monitoring model.

[0083] Additionally, the report may be a measure of the performance of the monitored model (referred to as model performance), such as the confidence or probability indicating the accuracy of the predictions of the monitored model.

[0084] In addition, if the monitored measurements continue to pass within the time window, the report is a status confirmation of the monitored model and does not trigger retraining or model switching. Otherwise, the report is a request for model retraining or model switching.

[0085] An example of signaling between the gNB 20 and the UE 10 is described in detail as follows. The gNB 20 configures the UE 10 by providing a configuration to the UE 10. In response to the configuration, the UE 10 monitors the monitored model using the monitoring model. The gNB 20 may send the configuration to the UE 10 via a downlink control information (DCI) signal, a radio resource control (RRC) signal, or a media access control (MAC) control element (CE).

[0086] The configuration may include a monitoring time window or a monitoring period.

[0087] The configuration may include a monitoring mode. The monitoring mode indicates whether the monitoring is performed by the monitoring ML model or by collecting real data to compare with the output of the monitored ML model. In one embodiment, the monitoring mode indicates that the UE 10 is monitored by another AI / ML model (i.e., monitored by the monitoring model). In some embodiments, the monitoring mode indicates that the UE 10 is monitored by collecting real data without the help of the monitoring model.

[0088] In addition, when the monitoring model is provided by gNB 20 or a third node, the default setting of the monitoring mode is to monitor using the configured monitoring model.

[0089] In addition, when the monitoring model is not provided by gNB 20 and / or the third node, the default setting of the monitoring mode is to monitor by collecting the real data without the assistance of the monitoring model.

[0090] The monitoring mode, the monitored model and / or the monitoring model may be activated / deactivated jointly or separately. An activation signal for model activation (e.g., activating the monitoring mode, the monitored model and / or the monitoring model) or a deactivation signal for model deactivation (e.g., deactivating the monitoring mode, the monitored model and / or the monitoring model) may be carried in a DCI signal, an RRC signal or a MAC-CE.

[0091] In one embodiment, activating the monitoring model and activating the monitoring mode are two different operations. Once the monitoring model is activated, the monitored model is in the monitoring mode.

[0092] Post-processing of AI / ML models, such as the monitored model and / or monitoring model, may include fine-tuning, model quantization, model distillation, model pruning, etc. The post-processing, for example, may further include adjusting or reducing the complexity of the AI / ML model or customizing the AI / ML model.

[0093] In one embodiment, the monitored model and the monitoring model are integrated into one model and downloaded from the gNB 20 to the UE 10 via the air interface. The gNB configures the UE for post-processing via at least one of RRC signaling, MAC-CE or DCI. Please note that at least one of {monitored model, monitoring model} can be obtained via post-processing of the ML model. Figures 4 to 11 The configuration signaling of the {monitored model and / or the monitoring model} obtained by post-processing the unilateral model and the bilateral model in the example described in the example is at least one of {RRC, MAC-CE or DCI signaling}.

[0094] In one embodiment, the UE 10 performs post-processing on the monitored model, that is, only the monitored model is obtained on the UE 10 through post-processing.

[0095] In one embodiment, the UE 10 performs post-processing on the monitoring model, that is, only the monitoring model is obtained on the UE 10 through post-processing.

[0096] In one embodiment, the UE 10 performs post-processing on the monitoring model and the monitored model. That is, the monitoring model and the monitored model are both obtained on the UE 10 through post-processing.

[0097] In addition, gNB 20 performs the post-processing.

[0098] In addition, the third node performs the post-processing.

[0099] In addition, the monitored model and the monitoring model are both trained on the third node. The monitored model is deployed and activated on the UE, and the monitoring model is deployed and activated on the gNB:

[0100] Reference Figure 5 , the monitored model is deployed and activated on UE 10, and the monitoring model is deployed and activated on gNB 20.

[0101] The monitored model may be trained or deployed on the gNB 20 or a third node. The procedure of the embodiment is described in detail as follows:

[0102] The monitored model is downloaded to the UE 10 .

[0103] The monitored model is deployed on the UE 10 .

[0104] The deployment is acknowledged by the UE 10, for example, by sending a message or a Hybrid Automatic Repeat Request (HARQ) Acknowledgement (ACK) or Negative Acknowledgement (NACK).

[0105] The monitored model is activated by UE 10 or gNB 20.

[0106] After a period of time, the output of the monitored model is reported to the gNB 20. For example, the output of the monitored model may include CSI, location information (coordinates) and / or beam information (beam index, RSRP).

[0107] The monitoring mode and / or the monitoring model is activated by gNB 20.

[0108] The UE 10 reports the input and output of the monitored model to the gNB 20. The input and output of the monitored model are further processed in the gNB 20 and input to the monitoring model.

[0109] In one embodiment, the report may include at least one of {real data, input of the monitored model, output of the monitored model, auxiliary information of the monitoring model}.

[0110] An example of signaling between the gNB 20 and the UE 10 is described in detail as follows.

[0111] The monitoring mode, the monitored model and / or the monitoring model may be activated / deactivated jointly or separately. An activation signal for model activation (e.g., activating the monitoring mode, the monitored model and / or the monitoring model) or a deactivation signal for model deactivation (e.g., deactivating the monitoring mode, the monitored model and / or the monitoring model) may be carried in a DCI signal, an RRC signal or a MAC-CE.

[0112] The auxiliary information sent from the UE 10 to the monitoring model of the gNB 20 may include at least one of the following: the current channel condition, L1-RSRP, L1-SINR, etc. RSRP stands for Reference Signal Received Power (RSRP), and SINR stands for Signal-to-Interference plus Noise Ratio (SINR). The monitored model is activated in the gNB and the monitoring model is activated in the UE:

[0113] Reference Figure 6 , the monitored model is activated in gNB 20 and the monitoring model is activated in UE 10. The monitored model can be trained or deployed on gNB 20 or a third node.

[0114] The procedures of the embodiment are described in detail below.

[0115] The monitoring model is downloaded to the UE 10 .

[0116] The monitoring model is deployed on the UE 10 .

[0117] The deployment is acknowledged by the UE 10, for example, by sending a message or a Hybrid Automatic Repeat Request (HARQ) Acknowledgement (ACK) or Negative Acknowledgement (NACK) to report whether the deployment was successful.

[0118] When the deployment is successful, the monitoring mode and / or the monitoring model is activated by gNB 20.

[0119] If the deployment is unsuccessful, the gNB 20 may redeploy the monitoring model or deploy another monitoring model.

[0120] After a period of time, the output of the monitoring model is reported to the gNB 20, such as location information (coordinates), beam information (beam index, RSRP) and confidence.

[0121] An example of signaling between the gNB 20 and the UE 10 is described in detail as follows.

[0122] The monitoring mode, the monitored model and / or the monitoring model may be activated / deactivated jointly or separately. An activation signal for model activation (e.g., activating the monitoring mode, the monitored model and / or the monitoring model) or a deactivation signal for model deactivation (e.g., deactivating the monitoring mode, the monitored model and / or the monitoring model) may be carried in a DCI signal, an RRC signal or a MAC-CE.

[0123] said reporting during said monitoring mode. The monitored model is activated on the gNB and the monitoring model is activated on the gNB:

[0124] Reference Figure 7 , the deployed monitored model and monitoring model are both deployed on gNB 20.

[0125] The monitored model may be trained or deployed on gNB 20 or a third node.

[0126] The procedures of the embodiment are described in detail as follows:

[0127] The gNB 20 notifies the UE 10 that the monitored model has been activated.

[0128] The UE 10 reports the input of the monitored model to the gNB 20.

[0129] The gNB 20 notifies the UE 10 that the monitoring model has been activated.

[0130] The UE 10 reports the input of the monitoring model to the gNB 20.

[0131] In some examples, the input to the monitored model and the input to the monitoring model may be the same.

[0132] In some examples, the input to the monitored model and the input to the monitoring model may be different. For example, the input to the monitored model and the input to the monitoring model may have different numbers of input beam indices and / or L1-RSRPs.

[0133] An example of signaling between the gNB 20 and the UE 10 is described in detail as follows.

[0134] The monitoring mode, the monitored model and / or the monitoring model may be activated / deactivated jointly or separately. An activation signal for model activation (e.g., activating the monitoring mode, the monitored model and / or the monitoring model) or a deactivation signal for model deactivation (e.g., deactivating the monitoring mode, the monitored model and / or the monitoring model) may be carried in a DCI signal, an RRC signal or a MAC-CE.

[0135] As an example, the report of the monitored model during the monitoring mode may be auxiliary information, including at least one of {Time of Arrival (TOA), Time Difference of Arrival (TDOA), channel condition, Doppler, L1-SINR}.

[0136] In addition, the AI / ML model can be run on the UE 10 / gNB 20 and monitored by a third node.

[0137] At least one of the monitoring model and the monitored model is trained at the third node.

[0138] The monitoring model and the monitored model are respectively transmitted to the UE 10 / gNB 20.

[0139] The UE 10 requests the monitoring model and / or the monitored model from the third node.

[0140] The third node may send the AI / ML model (monitoring model and / or the monitored model) to the UE 10, and the UE 10 sends the confirmation to the third node.

[0141] The gNB 20 requests the monitoring model and / or the monitored model from the third node. The third node may send the AI / ML model (monitoring model and / or the monitored model) to the gNB 20, and the gNB 20 sends the confirmation to the third node.

[0142] In addition, if the monitored model is located at the third node, the UE 10 may send the report to the third node instead of the gNB 20.

[0143] In addition, if the monitored model is located at the third node, the UE 10 can send the report to the gNB 20. Monitor the inputs and outputs of AI / ML models:

[0144] As an example, the input and output of the monitoring model have the same quantity / metric as the monitored model. For example, the monitoring model and the monitored model both output a prediction of the best beam in the time domain or the spatial domain from the same input. For example, the monitoring model and the monitored model both output the position of the UE 10 from the same input information (e.g., reference signal, time difference of arrival (TDOA), angle of arrival (AOA), channel impulse response (CIR)).

[0145] The difference between the monitoring model and the monitored model is a difference in complexity level. For example, the monitoring ML model has a first complexity level, the monitored ML model has a second complexity level, and the first complexity level is greater than the second complexity level.

[0146] As an example, the monitored model is a low complexity model, e.g., after post-processing. Typically, the monitoring model with high complexity has advanced performance. The UE 10 may execute AI / ML models of different complexity levels according to the UE capabilities of the UE 10. The UE 10 may frequently run low complexity AI / ML models, and less frequently run high complexity models only for model monitoring.

[0147] In addition, the UE 10 may deploy a low complexity model as a monitored model. The high complexity model runs on the gNB 20 as a monitoring model.

[0148] Another example is model generalization:

[0149] A generalized model is referred to as an AI / ML model that can handle all unseen data subsets. Typically, a generalized AI / ML model has acceptable performance in a range of settings and scenarios. An AI / ML model for a specific scenario has better performance in that scenario. In order to measure the performance of a generalized AI / ML model in a specific setting, an AI / ML model for a specific scenario can be selected as the monitoring model or the baseline model in these settings.

[0150] The generalized AI / ML model can be deployed on the UE 10 or gNB 20 for interference. And the AI / ML model for specific scenarios can be deployed on the UE 10 or gNB 20 to measure the generalization ability of the generalized AI / ML model in these scenarios. If the output difference of the two models is too large, for example, greater than a threshold, the generalized AI / ML model is considered infeasible. Monitoring the output of AI / ML models: Synthetic data used as a substitute for real data

[0151] The monitoring mode indicates whether the monitoring is performed by the monitoring ML model or by collecting real data to compare with the output of the monitored ML model. The monitoring ML model generates synthetic data from auxiliary information, and in the evaluation, the synthetic data replaces the real data to compare with the output of the monitored ML model. The auxiliary information includes at least one of a channel condition, a beam index, a reference signal received power (RSRP), a signal to interference plus noise ratio (SINR), and a channel impulse response (CIR).

[0152] The real data may be difficult to obtain (eg, the real position of the UE 10) or cumbersome to acquire (eg, the best beam). Model monitoring usually works in a responsive manner. That is, after the performance degradation occurs, the model becomes unreliable.

[0153] The true value data may be replaced by synthetic data generated by an artificial intelligence / machine learning (AI / ML) model. The input of the AI / ML model may be the auxiliary information (e.g., channel status, beam index, L1-RSRP, L1-SINR, CIR), data / noise reference signal reported by a user equipment (UE). The output of the AI / ML model is the synthetic data.

[0154] The input of the AI / ML model may be historical data, such as the historical measurement results or monitoring results. The output of the AI / ML model may be the performance of the AI / ML model at the current time. Monitoring AI / ML model outputs: Confidence.

[0155] The input of the AI / ML model may be the auxiliary information (e.g., channel condition, beam index, L1-RSRP, L1-SINR, CIR), data / noise reference signal reported by UE. The output of the AI / ML model is the synthesized data. The output of the monitoring model is confidence.

[0156] In one example, the historical data may be the confidence level, the AI / ML model prediction confidence level, indicating whether the monitored model is likely to work properly within at least the subsequent time unit. Bilateral Model:

[0157] In one embodiment, the monitoring model comprises a bilateral model and has a first monitoring sub-model operating in a first model deployment device and a second monitoring sub-model operating in a second model deployment device.

[0158] The monitored model includes another bilateral model and has a first monitored sub-model operating in the first model deployment device and a second monitored sub-model operating in the second model deployment device.

[0159] The first model deployment device includes a UE, and the second model deployment device includes a base station.

[0160] In one embodiment, the monitoring is performed to monitor an output of the first monitoring submodel and an output of the first monitored submodel, and the evaluating is performed to evaluate the difference between the output of the first monitoring submodel and the output of the first monitored submodel.

[0161] In one embodiment, the monitoring is performed to monitor an output of the second monitoring submodel and an output of the second monitored submodel, and the evaluating is performed to evaluate a difference between the output of the second monitoring submodel and the output of the second monitored submodel.

[0162] The difference between the output of the first monitoring submodel and the output of the first monitored submodel or the difference between the output of the second monitoring submodel and the output of the second monitored submodel is calculated using at least one of mean square error (MSE), normalized mean square error (NMSE), cosine similarity, confidence and accuracy. Bilateral model for localization:

[0163] For the positioning case, the bilateral model works as follows: the first part (the first monitored sub-model NN1_1 or the first monitoring sub-model NN2_1) extracts features (time difference, angle of arrival (AOA), time of arrival (TOA), time difference of arrival (TDOA), etc.) from the input (channel impulse response / reference signal). The features are sent to the gNB 20 for further processing to obtain the output (UE 10 location information / coordinates).

[0164] The first monitored sub-model NN1_1 is coupled to the second monitored sub-model NN1_2. The first monitoring sub-model NN2_1 is coupled to the second monitoring sub-model NN2_2.

[0165] Compared with the first monitoring sub-model NN2_1 and the second monitoring sub-model NN2_2, the first monitored sub-model NN1_1 and the second monitored sub-model NN1_2 have better performance, for example, have more complex model structures and / or higher complexity.

[0166] For the bilateral model, the model monitoring may be performed separately.

[0167] Figure 8 An example of a bilateral model for positioning is shown. A second bilateral model (first monitoring submodel NN2_1 and second monitoring submodel NN2_2) used as the monitoring model is deployed to monitor the first bilateral model (first monitored submodel NN1_1 and second monitored submodel NN1_2) used as the monitored model.

[0168] The monitoring may be performed using at least one of the following methods.

[0169] The monitoring may be performed at the outputs of the first monitored sub-model NN1_1 and the first monitoring sub-model NN2_1. The measured key performance indicator (KPI) may be at least one of the following: MSE, NMSE, cosine similarity, confidence, and accuracy.

[0170] The monitoring may be performed at the outputs of the first monitoring sub-model NN2_1 and the second monitoring sub-model NN2_2. The measured KPI may be at least one of the following: MSE, NMSE, cosine similarity, confidence, and accuracy.

[0171] Alternatively, at least one of {the first monitoring sub-model NN2_1 and the second monitoring sub-model NN2_2} is deployed at a third node.

[0172] In another example, for the bilateral model, the model monitoring may be jointly performed at the UE 10 side.

[0173] In one embodiment, the second monitoring sub-model and the second monitored sub-model are integrated into one sub-model.

[0174] Fig. 9A bilateral model for positioning is shown. A bilateral model (a first monitoring submodel NN2_1 and a second submodel NN12) is deployed to monitor another bilateral model (a first monitored submodel NN1_1 and a second submodel NN12). On the gNB side, the second submodel NN12 can process the outputs of the first monitored submodel NN1_1 and the second monitored submodel NN1_2.

[0175] The monitoring may be performed using at least one of the following methods.

[0176] The monitoring may be performed on a difference between an output of the first monitored submodel NN1_1 and an output of the second monitoring submodel NN2_2. The KPI for measuring the difference between an output of the first monitored submodel NN1_1 and an output of the second monitoring submodel NN2_2 may be at least one of the following: MSE, NMSE, cosine similarity, confidence, and accuracy.

[0177] The monitoring may be performed on a difference between an output of the second submodel NN12 obtained from the output of the first monitored submodel NN1_1 and an output of the second submodel NN12 obtained from the second monitoring submodel NN2_2. The KPI for measuring the difference may be at least one of: MSE, NMSE, cosine similarity, confidence, and accuracy.

[0178] Alternatively, at least one of the second sub-models NN12 is deployed at a third node.

[0179] In another example, for the bilateral model, the monitoring may be performed jointly at the gNB side.

[0180] In one embodiment, the first monitoring sub-model and the first monitored sub-model may be integrated into one sub-model.

[0181] Fig.10 A bilateral model for positioning is shown. A bilateral model (a first submodel NN3_1 and a second monitoring submodel NN3_2') is deployed to monitor another bilateral model (a first submodel NN3_1 and a second monitored submodel NN3_2). On the gNB side, the second monitoring submodel NN3_2 receives the output of the first submodel NN3_1 as an input of the second monitoring submodel NN3_2, and the second monitored submodel NN3_2' receives the output of the first submodel NN3_1 as an input of the second monitored submodel NN3_2'. The second monitored submodel NN3_2' is used as a monitoring model to monitor the performance of the second monitoring submodel NN3_2.

[0182] The process of the embodiment is described in detail as follows:

[0183] The gNB 20 notifies the UE 10 of the deployment of the second monitored sub-model NN3_2' as the monitoring model. Monitoring of the first sub-model NN3_1 and the second monitoring sub-model NN3_2 is performed at the gNB 20.

[0184] In another example, the second monitored sub-model NN3_2' is deployed at a third node. The output of the first sub-model NN3_1 is reported to the gNB 20 and the third node simultaneously. Channel State Information (CSI) Feedback Compression:

[0185] In one embodiment, the monitoring model includes a first autoencoder for reporting CSI, the first monitoring sub-model serves as a first encoder of the first autoencoder for compressing the CSI, and the second monitoring sub-model serves as a first decoder of the first autoencoder for decompressing the compressed CSI from the first encoder.

[0186] In one embodiment, the monitored model includes a second autoencoder for reporting CSI, the first monitored sub-model is used as a second encoder of the second autoencoder for compressing CSI, and the second monitored sub-model is used as a second decoder of the second autoencoder for decompressing the compressed CSI from the second encoder.

[0187] In one embodiment, the first autoencoder and the second autoencoder report CSI according to a configured monitoring period. In one embodiment, the monitoring period is reported by the UE. In one embodiment, the base station determines the monitoring period according to UE capabilities.

[0188] For the bilateral model, such as the autoencoder, a second model is deployed to measure the performance of the first model.

[0189] In one embodiment, the first model and the second model may have different complexities.

[0190] Alternatively, the first model and the second model may have different encoder output quantization levels. For example, the monitoring ML model has a first quantization level, the monitored ML model has a second quantization level, and the first quantization level is greater than the second quantization level.

[0191] Fig.11Two autoencoder models for CSI feedback compression and CSI feedback are shown. One autoencoder (referred to as the first autoencoder) may include an encoder A10 and a decoder A11. The other autoencoder (referred to as the second autoencoder) may include an encoder A20 and a decoder A21.

[0192] The first autoencoder may be a more complex model, while the second autoencoder is a simplified model. The first autoencoder may be a touchstone or benchmark for the performance of the second autoencoder.

[0193] Typically, the raw CSI and / or eigen vector needs to be reported by the UE with high accuracy as the ground truth for model monitoring and / or data collection. In this way, two autoencoder models are deployed and the ground truth is no longer needed in model monitoring. Therefore, the feedback overhead of reporting the raw CSI and / or eigen vector as the ground truth for model monitoring and / or data collection is reduced.

[0194] The principle for determining an AI / ML model as the touchstone (monitoring model) may be a stricter performance metric, such as a stricter cosine similarity, minimum mean square error (MMSE) / throughput, and / or prediction accuracy (e.g., 95% prediction accuracy).

[0195] As another example, the second model or both models may be obtained, for example, by post-processing the same AI / ML model. Thus, initially, a model is downloaded from the gNB 20 to the UE 10 if the autoencoder model is trained at the gNB 20.

[0196] The compression vector is reported by the UE 10. The process of configuring the UE 10 to run two AI / ML models is described in detail below. The gNB 20 may configure the UE 10, and the UE 10 reports the two compression vectors.

[0197] The UE 10 reports the compression vector once. When radio resources are limited, the UE 10 reports the compression vector according to a priority rule.

[0198] In one embodiment, when the plurality of compressed vectors are reported and overlapped, the compressed vector with a larger size is discarded. When the compressed vectors have the same size, the compressed vector with a larger age is discarded. Alternatively, the UE 10 randomly discards a vector.

[0199] Alternatively, the compressed vector a with a smaller size is discarded to ensure performance. Since the smaller the compression ratio, the better the performance. When the compressed vectors have the same size, the compressed vector with a larger age is discarded. Alternatively, the UE 10 randomly discards a vector.

[0200] The monitoring period is the UE capability of the UE 10. The monitoring period may be reported by the UE 10. Alternatively, the gNB 20 may explicitly determine the monitoring period based on the UE capability (storage, central processing unit (CPU), memory, floating point operations per second (FLOPS) ...) of the UE 10. After receiving the UE capability from the UE 10, the gNB 20 configures the monitoring period.

[0201] Alternatively, the gNB 20 configures the monitoring period regardless of the UE capability of the UE 10.

[0202] In one embodiment, the UE 10 reports the compression vectors at different periods.

[0203] The monitoring period is the UE capability of the UE 10. The monitoring period may be reported by the UE 10. Alternatively, the gNB 20 may explicitly determine the monitoring period based on the UE capability (storage, CPU, memory, FLOPS, ...) of the UE 10. After receiving the UE capability from the UE 10, the gNB 20 configures the monitoring period.

[0204] Alternatively, the gNB 20 configures the monitoring period regardless of the UE capability of the UE 10.

[0205] Please note that the method in this embodiment is not limited to the model monitoring. When multiple compression vectors must be reported with limited resources, the UE 10 reports the compression vectors according to a priority rule.

[0206] In one embodiment, the compressed vector with a larger age is discarded. When the compressed vectors have the same age, the compressed vector with a larger size is discarded. Alternatively, the UE 10 randomly discards a vector.

[0207] Note that the deployment of the monitoring model and the monitored model for the CSI feedback compression may follow a similar approach to the bilateral positioning model, such as being performed jointly or separately on the gNB 20 / UE 10 side, or even in a third node. The relevant signaling may be reported to the third node or reported to the gNB 20 and forwarded to the third node. ML-assisted non-AI (applicable to all use cases):

[0208] In another example, the monitoring model outputs the synthetic data in place of the true value. The synthetic data is generated for further processing. One use of the synthetic data is model monitoring.

[0209] The synthesized data is generated according to auxiliary information, for example, current channel conditions, indoor environment, outdoor environment, moving speed of the UE 10, Doppler frequency shift, etc.

[0210] The auxiliary information and synthetic data can be collected and sent to the node as hyperparameters of an AI / ML model. In one example, the AI / ML model is the monitoring model that can generate the synthetic data.

[0211] When the monitoring model is deployed at the UE 10, the UE 10 may obtain the auxiliary information used for the monitoring model, such as the current channel condition.

[0212] When the monitoring model is deployed at the gNB 20 / third node, the UE 10 may report the auxiliary information to the gNB 20 / third node.

[0213] In the example of training a monitored model for providing positioning, it is difficult to obtain the true value of the real location. The UE 10 can approach the positioning reference unit (PRU) or a tag to obtain its true location at a time. The reference is the PRU. The UE 10 can only monitor with reference to some tags or PRUs. This limits the model monitoring at the UE 10. An AI / ML model for generating data (referred to as a data generating neural network) can be deployed to generate the input data (e.g., the CIR, TOA, reference signal) and output data. The input data and the output data of the AI / ML model for generating data (referred to as a data generating neural network) are used by the monitoring model as corresponding synthetic data to replace the true value to monitor whether the monitored model works properly using the synthetic data.

[0214] For example, the input data in the synthesized data may include one or more of CIR, reference signal (RS), TDOA, AOA, angle of departure (AOD), and channel condition. For example, the output data in the synthesized data may include location information. The synthesized data includes input data for the monitoring model, and the input data is associated with the output data of the monitoring model. For example, the format of the synthesized data includes: {Input [CIR / RS / TDOA / AOA / AOD / channel status], output [location information]}.

[0215] In the example of training a supervised model for generating beam management data, collecting ground truth data is both resource intensive and time consuming.

[0216] For example, the input data in the synthesized data may include one or more subsets of all beams. For example, the output data in the synthesized data may include one or more predicted beams.

[0217] The synthesized data includes the input data associated with the output data, for example in the following format:

[0218] {Input [a subset of all beams], output [a predicted beam]} or {Input [measured values ​​of a subset of all beams], output [measured values ​​of the predicted beams in time or space]}. {Input [the measured values ​​of a subset of all beams and the beam IDs of these subset beams], output [measured values ​​of the predicted beams in time or space and the beam IDs of these predicted beams]}. Monitoring plan:

[0219] The gNB 20 configures the UE 10 by providing a configuration about one or more monitoring schemes to the UE 10. The configuration is at least one of {Radio Resource Control (RRC) signaling, Medium Access Control (MAC)-Control Element (CE) or Downlink Control Information (DCI) signaling}. In response to the configuration, the UE 10 uses one or more monitoring schemes. The one or more monitoring schemes may be selected from one or more of the following: Collect and / or report said truth values; Collect or generate and / or report synthetic data to replace true values; and • Using the same type of inputs and / or outputs of the monitoring model and the monitored model.

[0220] More specifically, the monitoring ML model monitors the first ML model according to an activated monitoring scheme among a plurality of monitoring schemes, and the plurality of monitoring schemes include: Comparing the output of the monitored ML model with the ground truth for the monitored ML model; comparing the output of the monitored ML model with synthetic data used to replace the true value; and Comparing the output of the monitored ML model with the output of the monitoring model based on the same inputs of the monitored ML model and the monitoring model.

[0221] One of the multiple monitoring schemes is activated as the activated monitoring scheme according to the configuration. The configuration may be carried in a DCI signal, an RRC signal or a MAC CE.

[0222] The configuration is configured by RRC signal or MAC-CE and activated by DCI signaling.

[0223] The configuration is configured by RRC signal or MAC-CE and deactivated by DCI signaling. For multiple monitoring models, the specific positions of the monitoring model and the monitored model can be Figures 4 to 11 Any combination of some of all of the described examples. Multiple Models:

[0224] In one embodiment, the monitoring model includes multiple component monitoring models instead of a single model, similar to the random forest in ML. The outputs of the multiple component monitoring models are mathematically processed, such as by averaging, maximizing, minimizing, etc., as the benchmark of the component monitoring model. The benchmark is regarded as the output of the monitoring model or the result of the monitoring. The result of the monitoring model is obtained by averaging or voting. Voting means that the majority wins. When most monitoring models (or most monitoring sub-models) indicate that the monitored model has failed, the monitored model is determined to have failed. Alternatively, if at least one solution indicates that the monitored model has failed, the monitored model is determined to have failed. Alternatively, if all solutions indicate that the monitored model has failed, the monitored model is determined to have failed.

[0225] As configured by the gNB 20 or the third node in the configuration, the monitoring model may be configured to include only one component monitoring model or multiple component monitoring models according to the configuration. The reference of the monitoring model may be carried in a DCI signal, an RRC signal, or a MAC CE. The configuration may be carried in a DCI signal, an RRC signal, or a MAC CE to notify the UE 10.

[0226] By default, the configuration is for a single AI / ML model as a monitor model.

[0227] Fig.12 7 is a block diagram of an example system 700 for wireless communication according to an embodiment of the present disclosure. The embodiments described herein may be implemented into the system using any suitably configured hardware and / or software. Fig.12The system 700 is shown and includes a radio frequency (RF) circuit 710, a baseband circuit 720, a processing unit 730, a memory / storage device 740, a display 750, a camera 760, a sensor 770, and an input / output (I / O) interface 780 coupled to each other as shown.

[0228] The processing unit 730 may include circuits, such as but not limited to one or more single-core or multi-core processors. The processor may include any combination of general-purpose processors and special-purpose processors, such as a graphics processor and an application processor. The processor may be coupled to the memory / storage device and configured to execute instructions stored in the memory / storage device to enable various applications and / or operating systems running on the system to run.

[0229] The radio control functions may include, but are not limited to, signal modulation, encoding, decoding, radio frequency offset, etc. In some embodiments, the baseband circuit may provide communications compatible with one or more radio technologies. For example, in some embodiments, the baseband circuit may support communications with 5G NR, Long Term Evolution (LTE), Evolved Universal Terrestrial Radio Access Network (EUTRAN) and / or other wireless metropolitan area networks (WMAN), wireless local area networks (WLAN), and wireless personal area networks (WPAN). An embodiment in which the baseband circuit is configured to support radio communications of more than one wireless protocol may be referred to as a multi-mode baseband circuit. In various embodiments, the baseband circuit 720 may include circuits for operating signals that are not strictly considered to be in the baseband frequency. For example, in some embodiments, the baseband circuit may include circuits for operating signals having an intermediate frequency, which is between the baseband frequency and the radio frequency.

[0230] In various embodiments, the system 700 may be a mobile computing device, such as, but not limited to, a laptop computing device, a tablet computing device, a netbook, an ultrabook, a smart phone, etc. In various embodiments, the system may have more or fewer components and / or a different architecture. Where appropriate, the methods described herein may be implemented as a computer program. The computer program may be stored on a storage medium, such as a non-transitory storage medium.

[0231] The described embodiments of the present disclosure are a combination of techniques / processes that may be adopted in 3GPP specifications to create a final product.

[0232] If the software functional unit is implemented, used and sold as a product, it can be stored in a readable storage medium in a computer. Based on this understanding, the technical solution proposed in the present disclosure can be basically or partially implemented in the form of the software product. Alternatively, a part of the technical solution that is beneficial to the conventional technology can be implemented in the form of the software product. The software product in the computer is stored in a storage medium, including a plurality of commands for a computing device (such as a personal computer, a server or a network device) to run all or some of the steps disclosed in the embodiment of the present disclosure. The storage medium includes a USB flash drive, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a floppy disk or other types of media capable of storing program code.

[0233] The present disclosure provides a monitoring method for monitoring an ML model. The invention provides an embodiment for solving problems in monitoring an AI / ML model. In some embodiments of the present disclosure, the monitoring of an AI / ML model is no longer limited to the time required to collect true value data, and the AI / ML model can be monitored in a timely manner.

[0234] The embodiments of the present disclosure may be applied to evaluate model generalization, including monitoring and evaluating the generalized AI / ML model.

[0235] In some embodiments of the present disclosure, synthetic truth data is synthesized and treated as truth data to assist in active switching or pre-training of AI / ML models, thereby improving system performance.

[0236] While the present disclosure has been described in conjunction with what is considered to be the most practical and preferred embodiments, it is to be understood that the present disclosure is not limited to the disclosed embodiments but is intended to cover various arrangements made without departing from the scope of the broadest interpretation of the appended claims.

Claims

1. A monitoring method for monitoring a machine learning (ML) model, which may be performed in at least one wireless communication device, comprising: performing a cellular communication task using the first ML model as a monitored ML model; Using a second ML model as a monitoring ML model to monitor the first ML model; as well as Evaluate performance of the monitored ML model based on the monitoring.

2. The method according to claim 1, characterized in that The cellular communication task includes one or more of channel state information CSI reporting, beam prediction in the time domain, beam prediction in the spatial domain, and positioning of a user equipment UE.

3. The method according to claim 1, characterized in that: A monitored model deployment device that deploys and activates the monitored ML model is different from a monitored model training device that trains the monitored ML model; and The monitored ML model is downloaded from the monitored model training device to the monitored model deployment device.

4. The method according to claim 3, characterized in that The monitored ML model is deployed and activated at one of a user equipment UE, a base station or a third node; and The monitoring ML model is trained at one of the user equipment, the base station or the third node.

5. The method according to claim 4, characterized in that The monitored ML model is activated by a downlink control information DCI signal, a radio resource control RRC signal or a medium access control MAC control element CE.

6. The method according to claim 1, characterized in that The monitored model deployment device that deploys and activates the monitored ML model is different from the model evaluation device that performs the evaluation; and The monitoring result is reported from the monitored model deployment device to the model evaluation device.

7. The method according to claim 6, characterized in that The monitored ML model is deployed and activated at one of a user equipment UE, a base station or a third node; and The evaluating is performed at one of the user equipment, the base station or the third node.

8. The method according to claim 6, characterized in that The result of the monitoring includes at least one combination of an input and an output of the monitored ML model.

9. The method according to claim 1, characterized in that: The monitoring model deployment device that deploys and activates the monitoring ML model is different from the monitoring model training device that trains the monitoring ML model; and The monitoring ML model is downloaded from the monitoring model training device to the monitoring model deployment device.

10. The method according to claim 9, characterized in that The monitoring ML model is deployed and activated at one of the user equipment UE, the base station or the third node; and The monitoring ML model is trained at one of the user equipment, the base station or the third node.

11. The method according to claim 10, characterized in that The monitoring ML model is activated by a DCI signal, an RRC signal or a MAC CE.

12. The method according to claim 1, characterized in that The monitoring model deployment device that deploys and activates the monitoring ML model is different from the model evaluation device that performs the evaluation; and The monitoring result is reported from the monitoring model deployment device to the model evaluation device.

13. The method according to claim 12, characterized in that The monitoring ML model is deployed and activated at one of the user equipment UE, the base station or the third node; and The evaluating is performed at one of the user equipment, the base station or the third node.

14. The method according to claim 12, characterized in that The result of the monitoring includes at least one combination of an input and an output of the monitoring ML model.

15. The method according to claim 1, characterized in that The monitoring mode indicates whether the monitoring is performed by the monitoring ML model or by collecting ground truth data to compare with the output of the monitored ML model.

16. The method according to claim 15, characterized in that The monitoring ML model generates synthetic data from the auxiliary information, and the synthetic data replaces the true data in the evaluation to be compared with the output of the monitored ML model.

17. The method according to claim 16, characterized in that The auxiliary information includes at least one of channel status, beam index, reference signal received power RSRP, signal to interference plus noise ratio SINR, channel impulse response CIR, indoor environment, outdoor environment, UE moving speed and Doppler frequency shift.

18. The method according to claim 16, characterized in that The synthesized data includes input data for the monitoring model, and the input data of the monitoring model is associated with output data of the monitoring model.

19. The method according to claim 1, characterized in that The monitoring ML model has a first complexity level, the monitored ML model has a second complexity level, and the first complexity level is greater than the second complexity level.

20. The method according to claim 1, characterized in that The monitoring model comprises a scenario-specific ML model, and the monitored model comprises a generalized ML model.

21. The method according to claim 1, characterized in that Both the monitoring model and the monitored model output predictions of the best beam in the time or spatial domain from the same input.

22. The method according to claim 1, characterized in that Both the monitoring model and the monitored model output UE positions from the same input.

23. The method according to claim 1, characterized in that The monitoring model includes a bilateral model and has a first monitoring sub-model operating in a first model deployment device and a second monitoring sub-model operating in a second model deployment device; and The monitored model includes another bilateral model and has a first monitored sub-model operating in the first model deployment device and a second monitored sub-model operating in the second model deployment device.

24. The method according to claim 23, characterized in that The first model deployment device includes a user equipment UE, and the second model deployment device includes a base station.

25. The method according to claim 23, characterized in that performing said monitoring to monitor an output of said first monitoring sub-model and an output of said first monitored sub-model, and performing said evaluating to evaluate a difference between said output of said first monitoring sub-model and said output of said first monitored sub-model; or The monitoring is performed to monitor an output of the second monitoring sub-model and an output of the second monitored sub-model, and the evaluating is performed to evaluate a difference between the output of the second monitoring sub-model and the output of the second monitored sub-model.

26. The method according to claim 25, characterized in that The first monitoring sub-model and the first monitored sub-model are integrated into one sub-model; or The second monitoring sub-model and the second monitored sub-model are integrated into one sub-model.

27. The method according to claim 25, characterized in that The difference between the output of the first monitoring submodel and the output of the first monitored submodel or the difference between the output of the second monitoring submodel and the output of the second monitored submodel is calculated using at least one of the mean square error MSE, the normalized mean square error NMSE, the cosine similarity, the confidence and the accuracy.

28. The method according to claim 23, characterized in that The monitoring model includes a first autoencoder for reporting CSI, the first monitoring sub-model is used as a first encoder of the first autoencoder for compressing CSI, and the second monitoring sub-model is used as a first decoder of the first autoencoder for decompressing the compressed CSI from the first encoder; and The monitored model includes a second autoencoder for reporting CSI, the first monitored sub-model is used as a second encoder of the second autoencoder for compressing CSI, and the second monitored sub-model is used as a second decoder of the second autoencoder for decompressing the compressed CSI from the second encoder.

29. The method according to claim 28, characterized in that The monitoring ML model has a first quantization level, the monitored ML model has a second quantization level, and the first quantization level is greater than the second quantization level.

30. The method according to claim 28, characterized in that The first autoencoder and the second autoencoder report CSI according to a configured monitoring period.

31. The method according to claim 30, characterized in that The monitoring period is reported by the UE; or The base station determines the monitoring period according to the UE capability.

32. The method according to claim 1, characterized in that The monitoring ML model monitors the first ML model according to an activated monitoring scheme among a plurality of monitoring schemes, and the plurality of monitoring schemes include: comparing an output of the monitored ML model to a ground truth for the monitored ML model; comparing the output of the monitored ML model with synthetic data used to replace the true value; and Based on the same input of the monitored ML model and the monitoring model, an output of the monitored ML model is compared with an output of the monitoring model.

33. The method according to claim 32, characterized in that One of the plurality of monitoring schemes is activated as the activated monitoring scheme according to the configuration.

34. The method according to claim 32, characterized in that The configuration is carried in a DCI signal, an RRC signal or a MACCE.

35. The method according to claim 32, characterized in that The configuration is configured by RRC signal or MAC-CE and activated by DCI signaling.

36. The method according to claim 32, characterized in that The configuration is configured by RRC signal or MAC-CE and deactivated by DCI signaling.

37. The method according to claim 1, characterized in that The monitoring model includes a plurality of component monitoring models, and outputs of the plurality of component monitoring models are mathematically processed as a benchmark of the component monitoring model.

38. The method according to claim 1, characterized in that The monitoring model may be configured to include only one component monitoring model or multiple component monitoring models depending on the configuration.

39. The method according to claim 37, characterized in that The configuration is carried in a DCI signal, an RRC signal or a MACCE.

40. A wireless communication device, comprising: A processor configured to call and run a computer program stored in a memory so that a device equipped with the processor executes the method according to any one of claims 1 to 39.

41. A chip, comprising: A processor configured to call and run a computer program stored in a memory so that a device equipped with the chip executes a method as claimed in any one of claims 1 to 39.

42. A computer-readable storage medium having a computer program stored therein, wherein the computer program causes a computer to execute the method according to any one of claims 1 to 39.

43. A computer program product comprising a computer program, wherein the computer program causes a computer to perform the method of any one of claims 1 to 39.

44. A computer program, wherein the computer program causes a computer to perform the method of any one of claims 1 to 39.

Images