Method and apparatus for AI / ML model monitoring

Abstract

A method executed by at least one processor in a user equipment (UE) includes receiving a set of resources from a base station, performing a first measurement of the set of resources based on a legacy mode that does not use an artificial intelligence / machine learning (AI / ML) model to generate a first output, performing a second measurement of the set of resources based on the AI / ML model to generate a second output, and reporting results corresponding to the first output and the second output to the base station.

Description

[Technical Field]

[0001] CROSS-REFERENCE TO RELATED APPLICATIONS This application claims priority to U.S. Provisional Application No. 63 / 421,724, filed November 2, 2022, the disclosure of which is incorporated herein by reference in its entirety.

[0002] Apparatus and methods consistent with example embodiments of the present disclosure relate to artificial intelligence machine learning (AI / ML) model performance monitoring, and more particularly, to AI / ML model performance monitoring for Radio Access Network (RAN) procedures. [Background technology]

[0003] Currently, a 3GPP® framework for AI / ML models of the air-interface is being considered, addressing each target use case with respect to aspects such as performance, complexity, and potential specification impacts. Some of these use cases focus on channel state information (CSI) feedback enhancements (e.g., reduced overhead, improved accuracy, prediction), beam management (e.g., beam prediction in the time and / or spatial domain for reduced overhead and latency, improved beam selection accuracy), and enhanced positioning accuracy for different scenarios, including, for example, those with heavy non-line-of-sight (NLOS) conditions.

[0004] These use cases can be categorized into sub-use cases for characterization and baseline performance evaluations. The AI / ML model approaches for the selected sub-use cases should be diverse enough to support various requirements for gNB-UE cooperation levels. The selection of use cases can be targeted at the formulation of a framework for applying AI / ML models to the air interface for these and other use cases.

[0005] AI / ML model, terminology, and descriptions are developed to identify common and specific characteristics for framework exploration, including characterizing the definition stages and associated complexity of AI / ML model-related algorithms, such as model generation (e.g., model training (including input / output, pre / post-processing, online / offline, as appropriate), model validation, model testing), and inference operations (e.g., input / output, pre / post-processing, as appropriate).

[0006] The framework study can identify various levels of collaboration between the UE and gNB relevant to the selected use case, including, for example, (1) no collaboration: only implementation-based AI / ML model algorithms without information exchange for comparison purposes, and (2) various levels of UE / gNB collaboration covering separate or joint ML operations. The framework study can further characterize the lifecycle management of the AI / ML model (e.g., model training, model deployment, model inference, model monitoring, model update).

[0007] The framework study can utilize datasets for training, validation, testing, and inference, and can identify common notations and terminology for functions, procedures, and interfaces related to AI / ML models.

[0008] In the related art, specific details and implementation forms of AI / ML model performance monitoring applicable to radio access network (RAN) procedures are unknown or not defined. In particular, after a certain period of time, an AI / ML model may require further training or improvement. Therefore, a procedure for comparing the performance of an AI / ML model with conventional methods is not defined. Summary of the Invention [Problem to be solved by the invention]

[0009] According to embodiments, systems and methods are provided for implementing mechanisms for AI / ML model performance monitoring in RAN (e.g., applicable to 3GPP NR). [Means for solving the problem]

[0010] According to an example embodiment, a method performed by at least one processor in a user equipment (UE) includes receiving a set of resources from a base station. The method includes performing a first measurement of the set of resources based on a legacy mode that does not use an artificial intelligence / machine learning (AI / ML) model to generate a first output. The method includes performing a second measurement of the set of resources based on the AI / ML model to generate a second output. The method includes reporting results corresponding to the first output and the second output to the base station.

[0011] According to an example embodiment, a UE includes at least one memory configured to store computer program code and at least one processor configured to access the at least one memory and operate as instructed by the computer program code, the computer program code including first receiving code configured to cause at least one of the at least one processor to receive a set of resources from a base station, first execution code configured to cause at least one of the at least one processor to perform a first measurement of the set of resources based on a legacy mode that does not use an artificial intelligence / machine learning (AI / ML) model to generate a first output, second execution code configured to cause at least one of the at least one processor to perform a second measurement of the set of resources based on the AI / ML model to generate a second output, and first reporting code configured to cause at least one of the at least one processor to report results corresponding to the first output and the second output to the base station.

[0012] According to an example embodiment, a non-transitory computer-readable medium is proposed having instructions stored thereon that, when executed by a processor in a UE, cause the processor to perform a method including receiving a set of resources from a base station, performing a first measuring of the set of resources based on a legacy mode that does not use an artificial intelligence / machine learning (AI / ML) model to generate a first output, performing a second measuring of the set of resources based on the AI / ML model to generate a second output, and reporting results corresponding to the first output and the second output to the base station.

[0013] Additional aspects will be set forth in part in the description that follows, and in part will be apparent from the description, or may be learned by practice of the presented embodiments of the present disclosure.

[0014] Features, aspects, and advantages of certain exemplary embodiments of the present disclosure are described below with reference to the accompanying drawings, in which like reference numerals refer to like elements. [Brief explanation of the drawings]

[0015] [Figure 1] FIG. 1 is a diagram of an example network device according to various embodiments of the present disclosure.

[0016] [Figure 2] FIG. 2 is a schematic diagram of an exemplary wireless communication system, in accordance with various embodiments of the present disclosure.

[0017] [Figure 3] FIG. 3 is an example time chart of the flow of events on the user equipment (UE) side, according to one or more embodiments of the present disclosure.

[0018] [Figure 4] 4(A) and 4(B) illustrate two operation modes (processing modes) configured in a UE in accordance with one or more embodiments of the present disclosure.

[0019] [Figure 5] FIG. 5 illustrates a flowchart of one embodiment of performing an AI / ML model supervision process in accordance with one or more embodiments of the present disclosure. DETAILED DESCRIPTION OF THE INVENTION

[0020] The following detailed description of exemplary embodiments refers to the accompanying drawings. The foregoing disclosure provides illustration and description, but is not intended to be exhaustive or to limit implementations to the precise form disclosed. Modifications and variations are possible in light of the above disclosure or may be acquired from practice of implementations. Moreover, one or more features or components of one embodiment may be incorporated into or combined with another embodiment (or one or more features of another embodiment). Furthermore, in the flowcharts and descriptions of operations provided below, it is understood that one or more operations may be omitted, one or more operations may be added, one or more operations may be performed concurrently (at least in part), or the order of one or more operations may be interchanged.

[0021] It will be apparent that the systems and / or methods described herein may be implemented in various forms, including hardware, firmware, or a combination of hardware and software. The actual specialized control hardware or software code used to implement the systems and / or methods is not intended to limit the implementation. Accordingly, the operation and behavior of the systems and / or methods are described herein without reference to specific software code. It will be understood that software and hardware can be designed to implement the systems and / or methods based on the description herein.

[0022] Although particular combinations of features are recited in the claims and / or disclosed herein, these combinations are not intended to limit the disclosure of possible implementations. Indeed, many of these features can be combined in ways not specifically recited in the claims and / or disclosed herein. Although each dependent claim listed below may depend directly on only one claim, the disclosure of possible implementations includes each dependent claim in combination with every other claim in the claim set.

[0023] No element, act, or instruction used herein should be construed as critical or essential unless expressly stated as such. Also, as used herein, the articles "a" and "an" are intended to include one or more items and may be used interchangeably with "one or more." Where only one item is intended, the term "one" or similar language is used. Also, as used herein, terms such as "has," "have," "having," "include," and "including" are intended to be open-ended terms. Furthermore, the phrase "based on" is intended to mean "based at least in part on," unless expressly stated otherwise. Furthermore, phrases such as "at least one of A and B" or "at least one of A or B" should be understood to include A only, B only, or both A and B.

[0024] Embodiments of the present disclosure relate to monitoring AI / ML models and taking actions based on the results of such monitoring (actions taken based on the monitoring output to improve or maintain system performance (e.g., throughput)). The AI / ML models may be used for inference on the UE side, the network side (e.g., in the gNB or core network), or both the UE and network sides. The AI / ML models may be used in the gNB, which does not exclude the case where the AI / ML models are used in another node in the network that also implements embodiments and methods of the present disclosure.

[0025] 1 is a diagram of an exemplary device for performing a translation service. Device 100 may correspond to any type of known computer, server, or data processing device. For example, device 100 may include a processor, a personal computer (PC), a printed circuit board (PCB) containing computing device, a minicomputer, a mainframe computer, a microcomputer, a telephonic computing device, a wired / wireless computing device (e.g., a smartphone, a personal digital assistant (PDA)), a laptop, a tablet, a smart device, or any other similarly functional device.

[0026] In some embodiments, as shown in FIG. 1, device 100 may include a set of components such as a processor 120, a memory 130, a storage component 140, an input component 150, an output component 160, and a communication interface 170.

[0027] Bus 110 may comprise one or more components that enable communication within (between) the set of components of device 100. For example, bus 110 may be a communications bus, a crossover bar, a network, etc. Although bus 110 is shown in FIG. 1 as a single line, bus 110 may be implemented using (providing multiple connections to) multiple (two or more) connections within the set of components of device 100. The present disclosure is not limited in this respect.

[0028] Device 100 may include one or more processors, such as processor 120. Processor 120 may be implemented in hardware, firmware, and / or a combination of hardware and software. For example, processor 120 may comprise a central processing unit (CPU), graphics processing unit (GPU), accelerated processing unit (APU), microprocessor, microcontroller, digital signal processor (DSP), field programmable gate array (FPGA), application-specific integrated circuit (ASIC), general-purpose single-chip or multi-chip processor or other programmable logic device, discrete gate or transistor logic circuitry, discrete hardware components, or any combination thereof designed to perform the functions described herein. A general-purpose processor may be a microprocessor, or any conventional processor, controller, microcontroller, or state machine. Processor 120 may also be implemented as a combination of computing devices, such as a combination of a DSP and a microprocessor, multiple microprocessors, one or more microprocessors in conjunction with a DSP core, or any other such configuration. In some embodiments, particular processes and methods may be performed by circuitry that is specific to a given function.

[0029] The processor 120 may control the overall operation of the device 100 and / or a set of components of the device 100 (e.g., the memory 130, the storage component 140, the input component 150, the output component 160, and the communication interface 170).

[0030] Device 100 may further comprise memory 130. In some embodiments, memory 130 may comprise random access memory (RAM), read-only memory (ROM), electrically erasable programmable ROM (EEPROM), flash memory, magnetic memory, optical memory, and / or another type of dynamic or static storage device. Memory 130 may store information and / or instructions for use (e.g., execution) by processor 120.

[0031] Storage component 140 of device 100 may store information and / or computer-readable instructions and / or code related to the operation and use of device 100. For example, storage component 140 may include a hard disk (e.g., a magnetic disk, an optical disk, a magneto-optical disk, and / or a solid-state disk), a compact disk (CD), a digital versatile disk (DVD), a universal serial bus (USB) flash drive, a Personal Computer Memory Card International Association (PCMCIA) card, a floppy disk, a cartridge, a magnetic tape, and / or another type of non-transitory computer-readable medium, along with a corresponding drive.

[0032] Device 100 may further comprise input component 150. Input component 150 may include one or more components that enable device 100 to receive information via user input (e.g., a touchscreen, a keyboard, a keypad, a mouse, a stylus, a button, a switch, a microphone, a camera, etc.), etc. Alternatively or additionally, input component 150 may include sensors for sensing information (e.g., a global positioning system (GPS) component, an accelerometer, a gyroscope, an actuator, etc.).

[0033] Output component 160 of device 100 may include one or more components that may provide output information from device 100 (e.g., a display, a liquid crystal display (LCD), a light emitting diode (LED), an organic light emitting diode (OLED), a haptic feedback device, a speaker, etc.).

[0034] Device 100 may further comprise a communication interface 170. Communication interface 170 may include a receiver component, a transmitter component, and / or a transceiver component. Communication interface 170 may enable device 100 to establish connections and / or transfer communications with other devices (e.g., a server, another device). Communication may be achieved via a wired connection, a wireless connection, or a combination of wired and wireless connections. Communication interface 170 may enable device 100 to receive information from and / or provide information to another device. In some embodiments, communication interface 170 may provide for communication with another device over a network such as a local area network (LAN), a wide area network (WAN), a metropolitan area network (MAN), a private network, an ad hoc network, an intranet, the Internet, an optical fiber-based network, a cellular network (e.g., a fifth generation (5G) network, a long term evolution (LTE) network, a third generation (3G) network, a code division multiple access (CDMA) network, etc.), a public land mobile network (PLMN), a telephone network (e.g., a public switched telephone network (PSTN)), etc., and / or a combination of these or other types of networks. Alternatively or additionally, communication interface 170 may provide for communication with another device over a device-to-device (D2D) communication link such as FlashLinQ, WiMedia, Bluetooth, ZigBee, Wi-Fi, LTE, 5G, etc. In other embodiments, communication interface 170 may include an Ethernet interface, an optical interface, a coaxial interface, an infrared interface, a radio frequency (RF) interface, etc.

[0035] Device 100 may be included in core network 240 and may perform one or more processes described herein. Device 100 may perform operations based on processor 120 executing computer-readable instructions and / or code, which may be stored by a non-transitory computer-readable medium, such as memory 130 and / or storage component 140. A computer-readable medium may refer to a non-transitory memory device. A memory device may include memory space within a single physical storage device and / or memory space distributed across multiple physical storage devices.

[0036] Computer readable instructions and / or code may be loaded into memory 130 and / or storage component 140 from another computer readable medium or from another device via communication interface 170. The computer readable instructions and / or code stored in memory 130 and / or storage component 140, when executed by processor 120, may cause device 100 to perform one or more processes described herein.

[0037] Alternatively, or in addition, hardwired circuitry may be used in place of or in combination with software instructions to implement one or more processes described herein. Thus, the embodiments described herein are not limited to any specific combination of hardware circuitry and software.

[0038] The number and arrangement of components shown in Figure 1 are provided as an example. In practice, there may be additional, fewer, different, or differently arranged components than those shown in Figure 1. Furthermore, two or more components shown in Figure 1 may be implemented within a single component, or a single component shown in Figure 1 may be implemented as multiple distributed components. Additionally or alternatively, a set of components shown in Figure 1 may perform one or more functions that are described as being performed by another set of components shown in Figure 1.

[0039] 2 illustrates an example wireless communication system in accordance with various embodiments of the present disclosure. The wireless communication system 200 (which may also be referred to as a wireless wide area network (WWAN)) may include one or more user equipment (UE) 210, one or more base stations 220, at least one transport network 230, and at least one core network 240. The device 100 (FIG. 1) may be incorporated into the UE 210 or the base station 220.

[0040] One or more UEs 210 can access at least one core network 240 and / or IP services 250 via connections to one or more base stations 220 over the RAN domain 224 and through at least one transport network 230. Examples of UEs 210 may include a cellular phone, a smartphone, a Session Initiation Protocol (SIP) phone, a laptop, a personal digital assistant (PDA), a satellite radio, a global positioning system (GPS), a multimedia device, a video device, a digital audio player (e.g., an MP3 player), a camera, a game console, a tablet, a smart device, a wearable device, a vehicle, an electric meter, a gas pump, a large or small kitchen appliance, a healthcare device, an implant, a sensor / actuator, a display, or any other similarly functioning device. Some of the one or more UEs 210 may be referred to as Internet-of-Things (IoT) devices (e.g., parking meters, gas pumps, toasters, vehicles, heart monitors, etc.). One or more UEs 210 may also be referred to as a station, a mobile station, a subscriber station, a mobile unit, a subscriber unit, a wireless unit, a remote unit, a mobile device, a wireless device, a wireless communication device, a remote device, a mobile subscriber station, an access terminal, a mobile terminal, a wireless terminal, a remote terminal, a handset, a user agent, a mobile agent, a client, or some other suitable terminology.

[0041] One or more base stations 220 may communicate wirelessly with one or more UEs 210 via the RAN domain 224. Each base station of the one or more base stations 220 may provide communication coverage to one or more UEs 210 located within the geographic coverage area of ​​the base station 220. In some embodiments, as shown in FIG. 2, a base station 220 may transmit one or more beamformed signals to one or more UEs 210 in one or more transmit directions. One or more UEs 210 may receive the beamformed signals from the base station 220 in one or more receive directions. Alternatively or additionally, one or more UEs 210 may transmit beamformed signals to the base station 220 in one or more transmit directions. The base station 220 may receive the beamformed signals from one or more UEs 210 in one or more receive directions.

[0042] The one or more base stations 220 may include macrocells (e.g., high-power cellular base stations) and / or small cells (e.g., low-power cellular base stations). Small cells may include femtocells, picocells, and microcells. The base station 220, whether a macrocell or a large cell, may include and / or may be referred to as an access point (AP), an evolved (or evolved Universal Terrestrial Radio Access Network (E-UTRAN)) Node B (eNB), a next-generation Node B (gNB), or any other type of base station known to those skilled in the art.

[0043] One or more base stations 220 may be configured to interface (e.g., establish connections, transfer data, etc.) with at least one core network 240 through at least one transport network 230. In addition to other functions, one or more base stations 220 may perform one or more of the following functions: forwarding data received from one or more UEs 210 (e.g., uplink data) to the at least one core network 240 via the at least one transport network 230; forwarding data received from the at least one core network 240 (e.g., downlink data) to one or more UEs 210 via the at least one transport network 230.

[0044] The transport network 230 may transport data (e.g., uplink data, downlink data) and / or signaling between the RAN domain 224 and the CN domain 244. For example, the transport network 230 may provide one or more backhaul links between one or more base stations 220 and at least one core network 240. The backhaul links may be wired or wireless.

[0045] The core network 240 may be configured to provide one or more services (e.g., enhanced mobile broadband (eMBB), ultra-reliable low latency communications (URLLC), massive machine type communications (mMTC), etc.) to one or more UEs 210 connected to the RAN domain 224 via the TN domain 234. As an example, the core network 240 performs translation services. Alternatively or additionally, the core network 240 may serve as an entry point for IP services 250. The IP services 250 may include the Internet, an intranet, an IP multimedia subsystem (IMS), streaming services (e.g., video, audio, games, etc.), and / or other IP services.

[0046] The AI / ML model may be used in various scenarios. For example, a double-sided AI / ML model may be used on the gNB side for encoder functions (e.g., compression, CSI compression, etc.) and on the UE side for decoder functions (e.g., decompression, etc.). In another example, an AI / ML model may be used on the UE side to predict a set of beams in the time domain and / or the spatial domain. For example, the UE may measure a first set of beams and use the measurements as input to the AI / ML model to predict a second set of beams. In another example, an AI / ML model may be used on the network side to predict a UE location using a set of measurements that may be reported by the UE. To perform a specific inference in a scenario, the UE and / or the network may perform measurements (e.g., measurements of reference signals) to predict the best set of resources (e.g., beams). If the inference is performed on the network side, the measurement results may be reported by the UE to the network, or the measurements may be performed on the network side.

[0047] The quality of the AI / ML model inference can be monitored. The quality of the AI / ML model inference can be determined by the system's key performance indicators (KPIs) when the inference is used in a system (e.g., in the case of a transmission scheme). For example, a UE can receive a set of beams (e.g., 64 beams), measure a subset of the received beams (e.g., 8 beams), infer the best beam for DL ​​transmission, and inform the gNB of this beam. The gNB can use this beam for transmission to the UE. The UE can compare the system KPIs (e.g., block error rate (BLER), throughput, spectral efficiency) when the inferred beam is used with the KPIs when the beam determined using a legacy method is used. If the KPIs of the beam determined using the legacy method are higher than the KPIs of the inferred beam, it can be determined that the AI / ML model may require further training or refinement.

[0048] In another example, the quality of the model inference can be measured by comparing the accuracy of the inference against the accuracy of a reference signal. For example, the CSI at the output of an AI / ML model-based decoder in the gNB can be compared to the CSI estimated and reported by the UE using legacy methods. A predefined metric (e.g., mean square error (MSE), a metric for measuring the difference) can be used to determine the accuracy of the AI / ML model. Other metrics can also be used to monitor the performance of the AI / ML model. For example, the metric can be the difference in reference signal received power (RSRP), etc.

[0049] In some embodiments, the mode of a procedure (e.g., transmission mode) may be determined by whether an AI / ML model is used for the procedure and further by the AI / ML model ID. For example, in a first transmission mode, the MCS may be determined using a legacy method without an AI / ML model, in a second transmission mode, the MCS may be determined using CSI from AI / ML model #0, and in a third mode, the MCS may be determined using CSI from AI / ML model #1.

[0050] System KPIs can be evaluated in the UE or the gNB. In some examples, the UE can calculate BLER, throughput, etc. and feed back the KPIs and / or functions of the KPIs (e.g., the difference between the KPIs) to the gNB. In other examples, the gNB can calculate the KPIs using feedback from the UE. For example, the gNB can calculate the KPIs using hybrid automatic repeat request (HARQ) feedback from the UE. Model accuracy evaluation may be a capability or function of the UE. For example, the UE can be configured to perform model performance evaluation.

[0051] According to some embodiments, AI / ML model monitoring may include multiple phases. A sample time diagram of the flow of events at the UE side is shown in FIG. 3. In the first phase, after receiving a set of resources (e.g., a reference signal) from the base station, the UE may generate multiple outputs. The outputs may be measurements of, for example, RSRP, CSI, channel impulse response (CIR), etc. of a beam. Although two outputs are shown in FIG. 3, the number of outputs may be one, three, or more. One output may be determined using a legacy method, and another output may be determined using an AI / ML model to generate inferences. In some examples, two or more AI / ML model outputs corresponding to different models may be generated.

[0052] The UE may use reference signal measurements to generate the output. For example, for CSI estimation, the UE may use CSI-RS resources. The legacy output may be a CSI estimate, and the AI / ML model output may be an AI / ML model encoder output, which may represent a compressed version of the estimated CSI. In this example, the same reference signal may be used to generate both outputs. In another example, the legacy output may be the best N downlink beams estimated using a reference signal, e.g., a synchronization signal block (SSB) or a CSI-RS. The AI / ML model output may be the best K beams (e.g., N may equal K) estimated using a subset of the reference signal. In a third example, the legacy output may be a time-of-arrival estimate using a position reference signal (PRS), and the AI / ML model output may be a time-of-arrival inference using a CIR. In some examples, the AI / ML model output may not be at the UE side (e.g., the UE does not perform inference). The UE may generate one or more outputs (e.g., time-of-arrival estimate and CIR) using one or more reference signals.

[0053] In the second phase, the output may be reported to the network. In the third phase, the gNB may perform transmissions (e.g., data transmissions) to the UE in the first and second modes. The number of modes is not limited to two. Transmission parameters for the first mode may be determined from the first output, and transmission parameters for the second mode may be determined from the second output, which may be referred to as association. Using the above example, the first mode may use a modulation coding scheme (MCS) determined from legacy CSI output, and the second mode may use an MCS determined from CSI inference of the AI / ML model. The final inference may be performed at the UE or gNB side. The UE may be instructed on resources for receiving different transmission modes. The UE may then calculate KPIs (e.g., BLER, throughput, etc.) for the different transmission modes and feed the KPIs back to the network. The KPIs may be fed back separately or in the same report. Depending on the specific use case used, one or more of these phases may not be present. For example, in a positioning use case, the network can determine the UE location based on the output fed back from the UE.

[0054] In some embodiments, the UE may be configured to report the output results in at least one of a periodic, aperiodic, and semi-static manner. For reporting, the UE may be configured with one or more of the following: one or more signals for calculating the reference output (legacy output), One or more signals that can be used as input to an AI / ML model (the AI / ML model can be on the UE side, the network side, or both); Reporting resources (e.g., time and frequency resources, channels used (e.g., physical uplink control channel (PUCCH), physical uplink shared channel (PUSCH), or medium access control (MAC) CE), reporting format, reporting quantity (e.g., RSRP, etc.)), Mapping between reports and signals, Mapping between AI / ML models and signals, and Mapping between reports and AI / ML models

[0055] The mapping between transmission modes and the powers associated with those modes may be indicated explicitly and / or implicitly to the UE using one or more, or a combination of, the following: Codepoints in the Downlink Control Information (DCI) may indicate a mapping, e.g., 00: transmission associated with legacy output, 01: transmission associated with AI / ML model #0, 10: transmission associated with AI / ML model #1. Radio network temporary identifier (RNTI) Time / frequency resources Bandwidth part (BWP) Physical downlink control channel (PDCCH) parameters (e.g., control resource set (CORESET) ID, search space ID)

[0056] The data used in the transmission modes (e.g., the first mode transmission and the second mode transmission in FIG. 3) may be the same. In some examples, HARQ may be disabled. In some examples, the KPIs may be averaged and / or filtered.

[0057] In some embodiments, the UE may be provided with a configuration for AI / ML model monitoring. This configuration may include parameters for AI / ML model inference (e.g., model inputs, resources on which the inputs are measured, AI / ML model ID, use case, time intervals at which monitoring may be performed, etc.) and / or parameters required for reporting. Exemplary use cases include, but are not limited to, using AI / ML models for CSI compression or beam prediction. Monitoring may be performed periodically, semi-statically, or aperiodically. Reporting may be periodically, semi-statically, or aperiodically. Reporting may be output reporting and / or KPI reporting.

[0058] An output report configuration can include a particular output type (e.g., legacy output, output from a particular AI / ML model), a reference signal from which the output is derived, a time window from which the output is derived, etc. A KPI report configuration can include a KPI type (e.g., BLER, throughput, etc.), a transmission to which the KPI is associated (e.g., the association may be by defining the resource to which the transmission occurs), an AI / ML model ID, etc.

[0059] According to some embodiments, the UE may operate or be configured to operate in a first mode as shown in FIG. 4(A).

[0060] For example, as shown in Figure 4(A), a UE may receive an instruction to activate model monitoring. This instruction may be carried in a MAC Control Element (CE) or DCI. Information included in the MAC CE may indicate which AI / ML model to monitor (e.g., model ID), which KPIs to use, whether filtering is used, filter parameters, observation length, etc. In some examples, the monitoring parameters may be configured, and the DCI and / or MAC CE may be used to indicate the specific configuration.

[0061] KPIs for a first mode of operation may be calculated and stored. For example, the monitoring interval per mode may be one discontinuous reception (DRX) period. KPIs for a second mode of operation may be calculated and stored. The KPIs may be fed back in the same report or in a different report. Reporting configuration may be performed by radio resource control (RRC). For example, the time instance at which the KPI report is sent may be known and may be used as a reference for the time of monitoring activation.

[0062] In another example, as shown in FIG. 4(B), KPI reporting can be configured without explicit monitoring activation. For example, KPIs may be reported periodically. Resources for different operation modes can be configured according to the periodic KPI reporting. For example, the times at which first mode transmission and second mode transmission occur may be known to the UE based on the KPI reporting (e.g., the UE performs first mode transmission and second mode transmission at a timing before KPI reporting is scheduled). In some examples, there may be separate KPI reports for different transmission modes.

[0063] According to some embodiments, the model monitoring scheme can be changed depending on whether any of the monitored KPIs are greater than or less than a configured threshold for a certain period of time. For example, after model selection / switching, the resulting KPIs may deteriorate because parameter adaptation may not be sufficient. Therefore, the UE or the network can configure a shorter monitoring period, and after a certain period of time, the model monitoring period can be shortened. In another example, if the BLER is greater than a configured threshold for a certain period of time, the network or the UE can configure the monitoring period to a shorter value, so that a model switch can be performed as soon as possible when any of the KPIs deteriorate below the threshold.

[0064] FIG. 5 shows a flowchart of one embodiment of an AI / ML process 500. The process may begin at operation S502, where a set of resources is received from a base station. The set of resources may be one or more beams or CSI-RS resources. The process proceeds to operation S504, where a first measurement of the set of resources is performed based on a legacy mode that does not use an AI / ML model to generate a first output. The process proceeds to operation S506, where a second measurement of the set of resources is performed based on an AI / ML model to generate a second output. For example, if one or more resources are beams, the legacy mode may measure each receive beam (e.g., 64), and the AI / ML model may measure a subset of the receive beams (e.g., 8) to predict the best beam. As another example, if one or more resources are CSI-RS resources, the legacy mode may provide estimated CSI as a first output, and the AI / ML model may provide another estimated CSI as a second output obtained from an input of a condensed version of the CSI-RS resources. The process continues to operation S508, where results corresponding to the first output and the second output are reported to the base station. The results may be, for example, measurements from which the base station determines the best beam. In another example, the UE may determine the best beam based on the first output reported to the base station and the best beam based on the second output. For example, using a legacy method, the UE may determine a first set of beams as the best beam, and using an AI / ML model, the UE may determine a second set of beams as the best beam, and the first set of beams and the second set of beams are reported to the base station.

[0065] The process proceeds to operation S510, where the UE receives resources from the base station in a first transmission mode. The process proceeds to operation S512, where the UE receives resources from the base station in a second transmission mode. For example, in the first transmission mode, the base station can transmit reference signals according to a first beam set determined using a legacy method, and in the second transmission mode, the base station can transmit reference signals according to a second beam set determined using an AI / ML model. The process proceeds to operation S514, where the UE transmits a report to the base station. For example, the report may be a KPI report reporting results of measurements from the first transmission mode and the second transmission mode.

[0066] The foregoing disclosure provides illustration and description, but is not intended to be exhaustive or to limit implementations to the precise forms disclosed. Modifications and variations are possible in light of the above disclosure or may be acquired from practice of implementations.

[0067] Some embodiments may relate to systems, methods, and / or computer-readable media at any possible technical detail level of integration. Additionally, one or more of the above components described above may be implemented as instructions stored on a computer-readable medium and executable by at least one processor (and / or may include at least one processor). The computer-readable medium may include computer-readable non-transitory storage medium(s) having computer-readable program instructions for causing a processor to perform operations.

[0068] A computer-readable storage medium may be a tangible device that can hold and store instructions for use by an instruction-execution device. A computer-readable storage medium may be, for example, but is not limited to, an electronic storage device, a magnetic storage device, an optical storage device, an electromagnetic storage device, a semiconductor storage device, or any suitable combination of the foregoing. A non-exhaustive list of more specific examples of computer-readable storage media includes portable computer diskettes, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), static random access memory (SRAM), portable compact disc read-only memory (CD-ROM), digital versatile disc (DVD), memory stick, floppy disk, mechanically encoded devices such as punch cards or ridge-in-groove structures with instructions recorded thereon, and any suitable combination of the foregoing. As used herein, computer-readable storage media should not be construed as being transitory signals per se, such as radio waves or other freely propagating electromagnetic waves, electromagnetic waves propagating through a waveguide or other transmission medium (e.g., light pulses passing through a fiber optic cable), or electrical signals transmitted through wires.

[0069] The computer-readable program instructions described herein can be downloaded from a computer-readable storage medium to each computing / processing device or to an external computer or external storage device via a network, such as the Internet, a local area network, a wide area network, and / or a wireless network. The network can include copper transmission cables, optical fiber transmissions, wireless transmissions, routers, firewalls, switches, gateway computers, and / or edge servers. A network adapter card or network interface in each computing / processing device receives the computer-readable program instructions from the network and forwards the computer-readable program instructions for storage in a computer-readable storage medium in the respective computing / processing device.

[0070] The computer-readable program code / instructions for carrying out operations may be either source code or object code written in any combination of one or more programming languages, including assembler instructions, instruction-set-architecture (ISA) instructions, machine instructions, machine-dependent instructions, microcode, firmware instructions, state-setting data, configuration data for integrated circuits, or object-oriented programming languages ​​such as Smalltalk, C++, and procedural programming languages ​​such as the "C" programming language or similar programming languages. The computer-readable program instructions may execute entirely on the user's computer, partially on the user's computer, as a stand-alone software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In the latter scenario, the remote computer may be connected to the user's computer via any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computer (e.g., via the Internet using an Internet Service Provider). In some embodiments, an electronic circuit, including, for example, a programmable logic circuit, a field programmable gate array (FPGA), or a programmable logic array (PLA), may execute computer-readable program instructions by utilizing state information of the computer-readable program instructions to personalize the electronic circuit to perform an aspect or operation.

[0071] These computer-readable program instructions may be provided to a processor of a general-purpose computer, a special-purpose computer, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute on the processor of the computer or other programmable data processing apparatus, create means for performing the functions / acts specified in one or more blocks of the flowcharts and / or block diagrams. These computer-readable program instructions may also be stored on a computer-readable storage medium that can direct a computer, programmable data processing apparatus, and / or other device to function in a particular manner, such that the computer-readable storage medium on which the instructions are stored includes a product containing instructions that implement aspects of the functions / acts specified in one or more blocks of the flowcharts and / or block diagrams.

[0072] The computer-readable program instructions may also be loaded onto a computer, other programmable data processing apparatus, or other device to cause the computer, other programmable apparatus, or other device to execute a series of operational steps to create a computer-implemented process, such that the instructions executing on the computer, other programmable apparatus, or other device perform the functions / acts specified in one or more blocks of the flowcharts and / or block diagrams.

[0073] The flowcharts and block diagrams in the figures illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer-readable media according to various embodiments. In this regard, each block in the flowcharts or block diagrams may represent a portion of a microservice(s), module, segment, or instruction set, which includes one or more executable instructions for implementing the specified logical function(s). The methods, computer systems, and computer-readable media may include additional, fewer, different, or differently arranged blocks than depicted in the figures. In some alternative implementations, the functions noted in the blocks may occur out of the order depicted in the figures. For example, two blocks shown in succession may actually be executed concurrently or substantially concurrently, or the blocks may sometimes be executed in the reverse order, depending on the functionality involved. It should also be noted that each block of the block diagrams and / or flowchart diagrams, and combinations of blocks in the block diagrams and / or flowchart diagrams, can be implemented by a dedicated hardware-based system that performs the specified functions or operations or executes a combination of dedicated hardware and computer instructions.

[0074] It will be apparent that the systems and / or methods described herein may be implemented in various forms, including hardware, firmware, or a combination of hardware and software. The actual specialized control hardware or software code used to implement these systems and / or methods is not intended to limit the implementation. Thus, the operation and behavior of the systems and / or methods are described herein without reference to specific software code, and it will be understood that software and hardware can be designed to implement the systems and / or methods based on the description herein.

[0075] Abbreviation: AI / ML Artificial Intelligence / Machine Learning BLER Block Error Rate BWP Bandwidth section CE Control Elements CORESET Control resource set CQI Channel Quality Indicator CIR Channel Impulse Response CRI CSI-RS Resource Indicator CSI Channel State Information CSI-RS Channel State Information Reference Signal CSI-RSRP CSI reference signal received power CSI-RSRQ CSI reference signal reception quality CSI-SINR CSI signal to interference and noise ratio DCI Downlink Control Information DL Downlink DM-RS demodulation reference signal DRX Discontinuous Reception HARQ Hybrid Automatic Repeat Request KPI Key Performance Indicator L1-RSRP Layer 1 Reference Signal Received Power LI Layer Indicator MAC Media Access Control MCS Modulation and Coding Scheme MSE Mean Squared Error NLOS non-line-of-site NR New Radio PDCCH Physical Downlink Control Channel PDSCH Physical Downlink Shared Channel PSS primary synchronization signal PUCCH Physical Uplink Control Channel PUSCH Physical Uplink Shared Channel QCL Quasi-collocation PMI Precoding Matrix Indicator PRB Physical Resource Block PRG Precoding Resource Block Group PRS Positioning Reference Signal RAN Radio Access Network RB Resource Block RBG Resource Block Group RNTI Radio Network Temporary Identifier RI Rank Indicator RRC Radio Resource Control RS reference signal SID Consideration Items Description SS Sync Signal SSB sync signal block SSS secondary synchronization signal SS-RSRP SS reference signal received power SS-RSRQ SS Reference Signal Reception Quality SS-SINR SS signal to interference and noise ratio TCI Transmit Configuration Indicator TDM time division multiplexing UE User Equipment UL Uplink

[0076] The above disclosure also encompasses the embodiments listed below.

[0077] (1) A method executed by at least one processor of a user equipment (UE), the method including: receiving a set of resources from a base station; performing a first measurement of the set of resources based on a legacy mode that does not use an artificial intelligence / machine learning (AI / ML) model to generate a first output; performing a second measurement of the set of resources based on the AI / ML model to generate a second output; and reporting results corresponding to the first output and the second output to the base station.

[0078] (2) The method according to (1), wherein the set of resources from the base station includes one or more channel state information reference signal (CSI-RS) resources, the first output is estimated CSI, and the second output is another estimated CSI obtained from an input of a compressed version of the CSI-RS resources.

[0079] (3) The method of (2) above, further comprising: receiving, based on the report, from the base station during a first transmission mode, one or more CSI-RS resources using a first modulation and coding scheme (MCS) determined based on the first output; receiving from the base station during a second transmission mode, one or more CSI-RS resources using an MCS determined based on the second output; determining a first set of key performance indicators (KPIs) corresponding to the first transmission mode; determining a second set of KPIs corresponding to the second transmission mode; and reporting the first set of KPIs and the second set of KPIs to the base station.

[0080] (4) The method described in (1) above, wherein the set of resources from the base station includes a plurality of beams, the first output includes a reference signal received power (RSRP) of each of the plurality of beams, the second output includes the RSRP of a subset of the plurality of beams, N best beams are determined based on the first output, and M best beams are determined based on the second output, where N and M are integers greater than 0.

[0081] (5) The method described in (4) above, further comprising determining N best beams based on the first output and determining M best beams based on the second output, and reporting the results to the base station comprises reporting the N best beams and the M best beams.

[0082] (6) The method described in (4) above, in which the base station determines the N best beams and the M best beams based on the reports.

[0083] (7) The method of (4) above, further comprising: performing a third measurement of the N best beams based on the legacy mode during a first transmission mode; performing a fourth measurement of the M best beams based on an AI / ML model during a second transmission mode; determining a first set of key performance indicators (KPIs) based on the third measurement; determining a second set of KPIs based on the fourth measurement; and reporting the first set of KPIs and the second set of KPIs to a base station.

[0084] (8) The method described in (1) above, wherein the set of resources includes one or more positioning reference signals (PRS) and a channel impulse response (CIR), the first output is a time of arrival estimation for the UE using one or more PRSs, the second output is a time of arrival inference using the CIR, and the base station determines the location of the UE based on the report.

[0085] (9) The method according to (1), further comprising receiving a measurement initiation signal from the base station, and performing the first measurement and the second measurement based on the reception of the measurement initiation signal.

[0086] (10) The method described in (1) above, wherein reporting of results corresponding to the first output and the second output is performed at a predetermined timing, and the first measurement and the second measurement are performed at a timing according to the predetermined timing.

[0087] (11) The method according to (1), further comprising receiving, from the base station, an indication of which AI / ML model to use from the plurality of AI / ML models for the second measurement.

[0088] (12) A user equipment (UE) comprising: at least one memory configured to store computer program code; and at least one processor configured to access the at least one memory and operate as instructed by the computer program code, the computer program code including: first receiving code configured to cause at least one of the at least one processor to receive a set of resources from a base station; first execution code configured to cause at least one of the at least one processor to perform a first measurement of the set of resources based on a legacy mode that does not use an artificial intelligence / machine learning (AI / ML) model to generate a first output; second execution code configured to cause at least one of the at least one processor to perform a second measurement of the set of resources based on the AI / ML model to generate a second output; and first reporting code configured to cause at least one of the at least one processor to report results corresponding to the first output and the second output to the base station.

[0089] (13) The UE described in (12) above, wherein the set of resources from the base station includes one or more channel state information reference signal (CSI-RS) resources, the first output is estimated CSI, and the second output is another estimated CSI obtained from an input of a compressed version of the CSI-RS resources.

[0090] (14) Computer program code includes a second receive code, a third receive code, a first decision code, a second decision code, and a second report code, wherein, based on the report, the second receive code is configured to cause at least one of the at least one processor to receive one or more CSI-RS resources from the base station during a first transmission mode using a first modulation and coding scheme (MCS) determined based on the first output, and the third receive code is configured to cause at least one of the at least one processor to receive one or more CSI-RS resources from the base station during a second transmission mode using an MCS determined based on the second output. The UE described in (12) above, wherein the UE is configured to cause at least one of the at least one processor to receive a number of CSI-RS resources, the first decision code being configured to cause at least one of the at least one processor to determine a first set of key performance indicators (KPIs) corresponding to the first transmission mode, the second decision code being configured to cause at least one of the at least one processor to determine a second set of KPIs corresponding to the second transmission mode, and the second reporting code being configured to cause at least one of the at least one processor to report the first set of KPIs and the second set of KPIs to the base station.

[0091] (15) A UE as described in (12) above, wherein the set of resources from the base station includes a plurality of beams, the first output includes a reference signal received power (RSRP) of each of the plurality of beams, the second output includes the RSRP of a subset of the plurality of beams, N best beams are determined based on the first output, and M best beams are determined based on the second output, where N and M are integers greater than 0.

[0092] (16) The UE described in (15) above, wherein the computer program code further includes a first decision code configured to cause at least one of the at least one processor to determine N best beams based on the first output, and a second decision code configured to cause at least one of the at least one processor to determine M best beams based on the second output, and the first reporting code is further configured to cause at least one of the at least one processor to report the results to the base station, including reporting the N best beams and the M best beams.

[0093] (17) The UE described in (15) above, wherein the base station determines the N best beams and the M best beams based on the report.

[0094] (18) The UE described in (15) above, wherein the computer program code further includes: a third execution code configured to cause at least one of the at least one processor to perform a third measurement of the N best beams based on the legacy mode during the first transmission mode; a fourth execution code configured to cause at least one of the at least one processor to perform a fourth measurement of the M best beams based on the AI / ML model during the second transmission mode; a first decision code configured to cause at least one of the at least one processor to determine a first set of key performance indicators (KPIs) based on the third measurement; a second decision code configured to cause at least one of the at least one processor to determine a second set of KPIs based on the fourth measurement; and a second reporting code configured to cause at least one of the at least one processor to report the first set of KPIs and the second set of KPIs to the base station.

[0095] (19) A UE as described in (12) above, wherein the set of resources includes one or more positioning reference signals (PRS) and a channel impulse response (CIR), the first output is an estimated time of arrival of the UE using one or more PRS, the second output is an inferred time of arrival using the CIR, and the base station determines the location of the UE based on the report.

[0096] (20) A non-transitory computer-readable medium having instructions stored thereon that, when executed by a processor in a user equipment (UE), cause the processor to perform a method including receiving a set of resources from a base station; performing a first measurement of the set of resources based on a legacy mode that does not use an artificial intelligence / machine learning (AI / ML) model to generate a first output; performing a second measurement of the set of resources based on the AI / ML model to generate a second output; and reporting results corresponding to the first output and the second output to the base station.

Claims

1. 1. A method executed by at least one processor of a user equipment (UE), the method comprising: receiving a set of resources from a base station; performing a first measurement of the set of resources based on a legacy mode that does not use an artificial intelligence / machine learning (AI / ML) model to generate a first output; performing a second measurement of the set of resources based on the AI / ML model to generate a second output; and reporting results corresponding to the first output and the second output to the base station.

2. the set of resources from the base station includes one or more channel state information reference signal (CSI-RS) resources; the first output is an estimated CSI; The method of claim 1 , wherein the second output is another estimated CSI obtained from an input of a condensed version of the CSI-RS resource.

3. Based on the above report, receiving the one or more CSI-RS resources from the base station during a first transmission mode using a first modulation and coding scheme (MCS) determined based on the first output; receiving the one or more CSI-RS resources from the base station during a second transmission mode using an MCS determined based on the second output; determining a first set of key performance indicators (KPIs) corresponding to the first transmission mode; determining a second set of KPIs corresponding to the second transmission mode; The method of claim 2 , further comprising: reporting the first set of KPIs and the second set of KPIs to the base station.

4. the set of resources from the base station includes a plurality of beams; the first output includes a reference signal received power (RSRP) for each of the plurality of beams; the second output includes RSRPs of a subset of the plurality of beams; determining N best beams based on the first output; determining M best beams based on the second output; The method of claim 1 , wherein N and M are integers greater than 0.

5. The method includes determining the N best beams based on the first output; determining the M best beams based on the second output; The method of claim 4 , wherein reporting the results to the base station includes reporting the N best beams and the M best beams.

6. The method of claim 4 , wherein the base station determines the N best beams and the M best beams based on the report.

7. performing, during a first transmission mode, a third measurement of the N best beams based on the legacy mode; and performing a fourth measurement of the M best beams based on the AI / ML model during a second transmission mode; determining a first set of key performance indicators (KPIs) based on the third measurements; determining a second set of KPIs based on the fourth measurement; and The method of claim 4 , further comprising: reporting the first set of KPIs and the second set of KPIs to the base station.

8. the set of resources includes one or more positioning reference signals (PRS) and a channel impulse response (CIR); the first output is an estimated time of arrival of the UE using the one or more PRSs; the second output is an arrival inference time using the CIR; The method of claim 1 , wherein the base station determines the location of the UE based on the report.

9. receiving a measurement initiation signal from the base station; The method of claim 1 , further comprising: performing the first measurement and the second measurement based on the reception of the measurement activation signal.

10. the reporting of the results corresponding to the first output and the second output is performed at a predetermined timing; The method of claim 1 , wherein the first measurement and the second measurement are performed at times according to the predetermined timing.

11. 10. The method of claim 1, further comprising receiving from the base station an indication of which AI / ML model from a plurality of AI / ML models to use for the second measurement.

12. A user equipment (UE), at least one memory configured to store computer program code; at least one processor configured to access the at least one memory and to operate as instructed by the computer program code, the computer program code comprising: first receiving code configured to cause at least one of the at least one processor to receive a set of resources from a base station; first execution code configured to cause at least one of the at least one processor to perform a first measurement of the set of resources based on a legacy mode that does not use an artificial intelligence / machine learning (AI / ML) model to generate a first output; second executable code configured to cause at least one of the at least one processor to perform a second measurement of the set of resources based on the AI / ML model to generate a second output; and a first reporting code configured to cause at least one of the at least one processor to report a result corresponding to the first output and the second output to the base station.

13. the set of resources from the base station includes one or more channel state information reference signal (CSI-RS) resources; the first output is an estimated CSI; The user equipment of claim 12, wherein the second output is another estimated CSI obtained from an input of a condensed version of the CSI-RS resource.

14. the computer program code comprising: a second receiving code, a third receiving code, a first determining code, a second determining code, and a second reporting code; Based on the above report, the second receive code is configured to cause at least one of the at least one processor to receive the one or more CSI-RS resources from the base station during a first transmission mode using a first modulation and coding scheme (MCS) determined based on the first output; the third receive code is configured to cause at least one of the at least one processor to receive the one or more CSI-RS resources from the base station during a second transmission mode using an MCS determined based on the second output; the first determination is configured to cause at least one of the at least one processor to determine a first set of key performance indicators (KPIs) corresponding to the first transmission mode; the second determining is configured to cause at least one of the at least one processor to determine a second set of KPIs corresponding to the second transmission mode; the second reporting is configured to cause at least one of the at least one processor to report the first set of KPIs and the second set of KPIs to the base station.

13. The user equipment of claim 12.

15. the set of resources from the base station includes a plurality of beams; the first output includes a reference signal received power (RSRP) for each of the plurality of beams; the second output includes RSRPs of a subset of the plurality of beams; determining N best beams based on the first output; determining M best beams based on the second output; The user equipment of claim 12 , wherein N and M are integers greater than 0.

16. the computer program code comprising: a first decision code configured to cause at least one of the at least one processor to determine the N best beams based on the first output; and a second determination configured to cause at least one of the at least one processor to determine the M best beams based on the second output; the first reporting code is further configured to cause at least one of the at least one processor to report the results to the base station, including reporting the N best beams and the M best beams.

16. The user equipment of claim 15.

17. The user equipment of claim 15 , wherein the base station determines the N best beams and the M best beams based on the report.

18. the computer program code comprising: third execution code configured to cause at least one of the at least one processor to perform, during a first transmission mode, a third measurement of the N best beams based on the legacy mode; and fourth execution code configured to cause at least one of the at least one processor to perform, during a second transmit mode, a fourth measurement of the M best beams based on the AI / ML model; and a first determination configured to cause at least one of the at least one processor to determine a first set of key performance indicators (KPIs) based on the third measurement; a second determination configured to cause at least one of the at least one processor to determine a second set of KPIs based on the fourth measurement; a second reporting configured to cause at least one of the at least one processor to report the first set of KPIs and the second set of KPIs to the base station.

16. The user equipment of claim 15.

19. the set of resources includes one or more positioning reference signals (PRS) and a channel impulse response (CIR); the first output is an estimated time of arrival of the UE using the one or more PRSs; the second output is an arrival inference time using the CIR; The user equipment of claim 12 , wherein the base station determines a location of the UE based on the report.

20. A non-transitory computer-readable medium having stored thereon instructions that, when executed by a processor in a user equipment (UE), cause the processor to perform a method, the method comprising: receiving a set of resources from a base station; performing a first measurement of the set of resources based on a legacy mode that does not use an artificial intelligence / machine learning (AI / ML) model to generate a first output; performing a second measurement of the set of resources based on the AI / ML model to generate a second output; and reporting a result corresponding to the first output and the second output to the base station.

Images