Method of advanced measurement signaling

Abstract

The present disclosure describes methods of advanced measurement signaling by configuring a set of ML model performance indication types for standalone model as single model and non-standalone model as multi-model / composite / ensemble model in a wireless communication system including base station e.g., gNB, TN, NTN and mobile station e.g., UE. In AI / ML model is applied to radio access network, signaling overhead can be significantly increased for model status feedback signaling. Therefore, model operation e.g., model training / inferencing / monitoring / updating can be set up between network and UE by using a set of types of data format for model measurement indication.

Description

[0001] 202404079

[0002] 1

[0003] Description

[0004] Method of advanced measurement signaling

[0005] TECHNICAL FIELD

[0006] The present disclosure relates to AI / ML based model operation with model measurement indication signaling, where techniques for pre-configuring and signaling the specific information about a set of types of data format for model measurement indication applicable to radio access network are presented.

[0007] BACKGROUND

[0008] In 3GPP (Third Generation Partnership Project), one of the selected study items as the approved Release 18 package is AI / ML (artificial intelligence / machine learning) as described in the related document (RP-213599) addressed in 3GPP TSG (Technical Specification Group) RAN (Radio Access Network) meeting #94e. The official title of AI / ML study item is “Study on AI / ML for NR Air Interface”. The goal of this study item is to identify a common AI / ML framework and areas of obtaining gains using AI / ML based techniques with use cases. According to 3GPP, the main objective of this study item is to study AI / ML framework for air-interface with target use cases by considering performance, complexity, and potential specification impact. In particular, AI / ML model, terminology and description to identify common and specific characteristics for framework are included as one of key work scopes. Regarding AI / ML framework, various aspects are under consideration for investigation and one of key items is about lifecycle management of AI / ML model where multiple stages are included as mandatory for model training, model deployment, model inference, model monitoring, model updating etc.

[0009] Also in 3GPP, two-sided (AI / ML) model is defined as a paired AI / ML model(s) over which joint inference is performed, where joint inference comprises AI / ML Inference whose inference is performed jointly across the UE and the network. Also for one-sided (AI / ML) model, UE-side (AI / ML) model is defined as an AI / ML model 202404079

[0010] 2 whose inference is performed entirely at the UE and network(NW)-side (AI / ML) model is defined as an AI / ML model whose inference is performed entirely at the network. Currently, AI / ML specification work is at the stage of work item discussion for Release 19. Earlier, in 3GPP TR 37.817 for Release 17, titled as Study on enhancement for Data Collection for NR and EN-DC, UE (user equipment) mobility was also considered as one of AI / ML use cases and one of scenarios for model training / inference is that both functions are located within RAN node. Followingly, in Release 18 the new work item of “Artificial Intelligence (AI)ZMachine Learning (ML) for NG-RAN” was initiated to specify data collection enhancements and signaling support within existing NG-RAN interfaces and architecture. For the above active standardization works, RAN-based AI / ML model is considered very significant for both network and UE to meet any desired model operations (e.g., model training, inference, selection, switching, update, monitoring, etc.). Model information can be signaled to pair both network-side and UE-side models for various lifecycle management (LCM) operations. However, signaling overhead indicating model information can be very high especially when model based LCM is processed between base station (BS / gNB) and multiple UEs. In LCM, model training is one of the most important parts for model deployment and currently there is no specification defined for signaling methods and network-UE behaviors so as to identify the required dataset when model updating / re-training as any activated model can be also impacted due to model / data drift. When ML condition changes, the enabled AI / ML model(s) can be impacted for model performance due to data / model drift. In this case, model re-training / updating can be executed.

[0011] US 2021326701 A1 describes the method of transmitting the measurements to other node for neural network training and a method of reporting a UE capability to a server, and configuring neural network parameters.

[0012] US 2022400373A1 shows a method performed by a UE transmitting, to a BS, UE capability information indicating at least one radio capability of the UE and at least one machine learning (ML) capability of the UE and receiving, from the BS based on the UE capability information, ML configuration information indicating at least one neural network function. 202404079

[0013] 3

[0014] US 2022330012A1 shows performance of a ML procedure based on at least one initial ML capability of the UE and determination of at least one updated ML capability for the ML procedure corresponding to an update to the at least one initial ML capability.

[0015] WO2023272718A1 shows UE indicates a support for an end-to-end multi-block machine learning application with block of the multi-block machine learning application.

[0016] US 2022377844A1 shows network transmission of an ML model training request to activate the ML model training and device transmission, based on receiving the ML model training request, ML model training results indicative of a trained ML model.

[0017] The present disclosure solves the cited problem above by the proposed embodiments and describes methods of advanced measurement signaling by configuring a set of ML model performance indication types for standalone model as single model and non-standalone model as multi-model / composite / ensemble model in a wireless communication system including base station e.g., gNB, TN, NTN and mobile station e.g., UE. In AI / ML model is applied to radio access network, signaling overhead can be significantly increased for model status feedback signaling. Therefore, model operation e.g., model training / inferencing / monitoring / updating can be set up between network and UE by using a set of types of data format for model measurement indication.

[0018] In the first embodiment , the method of advanced measurement signaling by configuring a set of ML model performance indication types for standalone model as single model and non-standalone model as multi-model / composite / ensemble model in a wireless communication system is characterized by comprising the steps, pre-defining mapping relation between model identifiers and model measurement indications (MMIs) of different types; generating data formats of indicating model performance measure with different types; categorizing ML models as standalone model and non-standalone model; 202404079

[0019] 4

[0020] Creating a new medium access control control element (MAC CE) to carry MM I value with the associated information for model measurement reporting.

[0021] In some embodiments of the method according to the first aspect, the method is characterized by that, that MMI represents model performance or model operation status in association with different Life Cycle Management (LCM) phases (such as training, inferencing, monitoring, etc.).

[0022] In some embodiments of the method according to the first aspect, the method is characterized by that, that the different LCM phases are training phases and / or inferencing and / or monitoring phases.

[0023] In some embodiments of the method according to the first aspect, the method is characterized by that, that MMI is represented with the configured number of bits so that a finite number of value ranges are supported.

[0024] In some embodiments of the method according to the first aspect, the method is characterized by that, that configuration information about MMI types and any associated mapping relation information such as model ID and MMI types is also sent via system information or dedicated radio resource control (RRC) signaling.

[0025] In some embodiments of the method according to the first aspect, the method is characterized by that, that MMI value set is represented with the configured number of bits so that a finite number of value ranges are associated with different UE ML capabilities supporting ML model operation.

[0026] In some embodiments of the method according to the first aspect, the method is characterized by that, that MMI value set of different types is determined by considering the reported UE ML capability and / or ML condition.

[0027] In some embodiments of the method according to the first aspect, the method is characterized by that, that MMI values is carried in the MAC CE, which is identified by specific logical channel identifiers (LCIDs). 202404079

[0028] 5

[0029] In some embodiments of the method according to the first aspect, the method is characterized by that, that MMI reporting with the associated type is included via a specific LCID value assigned.

[0030] In some embodiments of the method according to the first aspect, the method is characterized by that, that LCID can indicate a specific MMI type information among different types of MMI measure in association with ML model.

[0031] In some embodiments of the method according to the first aspect, the method is characterized by that, that the actual LCID values for MMI reporting can vary based on the specific implementation and configuration.

[0032] In some embodiments of the method according to the first aspect, the method is characterized by that, that one or more number of tables is configured to set different types of ML model performance indication measure.

[0033] In some embodiments of the method according to the first aspect, the method is characterized by that, that the specific data format to be generated for model performance indication measure is configured by network side so that this configuration information is provided to UE via system information or dedicated RRC signaling.

[0034] In some embodiments of the method according to the first aspect, the method is characterized by that, that there are multiple tables about different types of ML model performance indication measure by providing to UE selectively.

[0035] In some embodiments of the method according to the first aspect, the method is characterized by that, that Type 1 supports data format of indicating model performance measure for standalone or single model. 202404079

[0036] 6

[0037] In some embodiments of the method according to the first aspect, the method is characterized by that, that Type 1 based MMI represents specific MMI value for standalone / single model.

[0038] In some embodiments of the method according to the first aspect, the method is characterized by that, that Type 2-1 supports data format of indicating model performance measures of each individual subsidiary / partitioned models from multi-model.

[0039] In some embodiments of the method according to the first aspect, the method is characterized by that, that each subsidiary / partitioned models from multi-model is associated with the assigned model identifiers, respectively with mapping relation between model identifiers and MMI values of each subsidiary / partitioned models for Type 2-1.

[0040] In some embodiments of the method according to the first aspect, the method is characterized by that, that the number of subsidiary / partitioned models from multi-model can vary depending on different ML applications or deployment scenarios with specific ML model characteristics.

[0041] In some embodiments of the method according to the first aspect, the method is characterized by that, that Type 2-2 supports data format of indicating model performance measures of groupwise models by grouping subsidiary / partitioned models from multi-model.

[0042] In some embodiments of the method according to the first aspect, the method is characterized by that, that groupwise models represents model performance or model operation status in association with different LCM phases with the configured number of bits of MMI value.

[0043] In some embodiments of the method according to the first aspect, the method is characterized by that, that the averaged MMI values of each models in a group or end-to-end groupwise model measurement for measuring groupwise model is implemented for Type 2-2. 202404079

[0044] 7

[0045] In some embodiments of the method according to the first aspect, the method is characterized by that, that each groupwise models is associated with the assigned groupwise model identifiers, respectively with mapping relation between model identifiers and MMI values of each groupwise models for Type 2-2.

[0046] In some embodiments of the method according to the first aspect, the method is characterized by that, that Type 2-3 supports data format of indicating model performance measure of averaging performance of all subsidiary / partitioned models from multi-model.

[0047] In some embodiments of the method according to the first aspect, the method is characterized by that, that Type 2-3 represents model performance or model operation status as the averaged MMI value across different models within multi-model in association with different LCM phases with the configured number of bits of MMI value.

[0048] In some embodiments of the method according to the first aspect, the method is characterized by that, that the averaged MMI value across different models within multi-model is implemented for end-to-end model measurement with mapping relation between model identifiers of multi-model and the averaged MMI value for Type 2-3.

[0049] In some embodiments of the method according to the first aspect, the method is characterized by that, that Type 2-4 supports data format of indicating model performance measure of delta performance value of subsidiary / partitioned models from multi-model based on the averaged performance measure of all subsidiary / partitioned models from multi-model.

[0050] In some embodiments of the method according to the first aspect, the method is characterized by that, that Type 2-4 represents model performance or model operation status as the difference between the averaged MMI value across different 202404079

[0051] 8 models within multi-model and the MMI value of each subsidiary and / or partitioned models, which could be a delta measurement value, in association with different LCM phases with the configured number of bits of MMI value.

[0052] In some embodiments of the method according to the first aspect, the method is characterized by that, that each subsidiary / partitioned models is associated with the assigned model identifiers, respectively with mapping relation between model identifiers and MMI values of delta measurement for Type 2-4.

[0053] In some embodiments of the method according to the first aspect, the method is characterized by that, that standalone model is operated as single model itself for ML processing.

[0054] In some embodiments of the method according to the first aspect, the method is characterized by that, that non-standalone model consists of two or more subsidiary / partitioned models for ML processing.

[0055] In some embodiments of the method according to the first aspect, the method is characterized by that, that measurement criteria for model performance vary depending on implementation-specific such as prediction accuracy or statistical data distribution or any other performance metric.

[0056] In some embodiments of the method according to the first aspect, the method is characterized by that, that the number of subsidiary / partitioned models for non-standalone model vary depending on different ML use cases and / or deployment scenarios along with the associated model characteristics.

[0057] In some embodiments of the method according to the first aspect, the method is characterized by that, that subsidiary / partitioned models is for independent functional processing or dependent functional processing to each other.

[0058] In some embodiments of the method according to the first aspect, the method is characterized by that, that multiple subsidiary / partitioned models with 202404079

[0059] 9 non-standalone model is grouped together so that groupwise model is measured for model performance.

[0060] In some embodiments of the method according to the first aspect, the method is characterized by that, that group model identifier along with the associated subsidiary / partitioned model identifiers (e.g., when multiple subsidiary / partitioned models is grouped together) is assigned so that each entities related to ML model operation is aligned about groupwise model status with model performance measure.

[0061] In some embodiments of the method according to the first aspect, the method is characterized by that, that models is grouped so as to measure / monitor groupwise functional processing block.

[0062] In some embodiments of the method according to the first aspect, the method is characterized by that, that one or more groupwise models is used for multiple applications operated by ML model.

[0063] In some embodiments of the method according to the first aspect, the method is characterized by that, that a higher MM I value indicates better model performance or operation status and a lower MMI value indicates worse model performance or operation status.

[0064] In some embodiments of the method according to the first aspect, the method is characterized by that, that any configured threshold related to MMI value is used to determine model re-training, switching, updating, or de-activation.

[0065] In some embodiments of the method according to the first aspect, the method is characterized by that, that MMI value is calculated with model performance measure with the configured MMI type.

[0066] In some embodiments of the method according to the first aspect, the method is characterized by that, that a specific LCID for MMI reporting is identified for use so 202404079

[0067] 10 that MMI value(s) and the associated type with model identifier information is carried via Layer-1 (L1 ) or Layer-2 (L2) signaling.

[0068] In some embodiments of the method according to the first aspect, the method is characterized by that, that MMI values is carried in the MAC CE, which is identified by specific LCIDs.

[0069] In some embodiments of the method according to the first aspect, the method is characterized by that, that MMI reporting with the associated type is included via a specific LCID value assigned.

[0070] In some embodiments of the method according to the first aspect, the method is characterized by that, that LCID can indicate a specific MMI type information among different types of MMI measure in association with ML model.

[0071] In some embodiments of the method according to the first aspect, the method is characterized by that, that the actual LCID values for MMI reporting can vary based on the specific implementation and configuration.

[0072] In some embodiments of the method according to the first aspect, the method is characterized by that, that model performance measurement is configured to be periodic or non-periodic so as to apply different types of model performance indication.

[0073] In some embodiments of the method according to the first aspect, the method is characterized by that, that performance measure with different types of model performance indication is associated with specific periodicity or triggering method using threshold value.

[0074] In some embodiments of the method according to the first aspect, the method is characterized by that, that any specific type(s) of model performance indication is decided by network side or UE side autonomously (e.g., using 1 -bit indication). 202404079

[0075] 11

[0076] In some embodiments of the method according to the first aspect, the method is characterized by that, that one or more ML models is activated or inactive for support depending on different ML use cases and / or deployment scenarios along with the associated model characteristics.

[0077] In some embodiments of the method according to the first aspect, the method is characterized by that, that target model(s) for model performance measure at UE is in different LCM phases such as data collection, (re-)training, inferencing, updating, switching, monitoring.

[0078] In some embodiments of the method according to the first aspect, the method is characterized by that, that on network side a set of ML model performance indication types for standalone model (e.g., single model) and non-standalone model (e.g., multi-model such as composite / ensemble model) are configured depending on different ML use cases and / or deployment scenarios along with the associated model characteristics.

[0079] In some embodiments of the method according to the first aspect, the method is characterized by that, that configuration information about model performance measurement is sent to UE via system information or dedicated RRC signaling.

[0080] In some embodiments of the method according to the first aspect, the method is characterized by that, that any target model(s) is identified for performance measure so that the configured type of model performance indication is applied to one or more models.

[0081] In some embodiments of the method according to the first aspect, the method is characterized by that, that feedback message is sent to network side by indicating the specific type(s) used for model performance indication via L1 / L2 or RRC signaling.

[0082] According to a second aspect, the present disclosure relates to an apparatus for advanced measurement signaling by configuring a set of ML model performance indication types for standalone model as single model and non-standalone model as 202404079

[0083] 12 multi-model / composite / ensemble model in a the wireless communication systems, the apparatus comprising a wireless transceiver, a processor coupled with a memory in which computer program instructions are stored, said instructions being configured to implement steps of the claims according to the fist aspectl to 55.

[0084] According to a third aspect, the present disclosure relates to a User Equipment comprising an apparatus according to the second aspect.

[0085] According to a fourth aspect, the present disclosure relates to a gNB comprising an apparatus according to the second aspect.

[0086] According to a fifth aspect, the present disclosure relates to a wireless communication system for of advanced measurement signaling by configuring a set of ML model performance indication types for standalone model as single model and non-standalone model as multi-model / composite / ensemble model, wherein the wireless communication systems comprises user equipment according to claim 57, gNB according to claim 58, whereby the user equipment according to the third aspect, gNB according to the fourth aspect, whereby the user Equipment and the gNB each comprises a processor coupled with a memory in which computer program instructions are stored, said instructions being configured to implement steps of the according to the first aspect.

[0087] BRIEF DESCRIPTION OF THE DRAWINGS

[0088] Figure 1 is an exemplary table of different types of ML model performance indication measure.

[0089] Figure 2 is an exemplary block diagram of standalone and non-standalone models.

[0090] Figure 3 is an exemplary block diagram of groupwise model. 202404079

[0091] 13

[0092] Figure 4 is an exemplary description of Type 1 model measurement indication.

[0093] Figure 5 is an exemplary description of Type 2-1 model measurement indication.

[0094] Figure 6 is an exemplary description of Type 2-2 model measurement indication.

[0095] Figure 7 is an exemplary description of Type 2-3 model measurement indication.

[0096] Figure 8 is an exemplary description of Type 2-4 model measurement indication.

[0097] Figure 9 is an exemplary description of table of new MAC CE.

[0098] Figure 10 is an exemplary flow chart of configuring model performance indication types at network side.

[0099] Figure 11 is an exemplary flow chart of measuring the configured model performance type(s).

[0100] Figure 12 is an exemplary signaling flow of measuring model performance using the configured measurement type.

[0101] DETAILED DESCRIPTION

[0102] The detailed description set forth below, with reference to annexed drawings, is intended as a description of various configurations and is not intended to represent the only configurations in which the concepts described herein may be practiced. The detailed description includes specific details for the purpose of providing a thorough understanding of the various concepts. However, it will be apparent to those skilled in the art that these concepts may be practiced without these specific details. In particular, although terminology from 3GPP 5G NR may be used in this disclosure to exemplify embodiments herein, this should not be seen as limiting the scope of the invention. 202404079

[0103] 14

[0104] Some of the embodiments contemplated herein will now be described more fully with reference to the accompanying drawings. Other embodiments, however, are contained within the scope of the subject matter disclosed herein, the disclosed subject matter should not be construed as limited to only the embodiments set forth herein; rather, these embodiments are provided by way of example to convey the scope of the subject matter to those skilled in the art.

[0105] Generally, all terms used herein are to be interpreted according to their ordinary meaning in the relevant technical field, unless a different meaning is clearly given and / or is implied from the context in which it is used. All references to a / an / the element, apparatus, component, means, step, etc. are to be interpreted openly as referring to at least one instance of the element, apparatus, component, means, step, etc., unless explicitly stated otherwise. The steps of any methods disclosed herein do not have to be performed in the exact order disclosed, unless a step is explicitly described as following or preceding another step and / or where it is implicit that a step must follow or precede another step. Any feature of any of the embodiments disclosed herein may be applied to any other embodiment, wherever appropriate. Likewise, any advantage of any of the embodiments may apply to any other embodiments, and vice versa. Other objectives, features and advantages of the enclosed embodiments will be apparent from the following description.

[0106] In some embodiments, a more general term “network node” may be used and may correspond to any type of radio network node or any network node, which communicates with a UE (directly or via another node) and / or with another network node. Examples of network nodes are NodeB, MeNB, ENB, a network node belonging to MCG or SCG, base station (BS), multi-standard radio (MSR) radio node such as MSR BS, eNodeB, gNodeB, network controller, radio network controller (RNC), base station controller (BSC), relay, donor node controlling relay, base transceiver station (BTS), access point (AP), transmission points, transmission nodes, RRU, RRH, nodes in distributed antenna system (DAS), core network node (e.g. Mobile Switching Center (MSC), Mobility Management Entity (MME), etc), Operations & Maintenance (O&M), Operations Support System (OSS), Self Optimized Network (SON), positioning node (e.g. Evolved- Serving Mobile Location 202404079

[0107] 15

[0108] Centre (E-SMLC)), Minimization of Drive Tests (MDT), test equipment (physical node or software), etc.

[0109] In some embodiments, the non-limiting term user equipment (UE) or wireless device may be used and may refer to any type of wireless device communicating with a network node and / or with another UE in a cellular or mobile communication system. Examples of UE are target device, device to device (D2D) UE, machine type UE or UE capable of machine to machine (M2M) communication, PDA, PAD, Tablet, mobile terminals, smart phone, laptop embedded equipped (LEE), laptop mounted equipment (LME), USB dongles, UE category Ml, UE category M2, ProSe UE, V2V UE, V2X UE, etc.

[0110] Additionally, terminologies such as base station / gNodeB and UE should be considered non-limiting and do in particular not imply a certain hierarchical relation between the two; in general, “gNodeB” could be considered as device 1 and “UE” could be considered as device 2 and these two devices communicate with each other over some radio channel. And in the following the transmitter or receiver could be either gNodeB (gNB), or UE.

[0111] As will be appreciated by one skilled in the art, aspects of the embodiments may be embodied as a system, apparatus, method, or program product. Accordingly, embodiments may take the form of an entirely hardware embodiment, an entirely software embodiment (including firmware, resident software, micro-code, etc.) or an embodiment combining software and hardware aspects.

[0112] For example, the disclosed embodiments may be implemented as a hardware circuit comprising custom very-large-scale integration (“VLSI”) circuits or gate arrays, off-the-shelf semiconductors such as logic chips, transistors, or other discrete components. The disclosed embodiments may also be implemented in programmable hardware devices such as field programmable gate arrays, programmable array logic, programmable logic devices, or the like. As another example, the disclosed embodiments may include one or more physical or logical 202404079

[0113] 16 blocks of executable code which may, for instance, be organized as an object, procedure, or function.

[0114] Furthermore, embodiments may take the form of a program product embodied in one or more computer readable storage devices storing machine readable code, computer readable code, and / or program code, referred hereafter as code. The storage devices may be tangible, non- transitory, and / or non-transmission. The storage devices may not embody signals. In a certain embodiment, the storage devices only employ signals for accessing code

[0115] Any combination of one or more computer readable medium may be utilized. The computer readable medium may be a computer readable storage medium. The computer readable storage medium may be a storage device storing the code. The storage device may be, for example, but not limited to, an electronic, magnetic, optical, electromagnetic, infrared, holographic, micromechanical, or semiconductor system, apparatus, or device, or any suitable combination of the foregoing.

[0116] More specific examples (a non-exhaustive list) of the storage device would include the following: an electrical connection having one or more wires, a portable computer diskette, a hard disk, a random-access memory (“RAM”), a read-only memory (“ROM”), an erasable programmable read-only memory (“EPROM” or Flash memory), a portable compact disc readonly memory (“CD-ROM”), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing. In the context of this document, a computer readable storage medium may be any tangible medium that can contain or store a program for use by or in connection with an instruction execution system, apparatus, or device.

[0117] Code for carrying out operations for embodiments may be any number of lines and may be written in any combination of one or more programming languages including an object- oriented programming language such as Python, Ruby, Java, Smalltalk, C++, or the like, and conventional procedural programming languages, such as the “C” programming language, or the like, and / or machine languages such as assembly languages. The code may execute entirely on the user’s computer, partly 202404079

[0118] 17 on the user’s computer, as a stand-alone software package, partly on the user’s computer and partly on a remote computer or entirely on the remote computer or server. In the latter scenario, the remote computer may be connected to the user’s computer through any type of network, including a local area network (“LAN”), wireless LAN (“WLAN”), or a wide area network (“WAN”), or the connection may be made to an external computer (for example, through the Internet using an Internet Service Provider (“ISP”)).

[0119] Furthermore, the described features, structures, or characteristics of the embodiments may be combined in any suitable manner. In the following description, numerous specific details are provided, such as examples of programming, software modules, user selections, network transactions, database queries, database structures, hardware modules, hardware circuits, hardware chips, etc., to provide a thorough understanding of embodiments. One skilled in the relevant art will recognize, however, that embodiments may be practiced without one or more of the specific details, or with other methods, components, materials, and so forth. In other instances, well-known structures, materials, or operations are not shown or described in detail to avoid obscuring aspects of an embodiment. Reference throughout this specification to “one embodiment,” “an embodiment,” or similar language means that a particular feature, structure, or characteristic described in connection with the embodiment is included in at least one embodiment. Thus, appearances of the phrases “in one embodiment,” “in an embodiment,” and similar language throughout this specification may, but do not necessarily, all refer to the same embodiment, but mean “one or more but not all embodiments” unless expressly specified otherwise. The terms “including,” “comprising,” “having,” and variations thereof mean “including but not limited to,” unless expressly specified otherwise. An enumerated listing of items does not imply that any or all of the items are mutually exclusive, unless expressly specified otherwise. The terms “a,” “an,” and “the” also refer to “one or more” unless expressly specified otherwise.

[0120] Aspects of the embodiments are described below with reference to schematic flowchart diagrams and / or schematic block diagrams of methods, apparatuses, systems, and program products according to embodiments. It will be understood 202404079

[0121] 18 that each block of the schematic flowchart diagrams and / or schematic block diagrams, and combinations of blocks in the schematic flowchart diagrams and / or schematic block diagrams, can be implemented by code. This code may be provided to a processor of a general-purpose computer, special purpose computer, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, create means for implementing the fimctions / acts specified in the flowchart diagrams and / or block diagrams

[0122] The code may also be stored in a storage device that can direct a computer, other programmable data processing apparatus, or other devices to function in a particular manner, such that the instructions stored in the storage device produce an article of manufacture including instructions which implement the function / act specified in the flowchart diagrams and / or block diagrams.

[0123] The code may also be loaded onto a computer, other programmable data processing apparatus, or other devices to cause a series of operational steps to be performed on the computer, other programmable apparatus, or other devices to produce a computer implemented process such that the code which execute on the computer or other programmable apparatus provide processes for implementing the functions / acts specified in the flowchart diagrams and / or block diagrams.

[0124] The flowchart diagrams and / or block diagrams in the Figures illustrate the architecture, functionality, and operation of possible implementations of apparatuses, systems, methods, and program products according to various embodiments. In this regard, each block in the flowchart diagrams and / or block diagrams may represent a module, segment, or portion of code, which includes one or more executable instructions of the code for implementing the specified logical function(s).

[0125] It should also be noted that, in some alternative implementations, the functions noted in the block may occur out of the order noted in the Figures. For example, two blocks shown in succession may, in fact, be executed substantially concurrently, or 202404079

[0126] 19 the blocks may sometimes be executed in the reverse order, depending upon the functionality involved. Other steps and methods may be conceived that are equivalent in function, logic, or effect to one or more blocks, or portions thereof, of the illustrated Figures.

[0127] Although various arrow types and line types may be employed in the flowchart and / or block diagrams, they are understood not to limit the scope of the corresponding embodiments. Indeed, some arrows or other connectors may be used to indicate only the logical flow of the depicted embodiment. For instance, an arrow may indicate a waiting or monitoring period of unspecified duration between enumerated steps of the depicted embodiment. It will also be noted that each block of the block diagrams and / or flowchart diagrams, and combinations of blocks in the block diagrams and / or flowchart diagrams, can be implemented by special purpose hardware-based systems that perform the specified functions or acts, or combinations of special purpose hardware and code.

[0128] The description of elements in each figure may refer to elements of proceeding figures. Like numbers refer to like elements in all figures, including alternate embodiments of like elements.

[0129] The detailed description set forth below, with reference to the figures, is intended as a description of various configurations and is not intended to represent the only configurations in which the concepts described herein may be practiced. The detailed description includes specific details for the purpose of providing a thorough understanding of the various concepts. However, it will be apparent to those skilled in the art that these concepts may be practiced without these specific details. For instance, although 3GPP terminology, from e.g., 5G NR, may be used in this disclosure to exemplify embodiments herein, this should not be seen as limiting the scope of the present disclosure.

[0130] AI / ML Model is a data driven algorithm that applies AI / ML techniques to generate set of outputs based on set of inputs. 202404079

[0131] 20

[0132] AI / ML model delivery is a generic term referring to delivery of an AI / ML model from one entity to another entity in any manner. Note is An entity could mean network node / function (e.g., gNB, LMF, etc.), UE, proprietary server, etc.

[0133] AI / ML model Inference is a process of using trained AI / ML model to produce set of outputs based on set of inputs.

[0134] AI / ML model testing is a subprocess of training, to evaluate the performance of final AI / ML model using dataset different from one used for model training and validation. Differently from AI / ML model validation, testing does not assume subsequent tuning of the model.

[0135] AI / ML model training is a process to train an AI / ML Model [by learning the input / output relationship] in data driven manner and obtain the trained AI / ML Model for inference.

[0136] AI / ML model transfer is a delivery of an AI / ML model over the air interface in manner that is not transparent to 3GPP signalling, either parameters of model structure known at the receiving end or new model with parameters. Delivery may contain full model or partial model.

[0137] AI / ML model validation is a subprocess of training, to evaluate the quality of an AI / ML model using dataset different from one used for model training, that helps selecting model parameters that generalize beyond the dataset used for model training.

[0138] Data collection is a process of collecting data by the network nodes, management entity, or UE for the purpose of AI / ML model training, data analytics and inference.

[0139] Federated learning I federated training is a machine learning technique that trains an AI / ML model across multiple decentralized edge nodes e.g., UEs, gNBs each performing local model training using local data samples. The technique requires multiple interactions of the model, but no exchange of local data samples. 202404079

[0140] 21

[0141] Functionality identification is a process / method of identifying an AI / ML functionality for the common understanding between the NW and the UE. Note is Information regarding the AI / ML functionality may be shared during functionality identification. Where AI / ML functionality resides depends on the specific use cases and sub use cases.

[0142] Model activation means enable an AI / ML model for specific AI / ML-enabled feature.

[0143] Model deactivation means disable an AI / ML model for specific AI / ML-enabled feature.

[0144] Model download means Model transfer from the network to UE.

[0145] Model identification is A process / method of identifying an AI / ML model for the common understanding between the NW and the UE. The process / method of model identification may or may not be applicable and regarding the AI / ML model may be shared during model identification.

[0146] Model monitoring is A procedure that monitors the inference performance of the AI / ML model.

[0147] Model parameter update is Process of updating the model parameters of model. Model selection is the process of selecting an AI / ML model for activation among multiple models for the same AI / ML enabled feature. Model selection may or may not be carried out simultaneously with model activation.

[0148] Model switching is deactivating currently active AI / ML model and activating different AI / ML model for specific AI / ML-enabled feature.

[0149] Model update is process of updating the model parameters and / or model structure of model. 202404079

[0150] 22

[0151] Model upload is Model transfer from UE to the network.

[0152] Network-side (AI / ML) model is an AI / ML Model whose inference is performed entirely at the network.

[0153] Offline field data is the data collected from field and used for offline training of the AI / ML model.

[0154] Offline training is an AI / ML training process where the model is trained based on collected dataset, and where the trained model is later used or delivered for inference. Note is This definition only serves as guidance. There may be cases that may not exactly conform to this definition but could still be categorized as offline training by commonly accepted conventions.

[0155] Online field data is the data collected from field and used for online training of the AI / ML model.

[0156] Online training is an AI / ML training process where the model being used for inference) is (typically continuously) trained in (near) real-time with the arrival of new training samples. Note is the notion of (near) real-time vs. non real-time is context-dependent and is relative to the inference time-scale. This definition only serves as guidance.

[0157] There may be cases that may not exactly conform to this definition but could still be categorized as online training by commonly accepted conventions. Note is Fine-tuning / re-training may be done via online or offline training. This note could be removed when we define the term fine-tuning.

[0158] Reinforcement Learning (RL) is a process of training an AI / ML model from input (a.k.a. state) and feedback signal (a.k.a. reward) resulting from the model’s output (a.k.a. action) in an environment the model is interacting with. 202404079

[0159] 23

[0160] Semi-supervised learning is a process of training model with mix of labelled data and unlabelled data.

[0161] Supervised learning is a process of training model from input and its corresponding labels.

[0162] Two-sided (AI / ML) model is a paired AI / ML Model(s) over which joint inference is performed, where joint inference comprises AI / ML Inference whose inference is performed jointly across the UE and the network, i.e, the first part of inference is firstly performed by UE and then the remaining part is performed by gNB, or vice versa.

[0163] UE-side (AI / ML) model is an AI / ML Model whose inference is performed entirely at the UE.

[0164] Unsupervised learning is a process of training model without labelled data.

[0165] Proprietary-format models is ML models of vendor-Zdevice-specific proprietary format, from 3GPP perspective. They are not mutually recognizable across vendors and hide model design information from other vendors when shared.

[0166] Open-format models is ML models of specified format that are mutually recognizable across vendors and allow interoperability, from 3GPP perspective. They are mutually recognizable between vendors and do not hide model design information from other vendors when shared.

[0167] The disclosure is related to wireless communication system, which may be for example a 5G NR wireless communication system. More specifically, it represents a RAN of the wireless communication system, which is used exchange data with UEs via radio signals. For example, the RAN may send data to the UEs (downlink, DL), for instance data received from a core network (CN). The RAN may also receive data from the UEs (uplink, UL), which data may be forwarded to the CN. 202404079

[0168] 24

[0169] In the examples illustrated, the RAN comprises one base station, BS. Of course, the RAN may comprise more than one BS to increase the coverage of the wireless communication system. Each of these BSs may be referred to as NB, eNodeB (or eNB), gNodeB (or gNB, in the case of a 5G NR wireless communication system), an access point or the like, depending on the wireless communication standard(s) implemented.

[0170] The UEs are located in a coverage of the BS. The coverage of the BS corresponds for example to the area in which UEs can decode a PDCCH transmitted by the BS.

[0171] An example of a wireless device suitable for implementing any method, discussed in the present disclosure, performed at a UE corresponds to an apparatus that provides wireless connectivity with the RAN of the wireless communication system, and that can be used to exchange data with said RAN. Such a wireless device may be included in a UE. The UE may for instance be a cellular phone, a wireless modem, a wireless communication device, a handheld device, a laptop computer, or the like. The UE may also be an Internet of Things (loT) equipment, like a wireless camera, a smart sensor, a smart meter, smart glasses, a vehicle (manned or unmanned), a global positioning system device, etc., or any other equipment that may run applications that need to exchange data with remote recipients, via the wireless device.

[0172] The wireless device comprises one or more processors and one or more memories. The one or more processors may include for instance a central processing unit (CPU), a digital signal processor (DSP), a field-programmable gate array (FPGA), an application specific integrated circuit (ASIC), etc. The one or more memories may include any type of computer readable volatile and non-volatile memories (magnetic hard disk, solid-state disk, optical disk, electronic memory, etc.). The one or more memories may store a computer program product, in the form of a set of program-code instructions to be executed by the one or more processors to implement all or part of the steps of a method for exchanging data, performed at a UE’s side, according to any one of the embodiments disclosed herein. 202404079

[0173] 25

[0174] The wireless device can comprise also a main radio, MR, unit. The MR unit corresponds to a main wireless communication unit of the wireless device, used for exchanging data with BSs of the RAN using radio signals. The MR unit may implement one or more wireless communication protocols, and may for instance be a 3G, 4G, 5G, NR, WiFi, WiMax, etc. transceiver or the like. In preferred embodiments, the MR unit corresponds to a 5G NR wireless communication unit.

[0175] The detailed description set forth below, with reference to annexed drawings, is intended as a description of various configurations and is not intended to represent the only configurations in which the concepts described herein may be practiced. The detailed description includes specific details for the purpose of providing a thorough understanding of the various concepts. However, it will be apparent to those skilled in the art that these concepts may be practiced without these specific details. In particular, although terminology from 3GPP 5G NR may be used in this disclosure to exemplify embodiments herein, this should not be seen as limiting the scope of the invention.

[0176] The following explanation will provide the detailed description of the mechanism about pre-configuring and signaling the specific information about model online training by configuring a set of UE behaviors. AI / ML based techniques are currently applied to many different applications and 3GPP also started to work on its technical investigation to apply to multiple use cases based on the observed potential gains. AI / ML lifecycle can be split into several stages such as data collection / pre-processing, model training, model testing / validation, model deployment / update, model monitoring etc., where each stage is equally important to achieve target performance with any specific model(s). In applying AI / ML model for any use case or application, one of the challenging issues is to manage the lifecycle of AI / ML model. It is mainly because the data / model drift occurs during model deployment / inference and it results in performance degradation of AI / ML model.

[0177] Fundamentally, the dataset statistical changes occur after model is deployed and model inference capability is also impacted with unseen data as input. In a similar 202404079

[0178] 26 aspect, the statistical property of dataset and the relationship between input and output for the trained model can be changed with drift occurrence. In this context, model training or re-training is one of key issues for model performance maintenance as model performance such as inferencing and / or training is dependent on different model execution environment with varying configuration parameters.

[0179] To handle this issue, collaboration between UE and gNB is highly important to track model performance and re-configure model corresponding to different environments. AI / ML model needs model monitoring after deployment because model performance cannot be maintained continuously due to drift and update feedback is then provided to re-train / update the model or select alternative model. When AI / ML model enabled wireless communication network is deployed, it is then important to consider how to handle AI / ML model in activation with re-configuration for wireless devices under operations such as model training, inference, updating, etc. When one or multiple ML models are activated at UE, model performance update can be requested by network side so that target model performance can be supported. For reporting UE-sided model performance or model operation measurement, any specific data format of model performance update and the associated signaling need to be defined. In this method, a set of ML model performance indication types for standalone model (e.g., single model) and non-standalone model (e.g., multi-model such as composite / ensemble model) are configured where model measurement indication (MM I) represents model performance or model operation status in association with different LCM phases (such as training, inferencing, monitoring, etc.) and is represented with the configured number of bits so that a finite number of value ranges are supported.

[0180] Mapping relation between model identifiers and MMIs of different types is configured that can be also sent via system information or dedicated RRC signaling. MM I value set is represented with the configured number of bits so that a finite number of value ranges are associated with different UE ML capabilities supporting ML model operation. Specifically, MMI value set of different types is determined by considering the reported UE ML capability and / or ML condition. MMI values can be 202404079

[0181] 27 carried in the MAC Control Element (CE), which is identified by specific logical channel identifiers (LCIDs). MMI reporting with the associated type can be included via a specific LCID value assigned. LCID can indicate a specific MMI type information among different types of MMI measure in association with ML model. The actual LCID values for MMI reporting can vary based on the specific implementation and configuration.

[0182] Figure 1 shows an exemplary table of different types of ML model performance indication measure. In this example, one or more number of tables can be configured to set different types of ML model performance indication measure that can be used for feedback signaling between entities and / or on-device model management for ML operation via radio access network. The specific data format to be generated for model performance indication measure can be configured by network side and this configuration information can be provided to UE via system information or dedicated RRC signaling. If there are multiple tables about different types of ML model performance indication measure, those can be selectively sent to UE if applicable.

[0183] For example, Type 1 supports data format of indicating model performance measure for standalone or single model. Type 2 supports data format of indicating model performance measure for non-standalone or multi-model and there can be multiple subsidiary types. Type 2-1 supports data format of indicating model performance measures of each individual subsidiary / partitioned models from multi-model. Type 2-2 supports data format of indicating model performance measures of groupwise models by grouping subsidiary / partitioned models from multi-model. Type 2-3 supports data format of indicating model performance measure of averaging performance of all subsidiary / partitioned models from multi-model. Type 2-4 supports data format of indicating model performance measure of delta performance value of subsidiary / partitioned models from multi-model based on the averaged performance measure of all subsidiary / partitioned models from multi-model. 202404079

[0184] 28

[0185] Figure 2 shows an exemplary block diagram of standalone and non-standalone models. In this example, in high-level aspect two categories of ML models can be defined such as standalone model and non-standalone model where standalone model can be operated as single model itself for ML processing and non-standalone model can consist of two or more subsidiary / partitioned models for ML processing. In model performance measurement aspect, measurement criteria can vary depending on implementation-specific such as prediction accuracy or statistical data distribution or any other performance metric. Standalone model performance can be only measured using input / output values of target single model itself, and non-standalone model performance can be measured based on each subsidiary / partitioned models or groupwise models or the overall multi-model. The number of subsidiary / partitioned models for non-standalone model can vary depending on different ML use cases and / or deployment scenarios along with the associated model characteristics where subsidiary / partitioned models can be for independent functional processing or dependent functional processing to each other.

[0186] Figure 3 shows an exemplary block diagram of groupwise model. In this example, with non-standalone model having subsidiary / partitioned models, multiple subsidiary / partitioned models can be grouped together so that groupwise model can be measured for model performance. For example, when multiple subsidiary / partitioned models can be grouped together, group model identifier along with the associated subsidiary / partitioned model identifiers can be assigned so that each entities related to ML model operation can be aligned about groupwise model status with model performance measure. The criteria of model grouping can vary and one example is that models can be grouped so as to measure / monitor groupwise functional processing block. Or one or more groupwise models can be also used for multiple applications operated by ML model.

[0187] Figure 4 shows an exemplary description of Type 1 model measurement indication. In this example, Model measurement indication (MMI) can represent model performance or model operation status in association with different LCM phases 202404079

[0188] 29

[0189] (such as training, inferencing, monitoring, etc.) and can be represented with the configured number of bits so that a finite number of value ranges can be supported. For example, a higher MMI value indicates better model performance or operation status and a lower MMI value indicates worse model performance or operation status. Any configured threshold related to MMI value can be used to determine model re-training, switching, updating, or de-activation, etc. For example, type 1 based MMI can be indicated as MMI_MLs that represents specific MMI value for standalone / single model (MLs). There can be mapping relation between model identifier of standalone / single model and the MMI value for Type 1 .

[0190] Figure 5 shows an exemplary description of Type 2-1 model measurement indication. In this example, for non-standalone or multi-model, multiple subsidiary / partitioned models can be used and MMI_MU ( =1 ,2, ... ,K) can represent model performance or model operation status in association with different LCM phases (such as training, inferencing, monitoring, etc.) with the configured number of bits of MMI value so that a finite number of value ranges can be supported. The number of subsidiary / partitioned models (K) can vary depending on different ML applications or deployment scenarios with specific ML model characteristics. Each subsidiary / partitioned models can be also associated with the assigned model identifiers, respectively so that there can be mapping relation between model identifiers and MMI values with each subsidiary / partitioned models for Type 2-1 .

[0191] Figure 6 shows an exemplary description of Type 2-2 model measurement indication. In this example, for non-standalone or multi-model with multiple groupwise models, MMI_MG / ( / =1 ,2, ... ,L) can represent model performance or model operation status in association with different LCM phases (such as training, inferencing, monitoring, etc.) with the configured number of bits of MMI value so that a finite number of value ranges can be supported. When measuring groupwise model, the averaged MMI values of each models in a group can be implemented or end-to-end groupwise model measurement can be also considered. Each groupwise models can be also associated with the assigned groupwise model 202404079

[0192] 30 identifiers, respectively so that there can be mapping relation between model identifiers and MM I values with each groupwise models for Type 2-2.

[0193] Figure 7 shows an exemplary description of Type 2-3 model measurement indication. In this example, for non-standalone or multi-model, MMI_MA can represent model performance or model operation status as the averaged MM I value across different models within multi-model in association with different LCM phases (such as training, inferencing, monitoring, etc.) with the configured number of bits of MMI value so that a finite number of value ranges can be supported. When measuring non-standalone or multi-model, the averaged MMI value across different models within multi-model can be implemented for end-to-end model measurement. There can be mapping relation between model identifiers of multi-model and the averaged MMI value for Type 2-3.

[0194] Figure 8 shows an exemplary description of Type 2-4 model measurement indication. In this example, for non-standalone or multi-model, multiple subsidiary / partitioned models can be used and MMI_MDn(n=1 ,2,... ,N) can represent model performance or model operation status as the difference between the averaged MMI value across different models within multi-model and the MMI value of each subsidiary / partitioned models (e.g., delta measurement value) in association with different LCM phases (such as training, inferencing, monitoring, etc.) with the configured number of bits of MMI value so that a finite number of value ranges can be supported. Each subsidiary / partitioned models can be also associated with the assigned model identifiers, respectively so that there can be mapping relation between model identifiers and MMI values with delta measurement for Type 2-4.

[0195] Figure 9 shows an exemplary description of table of new MAC CE. In this example, a new MAC CE can be created to carry MMI value with the associated information for model measurement reporting. As MMI value is calculated with model performance measure with the configured MMI type, a specific LCID for MMI reporting can be identified for use so that MMI value(s) and the associated type with model identifier information can be carried via L1 / L2 signaling. MMI values can be 202404079

[0196] 31 carried in the MAC CE, which is identified by specific LCIDs. MMI reporting with the associated type can be included via a specific LCID value assigned. LCID can indicate a specific MMI type information among different types of MMI measure in association with ML model. The actual LCID values for MMI reporting can vary based on the specific implementation and configuration.

[0197] Figure 10 shows an exemplary flow chart of configuring model performance indication types at network side. In this example, on network side a set of ML model performance indication types for standalone model (e.g., single model) and non-standalone model (e.g., multi-model such as composite / ensemble model) are configured depending on different ML use cases and / or deployment scenarios along with the associated model characteristics. Configuration information about model performance measurement can be sent to UE via system information or dedicated RRC signaling. Model performance measurement can be also configured to be periodic or non-periodic so as to apply different types of model performance indication. With different types of model performance indication, performance measure can be associated with specific periodicity or triggering method using threshold value. Any specific type(s) of model performance indication can be decided by network side or UE side autonomously (e.g., using 1 -bit indication).

[0198] Figure 11 shows an exemplary flow chart of measuring the configured model performance type(s). In this example, one or more ML models can be activated or inactive for support depending on different ML use cases and / or deployment scenarios along with the associated model characteristics. Target model(s) for model performance measure at UE can be in different LCM phases such as data collection, (re-)training, inferencing, updating, switching, monitoring, etc.

[0199] After receiving configuration information, any target model(s) can then be identified for performance measure so that the configured type of model performance indication can be applied to one or more models. Based on model performance measure, feedback message can be sent to network side by indicating the specific type(s) used for model performance indication via L1 / L2 or RRC signaling if applicable. 202404079

[0200] 32

[0201] Figure 12 shows an exemplary signaling flow of measuring model performance using the configured measurement type. In this example, on network side a set of ML model performance indication types for standalone model (e.g., single model) and non-standalone model (e.g., multi-model such as composite / ensemble model) are configured depending on different ML use cases and / or deployment scenarios along with the associated model characteristics.

[0202] Configuration information about model performance measurement can be sent to UE via system information or dedicated RRC signaling. After receiving configuration information, any target model(s) can then be identified for performance measure so that the configured type of model performance indication can be applied to one or more models. Based on model performance measure, feedback message can be sent to network side by indicating the specific type(s) used for model performance indication.

Claims

20240407933Patent claims1 . 1 . A method of advanced measurement signaling by configuring a set of ML model performance indication types for standalone model as single model and non-standalone model as multi-model / composite / ensemble model in a wireless communication system, comprising:• Pre-defining mapping relation between model identifiers and MMIs of different types;• Generating data formats of indicating model performance measure with different types;• Categorizing ML models as standalone model and non-standalone model;• Creating a new MAC CE to carry MM I value with the associated information for model measurement reporting.

2. The method according to previous claim 1 , wherein MMI represents model performance or model operation status in association with different Life Cycle Management (LCM) phases (such as training, inferencing, monitoring, etc.).

3. The method according to previous claim 1 , wherein the different LCM phases are training phases and / or inferencing and / or monitoring phases.

4. The method according to one of the previous claims, wherein MMI is represented with the configured number of bits so that a finite number of value ranges are supported.

5. The method according to one of the previous claims, wherein configuration information about MMI types and any associated mapping relation information such as model ID and MMI types is also sent via system information or dedicated RRC signaling.

6. The method according to one of the previous claims, wherein MMI value set is represented with the configured number of bits so that a finite number of value20240407934 ranges are associated with different UE ML capabilities supporting ML model operation.

7. The method according to one of the previous claims, wherein MM I value set of different types is determined by considering the reported UE ML capability and / or ML condition.

8. The method according to one of the previous claims, wherein MM I values is carried in the MAC CE, which is identified by specific LCIDs.

9. The method according to one of the previous claims, wherein MM I reporting with the associated type is included via a specific LCID value assigned.

10. The method according to one of the previous claims, wherein LCID can indicate a specific MMI type information among different types of MMI measure in association with ML model.11 . The method according to one of the previous claims, wherein the actual LCID values for MMI reporting can vary based on the specific implementation and configuration.

12. The method according to one of the previous claims, wherein one or more number of tables is configured to set different types of ML model performance indication measure.

13. The method according to one of the previous claims, wherein the specific data format to be generated for model performance indication measure is configured by network side so that this configuration information is provided to UE via system information or dedicated RRC signaling.

14. The method according to one of the previous claims, wherein there are multiple tables about different types of ML model performance indication measure by providing to UE selectively.2024040793515. The method according to one of the previous claims, wherein Type 1 supports data format of indicating model performance measure for standalone or single model.

16. The method according to one of the previous claims, wherein Type 1 based MM I represents specific MMI value for standalone / single model.

17. The method according to one of the previous claims, wherein Type 2-1 supports data format of indicating model performance measures of each individual subsidiary / partitioned models from multi-model.

18. The method according to one of the previous claims, wherein each subsidiary / partitioned models from multi-model is associated with the assigned model identifiers, respectively with mapping relation between model identifiers and MMI values of each subsidiary / partitioned models for Type 2-1 .

19. The method according to one of the previous claims, wherein the number of subsidiary / partitioned models from multi-model can vary depending on different ML applications or deployment scenarios with specific ML model characteristics.

20. The method according to one of the previous claims, wherein Type 2-2 supports data format of indicating model performance measures of groupwise models by grouping subsidiary / partitioned models from multi-model.21 . The method according to one of the previous claims, wherein groupwise models represents model performance or model operation status in association with different LCM phases with the configured number of bits of MMI value.

22. The method according to one of the previous claims, wherein the averaged MMI values of each models in a group or end-to-end groupwise model measurement for measuring groupwise model is implemented for Type 2-2.2024040793623. The method according to one of the previous claims, wherein each groupwise models is associated with the assigned groupwise model identifiers, respectively with mapping relation between model identifiers and MMI values of each groupwise models for Type 2-2.

24. The method according to one of the previous claims, wherein Type 2-3 supports data format of indicating model performance measure of averaging performance of all subsidiary / partitioned models from multi-model.

25. The method according to one of the previous claims, wherein Type 2-3 represents model performance or model operation status as the averaged MMI value across different models within multi-model in association with different LCM phases with the configured number of bits of MMI value.

26. The method according to one of the previous claims, wherein the averaged MMI value across different models within multi-model is implemented for end-to-end model measurement with mapping relation between model identifiers of multi-model and the averaged MMI value for Type 2-3.

27. The method according to one of the previous claims, wherein Type 2-4 supports data format of indicating model performance measure of delta performance value of subsidiary / partitioned models from multi-model based on the averaged performance measure of all subsidiary / partitioned models from multi-model.

28. The method according to one of the previous claims, wherein Type 2-4 represents model performance or model operation status as the difference between the averaged MMI value across different models within multi-model and the MMI value of each subsidiary and / or partitioned models, which could be a delta measurement value, in association with different LCM phases with the configured number of bits of MMI value.2024040793729. The method according to one of the previous claims, wherein each subsidiary / partitioned models is associated with the assigned model identifiers, respectively with mapping relation between model identifiers and MMI values of delta measurement for Type 2-4.

30. The method according to one of the previous claims, wherein standalone model is operated as single model itself for ML processing.

31. The method according to one of the previous claims, wherein non-standalone model consists of two or more subsidiary / partitioned models for ML processing.

32. The method according to one of the previous claims, wherein measurement criteria for model performance vary depending on implementation-specific such as prediction accuracy or statistical data distribution or any other performance metric.

33. The method according to one of the previous claims, wherein the number of subsidiary / partitioned models for non-standalone model vary depending on different ML use cases and / or deployment scenarios along with the associated model characteristics.

34. The method according to one of the previous claims, wherein subsidiary / partitioned models is for independent functional processing or dependent functional processing to each other.

35. The method according to one of the previous claims, wherein multiple subsidiary / partitioned models with non-standalone model is grouped together so that groupwise model is measured for model performance.

36. The method according to one of the previous claims, wherein group model identifier along with the associated subsidiary / partitioned model identifiers (e.g., when multiple subsidiary / partitioned models is grouped together) is assigned so20240407938 that each entities related to ML model operation is aligned about groupwise model status with model performance measure.

37. The method according to one of the previous claims, wherein models is grouped so as to measure / monitor groupwise functional processing block.

38. The method according to one of the previous claims, wherein one or more groupwise models is used for multiple applications operated by ML model.

39. The method according to one of the previous claims, wherein a higher MMI value indicates better model performance or operation status and a lower MMI value indicates worse model performance or operation status.

40. The method according to one of the previous claims, wherein any configured threshold related to MMI value is used to determine model re-training, switching, updating, or de-activation.

41. The method according to one of the previous claims, wherein MMI value is calculated with model performance measure with the configured MMI type.

42. The method according to one of the previous claims, wherein a specific LCID for MMI reporting is identified for use so that MMI value(s) and the associated type with model identifier information is carried via L1 / L2 signaling.

43. The method according to one of the previous claims, wherein MMI values is carried in the MAC CE, which is identified by specific LCIDs.

44. The method according to one of the previous claims, wherein MMI reporting with the associated type is included via a specific LCID value assigned.

45. The method according to one of the previous claims, wherein LCID can indicate a specific MMI type information among different types of MMI measure in association with ML model.2024040793946. The method according to one of the previous claims, wherein the actual LCID values for MMI reporting can vary based on the specific implementation and configuration.

47. The method according to one of the previous claims, wherein model performance measurement is configured to be periodic or non-periodic so as to apply different types of model performance indication.

48. The method according to one of the previous claims, wherein performance measure with different types of model performance indication is associated with specific periodicity or triggering method using threshold value.

49. The method according to one of the previous claims, wherein any specific type(s) of model performance indication is decided by network side or UE side autonomously (e.g., using 1 -bit indication).

50. The method according to one of the previous claims, wherein one or more ML models is activated or inactive for support depending on different ML use cases and / or deployment scenarios along with the associated model characteristics.51 . The method according to one of the previous claims, wherein target model(s) for model performance measure at UE is in different LCM phases such as data collection, (re-)training, inferencing, updating, switching, monitoring.

52. The method according to one of the previous claims, wherein on network side a set of ML model performance indication types for standalone model (e.g., single model) and non-standalone model (e.g., multi-model such as composite / ensemble model) are configured depending on different ML use cases and / or deployment scenarios along with the associated model characteristics.2024040794053. The method according to one of the previous claims, wherein configuration information about model performance measurement is sent to UE via system information or dedicated RRC signaling.

54. The method according to one of the previous claims, wherein any target model(s) is identified for performance measure so that the configured type of model performance indication is applied to one or more models.

55. The method according to one of the previous claims, wherein feedback message is sent to network side by indicating the specific type(s) used for model performance indication via L1 / L2 or RRC signaling.

56. Apparatus for, the apparatus comprising a wireless transceiver, a processor coupled with a memory in which computer program instructions are stored, said instructions being configured to implement steps of the claims 1 to 55.

57. User Equipment comprising an apparatus according to claim 56.

58. gNB comprising an apparatus according to claim 56.

59. Wireless communication system for of advanced measurement signaling by configuring a set of ML model performance indication types for standalone model as single model and non-standalone model as multi-model / composite / ensemble model, wherein the wireless communication systems comprises user equipment according to claim 57, gNB according to claim 58, whereby the user Equipment and the gNB each comprises a processor coupled with a memory in which computer program instructions are stored, said instructions being configured to implement steps of the claims 1 to 55.

Images