Ensure consistency between training and inference phases via monitoring process

Abstract

An apparatus comprising at least one processor; and at least one non-transitory memory storing instructions that, when executed by the at least one processor, cause the apparatus to perform: sending a capability message, wherein the apparatus supports at least one enabled feature of machine learning, and wherein the capability message comprises an indication of a maximum number of performance monitoring procedures that can be processed by the apparatus for the at least one enabled feature; sending a report based at least in part on the sent capability message, wherein the report comprises: an identifier for a performance monitoring procedure, and associated information related to the performance monitoring procedure; and receiving an activation or selection command from a network based at least in part on the sent report, wherein the command is based at least in part on the identifier of the performance monitoring procedure.

Description

Cross-references to related applications

[0001] This application claims priority and benefit from U.S. Provisional Application No. 63 / 554268, filed February 16, 2024, the contents of which are incorporated herein by reference in their entirety. Technical Field

[0002] The exemplary, and not limiting, embodiments generally relate to machine learning, and more specifically to the consistency between training and inference. Background Technology

[0003] Artificial intelligence (AI, often also called machine learning ML, or even AI / ML) is used for many purposes in wireless networks such as cellular networks. Research into AI / ML technologies continues for wireless communications, including those using the NR air interface. Summary of the Invention

[0004] The following overview is intended as an example only. This overview is not intended to limit the scope of the claims.

[0005] According to one aspect, an example embodiment of an apparatus is provided, the apparatus comprising: at least one processor; and at least one non-transitory memory storing instructions, the instructions causing the apparatus, when executed by the at least one processor, to: send a capability message, wherein the apparatus supports at least one enabled feature for machine learning, and wherein the capability message includes an indication of a maximum number of performance monitoring processes that can be processed by the apparatus for the at least one enabled feature; send a report at least in part based on the sent capability message, wherein the report includes: an identifier for the performance monitoring process and associated information related to the performance monitoring process; and receive an activation or selection command from a network at least in part based on the sent report, wherein the command is at least in part based on the identifier of the performance monitoring process.

[0006] According to one aspect, an example embodiment of a method is provided, the method comprising: sending a capability message to a device, wherein the device supports at least one enabled feature for machine learning, and wherein the capability message includes an indication of a maximum number of performance monitoring processes that can be processed by the device for the at least one enabled feature; sending a report at least in part based on the sent capability message, wherein the report includes: an identifier for the performance monitoring process and associated information related to the performance monitoring process; and receiving an activation or selection command from a network at least in part based on the sent report, wherein the command is at least in part based on the identifier of the performance monitoring process.

[0007] According to one aspect, an example embodiment of an apparatus is provided, the apparatus comprising: components for transmitting a capability message, wherein the apparatus supports at least one enabled feature of machine learning, and wherein the capability message includes an indication of a maximum number of performance monitoring processes that the apparatus can process for the at least one enabled feature; components for transmitting a report at least in part based on the transmitted capability message, wherein the report includes: an identifier for the performance monitoring process and associated information related to the performance monitoring process; and components for receiving an activation or selection command from a network at least in part based on the transmitted report, wherein the command is at least in part based on the identifier of the performance monitoring process.

[0008] According to one aspect, a non-transitory program storage device readable by a device is provided, the non-transitory program storage device tangibly embodying an instruction program executable by the device to perform operations including: sending a capability message, wherein the device supports at least one enabled feature of machine learning, and wherein the capability message includes an indication of a maximum number of performance monitoring processes that the device can process for the at least one enabled feature; sending a report at least in part based on the sent capability message, wherein the report includes: an identifier for the performance monitoring process and associated information related to the performance monitoring process; and receiving an activation or selection command from a network at least in part based on the sent report, wherein the command is at least in part based on the identifier of the performance monitoring process.

[0009] According to one aspect, an example embodiment of an apparatus is provided, the apparatus comprising: at least one processor; and at least one non-transitory memory storing instructions which, when executed by the at least one processor, cause the apparatus to: receive a capability message from a user equipment, wherein the user equipment supports at least one enabled feature for machine learning, and wherein the capability message includes an indication of a maximum number of performance monitoring processes that can be processed by the user equipment for the at least one enabled feature; and, at least in part based on receiving the capability message, send a configuration to the user equipment, wherein the configuration is configured to enable the user equipment to report information related to the performance monitoring processes.

[0010] According to one aspect, an example embodiment of a method is provided, the method comprising: receiving a capability message from a user equipment, wherein the user equipment supports at least one enabled feature for machine learning, and wherein the capability message includes an indication of a maximum number of performance monitoring processes that can be processed by the user equipment for the at least one enabled feature; and sending a configuration to the user equipment, at least in part based on the receipt of the capability message, wherein the configuration is configured to enable the user equipment to report information related to the performance monitoring processes.

[0011] According to one aspect, an example embodiment of an apparatus is provided, the apparatus comprising: components for receiving a capability message from a user equipment, wherein the user equipment supports at least one enabled feature for machine learning, and wherein the capability message includes an indication of a maximum number of performance monitoring processes that can be processed by the user equipment for the at least one enabled feature; and components for sending a configuration to the user equipment, at least in part based on the received capability message, wherein the configuration is configured to enable the user equipment to report information related to the performance monitoring processes.

[0012] According to one aspect, a non-transitory program storage device readable by a device is provided, the non-transitory program storage device tangibly embodying an instruction program executable by the device to perform operations including: receiving a capability message from a user equipment, wherein the user equipment supports at least one enabled feature for machine learning, and wherein the capability message includes an indication of a maximum number of performance monitoring processes that can be processed by the user equipment for the at least one enabled feature; and sending a configuration to the user equipment, at least in part based on the received capability message, wherein the configuration is configured to enable the user equipment to report information related to the performance monitoring processes.

[0013] The subject matter of the independent claims is provided in several respects. The subject matter of the dependent claims also provides several other respects. Attached Figure Description

[0014] The foregoing aspects and other features are explained in the following description in conjunction with the accompanying drawings, wherein: Figure 1 This is a block diagram of one possible and non-limiting example system in which exemplary embodiments can be practiced; Figure 2 This is a diagram illustrating an example method; Figure 3 This is a diagram illustrating an example method; Figure 4 This is a diagram illustrating the example method. Detailed Implementation

[0015] The following abbreviations that may appear in the instruction manual and / or drawings are defined as follows: 3GPP Third Generation Partnership Project 5G (Fifth Generation) AI (Artificial Intelligence) AMF Access and Mobility Management Functions CE control elements CU Centralized Unit DU distribution unit eNB Evolved Node B (e.g., LTE base station) EN-DC LTE-NR Dual Connectivity FG Feature Group gNB is a base station used for 5G / NR, that is, a node that provides NR user plane and control plane protocol termination to the UE and connects to the 5G core network via the NG interface. I / F interface LCM Potential Consistency Model LTE Long Term Evolution MAC Media Access Control ML Machine Learning MME (Mobility Management Entity) ng or NG next generation NR New Radio N / W or NW network OTT over-the-top RAN (Radio Access Network) Rel version RLC Radio Link Control RRC Radio Resource Control RU radio unit Rx receiver SGW Service Gateway SMF Session Management Function TS Technical Specifications Tx transmitter UE (User Equipment) (e.g., wireless equipment, typically mobile equipment) Go to Figure 1 The figure shows a block diagram of a possible, non-limiting example of a wireless network 1 connected to a user equipment (UE) 10. Figure 1 The wireless network shows several network elements, including base station 70 and core network 90.

[0016] exist Figure 1 In this diagram, User Equipment (UE) 10 wirelessly communicates with base station 70 of Network 1 via radio link 11. UE 10 is a wireless communication device configured to access the network, such as a mobile device. UE 10 is shown as having one or more antennas 28. The ellipsis 2 indicates that there may be multiple UEs 10 wirelessly communicating with base station 70 via radio link. UE 10 includes one or more processors 13, one or more memories 15, and other circuitry 16. The other circuitry 16 may include one or more receivers (Rx) 17 and one or more transmitters (Tx) 18. One or more programs 12 are used to cause UE 10 to perform the operations described herein. For UE 10, the other circuitry 16 may include circuitry for user interface elements (not shown), such as a display.

[0017] As a network element of Network 1, base station 70 provides UE 10 with access to Network 1 and data network 91 via core network 90 (e.g., via the user plane function of core network 90). Therefore, base station 70 can be considered an access node or network device that provides UE 10 with access to Network 1. Base station 70 is shown having one or more antennas 58. Typically, base station 70 may be referred to as RAN node 70, although many people refer to it as gNB (gNode B, i.e., a base station for NR (New Radio)). However, there are many other examples of RAN nodes, including eNB (evolved Node B) or TRP (transmitter-receiver point). This document primarily uses the term TRP, and TRPs can have various implementations, such as a single TRP of a base station, a distributed unit, or a radio unit, where multiple such units can be coupled to a centralized unit.

[0018] Base station 70 (or a single TRP) includes one or more processors 73, one or more memories 75, and other circuitry 76. The other circuitry 76 includes one or more receivers (Rx) 77 and one or more transmitters (Tx) 78. One or more programs 72 are used to cause base station 70 to perform the operations described herein.

[0019] It should be noted that base station 70 can also be implemented using other wireless technologies, such as Wi-Fi (a wireless networking protocol used by devices to communicate without a direct cable connection). In the case of Wi-Fi, link 11 can be characterized as a wireless link.

[0020] Two or more base stations 70 communicate using, for example, multiple links 79. The multiple links 79 may be wired, wireless, or a combination of both, and may implement, for example, an Xn interface for 5G (fifth generation), an X2 interface for LTE (long-term evolution), or other suitable interfaces for other standards.

[0021] Network 1 may include a core network 90, such as one or more second network elements that include core network functions and provide connectivity to a data network 91 (e.g., a telephone network and / or a data communication network (e.g., the Internet)) via one or more links 81. The core network 90 includes one or more processors 93, one or more memories 95, and other circuitry 96. The other circuitry 96 includes one or more receivers (Rx) 97 and one or more transmitters (Tx) 98. One or more programs 92 are used to cause the core network 90 to perform the operations described herein.

[0022] The core network 90 can be a 5G core network. The core network 90 can implement or include multiple network functions (NFs) 99, and program 92 can include one or more NFs 99. The 5G core network can use hardware such as memory and processors, as well as virtualization layers. It can be a single standalone computing system, a distributed computing system, or a cloud computing system. As a network element of the core network, an NF 99 can be a container or virtual machine running on the hardware of the computing system(s) constituting the core network 90.

[0023] Core network functions for 5G can include access and mobility management functions provided by network function 99 (such as Access and Mobility Management Function (AMF)) and session management functions provided by network functions such as Session Management Function (SMF). For example, in LTE (Long Term Evolution) networks, core network functions for access and mobility management can be provided by MME (Mobility Management Entity) and / or SGW (Serving Gateway) functions, which route data to the data network. Many other possibilities also exist, such as... Figure 1 Examples shown include: AMF; SMF; MME; SGW; GMLC (Gateway Mobility Location Center); LMF (Location Management Function); UDM (Unified Data Management) / UDR (Unified Data Repository); NRF (Network Repository Function); and / or E-SMLC (Evolved Serving Mobility Location Center). These are merely exemplary core network functions that can be provided by core network 90, and it should be noted that core network 90 can provide both 5G and LTE core network functions simultaneously. Base station 70 is coupled to core network 90 via backhaul link 31. Base station 70 and core network 90 may include an NG (Next Generation) interface for 5G, an S1 interface for LTE, or other suitable interfaces for other radio access technologies communicating via backhaul link 31.

[0024] A computer-readable medium 94 is present in the data network 91. The computer-readable medium 94 contains instructions that, when downloaded and installed into the memory 15, 75, or 95 of the corresponding UE 10, base station 70, and / or core network element 90 and executed by the processor 13, 73, or 93, allow or cause the corresponding device to perform the corresponding actions described herein. The computer-readable medium 94 may be implemented in other forms, such as an optical disc or memory stick.

[0025] Programs 12, 72, and 92 contain instructions stored by one or more corresponding memories 15, 75, or 95. When executed by one or more corresponding processors 13, 73, or 93, these instructions allow or cause the corresponding device 10, 70, or 90 to perform the operations described herein. The computer-readable memories 15, 75, or 95 are circuits and can be of any type suitable for the local technical environment, and can be implemented using any suitable data storage technology, such as semiconductor-based memory devices, flash memory, firmware, magnetic storage devices and systems, optical storage devices and systems, fixed memory, and removable memory. The computer-readable memories 15, 75, and 95 can be components for performing storage functions. Processors 13, 73, and 93 are circuits and can be of any type suitable for the local technical environment. For example, by non-limiting example, these processors may include one or more of a general-purpose computer, a special-purpose computer, a microprocessor, one or more digital signal processors (DSPs), processors based on a multi-core processor architecture, and may also include special-purpose circuitry such as field-programmable gate arrays (FPGAs), application-specific integrated circuits (ASICs), signal processing devices, and other devices or combinations thereof. Processors 13, 73, and 93 may be components for causing the respective devices to perform functions such as those described herein. In particular, for any device having components for performing the functions described herein, the components may include at least one processor and at least one memory storing instructions that, when executed by the at least one processor, cause the device to perform the respective function.

[0026] Receivers 17, 77, and 97, and transmitters 18, 78, and 98 can implement wired and / or wireless interfaces. Receivers and transmitters can be collectively referred to as transceivers.

[0027] Network 1 enables network virtualization, which is the process of combining hardware and software network resources and functions into a single software-based managed entity (virtual network). Network virtualization involves platform virtualization, often combined with resource virtualization. Network virtualization is categorized as external, combining many networks or portions of networks into virtual units, or internal, providing network-like functionality to software containers on a single system. Note that the virtualized entities created by network virtualization (such as network function 99) are still implemented to some extent using hardware such as processors 73 and / or 93 and memory 75 and / or 95, and such virtualized entities also produce technical effects.

[0028] Typically, various embodiments of user equipment 10 may include, but are not limited to: wireless telephones (e.g., smartphones, mobile phones, cellular phones, Voice over Internet Protocol (VoIP) phones, and / or wireless local loop phones), tablet computers, portable computers, vehicles or in-vehicle devices for wireless V2X (vehicle-to-everything) communication, image capture devices such as digital cameras, gaming devices, music storage and playback devices, internet devices (including Internet of Things (IoT) devices), IoT devices with sensors and / or actuators for applications such as automation, portable units or terminals combining these functions, laptop embedded devices (LEE), laptop mounted devices (LME), Universal Serial Bus (USB) dongles, smart devices, wireless customer premises equipment (CPE), IoT devices, watches or other wearable devices, head-mounted displays (HMDs), vehicles, drones, medical devices and applications (e.g., remote surgery), industrial devices and applications (e.g., robots and / or other wireless devices operating in industrial and / or automated processing chain environments), consumer electronics devices, devices operating on commercial and / or industrial wireless networks, etc. In other words, UE 10 can be any terminal device capable of wireless communication. As an example and not a limitation, the UE may also be referred to as a communication device, terminal device (MT), subscriber station (SS), portable subscriber station, mobile station (MS), or access terminal (AT).

[0029] The 102nd RAN meeting, based on the study in FS_NR_AIML_Air [TR 38.843] regarding the application of AI / ML technology to the NR air interface, approved Rel-19 WI [RP-234039] concerning AI / ML for the NR air interface. This includes the following: Target in RP-234039 Provides standard support for the following aspects: - Beam management: DL Tx beam prediction for UE-side and NW-side models, including [RAN1 / RAN2]: Based on the measurement results of beam set B, spatial domain DL Tx beam prediction is performed on beam set A (“BM-Case 1”). Based on the historical measurement results of beam set B, perform time-domain DL Tx beam prediction on beam set A (“BM-Case 2”). ○ Specify the signaling / mechanisms (if any) required to facilitate LCM operation specific to beam management use cases. ○ Implement (multiple) methods to ensure consistency between training and inference regarding NW-side additional conditions (if identified) for inference at the UE. Note: Strive for a common framework design to support both BM-Case 1 and BM-Case 2. - Improved positioning accuracy, including [RAN1 / RAN2 / RAN3]: ○ Direct AI / ML localization: (First Priority) Case 1: UE-based positioning with UE-side model and direct AI / ML positioning (Second Priority) Case 2b: UE-assisted / LMF-based positioning with LMF side model and direct AI / ML positioning (First Priority) Case 3b: NG-RAN node-assisted localization with LMF side model and direct AI / ML localization ○ AI / ML assisted localization: (Second Priority) Case 2a: UE-assisted / LMF-based positioning with UE-side model and AI / ML-assisted positioning (First Priority) Case 3a: NG-RAN node-assisted localization with gNB test model and AI / ML-assisted localization ○ Specify the measurements, signaling / mechanisms (if any) required to facilitate LCM operation specific to the positioning accuracy enhancement use case. ○ Investigate and specify the signaling required for the measurement enhancement (if any). ○ Implement (multiple) methods to ensure consistency between training and inference regarding additional conditions (if identified) on the NW side when inferring at the UE for relevant location sub-use cases. … Based on the above, the following two points will be discussed: Beam management—DL Tx beam prediction for UE-side and NW-side models, including [RAN1 / RAN2]: ○ Implement (multiple) methods to ensure consistency between training and inference regarding NW-side additional conditions (if identified) for inference at the UE. Positioning accuracy is enhanced, including [RAN1 / RAN2 / RAN3]: ○ Implement (multiple) methods to ensure consistency between training and inference regarding additional conditions (if identified) on the NW side when inferring at the UE for relevant location sub-use cases. Rel-19 still requires further research and specification support to "implement (multiple) methods to ensure consistency between training and inference regarding NW-side additional conditions (if identified) for inference at the UE." The following is documented in TR 38.843 when discussing additional conditions.

[0030] 4.2.3 Additional conditions For features / FG that enable AI / ML Additional conditions This refers to any aspect of the UE capabilities assumed during model training but not belonging to the features / FG that enable AI / ML. This does not mean it must be specified. Additional conditions . Additional conditions These can be divided into two categories: additional conditions on the NW side and additional conditions on the UE side. Note: Whether it is necessary to affect the specifications is another issue.

[0031] For inference of the UE-side model, to ensure consistency between training and inference regarding additional NW-side conditions (if identified), the following options can be considered as potential methods (where feasible and necessary): - Model identifiers are used to align the additional conditions on the NW side and the UE side. - Train the model at NW and transmit it to the UE, where the model has already been trained under these additional conditions. - Provide the UE with information and / or instructions regarding additional conditions on the NW side. - Enhancing consistency through monitoring (UE and / or NW monitor the performance of candidate models / functions on the UE side to select models / functions) - Other methods are not excluded Note: The possibility that different methods can achieve the same function is not denied. Although this is not properly documented in TR 38.843, there are aspects that have been commonly referred to as additional conditions in previous discussions, and the following are considered possible examples: ○ Training dataset information ○ Site-related information (e.g., scene, location / TRP / area information, beam direction / codebook information) ○ Time-related information / timestamp information ○ gNB implementation information (explicit or implicit information about specific gNB implementation details) ○ UE Implementation Information (Explicit or implicit information about specific UE implementation details) ○ Statistical information (e.g., latency spread, angular spread, and line-of-sight / non-line-of-sight (LOS / NLOS) data) ○ Speed ​​and speed range information One aspect described in this article relates to “assisting consistency through monitoring (the UE and / or NW monitor the performance of candidate models / functions on the UE side to select a model / function)”, and can optionally consider additional conditions.

[0032] In the context of AI / ML associated with air interface use cases, additional conditions may include any factors assumed during model training but not reported in the UE capability report associated with AI / ML-enabled features / FG. It should be understood that ensuring consistency of NW-side additional conditions between training and inference can be addressed through various potential approaches, such as: - Model Identification: Keeps the models on the NW side and UE side aligned to achieve consistency regarding additional conditions.

[0033] - Train the model at the NW and transmit it to the UE: Train the model at the network (NW) and transmit it to the user equipment, where the model has been trained under additional conditions.

[0034] - Information / instructions from NW: Provide the UE with information and / or instructions regarding additional conditions on the NW side.

[0035] - Facilitate consistency through monitoring: The performance of candidate models / functions on the UE side is monitored by both the UE and NW to promote consistency and enable the selection of appropriate models or functions.

[0036] The features described in this article can be used in the fourth method above to build a framework that can handle additional conditions.

[0037] Also refer to Figure 2 This figure illustrates some features of an example embodiment. When the UE is connected to a network (NW) (as shown in 202), the UE can report a UE capability indication (as shown in 204), whereby the UE reports that it supports one or more ML-enabled features (e.g., beam prediction, CSI prediction). The UE is connected to the NW, and the UE capabilities (including N...) max The capability indication is also sent to the NW. For example, the capability indication might include the following: ○ Maximum number N of performance monitoring procedures (PMPs) that can be processed at the UE max , where N max Each of the performance monitoring processes (PMP) can be associated with a background ML model available at the UE (which is a logical model that can associate one or more physical ML models with the same logical model).

[0038] In one example embodiment, N can be determined based on the number of ML models that the UE wants to identify to the NW. max In other words, N maxThis reveals the maximum number of ML models that can be considered in the model identifier.

[0039] In another example embodiment, N can be determined based on hardware limitations when storing model parameters or a compiled version of the ML model (in other words, the trained ML model) at the UE. max In some examples, N max This can depend on the memory limitations of the UE in storing the model parameters associated with the trained ML model. In some examples, in N max Within the limitations, the UE can download new models from the OTT server / NW, and the UE may have to remove older ML models or update the model parameters of older models (if the downloaded model parameters are of an older model).

[0040] In another example embodiment, N can be determined based on limitations when monitoring active and inactive ML models at the UE. max "Active" can mean that the model is used for inference operations processed by the NW (to support the corresponding functions), while "inactive" can mean that the model is not directly used for inference operations processed by the UE, but the UE can (at least from time to time) perform model inference.

[0041] N max You can report by each ML-enabled feature, or across all ML-enabled features.

[0042] In one example embodiment, optional steps can be provided based on the received UE capabilities. NW can determine the number N of performance monitoring procedures (PMPs) used to configure the UE (where N is less than or equal to N). max ). Figure 2 This is illustrated in step 206. The NW can then anticipate tracking only N performance monitoring procedures (PMPs) of the UE. Each PMP can be identified by a bit field of ceil(log2(N)), which can be used as a PMP identifier (PMP ID). In the example embodiment, this number N can be determined by the AMF. In the example embodiment, this can be interpreted as determining the number of ML models the NW wishes to maintain in the UE's model identifier, and the PMP ID can refer to the model ID. As shown in step 208, the UE can receive a configuration containing information related to the number N of PMPs determined by the NW. Steps 206 and 208 are optional steps that provide control to the NW when defining the limits of PMPs the NW wishes to track.

[0043] As shown in 210, the UE can receive a configuration defined as reporting PMP-related information. The UE receives the feature configuration based on the reported ability to support ML features. This report may include the following: ● A bit field indicating the PMP ID; ● A bit or bit field indicating the report type; ● A bit field indicating at least one UE metric of the indicated PMP ID; and ● Other optional features.

[0044] For bit fields indicating the PMP ID, for example, ceil(log2(N)) or ceil(log2(N)) can be used. max Determine the dimensions. If the maximum number is N max The size of the indicator field can be determined by N. max Determined. For example, the identifier could be of size ceil(log2(N) max The bit field indicating the report type can point to one or more of the following parameters. Bits or bit fields indicating the report type can also reflect changes in the background model. For example, it can indicate whether the indicated PMP ID is associated with an older model (applicable to earlier reports on PMP IDs) or with a newer model (not applicable to earlier reports on PMP IDs). The model may change due to model downloads or updates. For the bit field indicating at least one UE metric for the indicated PMP ID, the UE metric can provide an assessment of the UE model's performance (associated with the ML model corresponding to the PMP ID) from the UE's perspective. The UE metric can contain a value or range determined by the UE based on one or more predefined values ​​for the UE, where the value provides a relative assessment / performance of the ML model when supporting features (or functions) that enable ML. For example, a relative assessment can be provided as follows:

[0045] In one variant, multiple parameters can be used instead of a single metric to define the UE metric for PMP ID, which is then reported by the UE.

[0046] Additionally (optionally), a bit field can be provided to indicate at least one NW metric for the indicated PMP ID. The NW metric provides an assessment of the NW's performance relative to the UE model (as associated with PMP ID monitoring). In the example, this parameter may be applicable when the UE is switching to a target cell and needs to report an earlier NW metric to the target cell. The NW metric may contain a value determined by predefined values ​​in the NW specification, where the value provides a relative assessment / performance of the ML model when supporting features (or functions) that enable ML. In the example, multiple parameters can be used instead of a single metric to define the NW metric for the PMP ID.

[0047] Additionally (optionally), a bit field indicating the configuration ID(s) associated with the PMP ID may be provided. The configuration ID may refer to the function that enables ML features (e.g., the CSI report configuration ID for enabling beam prediction features of ML).

[0048] In the example embodiment, the UE capability report (at step 2) may also carry some of the above information along with the PMP ID to the NW (where the size of the PMP ID is based on N). max (This is confirmed), and NW can regard it as an initial assessment made by the UE.

[0049] As shown in 212, in step 6, the UE may receive a configuration that enables it to report PMP-related information. As shown in 214, in step 7, inference operations for ML features may begin. The NW may initiate inference operations for one or more functions configured for the UE.

[0050] Regarding steps 8 and 9 (216, 218), step 8 becomes relevant only as an optional consideration if steps 3 to 4 are valid. For step 8, the UE can determine whether N is less than N0. max The UE selects N ML models and maps them to the PMP. In step 9, the UE can evaluate / monitor models of active and inactive functions based on available measurements accessible to the UE. For example, the UE can effectively evaluate model performance when the gNB frequently transmits a large number of DL RS. Furthermore, in some cases, the UE server (OTT) may have already sent some evaluations of new models, and these evaluations at the OTT can be taken from previous evaluations associated with the corresponding ML models.

[0051] Regarding the UE, when the UE is connected to the NW, the following situations may occur at any point in time: ● For one or more PMP IDs, i.e., for the applicable model representing one or more PMP IDs, the UE can perform performance monitoring or model evaluation and identify any changes associated with the background ML model (new or older).

[0052] ● Based on this monitoring or assessment, the UE can derive UE metrics for one or more PMP IDs.

[0053] ● If applicable, the UE can also determine the NW metric (based on the most recent value received from the NW).

[0054] ● If applicable, the UE can also identify functions associated with one or more PMP IDs (which may only be relevant when reporting changes related to the new model).

[0055] ● When updating an existing model or downloading a new model, the UE may have an initial assessment when determining the above parameters (this initial assessment may also be received from the entity that sent the model).

[0056] ● The UE can report one or more PMP IDs and corresponding parameters of PMPID based on reports initiated by the NW or by the UE.

[0057] ● In one variant, the UE can report the PMP ID and corresponding parameters as a UE-triggered MAC-CE command, in which the fields defined above are carried in the MAC-CE command.

[0058] Regarding NW, when a UE connects to an NW, the following situations may occur at any point in time: ● NW can perform performance monitoring / evaluation on one or more PMP IDs and can identify any changes associated with NW metrics.

[0059] ● NW can indicate to the UE the changes associated with the NW metric and the corresponding PMP ID.

[0060] ● In one variant, the UE can report the PMP ID and corresponding parameters as a UE-triggered MAC-CE command, in which the fields defined above are carried in the MAC-CE command.

[0061] ● Based on UE evaluation, NW can receive reports related to one or more of the N PMPs.

[0062] Step 10 at 220 shows that the UE can send a report containing model evaluations associated with one or more PMP IDs. The reported PMP-related information may include, for example, the PMP-ID and associated report type, UE metrics, NW metrics, and function IDs. Step 11 at 222 shows that the NW can perform internal performance evaluations on PMP IDs based on activated functions.

[0063] Based on the reported PMP-related information, the NW can initiate signaling by implicitly referencing the PMP-ID in the signaling indication to select, switch, activate, or deactivate the background ML model used at the UE. At the NW, similar to how the UE can perform performance monitoring and evaluation, the NW can determine the evaluation (NW metric) of one or more PMP IDs considered for the active function based on its own learning at the NW. As shown in step 12, this information can be further reported to the UE. As shown at 228 in step 14, the UE can update the NW metric associated with the corresponding PMP ID. Step 12 at 224 shows the UE receiving PMP-related information from the NW side, such as the PMP ID and NW metric. This can be referred to as model ID-based LCM, and in this case, PMPID can refer to the model ID. This can also be referred to as conditional processing, where the NW can use the reported performance indicator and associated function when determining the optimal background ML model given the NW situation.

[0064] In step 13 shown in 226, the NW can, for example, store PMP-related information under each corresponding PMP-ID. In addition to the parameters reported by the UE, the NW may also consider storing timestamp information, NW assumptions, configuration details, and other types of information useful when handling additional conditions. Here, the NW can also maintain past reports corresponding to a specific PMP-ID, provided that the reports within a PMP-ID are related to each other (i.e., related to the same background model).

[0065] Referring also to step 15 at 230 and step 16 at 232, with a full understanding of the performance associated with PMP ID and its past and present applicability, the NW can determine the optimal background model for the UE via PMP ID, both in terms of functionality and additional conditions. As shown in step 16, when there is an NW assumption that matches the NW additional conditions associated with the background model, signaling can consider PMP ID when processing the UE's background model.

[0066] The UE may retain only a limited number of ML models in the device, and the NW can implicitly control these ML models using the features described herein. When an ML model is changed or updated, the NW only needs to focus on the details of the latest ML model. There is no need to store unnecessary information about older versions. Since the UE can perform model performance evaluations via inactive model monitoring or through past learning, and signaling allows access to such information about UE model evaluations and provides a framework for updating these evaluations, the features described herein can be used to improve the handling of additional conditions through the monitoring process. For example, if the UE evaluates model UE-model_1 under a first set of NW assumptions (UE unknown) and reports the evaluation to the NW, and then the same UE model updates its evaluation under a second set of NW assumptions (UE unknown) and reports that evaluation to the NW, then over time the NW will have the UE's evaluations of UE-model_1 for different NW assumptions. Since such evaluations can also be derived for other models (models 2, 3, etc.) through a similar process, the NW can use the performance monitoring process ID to select or activate the ML model when matching NW additional conditions are used for inference.

[0067] An apparatus according to an example embodiment may be provided, comprising: at least one processor; and at least one non-transitory memory storing instructions that, when executed by the at least one processor, cause the apparatus to: send a capability message, wherein the apparatus supports at least one enabled feature for machine learning, and wherein the capability message includes an indication of a maximum number of performance monitoring processes that can be processed by the apparatus for the at least one enabled feature; send a report at least in part based on the sent capability message, wherein the report includes: an identifier for the performance monitoring process and associated information related to the performance monitoring process; and receive an activation or selection command from a network at least in part based on the sent report, wherein the command is at least in part based on the identifier of the performance monitoring process.

[0068] The performance monitoring process may be associated with a machine learning model. The command can be configured to enable the use of the machine learning model associated with the performance monitoring process. The identifier can be determined based on the maximum number of performance monitoring processes. The associated information may include at least one of the following: an indication of whether there is a relationship between the report and an earlier report associated with the same performance monitoring process; a user equipment metric determined at least partially at the device, wherein the user equipment metric provides a relative performance indication of the performance monitoring process; a network metric received from the network, wherein the network metric provides a relative performance indication of the performance monitoring process; or an identifier of at least one feature configuration associated with the performance monitoring process, wherein the feature configuration is received from the network to enable the machine learning feature. The device may include a user equipment, and the capability message may be a user equipment capability message of the user equipment; the maximum number of performance monitoring processes that can be processed by the device may be the maximum number of performance monitoring processes that can be processed by the user equipment when at least one enabled feature of machine learning is supported at the user equipment. The machine learning may include a background machine learning model, and the performance monitoring process may be associated with a background machine learning model available to the user equipment. The report may be configured or defined to the user equipment at least partially based on the capability message. When executed by the at least one processor, the instruction can cause the device to perform: receiving at least one function configuration based at least in part on the sent capability message; receiving a configuration configured to enable reporting of performance monitoring process-related information with the report, based at least in part on the sent capability message; initiating an inference operation for one or more functions configured to be configured to the user equipment using input from the network; monitoring machine learning performance related to active and inactive functions based on available measurements accessible to the device; evaluating machine learning performance related to active and inactive functions based on available measurements accessible to the device; receiving performance monitoring process information from the network and updating network metrics associated with the performance monitoring process; receiving a configuration from the network including information related to the number of performance monitoring processes determined by the network; determining when the number of performance monitoring processes in the information received from the network is less than the maximum number of performance monitoring processes that the device can process, and selecting the smaller number of models to map to the performance monitoring processes(s).

[0069] Also refer to Figure 3An example method may be provided, comprising: sending a capability message to a device, as shown in box 302, wherein the device supports at least one enabled feature for machine learning, and wherein the capability message includes an indication of a maximum number of performance monitoring procedures that the device can handle for the at least one enabled feature; sending a report, as shown in box 304, at least in part based on the sent capability message, wherein the report includes: an identifier for the performance monitoring procedure and information associated with the performance monitoring procedure; and receiving an activation or selection command from a network, as shown in box 306, at least in part based on the sent report, wherein the command is at least in part based on the identifier of the performance monitoring procedure. The performance monitoring procedure may be associated with a machine learning model. The command may be configured to enable the use of the machine learning model associated with the performance monitoring procedure. The identifier may be determined based on the maximum number of performance monitoring procedures. The associated information may include at least one of the following: an indication of whether there is a relationship between the report and an earlier report associated with the same performance monitoring process; a user equipment metric determined at least partially at the device, wherein the user equipment metric provides a relative performance indication of the performance monitoring process; a network metric received from the network, wherein the network metric provides a relative performance indication of the performance monitoring process; or an identifier of at least one feature configuration associated with the performance monitoring process, wherein the feature configuration is received from the network to enable the machine learning feature. The device may include a user equipment, and the capability message may be a user equipment capability message of the user equipment; the maximum number of performance monitoring processes that can be processed by the device may be the maximum number of performance monitoring processes that can be processed by the user equipment when at least one enabled feature supports machine learning at the user equipment. The machine learning may include a background machine learning model, and the performance monitoring process may be associated with a background machine learning model available to the user equipment. The report may be configured or defined to the user equipment at least partially based on the capability message.The method may further include: receiving at least one function configuration based at least in part on the sent capability message; receiving, at least in part on the sent capability message, a configuration configured to enable reporting of performance monitoring process-related information with the report; initiating an inference operation for one or more functions configured to be configured to the user device using input from the network; monitoring machine learning performance related to active and inactive functions based on available measurements accessible to the device; evaluating machine learning performance related to active and inactive functions based on available measurements accessible to the device; receiving performance monitoring process information from the network and updating network metrics associated with the performance monitoring process; and receiving from the network a configuration including information related to the number of performance monitoring processes determined by the network, determining when the number of performance monitoring processes in the information received from the network is less than the maximum number of performance monitoring processes that can be processed by the device, and selecting the smaller number of models to map to the performance monitoring processes(s).

[0070] An apparatus according to an example embodiment may be provided, the apparatus comprising: components for transmitting a capability message, wherein the apparatus supports at least one enabled feature of machine learning, and wherein the capability message includes an indication of a maximum number of performance monitoring processes that the apparatus can process for the at least one enabled feature; components for transmitting a report at least in part based on the transmitted capability message, wherein the report includes: an identifier for the performance monitoring process and information associated with the performance monitoring process; and components for receiving an activation or selection command from a network at least in part based on the transmitted report, wherein the command is at least in part based on the identifier of the performance monitoring process.

[0071] A non-transitory program storage device readable by a device may be provided, the non-transitory program storage device tangibly embodying an instruction program executable by the device to perform operations including: sending a capability message, wherein the device supports at least one enabled feature of machine learning, and wherein the capability message includes an indication of a maximum number of performance monitoring processes that the device can handle for the at least one enabled feature; sending a report at least in part based on the sent capability message, wherein the report includes: an identifier for the performance monitoring process and information associated with the performance monitoring process; and receiving an activation or selection command from a network at least in part based on the sent report, wherein the command is at least in part based on the identifier of the performance monitoring process.

[0072] An apparatus of an example embodiment may be provided, the apparatus comprising: at least one processor; and at least one non-transitory memory storing instructions which, when executed by the at least one processor, cause the apparatus to: receive a capability message from a user equipment, wherein the user equipment supports at least one enabled feature for machine learning, and wherein the capability message includes an indication of a maximum number of performance monitoring processes that the user equipment can process for the at least one enabled feature; and, at least in part based on receiving the capability message, send a configuration to the user equipment, wherein the configuration is configured to enable the user equipment to report information related to the performance monitoring processes.

[0073] The performance monitoring process in the performance monitoring process information can be associated with a machine learning model. When executed using the at least one processor, the instruction can cause the device to perform: receiving a report from the user equipment, wherein the report includes: an identifier for the performance monitoring process and associated information related to the performance monitoring process; determining, at least in part, an evaluation of one or more performance monitoring processes considered for use in the activity function based on machine learning at the device, based on the receipt of the report; sending information about the evaluation to the user equipment, wherein the information includes one or more performance monitoring process IDs and one or more network metrics for the one or more performance monitoring process IDs; storing the information under each corresponding performance monitoring process ID, wherein the storage may include storing at least one of the following: timestamp information, network hypotheses, configuration details, or other types of information for handling additional conditions; determining one or more background models for use by the user equipment, wherein the determination uses the performance monitoring process ID; determining the existence of network hypotheses as network additional conditions matching the associated background models; and based on the determined match, sending the performance monitoring process ID for use by the user equipment for the determined one or more background models.

[0074] Also refer to Figure 4An example method may be provided, comprising: receiving, as shown in box 402, a capability message from a user equipment, wherein the user equipment supports at least one enabled feature for machine learning, and wherein the capability message includes an indication of a maximum number of performance monitoring procedures that can be processed by the user equipment for the at least one enabled feature; and, as shown in box 404, sending a configuration to the user equipment, at least in part based on the receipt of the capability message, wherein the configuration is configured to enable the user equipment to report performance monitoring procedure-related information. The performance monitoring procedures in the performance monitoring procedure-related information may be associated with a machine learning model. The method may further include: receiving a report from the user equipment, wherein the report includes: an identifier for a performance monitoring process and associated information related to the performance monitoring process; determining an evaluation for one or more performance monitoring processes considered for use in the activity function, based at least in part on the receipt of the report, according to machine learning at the device; sending information about the evaluation to the user equipment, wherein the information includes one or more performance monitoring process IDs and one or more network metrics for the one or more performance monitoring process IDs; storing the information under each corresponding performance monitoring process ID, wherein the storage may include storing at least one of the following: timestamp information, network hypotheses, configuration details, or other types of information for handling additional conditions; determining one or more background models for use by the user equipment, wherein the determination may be indexed at least in part using the performance monitoring process IDs; determining the existence of network hypotheses as network additional conditions that match the associated background models; and based on the determination that the match exists, sending the determined performance monitoring process IDs of the one or more background models for use by the user equipment.

[0075] An apparatus according to an example embodiment may be provided, the apparatus comprising: means for receiving a capability message from a user equipment, wherein the user equipment supports at least one enabled feature of machine learning, and wherein the capability message includes an indication of a maximum number of performance monitoring processes that can be processed by the user equipment for the at least one enabled feature; and means for sending a configuration to the user equipment, at least in part based on receiving the capability message, wherein the configuration is configured to enable the user equipment to report information related to the performance monitoring processes.

[0076] A non-transitory program storage device readable by a device may be provided, the non-transitory program storage device tangibly embodying an instruction program executable by the device to perform operations including: receiving a capability message from a user equipment, wherein the user equipment supports at least one enabled feature of machine learning, and wherein the capability message includes an indication of a maximum number of performance monitoring processes that can be processed by the user equipment for the at least one enabled feature; and sending a configuration to the user equipment, at least in part based on receiving the capability message, wherein the configuration is configured to enable the user equipment to report information related to the performance monitoring processes.

[0077] An apparatus according to an example embodiment may be provided, comprising: components for the UE to report in a UE capability message the following: the maximum number of performance monitoring procedures (PMPs) that the UE can process when supporting at least one ML-enabled feature at the UE, wherein each performance monitoring procedure is associated with a background ML model available to the UE; and components for the UE to report in a report configured or defined to the UE, wherein the report carries at least an identifier for at least one performance monitoring procedure (PMP) and information associated with the at least one performance monitoring procedure, wherein the identifier is determined based on the maximum number of performance monitoring procedures, and the associated information includes one or more of the following: ● Is there any relationship between the latest report and earlier reports related to the same PMP? ● A UE metric determined at the UE, which provides the relative performance of the PMP; ● NW metrics received from NW at an earlier time, where the metric provides the relative performance of the PMP; ● Identifiers of one or more ML configurations associated with the PMP, wherein the ML configuration is received from the NW to enable ML features; and ● A component for receiving an activation or selection command from the NW by the UE, wherein the command is based on an identifier of at least one PMP, and the activation or selection enables the use of a background ML model associated with the at least one PMP.

[0078] As used in this article, the term “non-transient” refers to a limitation on the medium itself (i.e., tangible rather than signal-based), rather than a limitation on the persistence of data storage (e.g., RAM and ROM).

[0079] As used in this application, the term "circuit" may refer to one, more, or all of the following: (a) Hardware circuit implementation only (e.g., implementation only in analog and / or digital circuits); and (b) A combination of hardware circuitry and software, such as (if applicable): (i) A combination of analog and / or digital hardware circuitry with software / firmware; (ii) Any part of a hardware processor having software (including (multiple) digital signal processors, software, and (multiple) memories, which work together to enable a device such as a mobile phone or server to perform various functions); and (iii) A hardware circuit and / or processor, such as a microprocessor or part of a microprocessor, that requires software (e.g. firmware) for operation, but may be absent if operation does not require the software.

[0080] This definition of "circuit" applies to all uses of the term in this application, including its use in any claim. As another example, as used herein, the term "circuit" also covers only hardware circuitry or a processor (or processors), a portion of hardware circuitry or a processor, and its accompanying software and / or firmware implementation. The term "circuit" also covers, for example (as applicable to specific claim elements), baseband integrated circuits or processor integrated circuits used in mobile devices, or similar integrated circuits in servers, cellular network devices, or other computing or network devices.

[0081] It should be understood that the foregoing description is illustrative only. Various alternatives and modifications can be devised by those skilled in the art. For example, the features recited in the dependent claims can be combined with each other in any suitable combination. Furthermore, features of the different embodiments described above can be selectively combined to form new embodiments. Therefore, this description is intended to cover all such alternatives, modifications, and variations that fall within the scope of the appended claims.

Claims

1. An apparatus comprising: At least one processor; as well as At least one non-transitory memory storing instructions that, when executed using the at least one processor, cause the device to perform: Send a capability message, wherein the device supports at least one enabled feature for machine learning, and wherein the capability message includes an indication of the maximum number of performance monitoring processes that the device can process for the at least one enabled feature; Reports are sent based at least in part on the capability messages sent, wherein the reports include: Identifiers for the performance monitoring process, and Related information associated with the performance monitoring process; as well as Activation or selection commands are received from the network, at least in part based on the reports sent, wherein the commands are at least in part based on the identifier of the performance monitoring process.

2. The apparatus of claim 1, wherein the performance monitoring process is associated with a machine learning model.

3. The apparatus of claim 2, wherein the command is configured to enable the use of the machine learning model associated with the performance monitoring process.

4. The apparatus according to any one of claims 1 to 3, wherein the identifier is determined based on the maximum number of the performance monitoring processes.

5. The apparatus according to any one of claims 1 to 4, wherein the associated information includes at least one of the following: An indication of whether there is a relationship between the report and earlier reports associated with the same performance monitoring process; User equipment metrics determined at least in part at the device, wherein the user equipment metrics provide a relative performance indication of the performance monitoring process; Network metrics received from the network, wherein the network metrics provide a relative performance indication of the performance monitoring process; or An identifier for at least one functional configuration associated with the performance monitoring process, wherein the functional configuration is received from the network to enable machine learning features.

6. The apparatus according to any one of claims 1 to 5, wherein the apparatus includes a user equipment, and the capability message is a user equipment capability message of the user equipment, and wherein the maximum number of performance monitoring processes that can be processed by the apparatus is the maximum number of performance monitoring processes that can be processed by the user equipment when the user equipment supports the at least one enabled feature of the machine learning at the user equipment.

7. The apparatus of any one of claims 1 to 6, wherein the machine learning includes a background machine learning model, and wherein the performance monitoring process is associated with the background machine learning model available to the user device.

8. The apparatus according to any one of claims 1 to 7, wherein the report is configured or defined to the user equipment at least in part based on the capability message.

9. The apparatus according to any one of claims 1 to 8, wherein the instructions cause the apparatus to execute when executed using the at least one processor: At least one feature configuration is received, based at least in part on the sent capability message.

10. The apparatus according to any one of claims 1 to 9, wherein the instructions cause the apparatus to perform when executed using the at least one processor: Based at least in part on the sent capability messages, the receiver is configured to report information related to the performance monitoring process, along with the reports.

11. The apparatus according to any one of claims 1 to 9, wherein the instructions cause the apparatus to execute when executed using the at least one processor: Using input from the network, inference operations are initiated for one or more functions configured for the user equipment.

12. The apparatus according to any one of claims 1 to 10, wherein the instructions cause the apparatus to execute when executed using the at least one processor: The performance of machine learning in relation to active and inactive functions can be monitored based on available measurements accessible by the device.

13. The apparatus of claim 12, wherein the instructions cause the apparatus to perform when executed using the at least one processor: The performance of machine learning in relation to the active and inactive functions is evaluated based on the available measurements accessible by the device.

14. The apparatus according to any one of claims 1 to 13, wherein the instructions cause the apparatus to perform when executed using the at least one processor: Receive performance monitoring process information from the network; and Update the network metrics associated with the performance monitoring process.

15. The apparatus according to any one of claims 1 to 14, wherein the instructions cause the apparatus to execute when executed using the at least one processor: Configuration for receiving information from the network, including information related to the number of performance monitoring processes determined by the network.

16. The apparatus of claim 15, wherein the instructions cause the apparatus to perform when executed using the at least one processor: Determine when the number of performance monitoring processes in the information received from the network is less than the maximum number of performance monitoring processes that the device can process; and A smaller number of models are selected for mapping with the performance monitoring process.

17. A method comprising: Send a capability message to the device, wherein the device supports at least one enabled feature for machine learning, and wherein the capability message includes an indication of the maximum number of performance monitoring processes that can be processed by the device for the at least one enabled feature; Reports are sent based at least in part on the capability messages sent, wherein the reports include: Identifiers for the performance monitoring process, and Related information associated with the performance monitoring process; as well as Activation or selection commands are received from the network, at least in part based on the reports sent, wherein the commands are at least in part based on the identifier of the performance monitoring process.

18. An apparatus comprising: A component for sending capability messages, wherein the device supports at least one enabled feature for machine learning, and wherein the capability message includes an indication of the maximum number of performance monitoring processes that can be processed by the device for the at least one enabled feature; A component for sending a report based at least in part on the sent capability message, wherein the report includes: Identifiers for the performance monitoring process, and Related information associated with the performance monitoring process; as well as A component for receiving activation or selection commands from the network based at least in part on a sent report, wherein the commands are based at least in part on an identifier of the performance monitoring process.

19. An apparatus comprising: At least one processor; as well as At least one non-transitory memory storing instructions that, when executed using the at least one processor, cause the device to perform: Receive a capability message from a user equipment, wherein the user equipment supports at least one enabled feature for machine learning, and wherein the capability message includes an indication of the maximum number of performance monitoring processes that can be processed by the user equipment for the at least one enabled feature. as well as At least in part based on receiving the capability message, a configuration is sent to the user equipment, wherein the configuration is configured to enable the user equipment to report information related to the performance monitoring process.

20. The apparatus of claim 19, wherein the performance monitoring process in the performance monitoring process related information is associated with a machine learning model.

21. The apparatus according to any one of claims 19 to 20, wherein the instructions cause the apparatus to perform when executed using the at least one processor: Receive a report from the user equipment, wherein the report includes: Identifiers for the performance monitoring process; as well as Related information associated with the performance monitoring process.

22. The apparatus of claim 21, wherein the instructions, when executed using the at least one processor, cause the apparatus to perform: Based at least in part on the receipt of the report, an evaluation of one or more performance monitoring processes considered for use in the active functions is determined using machine learning at the device.

23. The apparatus of claim 22, wherein the instructions cause the apparatus to perform when executed using the at least one processor: Send information about the assessment to the user equipment, wherein the information includes one or more performance monitoring process IDs and one or more network metrics for the one or more performance monitoring process IDs.

24. The apparatus of claim 22, wherein the instructions cause the apparatus to perform when executed using the at least one processor: The information is stored under each corresponding performance monitoring process ID.

25. The apparatus of claim 24, wherein the storage further comprises storing at least one of the following: Timestamp information Network hypothesis, Configuration details, or Other types of information used to process additional conditions.

26. The apparatus according to any one of claims 19 to 25, wherein the instructions cause the apparatus to perform when executed using the at least one processor: One or more background models are determined for use by the user equipment, wherein the determination is made using the performance monitoring process ID.

27. The apparatus of claim 26, wherein the instructions cause the apparatus to perform when executed using the at least one processor: Identify network hypotheses that serve as additional conditions for matching the associated background model; and Based on the determined match, the performance monitoring process ID for the determined one or more background models is sent to the user equipment for use by the user equipment.

Images