Model function monitoring

Abstract

The embodiment of the invention relates to model function monitoring. A terminal device receives, from a network device, first information for configuring at least one time window for monitoring performance of an AI / ML model function. The terminal device receives second information indicating a first time window during which the performance of the AI / ML model function is higher than the performance level or a second time window during which the performance of the AI / ML model function is lower than the performance level. And the terminal device performs management operations associated with a plurality of AI / ML models configured to implement AI / ML model functions based on the second information. The solution for model function monitoring provided by the present disclosure can allow the network device to maintain reliable operation without exactly knowing which model is being used by the terminal device.

Description

Technical Field

[0001] Various example embodiments generally relate to the field of communications, and more specifically to terminal devices, network devices, methods, apparatuses, and computer-readable storage media for model function monitoring. Background Technology

[0002] With the development of communication technologies, model-based new radio (NR) air interfaces and resource allocation schemes have been studied. For example, in the 3GPP Release 18 (Rel-18) research project (SI), the goal was to explore the benefits of leveraging features to enhance the air interface, which enable improved support for AI / ML-based algorithms to enhance performance and / or reduce complexity / overhead.

[0003] As an example, in RAN#111, to distinguish between AI / ML models and the functions supported by AI / ML models, two different AI / ML-related identifier types (function identifier and model identifier) ​​are introduced. It is assumed that the model identifier uses a "model ID" during the identifier process, and that the function identifier uses a "function ID" (with or without an explicit model ID) during the identifier process. Summary of the Invention

[0004] Typically, exemplary embodiments of this disclosure provide a terminal device, network device, method, apparatus, and computer-readable storage medium for monitoring the functionality of AI / ML models. For example, exemplary embodiments of this disclosure provide solutions that allow network devices to maintain reliable AI / ML operation without knowing precisely which AI / ML model the terminal device is using.

[0005] In a first aspect, a terminal device is provided. The terminal device may include: at least one processor; and at least one memory storing instructions, which, when executed by the at least one processor, cause the terminal device to at least: receive first information from a network device for configuring at least one time window for monitoring the performance of an AI / ML model function; receive second information from the network device indicating that, during the first time window or the second time window, the performance of the AI / ML model function is above a performance level, and during the second time window, the performance of the AI / ML model function is below a performance level; and, based on the second information, perform management operations associated with a plurality of AI / ML models configured to implement the AI / ML model function.

[0006] In a second aspect, a network device is provided. The network device may include: at least one processor; and at least one memory storing instructions, which, when executed by the at least one processor, cause the network device to at least: send first information to an end device for configuring at least one time window for monitoring the performance of an AI / ML model function; determine a first time window or a second time window within the at least one time window, during which the performance of the AI / ML model function is above a performance level, and during the second time window, the performance of the AI / ML model function is below a performance level; and send second information to the end device indicating the first time window or the second time window.

[0007] In a third aspect, a method is provided. The method may include: at a terminal device, receiving first information from a network device for configuring at least one time window for monitoring the performance of an AI / ML model function; at the terminal device, receiving second information from the network device indicating that during the first time window or the second time window of the at least one time window, the performance of the AI / ML model function is above a performance level, and during the second time window, the performance of the AI / ML model function is below a performance level; and at the terminal device, based on the second information, performing management operations associated with multiple AI / ML models configured to implement the AI / ML model function.

[0008] In a fourth aspect, a method is provided. The method may include: at a network device, sending first information to a terminal device for configuring at least one time window for monitoring the performance of an AI / ML model function; at the network device, determining a first time window or a second time window among the at least one time window, during which the performance of the AI / ML model function is above a performance level, and during which the performance of the AI / ML model function is below a performance level; and at the network device, sending second information to the terminal device indicating at least one of the first or second time windows.

[0009] In a fifth aspect, an apparatus for a terminal device is provided. The apparatus may include: components for receiving first information from a network device for configuring at least one time window, the at least one time window being used to monitor the performance of an AI / ML model function; components for receiving second information from the network device, the second information indicating that during the first time window or the second time window of the at least one time window, the performance of the AI / ML model function is above a performance level, and during the second time window, the performance of the AI / ML model function is below a performance level; and components for performing management operations associated with a plurality of AI / ML models configured to implement the AI / ML model function based on the second information.

[0010] In a sixth aspect, an apparatus for a network device is provided. The apparatus may include: means for sending first information to a terminal device for configuring at least one time window, the at least one time window being used to monitor the performance of an AI / ML model function; means for determining at the network device either a first time window or a second time window among the at least one time window, during the first time window the performance of the AI / ML model function is above a performance level, and during the second time window the performance of the AI / ML model function is below a performance level; and means for sending second information to the terminal device indicating at least one of the first time window or the second time window.

[0011] In a seventh aspect, a non-transitory computer-readable medium is provided, comprising program instructions for causing a device to perform at least the method according to a third or fourth aspect.

[0012] In an eighth aspect, a computer program is provided, comprising instructions that, when executed by a device, cause the device to at least: receive first information from a network device for configuring at least one time window for monitoring the performance of an AI / ML model function; receive second information from the network device indicating that, during the first time window or the second time window of the at least one time window, the performance of the AI / ML model function is above a performance level, and during the second time window, the performance of the AI / ML model function is below a performance level; and perform management operations associated with a plurality of AI / ML models configured to implement the AI / ML model function based on the second information.

[0013] In a ninth aspect, a computer program is provided, comprising instructions that, when executed by a device, cause the device to at least: send first information to a terminal device for configuring at least one time window for monitoring the performance of an AI / ML model function; determine a first time window or a second time window among the at least one time window, during which the performance of the AI / ML model function is above a performance level and during which the performance of the AI / ML model function is below a performance level; and send second information to the terminal device indicating the first time window or the second time window.

[0014] In a tenth aspect, a terminal device is provided. The terminal device may include: a first receiving circuit configured to receive from a network device first information for configuring at least one time window, the at least one time window being used to monitor the performance of an AI / ML model function; a second receiving circuit configured to receive from the network device second information indicating that, during the first time window or the second time window of the at least one time window, the performance of the AI / ML model function is higher than a performance level, and during the second time window, the performance of the AI / ML model function is lower than a performance level; and an execution circuit configured to perform management operations associated with a plurality of AI / ML models configured to implement the AI / ML model function based on the second information.

[0015] In an eleventh aspect, a network device is provided. The network device may include: a first transmitting circuit configured to transmit to a terminal device first information for configuring at least one time window, the at least one time window being used to monitor the performance of an AI / ML model function; a determining circuit configured to determine either a first time window or a second time window within the at least one time window, wherein the performance of the AI / ML model function is above a performance level during the first time window, and the performance of the AI / ML model function is below a performance level during the second time window; and a second transmitting circuit configured to transmit to the terminal device second information indicating the first time window or the second time window.

[0016] It should be understood that the summary portion of this disclosure is not intended to identify key or essential features of the embodiments thereof, nor is it intended to limit the scope of this disclosure. Other features of this disclosure will become readily apparent from the following description. Attached Figure Description

[0017] Some exemplary embodiments will now be described with reference to the accompanying drawings, in which: Figure 1A An example network environment in which example embodiments of this disclosure may be implemented is shown; Figure 1BExample illustrations of AI / ML model function monitoring related to some embodiments of this disclosure are shown; Figure 2 Example signaling procedures for monitoring the functionality of AI / ML models according to some embodiments of this disclosure are shown; Figure 3 Another example signaling procedure for monitoring the functionality of an AI / ML model, according to some embodiments of this disclosure, is shown; Figure 4 Example flowcharts of methods implemented at a terminal device according to some example embodiments of the present disclosure are shown; Figure 5 Example flowcharts are shown of methods implemented at a network device according to some example embodiments of the present disclosure; Figure 6 Simplified example block diagrams of devices suitable for implementing embodiments of the present disclosure are shown; and Figure 7 Example block diagrams of example computer-readable media according to some embodiments of the present disclosure are shown; Throughout the accompanying drawings, the same or similar reference numerals denote the same or similar elements. Detailed Implementation

[0018] The principles of this disclosure will now be described with reference to some exemplary embodiments. It should be understood that these embodiments are described for illustrative purposes and to assist those skilled in the art in understanding and implementing this disclosure, and do not imply any limitation on the scope of this disclosure. The disclosure described herein can be implemented in various ways other than those described below.

[0019] In the following description and claims, unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this disclosure pertains.

[0020] References to "an embodiment," "an embodiment," "an example embodiment," etc., in this disclosure indicate that the described embodiment may include a particular feature, structure, or characteristic, but not every embodiment necessarily includes that particular feature, structure, or characteristic. Furthermore, such phrases do not necessarily refer to the same embodiment. Additionally, when a particular feature, structure, or characteristic is described in connection with an embodiment, it is believed that its influence on such feature, structure, or characteristic in conjunction with other embodiments is within the knowledge of those skilled in the art, whether explicitly described or not.

[0021] It is understood that although the terms “first” and “second”, etc., may be used herein to describe various elements, these elements should not be limited by these terms. These terms are used only to distinguish one element from another. For example, a first element may be referred to as a second element, and similarly, a second element may be referred to as a first element, without departing from the scope of the exemplary embodiments. As used herein, the term “and / or” includes any and all combinations of one or more of the listed terms.

[0022] The terminology used herein is for the purpose of describing particular embodiments only and is not intended to limit the exemplary embodiments. As used herein, the singular forms “a,” “an,” and “the” are also intended to include the plural forms unless the context clearly indicates otherwise. It will be further understood that the terms “comprising,” “including,” “having,” “having,” “containing,” and / or “comprising” as used herein specify the presence of stated features, elements, and / or components, etc., but do not exclude the presence or addition of one or more other features, elements, components, and / or combinations thereof. As used herein, “at least one of the following: ” and “at least one of ” and similar wording, wherein a list of two or more elements is combined by “and” or “or”, means at least any one of the elements, or at least any two or more of the elements, or at least all of the elements.

[0023] As used in this application, the term "circuit" may refer to one or more of the following: (a) Hardware circuit implementation only (e.g., implemented with purely analog and / or digital circuits), and (b) A combination of hardware circuitry and software, such as (if applicable): (i) A combination of (multiple) analog and / or digital hardware circuits and software / firmware, and (ii) Any part of a hardware processor having software (including (multiple) digital signal processors, software, and (multiple) memories, which work together to enable a device such as a mobile phone or server to perform various functions), and (c) Multiple hardware circuits and / or multiple processors that require software (e.g., firmware) to operate, such as being multiple microprocessors or part of multiple microprocessors, but the software may be absent when it is not required to operate.

[0024] This definition of "circuit" applies to all uses of the term in this application (including any claim). As another example, as used in this application, the term "circuit" also covers only hardware circuitry, or a processor (or multiple processors), or a portion of hardware circuitry or a processor and its accompanying software and / or firmware. The term "circuit" also covers (e.g., if applicable to a particular claim element) baseband integrated circuits or processor integrated circuits for mobile devices or similar integrated circuits in servers, cellular network devices, or other computing or networking devices.

[0025] As used herein, the term "communication network" refers to a network that conforms to any suitable communication standard, such as Long Term Evolution (LTE), LTE-Advanced (LTE-A), Wideband Code Division Multiple Access (WCDMA), High-Speed ​​Packet Access (HSPA), Narrowband Internet of Things (NB-IoT), etc. Furthermore, communication between terminal devices and network devices in a communication network can be performed according to any suitable generation of communication protocol, including but not limited to third-generation (3G), fourth-generation (4G), 4.5G, fifth-generation (5G) communication protocols and / or more. Embodiments of this disclosure can be applied to various communication systems. Given the rapid development of communications, there will certainly be future types of communication technologies and systems that embody the future types of this disclosure. This disclosure should not be construed as limiting its scope to the aforementioned systems.

[0026] As used herein, the term "network device" refers to a node in a communication network through which terminal devices access the network and receive services. Depending on the terminology and technology applied, network devices can refer to base stations (BS) or access points (APs), such as Node B (NodeB or NB), evolved Node B (eNodeB or eNB), NR NB (also known as gNB), Remote Radio Unit (RRU), Radio Head (RH), Remote Radio Head (RRH), relays, and low-power nodes (such as femtoseconds, picoseconds, etc.).

[0027] The term "terminal device" refers to any terminal device capable of wireless communication. By way of example and not limitation, a terminal device may also be referred to as a communication device, user equipment (UE), subscriber station (SS), portable subscriber station, mobile station (MS), or access terminal (AT). Terminal devices may include, but are not limited to, mobile phones, cellular phones, smartphones, Voice over IP (VoIP) phones, wireless local loop phones, tablet computers, wearable terminal devices, personal digital assistants (PDAs), portable computers, desktop computers, image capture terminal devices such as digital cameras, gaming terminal devices, music storage and return devices, in-vehicle wireless terminal devices, wireless endpoints, mobile stations, laptop embedded devices (LEEs), laptop-mounted devices (LMEs), USB dongles, smart devices, wireless customer premises equipment (CPEs), Internet of Things (IoT) devices, watches or other wearable devices, head-mounted displays (HMDs), vehicles, drones, medical devices and applications (e.g., remote surgery), industrial devices and applications (e.g., robots and / or other wireless devices operating in industrial and / or automated processing chain environments), consumer electronic devices, devices operating in commercial advertising, relay nodes, integrated access and backhaul (IAB) nodes and / or industrial wireless networks, etc. In the following description, the terms "terminal device," "communication device," "terminal," "user equipment," and "UE" are used interchangeably.

[0028] As used herein, the terms “resource,” “transmission resource,” “resource block,” “physical resource block” (PRB), “uplink (UL) resource,” or “downlink (DL) resource” can refer to any resource used to perform communication (e.g., communication between a terminal device and a network device), such as resources in the time domain, resources in the frequency domain, resources in the spatial domain, resources in the code domain, resources in a combination of more than one domain, or any other resource supporting the communication. In the following description, some exemplary embodiments of this disclosure will be illustrated using time-domain resources (e.g., subframes) as examples of transmission resources. Note that the exemplary embodiments of this disclosure are equally applicable to other resources in other domains.

[0029] The term "artificial intelligence and / or machine learning (AI / ML)" refers to a software-implemented method based on a mathematical algorithm or model that provides an inference function. Such a model is typically a mathematical algorithm that is trained using information and replicates the decisions an expert would make when given the same information. According to some embodiments, the AI / ML function can also provide data analysis. For example, an AI / ML training function associated with the model acquires data, runs the data through the AI / ML model and derives an associated loss, and adjusts the parameterization of the AI / ML model based on the calculated loss. Training methods can include supervised learning, unsupervised learning, and reinforcement learning, and training can be performed offline or continuously. The inference function can be one of several known categories, such as regression-based, clustering-based, association-based, or reward-based behavior, where appropriate training methods are applied.

[0030] Example applications of AI and / or ML include, but are not limited to: speech recognition; image processing / computer vision; natural language processing; information retrieval; personalization and recommendation; robotics; data analytics including predictive and prescriptive analysis; and use cases for the design and / or planning and / or optimization and / or configuration and / or control and / or management of communication systems and / or networks.

[0031] Example use cases can be, but are not limited to: - Use cases related to the physical layer of communication networks, such as modulation, coding, decoding, signal detection, channel estimation, prediction, compression, and interference mitigation; - Use cases related to the media access control layer of communication networks, such as multiple access and resource allocation (e.g., power control, scheduling, spectrum management). - Channel modeling; - Network optimization; - Cell capacity estimation in cellular networks; - Routing; - Resource management; - Data business management; - Security and anomaly detection; - Root cause analysis; - Transmission protocol design and optimization; - User / network / application behavior analysis / prediction; - Transport layer congestion control; - User experience modeling and optimization; - User mobility and location management; - Network slicing, network virtualization, and software-defined networking; - Compensation for nonlinear impairments in optical networks (e.g., visible light communication, fiber optic communication, and fiber-wireless convergence networks), and - Transmission quality estimation and optical performance monitoring in optical networks.

[0032] The term "AI / ML entity" specifies any network entity that contains one or more AI and / or ML capabilities. Example network entities include, but are not limited to: - Radio access network entities, such as base stations (e.g., cellular base stations, such as eNodeB in LTE and Advanced LTE networks and gNodeB used in 5G networks, as well as femtocells used in homes or business centers). - Relay station; - Control station (e.g., radio network controller, base station controller, network switching subsystem); - Access points in a local area network or self-organizing network; - Gateway and radio access network entities; - Network management entities (e.g., Operations, Administration and Management (OAM) entities); - Network automation system; - Distributed analytical entities, such as autonomous systems (D-SON); - Network functions (e.g., Network Data Analysis Function (NWDAF) as defined in the current 3GPP standard); - User Equipment (UE).

[0033] As mentioned above, Rel-18 3GPP initiated a study on AI / ML for the NR air interface, and the objectives were described in RP-213599. The goal of this study project is to explore the benefits of leveraging features to enhance the air interface, enabling improved support for AI / ML-based algorithms to improve performance and / or reduce complexity / overhead, considering several use cases to identify common AI / ML frameworks, including the functional requirements of the AI / ML architecture. The study should also identify areas where AI / ML models can improve the performance of air interface functionality. The specification impact will be evaluated to improve the overall understanding of what is needed to support AI / ML technologies for the air interface.

[0034] To better understand the terminology related to AI / ML technologies, Table 1 shows the RAN1 protocol for a list of terms used in AI / ML.

[0035] Regarding AI / ML models and model functionality, RAN1#111 makes the following working assumptions about the model types under consideration (as shown in Table 2), treating "proprietary models" and "open format models" as two separate model format categories for discussion in RAN1.

[0036] As can be seen from the perspective discussed in RAN1, RAN1 can assume that proprietary format models cannot be mutually identifiable across vendors, thus hiding model design information from other vendors when shared. RAN1 can also assume that open format models are mutually identifiable between vendors, and that they do not hide model design information from other vendors when shared.

[0037] Enabling open format models requires specification work to make them interoperable across devices from different vendors (e.g., via UE and network). AI / ML models may not be separate from the rest of the functionality that applies the AI / ML model to a specific decision-making (inference) process. These can include, for example, runtime instructions, input data preprocessing, and output data postprocessing algorithms. Open format models can support cross-vendor parameter updates and over-the-air training. An example of an open format for ML models is ONNX. If 3GPP specifies a new format for ML models, it is also considered an open format.

[0038] In the RAN1 discussion, both "proprietary format model" and "open format model" are considered physical models or "models" that are usually referred to as physical models. Physical models can be defined as compiled models for specific hardware, complete models for specific hardware, complete models with floating-point parameters, or functional and complete model structures.

[0039] As discussed in the background section, model identifiers and function identifiers are introduced to distinguish AI / ML models from the functions supported by AI / ML models. Tables 3 and 4 show descriptions of these two terms, respectively.

[0040] The network (NW) can activate, deactivate, or switch between different functions based on its function ID (each function uses a proprietary model at the UE). Alternatively, when available, the network can activate, deactivate, or switch between different functions based on a model ID combined with associated metadata. In both alternatives, at least the function ID (which can be a label identifying a given function or use case) needs to be specified in 3GPP to ensure UE-NW interoperability without requiring a bilateral protocol.

[0041] In some variations, a function can also be referred to as all or part of the logical model, where the logical model is merely an extension of the model or physical model and is primarily described by explicit datasets, nominal inputs, nominal ideal outputs, and other parameters. Additionally, the logical model can be described by conditions that the model must satisfy; these conditions can also be called applicability conditions (scenario, site, model usage conditions, etc.). Typically, the physical model can be separated from the logical model for different processing purposes, such as model transfer. In the following discussion, model or model ID can primarily refer to the physical model. However, as long as the logical model is not fully identified by a function or function ID, the model or model ID can also be referred to as the logical model.

[0042] In this way, model ID-based lifecycle management (LCM) (such as model activation, model deactivation, handover, and monitoring) can be handled by the UE implementation and proprietary mechanisms specific to the UE vendor, while function-based LCM (such as function activation, function deactivation, handover, and monitoring) is handled by NW / NG-RAN.

[0043] Function ID-based LCM should use any available model ID indicated by the UE during monitoring and in potential indications to the UE regarding the performance of detected functions. This will also enable separate LCM procedures for functions and models.

[0044] In practice, model identification may not always be supported (e.g., the network may not be able to interpret model metadata). Instead, UE vendors may prefer to treat model ID-based LCM as a UE-implementation specific matter (i.e., the UE can support more than one AI / ML model ID for a given function ID and decide to switch across these AI / ML models without affecting functionality). In this case, the network may have to rely on function-based LCM, where function IDs can be considered to perform function selection, switching, deactivation, and other relevant LCM aspects.

[0045] In scenarios where UE autonomously activates, selects, and switches AI / ML models for a given function enabled by the network, from the UE's perspective, since AI / ML models are implementation-specific, the UE may prefer to have degrees of freedom in selecting, activating, deactivating, and switching AI / ML models while still meeting the performance level for the enabled function (e.g., as defined in RAN4).

[0046] However, from the network's perspective, performance monitoring can be performed at the function ID level, and any changes caused by changes in the background ML model at the UE may not be visible at the network or may not be controllable by the network. If the overall performance of a function degrades or changes significantly over time (i.e., unreliable ML model inference), the network may initiate deactivation of the entire function (e.g., deactivation of channel state information (CSI) predictions) due to autonomous model selection, activation, switching, and updates at the UE side.

[0047] For example, UEs moving towards or within cities may experience different radio channels (such as rural and urban radio channels) that require specific adaptations for CSI prediction processes. This lack of adaptation will lead to packet loss. This situation should be minimized while still providing the UE with a good level of freedom to control its own AI / ML model.

[0048] Therefore, when model ID-based LCM is processed by the UE (i.e., the UE-side AI / ML model and related LCM steps are not visible to the network), the network needs to still be able to maintain reliable AI / ML operation for a given use case, sub-use case, ML feature or function.

[0049] This disclosure provides an example embodiment of a solution for monitoring AI / ML model functionality. According to an embodiment of this disclosure, a terminal device receives first information from a network device for configuring at least one time window for monitoring the performance of an AI / ML model function. The terminal device receives second information indicating either a first time window or a second time window, during which the performance of the AI / ML model function is above a performance level, and during the second time window, the performance of the AI / ML model function is below a performance level. Based on the second information, the terminal device performs management operations associated with multiple AI / ML models configured to implement AI / ML model functionality. It should be understood that the above process steps may work together, partially work together, or work independently of each other in the operational flow described in the next section.

[0050] The example embodiments for AI / ML model functionality monitoring provided in this disclosure can allow network devices to maintain reliable AI / ML operations without knowing exactly which AI / ML model an end device is using. The principles of this disclosure and some example embodiments are described in detail below with reference to the accompanying drawings.

[0051] For illustrative purposes, the following will refer to Figures 1A-7This disclosure describes the principles and exemplary embodiments of the present disclosure for monitoring the functionality of AI / ML models. However, it should be noted that these embodiments are given to enable those skilled in the art to understand the inventive concept of this disclosure and implement the solutions presented herein, and are not intended to limit the scope of this application in any way.

[0052] refer to Figure 1A The illustration shows an example network environment 100A that can implement example embodiments of the present disclosure. Network environment 100A may be part of a communication network, which includes terminal device 102 and network device 104.

[0053] like Figure 1A As shown, terminal device 102 can also be referred to as user equipment 102 or UE 102. Network device 104 can also be referred to as gNB 104. Terminal device 102 and network device 104 can communicate with each other (106). Terminal device 102 can support one or more AI / ML model IDs for a given function. For example, one or more AI / ML models can be used for CSI prediction. The network can know the overall performance degradation of CSI and determine the deactivation of CSI prediction. This is because background AI / ML model changes on the UE side may not be visible at the network or cannot be controlled by the network. For clarity, reference will be made to... Figure 1B Discuss the process.

[0054] refer to Figure 1B It illustrates example illustrations of AI / ML model function monitoring in relation to some embodiments of this disclosure. Figure 1B An example of UE autonomous model selection, activation, and switching for a given function is shown, which may lead to changes in inference performance over a period of time. Figure 1B As shown, UE 108 can correspond to terminal device 102, which can communicate with network devices. gNB 110 can correspond to network device 104, which can communicate with terminal devices.

[0055] gNB 110 can send (114) AI / ML function query (116) to UE 108. UE 108 can receive (112) AI / ML function query (116) from gNB 110. This signaling (112, 114, 116) can be used as a UE capability query. UE 108 can send (118) AI / ML function report (122) to gNB 110. gNB 110 can receive (120) AI / ML function report (122) from UE 108. This signaling (118, 120, 122) can be used as a UE capability report.

[0056] gNB 110 can select or determine (124) an AI / ML function (assuming function ID X is selected in this document). gNB 110 can send (128) the configuration (130) of the selected function ID X to UE 108. UE 108 can receive (126) the configuration (130) from gNB 110. UE 108 can determine (132) any one of the AI / ML models (such as AI / ML models N1, N2, ..., Nx). UE 108 can autonomously activate or deactivate (134) the AI / ML model.

[0057] Dashed box 136 illustrates the detailed process of functional performance monitoring. For example, for functional ID X, AI / ML model N1 (corresponding to functional ID X) can be used for inference (138). gNB 110 can monitor (140) functional performance. UE 108 can monitor (142) the performance of AI / ML models (such as AI / ML model N1). UE 108 can autonomously switch (144) AI / ML models. For example, switching AI / ML model N1 to AI / ML model N3. AI / ML model N3 can be used for inference (146).

[0058] gNB 110 can monitor (148) functional performance. gNB 110 can determine (150) poor average performance. gNB 110 can decide to deactivate function ID X. gNB 110 can send (154) deactivation (156) of function ID X to UE 108. UE 108 can receive (152) deactivation (156) of function ID X from gNB 110.

[0059] Therefore, it would be better if the network could still control the AI / ML model even when performance does not meet standards, rather than disabling the entire function. When LCM based on model ID is processed by the UE (i.e., the UE-side AI / ML model and related LCM steps are not visible to the network), it is necessary to allow the network to appropriately configure the UE for a given function ID X, which will... Figure 2 The discussion is ongoing.

[0060] refer to Figure 2 The diagram illustrates an example signaling process 200 for monitoring the performance of an AI / ML model according to some embodiments of the present disclosure. As shown, network device 104 sends (204) first information (206) to terminal device 102. This first information is used to configure at least one time window for monitoring the performance of the AI / ML model functionality. Terminal device 102 receives (202) first information (206) from network device 104.

[0061] Network device 104 determines (208) a first time window within at least one time window during which the performance of the AI / ML model function is above the performance level, or network device 104 determines (208) a second time window within at least one time window during which the performance of the AI / ML model function is below the performance level.

[0062] Network device 104 sends (212) second information (214) to terminal device 102. The second information (214) indicates a first time window or a second time window. Terminal device 102 receives (210) the second information (214) from network device 104. Terminal device 102 performs (216) management operations associated with multiple AI / ML models configured to implement AI / ML model functions.

[0063] In some example embodiments, management operations may include activating an AI / ML model among multiple AI / ML models. Management operations may also include deactivating an AI / ML model among multiple AI / ML models. Management operations may also include adjusting an AI / ML model among multiple AI / ML models. Management operations may also include switching between multiple AI / ML models and / or selecting among multiple AI / ML models.

[0064] By implementing Figure 2 This allows network devices to maintain reliable AI / ML operations without needing to know exactly which AI / ML model the end device is using. In other words, the network doesn't need to know precisely which AI / ML model the UE is using, but it can still control it if performance is insufficient.

[0065] refer to Figure 3 This illustrates another example signaling procedure 300 for AI / ML model performance monitoring according to some embodiments of this disclosure. It should be understood that... Figure 3 Example signaling procedure 300 in the example can be considered as Figure 2 An example of signaling procedure 200 in the example. Therefore, Figure 3 UE 302 in the text is Figure 2 An example of terminal device 102 in the example is that it can communicate with network devices. Figure 3 gNB 304 in the text is Figure 2 The example of network device 104 is shown, which can communicate with terminal devices. Additionally, core network device 306 and vendor database 308 can participate in some steps and signaling.

[0066] UE 302 can be configured (310) to use AI / ML assistance for use case function ID X. gNB 304 may want to (312) determine whether UE 302 is switching different AI / ML models for a given function ID X. At this point, gNB 302 may not be sure (or unaware) whether UE 302 will switch internally across multiple AI / ML model implementations (e.g., CNN and RNN, localized (cell-specific) or generic ML models (applicable to multiple cells), more accurate versus less accurate for better power savings).

[0067] In some example embodiments, gNB 304 may optionally use any tracking window data that may have been previously stored in core network 306 for a given UE and a given function ID. gNB 304 may send (314) a tracking window data request (318) for function ID X to core network device 306. Core network device 306 may receive (316) the tracking window data request (318) from gNB 304. Core network device 306 may send (322) a tracking window data response (324) for function ID X to gNB 304. gNB 302 may receive (320) the tracking window data response (324) for function ID X from core network device 306.

[0068] In some example embodiments, for a given model function (identified by an ID), the gNB 304 can use a time window called the “model-LCM-tracking window” (T), which enables tracking of UE-side model LCM-related changes. UE-side model LCM-related changes may include UE-autonomous model activation, model selection, model deactivation, model switching, model updates / fine-tuning, or other aspects.

[0069] In some example embodiments, the same aspects can be used when replacing the network. A test equipment (TE) can be used, and UE 302 can be regarded as a device under test (DUT). In this case, even if UE 302 may not move, changes in the propagation environment can be provided through channel emulation.

[0070] In some example implementations, one or more model-LCM-tracking windows may be considered during inference operations for a given function ID. The durations (T1, T2, ..., Tm) of the model-LCM-tracking window can be different from each other.

[0071] gNB 304 can generate (326) a configuration for the AI / ML model LCM by defining a tracking window configuration for the UE-side AI / ML model. gNB 304 can send (330) a configuration request (332) to UE 302. UE 302 can receive (328) the configuration request (332). The tracking window configuration may include aspects (1) and (2).

[0072] In some example embodiments, aspect (1) may include one or more AI / ML model LCM-tracking windows of duration T. There may be M such durations distinct over the entire time period S, such that T1 + T2 + … TM. From the perspective of UE behavior, UE 302 should track each period T... M These are considered as operations applicable to the ML model implementation under a given function ID X, and are independent of each other.

[0073] This allows the network to ensure that the data in each tracking window is different from the data in another tracking window, without the UE having to explicitly reveal which implementation it is using. For example, if UE 302 will perform a handover (HO) between cell 1 and cell 2 and uses an AI / ML model to predict the Reference Received Power (RSRP) in cell 1 and cell 2, it can use the following AI / ML model implementations: a cell 1-specific AI / ML model at T1, a generic AI / ML model for both cell 1 and cell 2 at T2, and a cell 2-specific AI / ML model at T3.

[0074] In some example embodiments, aspect (2) may include one or more events that cause the UE to perform an AI / ML model switch. For example, in mobility as an AI / ML use case, the UE 302 may decide to switch the AI / ML model when the execution conditions for a CHO (Conditional Handover) are met. Another example is AI / ML assistance for carrier aggregation features when the UE 302 switches from a general model for FR1 and FR2 frequencies to a specific ML model to perform FR2 measurements, for example, on the corresponding frequency band.

[0075] UE 302 may send a configuration response (334) (338) to gNB 304. gNB 304 may receive a configuration response (336) (338) from UE 302. Dashed box 340 illustrates the detailed process of functional performance monitoring (such as for function ID X). UE 302 may determine or detect (342) that the execution conditions for tracking window ID Tn are met. UE 302 may send (344) an indication (348) during time window Tn indicating the initialization of tracking for the AI / ML model. UE 302 may receive an indication (346) (348) from UE 302.

[0076] In some example implementations, the tracking window ID Tn can be configured with a defined duration (in milliseconds or seconds), and the UE will ensure that it uses a given AI / ML model function and tracking window ID with a set of enforcement criteria (e.g., for CHO events to allow the UE to switch between different AI / ML model functions in the source and target cells for a given function ID).

[0077] In this scenario, UE 302 can follow the tracking window based on this set of execution criteria. For example, if UE 302 will perform a HO (Hosting Operation) between cell 1 and cell 2 and use an AI / ML model to predict the RSRP (Resistance to Resistant Parameters) in both cells, it can use the following AI / ML model implementation: a cell 1-specific AI / ML model at T1, a generic AI / ML model for both cells 1 and 2 at T2, and a cell 2-specific AI / ML model at T3. Therefore, UE 302 effectively counts three tracking windows, but based on the execution conditions in the CHO configuration. UE 302 can label these as sub-tracking window IDs (e.g., in this case, window X has sub-tracking windows X.1, X.2, and X.3).

[0078] gNB 304 can track the performance of function ID X (350) for a defined duration. In this case, UE 302 can follow the tracking window based on the duration guided by the network (gNB 304). UE 302 can notify gNB 304 when the tracking window ID starts and ends by indicating to allow gNB 304 to synchronize its side.

[0079] In some example embodiments, signaling (344, 346, 348) can be omitted if the tracing window can be fully guided by the duration provided by gNB 304 (and is not under execution conditions).

[0080] In some example embodiments, the gNB 304 can track model performance by considering one or more Key Performance Indicators (KPIs). The gNB 304 can consider one or more KPIs to determine the performance of a given function. One or more KPIs may include accuracy, data throughput, block error rate (BLER), number of retransmissions, number of Hybrid Automatic Repeat Requests (HARQ), number of beam faults (BF), or number of radio link faults (RLF). One or more KPIs can be used for duration. Each of the model-LCM-tracking windows during (or T1+T2+…Tm) period.

[0081] In some example embodiments, when all tracking windows are exhausted (352), gNB 304 can summarize the history of the identified KPIs and arrange or aggregate them in different ways. For example, gNB 304 can arrange or aggregate historical KPIs based on the function ID and / or any other information available to UE 302 or from function meta-information (e.g., vendor ID, UE model ID, UE capabilities, etc.). As another example, gNB 304 can arrange or aggregate historical KPIs within an LCM-tracking window, a minimum-length LCM-tracking window, or other fixed time intervals. In another example, gNB 304 can use additional information (e.g., network load, time of day, UE 302 location, etc.) to tag or enrich historical KPIs. In yet another example, gNB 304 can collect and update statistical measures, means, deviations, distributions, etc., of KPIs. In yet another example, UE-vendor pairs in the AI / ML model information or UE vendor-specific information in the AI / ML model information can be updated for later retrieval.

[0082] In some example embodiments, if gNB 304 wishes to perform additional measurements on UE 302, it can provide UE 302 with a subsequent request for a different configuration with a tracking window ID duration / condition. For example, gNB 304 may be interested in an earlier specific tracking duration and may optionally emphasize UE 302's use of its connected AI / ML model. This is reflected in dashed box 356, which illustrates the initialization of another tracking sequence. gNB 304 may send a configuration request (360) (362) to UE 302. UE 302 may receive (358) the configuration request (362). UE 302 may send a configuration response (364) (368) to gNB 304. gNB 304 may receive (366) the configuration response (368) from UE 302.

[0083] In some example embodiments, if the summary of determined KPIs by gNB 304 is proven, and gNB 304 believes that UE 302 may not perform AI / ML model switching within a given tracking window, it can utilize a preferred window configuration request to set up a configuration containing the tracking window ID approved and confirmed by gNB 304. When performing the configuration, gNB 304 can indicate the tracking window ID, the preferred tracking window duration, and the sub-tracking window IDs.

[0084] In some example implementations, the gNB 304 can compare determined KPIs within the model-LCM-tracking window. As an example, this comparison can be based on best, worst, or average considerations or KPI distribution methods to observe outliers in KPI changes over time. As another example, the comparison can be based on other metrics (such as changes in difference over time) that can be derived from the determined KPIs.

[0085] In some example embodiments, when the KPIs determined across multiple model-LCM-tracking windows are within a specific level of performance variation (e.g., if accuracy is used as the KPI and each window varies within X% (where X is a value, such as X = 5, 10, etc.)), the KPIs do not change significantly from one another. Therefore, gNB 304 may not need to initiate any additional steps and can continue with the steps described above for future use of the functionality.

[0086] In some example implementations, when the KPIs determined across multiple model-LCM-tracking windows are not within a specific level of performance variation (e.g., if accuracy is used as the KPI and some windows are not within X% variation), the KPIs vary significantly from one another. gNB 304 can additionally derive best or worst model-LCM-tracking windows based on the determined KPIs.

[0087] In some example implementations, performance levels can be absolute or relative values. An absolute value compares a KPI value to an absolute threshold. A relative value compares a KPI change (relative to another KPI value, such as a previously measured KPI value for an AI / ML function) to a relative threshold (meaning performance improved or deteriorated by X%). Furthermore, KPI changes can exceed the threshold, depending on which KPI is measured, which can indicate performance improvement or degradation. For example, a 10% increase in data throughput indicates improved performance, while a 10% increase in BER indicates degraded performance.

[0088] This is reflected in dashed box 370, which illustrates the detailed process of configuring the preferred tracking window for the UE. gNB 304 may send (374) a configuration request (376) to UE 302 to configure the preferred tracking window. UE 302 may receive (372) the configuration request (376) from gNB 304. UE 302 may send (378) a configuration response (382) to gNB 304. gNB 304 may receive (380) the configuration response (382) from UE 302.

[0089] In some example embodiments, UE 302 may be allowed to perform only one operation (only one model switch) associated with the model-ID-based LCM within the model-LCM-tracking duration (T or T1 / T2… / Tm). In some example embodiments, when a performance change is identified for one or more model-LCM-tracking windows, gNB 304 may indicate a preferred (or non-preferred) model-LCM-tracking window to continue the associated functionality.

[0090] In some example embodiments, since UE 302 can know the exact LCM change during the indicated model-LCM-tracking window, UE 302 should either correct the LCM steps performed in the model-LCM-tracking window (if performance degradation is observed and the network is indicated as a non-preferred model-LCM-tracking window) or maintain the LCM steps performed in the model-LCM-tracking window (if performance gain is observed and the network is indicated as a preferred model-LCM-tracking window).

[0091] In some example implementations, if the AI / ML model is no longer used (due to a change in use case, or it is not in use, invalid, etc.), UE 302 will trigger an inactivity signaling instruction to gNB 304. This will instruct gNB 304 to stop the model-LCM-tracing window for that AI / ML model. In some example implementations, in a test setup, TE collects violations of KPIs across multiple model-LCM-tracing windows (KPIs not within a specific level of performance change), and if the number of intervals of violations exceeds a threshold (such as xx%, where xx is in the range of 0 to 100), the test can be considered a failure.

[0092] In some example embodiments, UE 302 can pause any management operations associated with multiple AI / ML models during at least one time window. As an example, UE 302 can pause activation, deactivation, and adjustment of AI / ML models.

[0093] In some example embodiments, gNB 304 may send (382) an update tracking window data request (386) to core network device 306 for a given function ID (such as function ID X). Core network device 306 may receive (384) the update tracking window data request (386) from gNB 304. Core network device 306 may send (390) an update tracking window data response (392) to gNB 304. gNB 304 may receive (388) the update tracking window data response (392) from core network device 306.

[0094] In some example embodiments, UE 302 may send (391) an update tracking window data request (393) to vendor database 308 for a given function ID and AI / ML model (such as function ID X and AI / ML model ID X). Vendor database 308 may receive the update tracking window data request (393) from UE 302. Vendor database 308 may send (395) an update tracking window data response (396) to UE 302 for a given function ID and AI / ML model. UE 302 may receive (394) the update tracking window data response (396).

[0095] In function-based LCM, AI / ML models do not need to be identified at the network level, and the UE can perform model-level LCM. Therefore, by implementing Figure 3 This allows the network to have awareness and / or interaction with model-level LCM, and therefore the network does not need to know exactly which AI / ML model the UE is using, but still has control if performance fails to meet standards. It also allows the network and UE to store summaries of KPIs that are executed during function-based LCM handover to test the AI / ML model, and thus allows the network to appropriately configure the UE for a given function ID.

[0096] It should be understood that although the above description involves network-side operations, it should also be understood that these can be performed by the UE. In this case, some additional steps may mean that the UE signals changes or requests to the network within the Model-LCM-Tracking Window. There is also the possibility that the network and UE operate in a digital twin manner, where both entities (UE and network) perform similar steps within the Model-LCM-Tracking Window operation.

[0097] refer to Figure 4 This illustrates an example flowchart 400 of a method implemented at a terminal device according to some example embodiments of the present disclosure. (The last part, "will be combined with...", appears to be a fragment and doesn't translate directly.) Figure 1A For reference.

[0098] At 402, terminal device 102 receives from network device 104 first information for configuring at least one time window for monitoring the performance of AI / ML model functions. In some example embodiments, the first information may include an identifier for the AI / ML model function. The first information may also include an identifier for a time window within at least one time window. The first information may also include the duration of the time window. The first information may also include criteria associated with the time window for enabling the terminal device to perform model switching between multiple AI / ML models.

[0099] At 404, terminal device 102 receives second information from network device 104. In some example embodiments, the second information may indicate a first time window within at least one time window during which the performance of the AI / ML model function is above a performance level. Alternatively, the second information may indicate a second time window within at least one time window during which the performance of the AI / ML model function is below a performance level.

[0100] At 406, based on the second information, the terminal device 102 performs management operations associated with multiple AI / ML models configured to implement AI / ML model functionality. In some example embodiments, the management operations may include activating one of the multiple AI / ML models. The management operations may also include deactivating one of the multiple AI / ML models. The management operations may also include adjusting one of the multiple AI / ML models. The management operations may also include switching between and / or selecting among the multiple AI / ML models.

[0101] In some example embodiments, based on the satisfaction of a determination criterion, terminal device 102 can determine at least one of the start point and end point or duration of a time window. Terminal device 102 can send third information to network device 104. The third information indicates the start point of the time window and indicates at least one of the end point and duration of the time window.

[0102] In some example embodiments, if the second information indicates a first time window, the terminal device 102 may perform a management operation by identifying a first AI / ML model used during the first time window among a plurality of AI / ML models as a suitable AI / ML model for implementing the functions of the AI / ML model.

[0103] In some example embodiments, if the second information indicates a second time window, the terminal device 102 may perform a management operation by identifying a second AI / ML model used during the second time window as not a suitable AI / ML model for implementing the functionality of the AI / ML model.

[0104] In some example embodiments, terminal device 102 may suspend any management operations associated with multiple AI / ML models during at least one time window. In some exemplary embodiments, if terminal device 102 determines that more than one AI / ML model among multiple AI / ML models is used in a time window within the at least one time window, terminal device 102 may determine more than one sub-time window corresponding to each of the more than one AI / ML models used in that time window. Terminal device 102 may send fourth information to network device 104. The fourth information indicates more than one sub-time window.

[0105] In some example implementations, at least one time window includes multiple time windows, and multiple AI / ML models are used in multiple time windows respectively.

[0106] In some example embodiments, if terminal device 102 determines that one of the multiple AI / ML models will no longer be used, terminal device 102 may send a fifth message to network device 104. This fifth message indicates that the AI / ML model is no longer in use.

[0107] In some example embodiments, at least one time window may include multiple time windows, and the multiple time windows have the same duration, or at least two of the multiple time windows have different durations.

[0108] refer to Figure 5 This illustrates an example flowchart 500 of a method implemented at a network device according to some example embodiments of the present disclosure. (The last part, "will be combined with...", appears to be a fragment and doesn't translate directly.) Figure 1A For reference.

[0109] At point 502, network device 104 sends first information to terminal device 102 for configuring at least one time window for monitoring the performance of AI / ML model functionality. In some example embodiments, the first information may include an identifier for the AI / ML model functionality. The first information may also include an identifier for a time window within at least one time window. The first information may also include the duration of the time window. The first information may also include criteria associated with the time window for enabling the terminal device to switch between multiple AI / ML models.

[0110] At point 504, network device 104 determines a first time window or a second time window. The first time window is a time window in which the performance of the AI / ML model function is above a performance level within at least one time window. The second time window is a time window in which the performance of the AI / ML model function is below a performance level within at least one time window.

[0111] At point 506, network device 104 sends second information to terminal device 102 indicating a first time window or a second time window. In some example embodiments, the second information may indicate a first time window within at least one time window during which the performance of the AI / ML model function is above a performance level. Otherwise, the second information may indicate a second time window within at least one time window during which the performance of the AI / ML model function is below a performance level.

[0112] In some example embodiments, terminal device 102 may use the second information to perform management operations associated with multiple AI / ML models configured to implement AI / ML model functionality. In some example embodiments, the management operations may include activating one of the multiple AI / ML models. The management operations may also include deactivating one of the multiple AI / ML models. The management operations may further include adjusting one of the multiple AI / ML models. The management operations may also include switching between and / or selecting among the multiple AI / ML models.

[0113] In some example embodiments, where the first information includes a standard, network device 104 may receive a third indication from terminal device 102. The third information may indicate at least one of the start point of the time window and the end point and duration of the time window.

[0114] In some example embodiments, network device 104 may receive fourth information from terminal device 102. The fourth information may indicate more than one sub-time window within a time window, wherein the more than one sub-time window corresponds to more than one AI / ML model used by the terminal device in the time window.

[0115] In some example embodiments, at least one time window may include multiple time windows. Network device 104 may determine at least one of a first time window or a second time window by: determining multiple KPI values ​​for KPIs associated with AI / ML model functionality for multiple time windows; determining that performance in the first time window is above a performance level based on a comparison of the multiple KPI values; or determining that performance in the second time window is below a performance level based on a comparison of the multiple KPI values.

[0116] In some example implementations, KPIs may include accuracy, data throughput, block error rate (BLER), number of retransmissions, number of hybrid automatic repeat requests (HARQ), number of beam faults (BF), or number of radio link faults (RLF) associated with AI / ML model functionality.

[0117] In some example embodiments, network device 104 may receive fifth information from terminal device 102. The fifth information indicates that an AI / ML model among multiple AI / ML models is no longer being used. In some example embodiments, at least one time window may include multiple time windows, and the multiple time windows may have the same duration, or at least two of the multiple time windows may have different durations.

[0118] In some example embodiments, network device 104 can determine that the performance of the AI / ML model function is above a performance level by determining that the performance change associated with a first time window is below a predetermined change level. In some example embodiments, network device 104 can determine that the performance of the AI / ML model function is below a performance level by determining that the performance change associated with a second time window is above a predetermined change level.

[0119] In some example implementations, at least one time window includes multiple time windows, and multiple AI / ML models are used in multiple time windows respectively.

[0120] By implementing methods 400 and 500, example embodiments for AI / ML model function monitoring can allow the network to have awareness and / or interaction with model-level LCM, and therefore the network does not need to know exactly which AI / ML model the UE is using, but still has control when performance does not meet standards. It can also allow the network and UE to store a summary of KPIs that are executed during function-based LCM handover to test the AI / ML model, and thus allow the network to appropriately configure the UE for a given function ID.

[0121] In some example embodiments, the apparatus capable of performing method 400 may include components for performing the corresponding steps of method 400. These components may be implemented in any suitable form. For example, the components may be implemented in a circuit or software module.

[0122] In some example embodiments, the apparatus may include: components for receiving first information from a network device for configuring at least one time window for monitoring the performance of an AI / ML model function; components for receiving second information from the network device, the second information indicating that during the first time window or the second time window of the at least one time window, the performance of the AI / ML model function is above a performance level, and during the second time window, the performance of the AI / ML model function is below a performance level; and components for performing management operations associated with a plurality of AI / ML models configured to implement the AI / ML model function based on the second information.

[0123] In some example embodiments, the management operation may include at least one of the following: activating an AI / ML model among multiple AI / ML models; deactivating an AI / ML model among multiple AI / ML models; or adjusting an AI / ML model among multiple AI / ML models.

[0124] In some example embodiments, the first information may include at least one of the following: an identifier of the AI / ML model function; an identifier of a time window within at least one time window; the duration of the time window; or a standard associated with the time window for enabling the terminal device to perform model switching between multiple AI / ML models.

[0125] In some example embodiments, the apparatus may further include: components for determining at least one of a start point and an end point of a time window or a duration of a time window based on determining that the criterion is met; and components for sending third information to a network device, the third information indicating at least one of a start point and an end point of a time window or a duration of a time window.

[0126] In some example embodiments, the second information may include at least one of the following: an identifier of a time window within at least one time window; an indication of whether the time window is characterized as a first time window or a second time window; or a performance metric of the AI / ML model's functionality during the time window.

[0127] In some example embodiments, the component for performing management operations may further include a component for identifying a first AI / ML model used during the first time window among a plurality of AI / ML models as a suitable AI / ML model for implementing the functions of the AI / ML model, based on receiving second information indicating a first time window.

[0128] In some example embodiments, the component for performing management operations may further include: a component for identifying a second AI / ML model used during the second time window among a plurality of AI / ML models as not a suitable AI / ML model for implementing the functionality of the AI / ML model, based on receiving second information indicating a second time window.

[0129] In some example embodiments, the apparatus may further include a component for suspending any management operations associated with multiple AI / ML models during at least one time window.

[0130] In some example embodiments, the apparatus may further include: a component for determining more than one sub-time window corresponding to each of the more than one AI / ML models used in the time window, based on determining that more than one AI / ML model is used in a time window within at least one time window; and a component for sending fourth information indicating the more than one sub-time window to a network device.

[0131] In some example embodiments, at least one time window may include multiple time windows, and multiple AI / ML models are used in multiple time windows respectively.

[0132] In some embodiments, the apparatus may further include components for performing other steps in some embodiments of method 400. In some embodiments, the components include at least one processor and at least one memory including computer program code, the at least one memory and the computer program code being configured to cause execution of the apparatus together with the at least one processor.

[0133] In some example embodiments, the apparatus capable of performing method 500 may include components for performing the corresponding steps of method 500. These components may be implemented in any suitable form. For example, the components may be implemented in a circuit or software module.

[0134] In some example embodiments, the apparatus may include: components for sending first information to a terminal device for configuring at least one time window for monitoring the performance of an AI / ML model function; components for determining a first time window or a second time window among the at least one time window, during which the performance of the AI / ML model function is above a performance level and during which the performance of the AI / ML model function is below a performance level; and components for sending second information to the terminal device, the second information indicating the first time window or the second time window.

[0135] In some example embodiments, the first information may include at least one of the following: an identifier of the AI / ML model function; an identifier of a time window within at least one time window; the duration of the time window; or a standard associated with the time window for enabling the terminal device to perform model switching between multiple AI / ML models.

[0136] In some example embodiments, the second information is used by the terminal device to perform management operations associated with multiple AI / ML models configured to implement AI / ML model functionality.

[0137] In some example embodiments, the management operation may include at least one of the following: activating an AI / ML model among multiple AI / ML models; deactivating an AI / ML model among multiple AI / ML models; or adjusting an AI / ML model among multiple AI / ML models.

[0138] In some example embodiments, the apparatus may further include: a component for receiving third indication information from a terminal device when the first information includes the standard, the third indication information indicating at least one of the start point of a time window, the end point of the time window, and the duration of the time window.

[0139] In some example embodiments, the apparatus may further include: a component for receiving fourth information from a terminal device, the fourth information indicating more than one sub-time window within a time window, wherein the more than one sub-time window corresponds to more than one AI / ML model used by the terminal device in the time window.

[0140] In some example embodiments, at least one time window may include multiple time windows, and the components for determining at least one of the first or second time windows may include: components for determining multiple KPI values ​​associated with the AI / ML model functionality for the multiple time windows; components for determining performance above a performance level in the first time window based on a comparison of the multiple KPI values; and components for determining performance below a performance level in the second time window based on a comparison of the multiple KPI values.

[0141] In some example implementations, KPIs may include at least one of the following: accuracy, data throughput, block error rate (BLER), number of retransmissions, number of hybrid automatic repeat request (HARQ), number of beam faults (BF), or number of radio link faults (RLF) associated with AI / ML model functionality.

[0142] In some embodiments, the apparatus may further include components for performing other steps in some embodiments of method 500. In some embodiments, the components include at least one processor and at least one memory including computer program code, the at least one memory and the computer program code being configured to cause execution of the apparatus together with the at least one processor.

[0143] refer to Figure 6 The diagram illustrates an example simplified block diagram of a device suitable for implementing embodiments of the present disclosure. Device 600 can be provided to implement a communication device, such as... Figure 1AThe terminal device 102 shown is illustrated. As shown, device 600 includes one or more processors 610, one or more memories 620 that can be coupled to processor 610, and one or more communication modules 640 that can be coupled to processor 610.

[0144] The communication module 640 is used for bidirectional communication. The communication module 640 has at least one antenna to facilitate communication. The communication interface can represent any interface necessary for communication with other network elements; for example, the communication interface can be wireless or wired to other network elements, or a software-based interface for communication.

[0145] As a non-limiting example, processor 610 can be any type suitable for a local technology network and can include one or more of the following: general-purpose computer, special-purpose computer, microprocessor, digital signal processor (DSP), and processor based on a multi-core processor architecture. Device 600 can have multiple processors, such as application-specific integrated circuit chips that are time-dependent on a clock of a synchronous main processor.

[0146] Memory 620 may include one or more non-volatile memories and one or more volatile memories. Examples of non-volatile memories include, but are not limited to, read-only memory (ROM) 624, electrically programmable read-only memory (EPROM), flash memory, hard disk, optical disc (CD), digital video disc (DVD), and other magnetic and / or optical storage. Examples of volatile memories include, but are not limited to, random access memory (RAM) 622 and other volatile memories that will not persist during power outages.

[0147] Computer program 630 includes computer-executable instructions that are executed by the associated processor 610. Program 630 may be stored in ROM 624. Processor 610 may perform any suitable actions and processes by loading program 630 into RAM 622.

[0148] The embodiments of this disclosure can be implemented by means of a program, enabling device 600 to perform as described in the reference. Figures 2 to 5 Any process discussed in this disclosure. Embodiments of this disclosure may also be implemented in hardware or a combination of software and hardware.

[0149] In some embodiments, program 630 may be tangibly contained in a computer-readable medium, which may be contained in device 600 (e.g., memory 620) or other storage device accessible to device 600. Device 600 may load program 630 from the computer-readable medium into RAM 622 for execution. The computer-readable medium may include any type of tangible non-volatile memory, such as ROM, EPROM, flash memory, hard disk, CD, DVD, etc. Figure 7 An example of a computer-readable medium 700 in the form of a CD or DVD is shown. A program 630 is stored on the computer-readable medium.

[0150] Generally, the various embodiments of this disclosure can be implemented in hardware or dedicated circuitry, software, logic, or any combination thereof. Some aspects may be implemented in hardware, while others may be implemented in firmware or software executable by a controller, microprocessor, or other computing device. Although various aspects of the embodiments of this disclosure are shown and described as block diagrams, flowcharts, or using some other graphical representation, it should be understood that, as non-limiting examples, the blocks, apparatuses, systems, techniques, or methods described herein may be implemented in hardware, software, firmware, dedicated circuitry or logic, general-purpose hardware or controllers or other computing devices, or some combination thereof.

[0151] This disclosure also provides at least one computer program product tangibly stored on a non-transitory computer-readable storage medium. The computer program product includes computer-executable instructions, such as instructions included in a program module that executes on a device on a target real or virtual processor, to perform the actions described above. Figure 4 or Figure 5 Methods 400 or 500 are described. Typically, a program module includes routines, programs, libraries, objects, classes, components, data structures, etc., that perform specific tasks or implement specific abstract data types. In various embodiments, the functionality of a program module can be combined or split among program modules as needed. The machine-executable instructions for a program module can execute on a local or distributed device. In a distributed device, a program module can reside on both local and remote storage media.

[0152] Program code used to perform the methods of this disclosure may be written in any combination of one or more programming languages. This program code may be provided to a processor or controller of a general-purpose computer, special-purpose computer, or other programmable data processing apparatus, such that when executed by the processor or controller, the program code enables the functions / operations specified in the flowcharts and / or block diagrams to be implemented. The program code may be executed entirely on a machine, partially on a machine, as a stand-alone software package, partially on a machine and partially on a remote machine, or entirely on a remote machine or server.

[0153] In the context of this disclosure, computer program code or related data may be carried by any suitable carrier to enable a device, apparatus, or processor to perform the various processes and operations described above. Examples of carriers include signals, computer-readable media, etc.

[0154] Computer-readable media can be computer-readable signal media or computer-readable storage media. Computer-readable media can include, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, apparatuses, or devices, or any suitable combination thereof. More specific examples of computer-readable storage media will include electrical connections having one or more wires, portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable optical disc read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination thereof. As used herein, the term “non-transient” is a limitation of the medium itself (i.e., tangible, not signaling), and not a limitation of data storage persistence (e.g., RAM versus ROM).

[0155] Furthermore, although operations are described in a specific order, this should not be construed as requiring such operations to be performed in the specific order shown or in sequential order, or to perform all shown operations to achieve the desired result. In some cases, multitasking and parallel processing may be advantageous. Similarly, while several specific implementation details are included in the above discussion, these should not be construed as limiting the scope of this disclosure, but rather as descriptions of features that may be specific to particular embodiments. Certain features described in the context of individual embodiments may also be implemented in combination in a single embodiment. Conversely, various features described in the context of a single embodiment may also be implemented individually or in any suitable sub-combination in multiple embodiments.

[0156] Although this disclosure has been described in language specific to structural features and / or methodological actions, it should be understood that the disclosure as defined in the appended claims is not necessarily limited to the specific features or actions described above. Rather, the specific features and actions described above are disclosed as exemplary forms for implementing the claims.

Claims

1. A terminal device, comprising: At least one processor; as well as At least one memory storing instructions that, when executed by the at least one processor, cause the terminal device to at least: Receive first information from a network device for configuring at least one time window, said at least one time window for monitoring the performance of an artificial intelligence / machine learning (AI / ML) model function; The network device receives second information indicating either a first time window or a second time window within the at least one time window, during which the performance of the AI / ML model function is above the performance level, and during the second time window, the performance of the AI / ML model function is below the performance level. as well as Based on the second information, management operations associated with multiple AI / ML models configured to implement the functions of the AI / ML models are performed.

2. The terminal device according to claim 1, wherein the management operation includes at least one of the following: Activate the AI / ML model among the multiple AI / ML models; Deactivate the AI / ML models among the multiple AI / ML models; or Adjust the AI / ML models among the multiple AI / ML models.

3. The terminal device according to claim 1 or 2, wherein the first information includes at least one of the following: The identifier of the AI / ML model function; The identifier of the time window in the at least one time window; The duration of the time window; or A standard associated with the time window for enabling the terminal device to perform model switching between the multiple AI / ML models.

4. The terminal device according to claim 3, wherein the terminal device is further configured to: Based on determining that the criterion is met, at least one of the following is determined: the start point of the time window and the end point of the time window, or the duration of the time window; and Send a third message to the network device, the third message indicating at least one of the start point of the time window and the end point of the time window or the duration of the time window.

5. The terminal device according to any one of claims 1 to 4, wherein the second information includes at least one of the following: The identifier of the time window in the at least one time window; An indication of whether the time window is represented as the first time window or the second time window; or A performance metric for the AI / ML model functionality during the time window.

6. The terminal device according to any one of claims 1 to 5, wherein the terminal device is configured to perform the management operation by: Based on the second information received indicating the first time window, the first AI / ML model among the plurality of AI / ML models used during the first time window is identified as a suitable AI / ML model for implementing the functions of the AI / ML model.

7. The terminal device according to any one of claims 1 to 5, wherein the terminal device is configured to perform the management operation by: Based on the second information received indicating the second time window, the second AI / ML model used during the second time window among the plurality of AI / ML models is identified as not a suitable AI / ML model for implementing the functions of the AI / ML model.

8. The terminal device according to claims 1 to 7, wherein the terminal device is further configured to suspend any management operations associated with the plurality of AI / ML models during the at least one time window.

9. The terminal device according to any one of claims 1 to 7, wherein the terminal device is further configured to: Based on determining that more than one of the plurality of AI / ML models is used within a time window in the at least one time window, more than one sub-time window is determined, each corresponding to one of the more than one AI / ML models used in the time window; and Send a fourth message to the network device indicating the more than one sub-time window.

10. The terminal device according to any one of claims 1 to 9, wherein the at least one time window comprises a plurality of time windows, and wherein the plurality of AI / ML models are used respectively in the plurality of time windows.

11. A network device, comprising: At least one processor; as well as At least one memory storing instructions that, when executed by the at least one processor, cause the network device to at least: Send first information to the terminal device for configuring at least one time window, the at least one time window being used to monitor the performance of the artificial intelligence / machine learning (AI / ML) model function; A first time window or a second time window is determined among the at least one time window, during which the performance of the AI / ML model function is above the performance level, and during the second time window, the performance of the AI / ML model function is below the performance level; as well as Send a second message to the terminal device, the second message indicating the first time window or the second time window.

12. The network device of claim 11, wherein the second information is used by the terminal device to perform management operations associated with a plurality of AI / ML models configured to implement the functionality of the AI / ML models. And the management operation mentioned therein includes at least one of the following: Activate the AI / ML model among the multiple AI / ML models; Deactivate the AI / ML models among the multiple AI / ML models; or Adjust the AI / ML models among the multiple AI / ML models.

13. The network device according to claim 11 or 12, wherein the first information includes at least one of the following: The identifier of the AI / ML model function; The identifier of the time window in the at least one time window; The duration of the time window; or A standard associated with the time window for enabling the terminal device to perform model switching between the multiple AI / ML models.

14. The network device of claim 13, wherein the network device is further configured to: If the first information includes the standard, third indication information is received from the terminal device, the third indication information indicating at least one of the start point and end point of the time window or the duration of the time window.

15. The network device according to any one of claims 11 to 14, wherein the second information includes at least one of the following: The identifier of the time window in the at least one time window; An indication of whether the time window is represented as the first time window or the second time window; or A performance metric for the AI / ML model functionality during the time window.

16. The network device according to any one of claims 11 to 15, wherein the network device is further configured to: The terminal device receives fourth information indicating more than one sub-time window within the at least one time window, wherein the more than one sub-time window corresponds to more than one AI / ML model used by the terminal device in the time window.

17. The network device according to any one of claims 11 to 16, wherein the at least one time window comprises a plurality of time windows, and the network device is configured to determine at least one of the first time window or the second time window by: For the multiple time windows, determine multiple KPI values ​​for key performance indicators (KPIs) associated with the functions of the AI / ML model; Based on the comparison of the multiple KPI values, it is determined that the performance in the first time window is higher than the performance level; or Based on the comparison of the multiple KPI values, it is determined that the performance in the second time window is lower than the performance level.

18. The network device of claim 17, wherein the KPI includes at least one of the following: The accuracy, data throughput, block error rate (BLER), number of retransmissions, number of Hybrid Automatic Repeat Requests (HARQ), number of beam faults (BF), or number of radio link faults (RLF) associated with the AI / ML model functionality.

19. A method comprising: At the terminal device, first information for configuring at least one time window is received from the network device, the at least one time window being used to monitor the performance of the artificial intelligence / machine learning (AI / ML) model function; At the terminal device, second information is received from the network device, the second information indicating a first time window or a second time window among the at least one time window, during which the performance of the AI / ML model function is above the performance level, and during the second time window, the performance of the AI / ML model function is below the performance level; as well as At the terminal device, based on the second information, management operations associated with multiple AI / ML models configured to implement the functions of the AI / ML models are performed.

20. A method comprising: At the network device, first information for configuring at least one time window is sent to the terminal device, the at least one time window being used to monitor the performance of the artificial intelligence / machine learning (AI / ML) model function; At the network device, a first time window or a second time window is determined among the at least one time window, during which the performance of the AI / ML model function is above the performance level, and during the second time window, the performance of the AI / ML model function is below the performance level; as well as At the network device, second information indicating at least one of the first time window or the second time window is sent to the terminal device.

21. A non-transitory computer-readable medium comprising program instructions that, when executed by a device, cause the device to perform at least the method according to claim 19 or 20.

Images