Method and apparatus for performance monitoring based on generative ai

Abstract

Various embodiments of this disclosure relate to methods and apparatus for performance monitoring based on generative AI. The method includes: receiving an ML model monitoring request, the request including an indication of a desired matching ratio for GenAI model output data and measurement data, and a query for the availability of GenAI model output data for monitoring; determining whether the queried GenAI model output data is locally available at a UE; if the queried data is unavailable, transmitting a request for GenAI model output data, wherein the request includes an indication of the desired matching ratio; receiving a pre-trained GenAI model or the requested GenAI model output data associated with the desired matching ratio; if the pre-trained GenAI model is received, generating GenAI model output data associated with the desired matching ratio; performing ML model performance monitoring using at least a portion of the generated data or at least a portion of the received data; and transmitting monitoring results.

Description

Technical Field

[0001] This disclosure pertains to the field of communications. Background Technology

[0002] Some abbreviations that can be found in the instruction manual and / or accompanying drawings are defined here as follows.

[0003] 6G sixth generation GenAI Generative Artificial Intelligence ML machine learning O-RAN Open Radio Access Network UE User Equipment NW network elements RAN Radio Access Network GI generalization information MF matching factor The transition to 6G networks presents a range of challenges due to increased complexity, growing data traffic, and the demand for personalized services. Traditional network management techniques are no longer sufficient, as 6G networks require faster speeds, lower latency, and the ability to support advanced applications. Open Radio Access Networks (O-RAN) offer a flexible architecture that facilitates the integration of hardware and software from diverse vendors, addressing some of these challenges. As networks continue to evolve, AI-driven solutions become crucial for ensuring seamless operation and optimizing performance. Generative AI (GenAI) is gaining significant attention in 6G networks to address complex issues such as resource allocation, traffic prediction, and security. Summary of the Invention

[0004] The following discloses aspects relating to the exemplary embodiments disclosed herein. It should be understood that these aspects are not intended to limit the scope of this disclosure. In fact, this disclosure may cover multiple aspects that may not be set forth below.

[0005] According to an embodiment, a user equipment (UE) (403) is provided, including at least one processor (503) and at least one memory (502) storing instructions that, when executed, cause the UE (403) to: receive a machine learning (ML) model monitoring request from a network element (NW), the request including an indication of a desired matching ratio of generative AI (GenAI) model output data and measurement data, and a query for the availability of generative AI (GenAI) model output data for monitoring; determine whether the queried generative AI (GenAI) model output data is locally available at the UE; and if the queried data is unavailable, transmit a request for generative AI (GenAI) model output data to a UE provider server or network element. The request includes an indication of a desired matching ratio between the output data and measurement data of a generative AI (GenAI) model; receiving a pre-trained generative AI (GenAI) model or requested generative AI (GenAI) model output data associated with the desired matching ratio between the output data and measurement data; if a pre-trained generative AI (GenAI) model is received, generating generative AI (GenAI) model output data associated with the desired matching ratio between the output data and measurement data; using at least a portion of the generated data or at least a portion of the received data to perform performance monitoring of the ML model; and transmitting the monitoring results to network elements.

[0006] Another embodiment provides a method performed by a user equipment (UE) (403), comprising: receiving a machine learning (ML) model monitoring request from a network element, the request including an indication of a desired matching ratio of generative AI (GenAI) model output data and measurement data, and a query for the availability of generative AI (GenAI) model output data for monitoring; determining whether the queried generative AI (GenAI) model output data is locally available at the UE; and if the queried data is unavailable, transmitting a request for generative AI (GenAI) model output data to a UE provider server or network element, wherein the request includes the generative AI (GenAI) model output data. The system provides an indication of the required matching ratio between data and measurement data, receives a pre-trained generative AI (GenAI) model or requested generative AI (GenAI) model output data associated with the required matching ratio between the generative AI (GenAI) model output data and the measurement data, generates generative AI (GenAI) model output data associated with the required matching ratio between the generative AI (GenAI) model output data and the measurement data when the pre-trained generative AI (GenAI) model is received, performs performance monitoring of the ML model using at least a portion of the generated data or at least a portion of the received data, and transmits the monitoring results to the network element (NW).

[0007] In another embodiment, a network element (402) is provided, comprising: at least one processor and at least one memory storing instructions that, when executed by the at least one processor, cause the network element to: send a machine learning (ML) performance monitoring request to a user device, the request including an indication of a desired match ratio between generative AI (GenAI) model output data and measurement data, and a query for the availability of generative AI (GenAI) model output data for monitoring; if the queried data is unavailable at the user device, receive from the user device a request for generative AI (GenAI) model output data, wherein the request includes an indication of a desired match ratio between the generative AI (GenAI) model output data and measurement data; transmit a pre-trained generative AI (GenAI) model or the requested generative AI (GenAI) model output data associated with the desired match ratio between the generative AI (GenAI) model output data and measurement data; receive monitoring results from the user device; and trigger model retraining if performance degradation is detected.

[0008] Another embodiment provides a method performed by a network element (402), comprising: sending a machine learning (ML) performance monitoring request to a user device, the request including an indication of a desired match ratio between generative AI (GenAI) model output data and measurement data, and a query for the availability of generative AI (GenAI) model output data for monitoring; if the queried data is unavailable at the user device, receiving from the user device a request for generative AI (GenAI) model output data, wherein the request includes an indication of a desired match ratio between the generative AI (GenAI) model output data and measurement data; transmitting a pre-trained generative AI (GenAI) model or the requested generative AI (GenAI) model output data associated with the desired match ratio between the generative AI (GenAI) model output data and measurement data; receiving monitoring results from the user device; and triggering ML model retraining if performance degradation is detected.

[0009] Furthermore, simulated data reduces the reliance on large-scale real-world data collection. The controlled variations in simulated data also support robust model evaluation.

[0010] This embodiment enables flexible deployment and supports ML monitoring on both the UE and NW sides. Furthermore, this approach aligns with ongoing 3GPP discussions (Release 19) regarding enhancements to ML monitoring. The proposed embodiment can be contributed to Release 20 as part of a data collection framework within the LCM (potentially for ML-enabled mobility use cases in Release 20, using simulated data generated by the UE rather than measured).

[0011] Details of exemplary embodiments are set forth in the accompanying drawings and the following description. Other features, objects, and advantages of this disclosure will be apparent from the specification, drawings, and claims. Attached Figure Description

[0012] The specific embodiments are described with reference to the accompanying drawings. The same numbers are used throughout the drawings to refer to features and components: Figure 1 This demonstrates the use of Generative AI (GenAI) technology to generate simulated data for performance monitoring of AI / ML-enabled modules, focusing on their characteristics and functions.

[0013] Figure 2 It provides a high-level representation based on the GenAI approach.

[0014] Figure 3 A workflow for ML-enabled feature monitoring using Generative AI (GenAI) in wireless communication networks is shown.

[0015] Figure 4AA flowchart of method 400A for performance monitoring of machine learning (ML) models performed on the user equipment (UE) side is shown.

[0016] Figure 4B A flowchart of method 400B for performance monitoring of machine learning (ML) models performed at network elements (NWs) is shown.

[0017] Figure 5 The signaling diagram between the user equipment (UE) and network elements for performing performance monitoring is shown.

[0018] Figure 6 A method for performance monitoring of use case-related machine learning (ML) models performed by a user device is shown.

[0019] Figure 7 A method for performance monitoring of use case-related machine learning (ML) models performed by network elements is shown.

[0020] Figure 8 A signaling diagram between user equipment, network elements, and user equipment (UE) provider servers for use case-related machine learning (ML) performance monitoring is shown in an example implementation of this topic.

[0021] Figure 9A The process of monitoring ML model performance at the UE provider server is illustrated.

[0022] Figure 9B The process of sending and utilizing simulated data for ML performance monitoring is further illustrated.

[0023] Figure 10 This is a simplified block diagram of a device for implementing exemplary embodiments of the present disclosure.

[0024] Figure 11 This is a simplified block diagram of a device for implementing exemplary embodiments of the present disclosure.

[0025] Figure 12 A flowchart (1200) is shown for a method (1200) for performance monitoring of use case-related machine learning (ML) models executed on the user equipment (UE) side.

[0026] Figure 13 A flowchart (1300) is shown for a method (1300) for performance monitoring of use case-related machine learning (ML) models executed on the network element (NW) side. Detailed Implementation

[0027] Example embodiments will now be described with reference to the accompanying drawings. The terminology used in this disclosure of the example embodiments shown in the drawings is not intended to be limiting. In the drawings, similar numerals denote similar elements.

[0028] It should be understood that although the terms "first," "second," etc., may be used herein to describe various elements, these elements should not be limited by these terms. These terms are only used to distinguish one element from another. For example, without departing from the scope of the exemplary embodiments, a first element may be referred to as a second element, and similarly, a second element may be referred to as a first element.

[0029] This specification may refer to "a," "an," or "some" embodiments in several places. This does not necessarily mean that each such reference is for the same embodiment(s), or that the feature applies only to a single embodiment. Individual features of different embodiments may also be combined to provide other embodiments.

[0030] As used herein, the singular forms “a,” “an,” and “the” are intended to include the plural forms as well, unless otherwise expressly stated. It will also be understood that, when used in this specification, the terms “comprising,” “including,” “containing,” and / or “including” specify the presence of the stated feature, integer, step, operation, element, and / or component, but do not preclude the presence or addition of one or more other features, integers, steps, operations, elements, components, and / or combinations thereof.

[0031] As used herein, whenever the phrase “at least one” or “one or more” precedes a list of elements, where the elements are connected by “and” or “or”, it means at least any one of the elements, or at least some of the elements, or all of the elements. As used herein, whenever the phrase “one of the following” precedes a list of elements, where the elements are connected by “and” or “or”, it means that only one of the elements exists at a given time, unless the context allows for the inclusion of more than one element. Unless the relevant context otherwise indicates, the use of the term “or” should be understood as “inclusive or” rather than “exclusive or”. Unless otherwise specifically stated, or otherwise understood in the context in which it is used, conditional language, such as “may” or “may”, is generally intended to convey that a particular embodiment may include, while other embodiments may not include, a particular feature, element, and / or step. Therefore, such conditional language is generally not intended to imply that one or more embodiments require a feature, element, and / or step in any way. It should be understood that when an element is referred to as “connected” or “coupled” to another element, it may be directly connected or coupled to the other element, or there may be an intermediate element. Furthermore, as used herein, “connected” or “coupled” can include wireless connections or couplings. As used herein, the term “and / or” includes any and all combinations and arrangements of one or more of the associated listed items. As used herein, the terms “at least one” and “one or more” mean “any one of at least one” and “any one of one or more”, respectively.

[0032] Unless otherwise defined, all terms used herein (including technical and scientific terms) shall have the same meaning as commonly understood by one of ordinary skill in the art to which this disclosure pertains. It will also be understood that terms such as those defined in common dictionaries shall be interpreted as having a meaning consistent with their meaning in the context of the relevant field, and shall not be interpreted in an idealized or overly formal sense unless expressly defined herein.

[0033] The accompanying drawings depict a simplified structure showing only some elements and functional entities, all of which are logical units whose implementations can differ from those shown. The connections shown are logical connections; actual physical connections may differ. Furthermore, all logical units described and depicted in the drawings include the software and / or hardware components required for the unit's operation. Additionally, each unit itself may implicitly include one or more components. These components may be operatively coupled to each other and configured to communicate with each other to perform the unit's functions.

[0034] As used herein, the term "circuit system" may refer to at least one of the following: a) Hardware circuit implementation only (such as implementations in analog, digital and / or quantum circuit systems); b) A combination of (multiple) hardware circuits and software, such as (if applicable): (i) a combination of (multiple) analog, digital and / or quantum hardware circuits with software / firmware, and (ii) any or all portions of (multiple) hardware processors (including (multiple) digital and / or quantum processors) and (multiple) memories having software, which work together to enable a device such as a mobile device, computing device or server to perform various functions); or c) Any or all of the hardware circuitry (e.g., firmware) required for operation, such as any or all of the components of a microprocessor, a processor, and / or a quantum processor, but the software may not exist if it is not required for operation.

[0035] This definition of circuit system applies to all uses of the term in this application (including in any claim). As another example, as used in this application, the term circuit system also covers implementations of hardware circuitry or processors (or processors) or a portion thereof and its (or their) accompanying software and / or firmware. For example, and if applicable to a particular claim element, the term circuit system also covers baseband integrated circuits or processor integrated circuits for mobile devices or similar integrated circuits in servers, cellular network devices or other computing or network devices.

[0036] However, in the following text, different example embodiments will be described using communication network architectures based on 3GPP standards for communication networks (such as 6G (sixth generation)) as examples of communication networks to which example embodiments can be applied, without limiting the example embodiments to such architectures.

[0037] Generative AI (GenAI) models are designed to learn the underlying structure of data to generate synthetic or non-real-world data that statistically simulates or resembles the distribution of the source data. GenAI also refers to a type of ML model that has the ability to generate new data based on patterns it learns, thus achieving data augmentation. Using GenAI can be beneficial in ML model performance monitoring because, for example, relying on ground truth data is expensive and resource-intensive / time-consuming in many cases. Furthermore, ground truth data is collected under certain conditions and may not be suitable as part of a performance monitoring process for testing the generalization ability of an ML model.

[0038] Figure 1 This paper presents a high-level overview of how generative AI (GenAI) technology can be used to generate simulated data for performance monitoring of AI / ML-enabled functions in wireless communication networks. Traditional methods of ML performance monitoring rely heavily on real-world measurement data, which can be expensive and time-consuming to collect. GenAI models enhance ML performance monitoring by leveraging simulated data generation. The AI / ML-enabled module (102) represents a machine learning model deployed in the network that requires continuous monitoring to ensure optimal performance. This module can be responsible for various tasks such as beam management, localization, or channel state information (CSI) prediction. To support its monitoring, data sources such as use case AI / ML modules (101) can be provided. Historical data of the inputs and outputs of use case AI / ML models (102) are used for simulated data generation to train and validate generative AI models. Figure 2 A general representation of a generative AI (GenAI)-based approach for generating simulated data to support ML performance monitoring in use cases within wireless communication networks is illustrated. Similarly, this approach leverages GenAI techniques to create simulated monitoring data when real-world data is constrained, unavailable, or difficult to collect under varying network conditions. This enables efficient, scalable, and cost-effective ML model evaluation without over-reliance on field-collected data.

[0039] A generative AI model (201) is responsible for generating simulated data that closely resembles real-world network conditions. It can be trained using historical or pre-collected data, ensuring that the simulated output maintains statistical consistency with actual network measurements. The model is adaptable and can be configured to generate data with varying levels of generalization based on specific monitoring needs. In the example, the GenAI model can be trained based on real radio measurement data and can be dynamically adapted to real-time network conditions. In an embodiment, the user equipment can receive retraining messages from network elements for retraining the ML model and performs the retraining using simulated monitoring data, at least in part.

[0040] The generalization control parameter (202) defines the similarity between the simulated data and the real-world measurement data. This parameter indicates whether the generated data matches specific real-world conditions or has several general real-world conditions.

[0041] Input data generated by GenAI can be fed into ML models deployed in the network, either at the user equipment (UE) (403) side or within the network element (NW) (402), to evaluate and monitor outputs based on the input. The ML performance monitoring module (204) monitors the performance of (multiple) ML-enabled features. If a performance degradation is detected, the UE can notify the network element and request further steps, such as initiating model retraining or parameter tuning.

[0042] This approach enables the generation of simulated data on demand, ensuring continuous and adaptive ML model evaluation while minimizing the need for real-world data collection. By leveraging generative AI technologies, this method aligns with 3GPP AI / ML model lifecycle management and provides a scalable solution for monitoring ML enablement capabilities in 5G and future 6G networks.

[0043] Figure 3 This paper illustrates a workflow for monitoring ML-enabled features using Generative AI (GenAI) in wireless communication networks. The process ensures the continuous monitoring, evaluation, and optimization of ML models deployed in user equipment (UE) (403) or network elements (NW) (402) using a combination of real-world data and simulated data generated by the generative AI module. This ensures a structured sequence, guaranteeing the effective collection, generation, validation, and utilization of data for ML model training, inference, and performance monitoring.

[0044] In this embodiment, ML management (301) is responsible for managing the entire ML lifecycle, including training, inference, and monitoring. It defines data collection strategies, determines the required level of generalization information (GI), and coordinates interactions between different system components. ML management (301) can be implemented at a network element (NW) (402) or a cloud-based AI / ML controller.

[0045] ML Management (301) initiates an ML model monitoring request, specifying: the target ML model to be monitored, the required level of accuracy for evaluation, the status of real-world data availability, and the level of generalization information (GI) generated from the simulated data (if required).

[0046] In this embodiment, real-world measurement data (302) can be collected from the network, including radio signal measurements, user mobility patterns, and environmental conditions. This data is used to train the ML model and serves as a reference for evaluating the performance of ML-driven network functions. Network elements (NW) (402) or user equipment (UE) (403) can access the stored real-world measurement data to determine if it is sufficient for ML model evaluation. If sufficient real-world data is available, it can be used directly for performance evaluation. If the measured real-world data is insufficient, the system can perform simulated data generation.

[0047] In this embodiment, when real-world measurement data is unavailable or insufficient, the generative AI module (303) generates simulated data. The generative AI module can be trained using historical real-world data and can create statistically accurate simulated samples. Those skilled in the art will note that the input or input-output data samples are used for ML performance monitoring.

[0048] The ML management (301) can request simulated data from the generative AI module (303), specifying the level of generalization information (GI). The generative AI module (303) can generate simulated data that is consistent with statistical patterns of real-world network conditions.

[0049] Simulated data can be tagged with metadata, including a GI level defining the similarity to real-world data, a matching factor quantifying the alignment with measured data, and a scenario classification specifying the network conditions associated with the data. In the example, simulated monitoring data can be generated based on a fidelity parameter indicating the level of matching between the simulated monitoring data and historical (previously real) monitoring data. Here, GI information is forwarded to the GENAI module. Optionally, a target fidelity can also be shared from data processing complementary to GI.

[0050] In this embodiment, data processing rules (304) manage how real-world and simulated data are filtered, validated, and used for ML model training. These rules can be implemented at a network element (NW) (402) or a centralized AI model controller. The network element (NW) (402) can validate received simulated data by comparing it with historical real-world data to ensure it meets quality and statistical accuracy thresholds and confirming data availability for ML performance monitoring.

[0051] If the simulated data is validated, it can be stored for future use, and the system trains the model. In an embodiment, use case ML training (305) can represent the training phase of developing and optimizing an ML model using real-world and simulated data. Training can occur at network elements (NW) (402) for full network optimization (e.g., beam management, resource allocation) and at user equipment (UE) (403) for device-specific intelligence (e.g., positioning, adaptive radio configuration), where the ML model can be trained and model parameters fine-tuned to ensure alignment with the generalization information (GI) level. Once training is complete, the trained ML model can be stored and ready for real-time inference.

[0052] In an embodiment, use case ML inference (306) may refer to the real-time execution of a trained ML model at a user equipment (UE) (403). The UE can use the trained ML model to process incoming network data and generate predictions or optimizations for network functions, such as beamforming for enhanced signal transmission, channel state estimation for adaptive radio conditions, and location-based optimization algorithms. The UE can continuously record the inference results to ensure optimal model execution.

[0053] In an embodiment, use case ML monitoring (307) can be performed at the user equipment (UE) (403) to evaluate the performance of the ML model.

[0054] In this embodiment, after receiving the ML performance report from the UE, the network element (NW) (402) can determine whether model retraining is required. The NW can detect changes in ML feature performance. If retraining is required, the system can loop back for model retraining.

[0055] This workflow ensures that ML models in wireless networks are continuously optimized by: comprehensive ML monitoring using both real-world and simulated data; using simulated data generation strategies based on generalization information (GI) levels to enhance adaptability; applying data validation rules before ML training to ensure high-quality input; performing ML model inference at the UE for device-level intelligence; and monitoring ML model performance and triggering retraining when needed.

[0056] By integrating simulated data based on generative AI into the ML lifecycle, this approach eliminates the reliance on large-scale real-world data collection, enhances continuous ML performance evaluation, and ensures AI-driven adaptability in wireless networks. The solution also aligns with the 3GPP AI / ML lifecycle management framework, providing scalable and efficient monitoring strategies for 5G and above.

[0057] Figure 4A A method for performance monitoring of a use-case-related machine learning (ML) model, performed by a user device, is illustrated. It should be understood that the method steps are shown for reference only, and the order of the method steps should not be construed as limiting. The method steps may include any additional steps in any order. Although method 400A can be implemented in any user device, an example method is provided with reference to user device 1200 for ease of explanation.

[0058] At step 402A, the User Equipment (UE) receives a request from the Network Element (NW) for performance monitoring of a use case-related ML model. This request includes an indication of the desired match ratio between the output data of the Generative AI (GenAI) model and the measurement data. The monitoring request includes an indication of the desired match ratio between simulated measurement data and measurement data obtained through measurement. The monitoring request may be received as a result of the Network Element identifying the need for use case-related ML model-based performance monitoring using simulated measurement data from the GenAI model.

[0059] At step 404A, the UE checks the availability of GenAI model output data and / or the ability to execute a GenAI model associated with the desired match ratio indicated by the generative AI (GenAI) model output data and measurement data. In another example, use case ML performance monitoring may not be performed at the UE due to the unavailability of measurement data. In yet another example, the UE may not perform monitoring because it does not have access to the use case-related machine learning (ML) model, or because it cannot achieve the desired match ratio when generating simulated measurement data.

[0060] At step 406A, the UE transmits a request for GenAI model output data to the network element if: the GenAI model output data is unavailable, and the UE cannot execute the GenAI model considering the required matching ratio. Therefore, in this case, the UE can send a request to the network element to generate simulated measurement data.

[0061] In step 408A, the UE transmits a request for the GenAI model to the network element if: the GenAI model output data is unavailable, and the UE is able to execute the GenAI model considering the required matching ratio. In such an example, the network element uses the GenAI model to generate simulated measurement data.

[0062] In one example, the UE can use at least a portion of the GenAI model output data to retrain the use case-related ML model.

[0063] In one example, the UE can detect performance degradation based on use case-related ML model performance monitoring, and implement the retention of use case-related ML models and / or notify network elements of performance degradation.

[0064] In one example, the UE can use at least a portion of the monitoring data to specifically update its ML model.

[0065] In the example, the monitoring data corresponds only to the input data of the ML model to which performance monitoring is to be performed, or to both the input and output data, and the monitoring data is processed by the ML model to derive the monitoring results.

[0066] In one example, monitoring data is used by the UE ML model to derive monitoring results.

[0067] In the example, the UE can check its ability to run a GenAI model locally or request the necessary simulated data generated by the GenAI model.

[0068] In the example, performance monitoring includes generating monitoring reports shared with NW.

[0069] In one example, monitoring results are analyzed at the UE to evaluate the performance of the ML model.

[0070] Figure 4B This illustrates a method for performance monitoring of use-case-related machine learning (ML) models performed by a network element. It should be understood that the method steps are shown for reference only, and the order of the method steps should not be construed as limiting. The method steps may include any additional steps in any order. Although method 400B can be implemented in any network element, an example method is provided with reference to network element 1300 for ease of explanation.

[0071] At step 402B, the network element (NW) identifies the need for performance monitoring of a use case-related machine learning (ML) model utilizing the output data of a generative artificial intelligence (GenAI) model, and determines the desired match ratio between the GenAI model output data and the measurement data. Measurement data can be acquired through measurement and can be used to generate performance monitoring for the use case-related machine learning (ML) model. In cases where measurement data is unavailable, simulated measurement data can be used to perform performance monitoring for the use case-related machine learning (ML) model, as explained in detail below.

[0072] At step 404B, the network element (NW) transmits a request to at least one user equipment (UE) for performance monitoring of the ML model related to the use case. The request includes an indication of the desired matching ratio between the GenAI model output data and the measurement data.

[0073] In step 406B, the network element (NW) receives a request for GenAI model output data or a request for the GenAI model from the user equipment. In response to receiving a request for GenAI model output data, in step 408B, the network element performs inference on the GenAI model to obtain GenAI model output data and transmits the GenAI model output data to the user equipment.

[0074] Furthermore, at step 410B, if the network element (NW) receives a request for the GenAI model, the network element (NW) transmits the GenAI model to the user equipment.

[0075] At step 412B, the network element (NW) receives information from the user equipment regarding: performance monitoring of the use case-related machine learning (ML) model associated with the transmitted GenAI model, and information on the required matching ratio of the GenAI model output data and measurement data, or the transmitted GenAI model output data.

[0076] In one example, if a performance degradation of the ML model relevant to the use case is detected, the network element (NW) can send an ML model retraining request along with a GI level indication.

[0077] In the example, NW determines the level of generalization used for monitoring by including a generalization indicator (GI), which specifies the degree to which the expected model to be tested during the monitoring process generalizes.

[0078] In one example, based on the UE's response indicating its ability to run GenAI, the NW chooses between sending the complete trained GenAI model to the UE or sending simulated data.

[0079] In one example, NW triggers UE-side model retraining or refinement by indicating the GI level.

[0080] In the example, the monitoring results are analyzed at NW to evaluate the performance of the ML model.

[0081] Figure 5 The signaling diagram between UE 5-1 and network element 5-2 for performing performance monitoring is shown. At step 502, UE 5-1 is configured for ML model training and ML model inference at step 504. At step 508, network element 5-2 may require performance monitoring with desired generalization and check the availability of monitoring data. At step 510, network element 5-2 may request UE 5-1 to perform ML monitoring (with GI indication) and may request data availability from UE 5-1. At step 512, UE 5-1 may utilize the indicated GI & check capability to check the availability of monitoring data, and at step 514, UE 5-1 may respond to network element 5-2 in response to the ML monitoring and request Gen AI data. At step 516, network element 5-2 may check the availability of the Gen AI model and prepare if it is unavailable. At step 518, network element 5-2 may send the Gen AI model generated for the data to UE 5-1. At step 520, UE 5-1 performs ML monitoring using the provided Gen AI model, and UE 5-1 may send the monitoring results to network element 5-2 at step 522. At step 524, network element 5-2 may send a retraining request with GI indication (optional) to UE 501, and at step 526, UE 501 may perform model retraining / refinement taking into account NW requests.

[0082] It should be noted that in some embodiments, although described in a sequential manner, Figure 4 and... Figure 5 The steps outlined in the figure can be performed in different orders or in parallel, and in some embodiments, Figure 4 and Figure 5 may be omitted. Figure 5 Some steps in the process.

[0083] Figure 6 A method for performance monitoring of a use-case-related machine learning (ML) model, performed by a user device, is illustrated. It should be understood that the method steps are shown for reference only, and the order of the method steps should not be construed as limiting. The method steps may include any additional steps in any order. Although method 600 can be implemented in any user device, example method 1000 is provided with reference to user device 1200 for ease of explanation.

[0084] At step 602, the user equipment (UE) receives a monitoring request from a network element for performance monitoring of a use-case-related machine learning (ML) model using simulated measurement data generated by generative artificial intelligence (GenAI). The monitoring request includes an indication of the desired match ratio between the simulated measurement data and the measurement data obtained from the measurement. The monitoring request can be received as a result of the network element identifying the need for performance monitoring of the use-case-related ML model using simulated measurement data from the GenAI model.

[0085] In step 604, it is determined that the UE cannot perform the requested monitoring. In one example, the UE may be unable to perform the required monitoring because it cannot run the GenAI model to generate simulated measurement data. In another example, use case ML performance monitoring may not be performed at the UE due to the unavailability of measurement data. In yet another example, the UE may not perform monitoring because it does not have access to the use case-related machine learning (ML) model, or because it cannot achieve the required match ratio when generating simulated measurement data.

[0086] In step 606, the UE transmits information to the network element regarding the user equipment's inability to perform the requested monitoring, as well as information about the user equipment's supplier server, so that the network element can request the supplier server to assist with the requested monitoring. Therefore, in this case, the UE can send a request to the UE supplier server to generate simulated measurement data, the request including the instruction. In such an example, the UE supplier server can receive a GenAI model from the network element, and the GenAI model can be used at the UE supplier server to generate simulated measurement data. In one example, the GenAI model can be received incrementally at the UE supplier server. Thus, the GenAI model to be sent is compared with the GenAI model available at the UE supplier server, and only the necessary updates to the GenAI model are sent. Furthermore, at the UE supplier server, use case-related machine learning (ML) model performance monitoring can be performed based on the generated simulated measurement data to generate performance monitoring results, and the performance monitoring results can be sent to the network element.

[0087] Figure 7 A method for performance monitoring of use-case-related machine learning (ML) models, performed by a network element, is illustrated. It should be understood that the method steps are shown for reference only, and the order of the method steps should not be construed as limiting. The method steps may include any additional steps in any order. Although method 700 can be implemented in any network element, an example method 700 is provided with reference to network element 1300 for ease of explanation.

[0088] At step 702, the network element identifies the need for performance monitoring of a use-case-related machine learning (ML) model using simulated measurement data generated by a generative artificial intelligence (GenAI) model, and the network element determines an indication of the desired matching ratio between the simulated measurement data of the GenAI model and the measurement data acquired by measurement. Measurement data can be acquired through measurement and can be used to generate performance monitoring based on the use-case-related machine learning (ML) model. In cases where measurement data is unavailable, simulated measurement data can be used to perform performance monitoring of the use-case-related machine learning (ML) model, as explained in detail below.

[0089] At step 704, the network element transmits a monitoring request to at least one user equipment (UE) for performance monitoring of a use case-related machine learning (ML) model using simulated measurement data.

[0090] At box 706, a response indicating that use case ML performance monitoring cannot be performed at the UE is received at the network element. Additionally, the response may indicate that the UE cannot run the GenAI model to generate simulated measurement results. Upon receiving the response at the network element, the network element transmits a request for generating simulated measurement data to the UE provider server at box 708. This request may include an indication indicating a desired matching ratio between the simulated measurement data from the GenAI model and the measurement data obtained from the measurement. Such indications(s) can be used at the UE provider server to generate accurate simulated measurement data.

[0091] Furthermore, after transmitting the request to the UE provider server, at box 710, the network element receives a request for GenAI from the UE provider server. In response, at box 712, the network element transmits the GenAI model to the UE provider server. As described above, the GenAI model is incrementally received at the UE provider server based on a comparison between the GenAI model to be transmitted and the GenAI models available at the UE provider server. The GenAI model can be used at the UE provider server to generate simulated measurement data. Finally, at box 714, the network element receives performance monitoring results. According to this topic, performance monitoring results are generated at the UE provider server based on simulated measurement data, which is generated based on an indication. As discussed above, the indication is used to indicate the desired matching ratio between the simulated measurement data of the GenAI model and the measurement data obtained by measurement.

[0092] Figure 8A signaling diagram is shown between user equipment 8-1, network element 8-2, and user equipment (UE) provider server 8-3 for use case-related machine learning (ML) performance monitoring in an example implementation of this topic.

[0093] At step 804, a monitoring request for use case-related machine learning (ML) model performance monitoring can be sent from network element 8-2 to UE 8-1. ML model performance monitoring can be performed using simulated measurement data from a generative artificial intelligence (GenAI) model. Furthermore, the monitoring request may include an indication of the desired matching ratio between the simulated measurement data of the GenAI model and the measurement data obtained from the measurement. In this example, measurement data obtained through previous measurements is typically used to perform use case-related ML model performance monitoring. If measurement data is unavailable, simulated measurement data can be used to perform use case-related ML model performance monitoring.

[0094] According to this topic, as a result of network element identification of the need for performance monitoring of use-case-related machine learning (ML) models that utilize simulated measurement data from GenAI models, a monitoring request can be sent.

[0095] Furthermore, in step 806, the UE sends a response to the network element. This response indicates that use case-related ML monitoring cannot be performed at the UE. In one example, the UE may be unable to perform use case-related ML monitoring due to the unavailability of monitoring data. In another example, the UE may not perform monitoring because the GenAI model cannot be run to generate simulated measurement data.

[0096] At step 808, the network element sends a request for ML model performance monitoring to the UE provider server 8-3. In this example, the request is sent along with an indication. At step 810, the UE provider server responds to the unavailability of measurement data and requests a GenAI model. The GenAI model can be used to generate simulated measurement data, based on which use case-related ML monitoring can be performed.

[0097] At step 812, the GenAI model is transmitted from the network element to the UE provider server. In the example embodiment, the GenAI model is transmitted incrementally. In incremental transmission, the version of the GenAI model available at the UE provider server is compared with the version of the GenAI model to be transmitted, and the difference between the two is transmitted to the UE provider server. This minimizes transmission overhead because it is not necessary to transmit the complete GenAI model to the UE provider server every time, and only the incremental / difference GenAI model needs to be transmitted to the UE provider server. The GenAI model is then used at the UE provider server to generate simulated monitoring data.

[0098] At step 814, the UE provider server performs use case-related ML monitoring using simulated monitoring data generated by the GenAI model. Furthermore, at step 816, the monitoring results are sent from the UE provider server to the network element. Additionally, in the example embodiment, the ML model can be trained using pre-collected data. This is shown as first step 802 in the flowchart. The pre-collected data may be based in part on actual measurement data or in part on simulated measurement data generated by the GenAI model.

[0099] At step 818, if a degradation in the performance results of the use case-related ML model is detected, retraining of the ML model on the UE provider server can be performed. At step 820, any updates regarding changes (e.g., changes to the ML model) can be transmitted to the network element.

[0100] Figure 9A The process of ML model performance monitoring at the UE provider server (901) is shown, focusing on requesting and receiving simulation data for monitoring purposes.

[0101] The network element (902) sends a request to the UE for ML performance monitoring. This request includes an indication of the desired matching ratio between the generative AI (GenAI) model output data and measurement data, and a query for the availability of the generative AI (GenAI) model output data used for monitoring. If the required real-world measurement data is unavailable or insufficient, the user equipment (903) can determine whether the queried generative AI (GenAI) model output data is available locally at the UE. If the queried generative AI (GenAI) model output data is not available locally, the user equipment (903) can transmit a request for generative AI (GenAI) model output data, including an indication of the desired matching ratio between the generative AI (GenAI) model output data and measurement data, to the UE provider server or the network element.

[0102] As described above, the user equipment (903) may transmit a request to the UE provider server (901). This request may include: a generalization information (GI) level, which defines the similarity between simulated data and real-world measurement data; information about the ML model to be monitored, including specific use cases (e.g., mobility management, location, or beam prediction); and additional parameters such as the target network scenario and the required level of accuracy for performance evaluation.

[0103] Upon receiving a request, the UE provider server (901) can process the request and assess whether the requested simulated data is available in its repository. The UE provider server (901) can check the available simulated data in its simulated data repository. If simulated data matching the requested parameters exists, the simulated data can be retrieved and prepared for transmission to the user equipment (903). If the requested simulated data is unavailable or not aligned with the specified generalization information (GI) level, the network element can send a GenAI model to the UE provider (901), and the UE provider (901) can use the trained generative AI model to generate new simulated data. Alternatively, the network element can generate simulated data, and the UE provider receives the simulated data to perform monitoring.

[0104] Subsequently, the UE provider server (901) can use a generative AI model trained on real-world measurement data to generate simulated monitoring data. The simulated data generation process involves: learning from real-world data stored in a simulated data repository; generating new simulated data samples based on a specified level of generalization information (GI); and labeling each simulated sample with a matching factor that quantifies the similarity between the generated data and real-world measurements.

[0105] Once the simulated data is generated at the network element, it can be prepared for transmission to the UE provider (901). The UE provider server (901) can transmit the simulated monitoring data to the network element (902). This transmission may include: simulated data samples formatted to resemble real-world network measurement data, a matching factor indicator that allows the network element (902) to assess the similarity between the data and real-world measurements, and metadata specifying the level of generalization information (GI) to ensure that the simulated data is aligned with the monitoring request parameters.

[0106] Upon receiving the simulated monitoring data, the UE provider (901) can perform ML model monitoring by feeding the simulated data into the ML model deployed at the user equipment (903), comparing the model's predictions with the expected results, identifying performance deviations, evaluating the model's generalization ability, and ensuring that the model performs well under different conditions. The monitoring results are shared with network elements.

[0107] If a performance degradation is detected, the network element (902) can initiate a model retraining request, and the UE provider (901) should update or retrain its ML model using additional simulated or real-world data via signaling.

[0108] This approach ensures continuous ML performance evaluation by eliminating reliance on real-world data collection while enabling on-demand generation of simulated data. Integration at the generalization information (GI) level allows for fine-tuning control over the use of simulated data, thereby improving the adaptability of ML models to various network conditions. By leveraging simulated data generation based on generative AI, this system provides an efficient and scalable solution for AI-driven wireless networks in 5G and beyond.

[0109] Figure 9B The process of sending and utilizing simulated data for ML performance monitoring is further illustrated, along with the interactions between the user equipment (903), the UE provider server (901), and the network element (902). This figure is in... Figure 9A The above section extends the discussion to detail how to generate, transmit, and utilize simulated data to evaluate the performance of ML models.

[0110] Here, the network element sends simulated data to the UE provider server (901). The UE provider can use the provided simulated data to perform monitoring. The UE provider (901) sends the results of this monitoring to the network element (902). The network element (902) can send a retraining request with a GI indication to the UE provider when needed. In response, the UE provider (901) can send the retraining update as an incremental update.

[0111] The simulated data can be formatted to resemble real-world network conditions and includes: simulated monitoring data tagged with metadata that describes the statistical properties of the data; a matching factor indicator that allows the UE provider (901) to determine the similarity between the simulated measurement data and the real-world measurement data; and a generalization information (GI) level that specifies the extent to which the simulated data has been adapted to new or extreme conditions for robust ML evaluation.

[0112] After receiving the simulated data from the network element (902), the UE provider (901) can evaluate the data used for ML performance monitoring. The UE provider can verify that the received data meets the required monitoring parameters, thereby ensuring that it is consistent with the expected operating conditions of the ML model. Subsequently, the network element (902) can send the verified simulated data to the user equipment (903). This transmission ensures that the UE (903) can access the appropriate monitoring data, enabling the UE to perform performance evaluation on its ML model.

[0113] The transmission may include: simulated data samples formatted for processing by the ML model at the UE (903), performance evaluation parameters that ensure the UE uses the correct subset of data for monitoring, and metadata defining the generalization level, allowing the UE to evaluate the model's generalization performance under different conditions.

[0114] The network element (902) can process the monitoring results received from the user equipment (903). Based on the reported performance evaluation, the network element (902) can determine whether the ML model needs to be retrained. If a performance degradation is detected, a retraining request can be triggered, instructing the UE provider server (901) to generate additional training data or update the ML model using a refined dataset. This process ensures that the UE can receive simulated data, process it for ML model evaluation, compare performance results, and transmit a performance report back to the network element (NW) (902).

[0115] Technicians will note that the user equipment can receive a pre-trained generative AI (GenAI) model or requested generative AI (GenAI) model output data associated with a desired matching ratio between the generative AI (GenAI) model output data and measurement data. When a pre-trained generative AI (GenAI) model is received, the user equipment can generate generative AI (GenAI) model output data associated with a desired matching ratio between the generative AI (GenAI) model output data and measurement data, use at least a portion of the generated data or at least a portion of the received data to perform performance monitoring of the ML model, and transmit the monitoring results to network elements.

[0116] Those skilled in the art will also note that the network element includes at least one processor and at least one memory storing instructions that, when executed by the at least one processor, cause the network element to implement corresponding features with respect to those features implemented by the user equipment. The network element sends a machine learning (ML) performance monitoring request to the user equipment, the request including an indication of a desired match ratio between generative AI (GenAI) model output data and measurement data, and a query for the availability of generative AI (GenAI) model output data for monitoring; if the queried data is unavailable at the user equipment, it receives a request from the user equipment for generative AI (GenAI) model output data, wherein the request includes an indication of a desired match ratio between the generative AI (GenAI) model output data and measurement data; transmits a pre-trained generative AI (GenAI) model or the requested generative AI (GenAI) model output data associated with the desired match ratio between the generative AI (GenAI) model output data and measurement data; and receives monitoring results from the user equipment. Additionally, this process ensures that the NW can receive queried simulated data from the UE provider server (901), transmit it to the UE for ML monitoring, process received evaluation reports, and trigger model retraining requests when necessary. By integrating generative AI-based simulated data generation into ML performance monitoring, this approach eliminates the need for large-scale real-world data collection, enhances continuous ML model evaluation, and ensures AI-driven adaptability in wireless networks. The solution also aligns with the 3GPP AI / ML lifecycle management framework, providing a scalable and efficient monitoring strategy for 5G and future 6G networks.

[0117] In one embodiment, the generative AI (GenAI) model is trained using real radio measurement data and dynamically adjusted based on real-time network conditions.

[0118] In one embodiment, the user equipment is configured to: receive a retraining request message with a GI indication for retraining an ML model from a network element, and to perform real retraining using at least partially simulated monitoring data.

[0119] In this embodiment, after determining that the required monitoring data is unavailable, the user equipment is instructed to: initiate a request for simulated data generation to the UE provider server or network element based on the generalization information (GI) level indicated in the monitoring request. In this embodiment, the GI level is used to control the generalization of the generated simulated monitoring data.

[0120] In one embodiment, if the performance is simulated, the UE notifies the network element and requests the model to be retrained.

[0121] In this embodiment, the request is generated based on historical ML model performance data and real-time network conditions.

[0122] In one embodiment, the processor determines, based on the UE capability, whether to transmit a pre-trained generative AI (GenAI) model or the requested generative AI (GenAI) model output data associated with a desired matching ratio between the generative AI (GenAI) model output data and the measurement data, wherein the UE capability is received from the UE as part of a response to the requesting UE.

[0123] In this embodiment, the data is simulated and includes an indication that matches the fidelity of the real data.

[0124] In this embodiment, the GI level is dynamically adjusted based on real-time network conditions and past model performance.

[0125] In one embodiment, the network element determines whether the UE is able to generate simulated monitoring data before sending the GenAI model.

[0126] In this embodiment, the monitoring results received from the UE are stored and analyzed to improve future AI / ML model performance evaluation.

[0127] In this embodiment, the network element dynamically adjusts the frequency of ML monitoring requests based on UE performance trends. Figure 10 This is a simplified block diagram of a device 1001 for implementing an example embodiment of the present disclosure. Device 1001 is an example of a device that can be configured to implement the various methods and processes described herein. Device 1001 may be a terminal device or a user equipment (UE).

[0128] Device 1001 includes a processor 1003 capable of controlling the operation of the device. Processor 1003 may also be referred to as a Central Processing Unit (CPU). Memory 1002 may include both read-only memory (ROM) and random access memory (RAM), and memory 1002 can provide instructions and data to processor 1003. Memory 1002 and processor 1003 may be operatively coupled. Memory 1002 may store computer-readable instructions / computer program code. Computer-readable instructions / computer program code may be pre-stored in memory 1002, or alternatively or additionally, they may be received by device 1001 via electromagnetic carrier signals and / or copied from a physical entity such as a computer program product. Execution of the computer-readable instructions by processor 1003 may enable device 1001 to implement the exemplary embodiments described herein, such as... Figures 4A to 9B , Figures 12 to 13 (If applicable) as shown.

[0129] As used herein, “memory” (also referred to as “computer-readable medium” or “multiple computer-readable media”) can be any non-transitory medium or multiple media or components that can contain, store, transmit, propagate or deliver instructions for use by or in conjunction with an instruction execution system, apparatus or device, such as a computer. As used herein, the term “non-transitory” is a limitation on the medium itself (i.e., tangible, not tactile) rather than a limitation on the persistence of data storage (e.g., RAM and ROM).

[0130] In addition, the circuit system 1006 may include a transmitter (TX) 1004 and a receiver (RX) 1005 that enable the device (1001) to transmit or receive data. The device (1001) may include (not shown) multiple antennas, transmitters, and receivers.

[0131] In some example embodiments, the device (1001) may include features that enable it to perform... Figures 4A to 9B and Figures 12 to 13The component is the one used for the steps / operations discussed herein. This component can be implemented in any suitable form. For example, the component can be implemented in a circuit system (e.g., a memory (1002) and a processor (1003) or a software module).

[0132] Figure 11 This is a simplified block diagram of a device 1101 for implementing an exemplary embodiment of the present disclosure. Device 1101 is an example of a device that can be configured to implement the various methods and processes described herein. Device 1101 may be a network element.

[0133] Device 1101 includes a processor 1103 capable of controlling the operation of the device. Processor 1103 may also be referred to as a central processing unit (CPU). Memory 1102 may include both read-only memory (ROM) and random access memory (RAM), and memory 1102 can provide instructions and data to processor 1103. Memory 1102 and processor 1103 may be operatively coupled. Memory 1102 may store computer-readable instructions / computer program code. Computer-readable instructions / computer program code may be pre-stored in memory 1102, or alternatively or additionally, they may be received by device 1101 via electromagnetic carrier signals and / or copied from a physical entity such as a computer program product. Execution of the computer-readable instructions by processor 1103 may enable device 1101 to implement the exemplary embodiments described herein, such as... Figures 4A to 9B , Figures 12 to 13 (If applicable) as shown.

[0134] As used herein, “memory” (also referred to as “computer-readable medium” or “multiple computer-readable media”) can be any non-transitory medium or multiple media or components that can contain, store, transmit, propagate or deliver instructions for use by or in conjunction with an instruction execution system, apparatus or device, such as a computer. As used herein, the term “non-transitory” is a limitation on the medium itself (i.e., tangible, not tactile) rather than a limitation on the persistence of data storage (e.g., RAM and ROM).

[0135] In addition, the circuit system 1106 may include a transmitter 1104 and a receiver 1105 that enable the device (1101) to transmit or receive data. The device (1101) may include (not shown) multiple antennas, transmitters and receivers.

[0136] In some example embodiments, the device (1101) may include features that enable it to perform, where applicable, such as Figures 4A to 9B , Figures 12 to 13The component is the one used for the steps / operations discussed herein. This component can be implemented in any suitable form. For example, the component can be implemented in a circuit system (e.g., a memory (1102) and a processor (1103) or a software module).

[0137] Figure 12 A flowchart (1200) of a method for performance monitoring of a use case-related machine learning (ML) model on the user equipment (UE) side is shown. At 1202, the UE receives a machine learning (ML) model monitoring request from a network element. The request includes an indication of the desired match ratio between the generative AI (GenAI) model output data and measurement data, and a query for the availability of the generative AI (GenAI) model output data used for monitoring.

[0138] At 1204, the user equipment determines whether the queried generative AI (GenAI) model output data is locally available at the UE. If the queried data is not available, the user equipment transmits a request for the generative AI (GenAI) model output data to the UE provider server or network element at 1206, wherein the request includes an indication of the required matching ratio between the generative AI (GenAI) model output data and the measurement data.

[0139] In response, the user equipment receives at 1208 either a pre-trained generative AI (GenAI) model or the requested generative AI (GenAI) model output data associated with the desired matching ratio of the generative AI (GenAI) model output data and the measurement data.

[0140] When a pre-trained generative AI (GenAI) model is received, the user device generates generative AI (GenAI) model output data at 1210 that is associated with the desired matching ratio between the generative AI (GenAI) model output data and the measurement data.

[0141] The user equipment performs performance monitoring of the ML model at 1212 using at least a portion of the generated data or at least a portion of the received data, and transmits the monitoring results to the network element (NW) at 1214.

[0142] Figure 13 A flowchart (1300) is shown for a method (1300) for performance monitoring of use case-related machine learning (ML) models executed on the network element (NW) side.

[0143] At 1302, the network element sends a machine learning (ML) performance monitoring request to the user device. The request includes an indication of the required matching ratio between the output data and measurement data of the generative AI (GenAI) model, as well as a query for the availability of the generative AI (GenAI) model output data used for monitoring.

[0144] At 1304, if the queried data is not available at the user equipment, the network element receives a request from the user equipment for the output data of the generative AI (GenAI) model, wherein the request includes an indication of the desired matching ratio between the output data of the generative AI (GenAI) model and the measurement data.

[0145] At 1306, the network element transmits a pre-trained generative AI (GenAI) model or the requested generative AI (GenAI) model output data associated with the desired matching ratio of the generative AI (GenAI) model output data and the measurement data, and at 1308, receives monitoring results from the user equipment.

[0146] Example embodiments of this disclosure have been disclosed in the accompanying drawings and specification. Although specific terminology has been used, it is used in a general and descriptive sense only and not for limiting purposes. It will be apparent to those skilled in the art that various modifications and variations can be made to the example embodiments disclosed herein that are consistent with this disclosure without departing from the spirit and scope of this disclosure. Other example embodiments consistent with this disclosure will become apparent from practice upon consideration of the specification and the description herein.

[0147] Furthermore, the various implementations of this disclosure can be described with reference to the following terms, and their features can be combined in any reasonable manner.

[0148] Clause 1. A user equipment (UE) comprising: at least one processor; and at least one memory storing instructions that, when executed by the at least one processor, cause the UE to: receive a machine learning (ML) model monitoring request from a network element (NW), the request including an indication of a desired matching ratio of generative AI (GenAI) model output data and measurement data, and a query for the availability of generative AI (GenAI) model output data for monitoring; determine whether the queried generative AI (GenAI) model output data is locally available at the UE; and if the queried data is unavailable, transmit a request for the generative AI (GenAI) model output data to a UE provider server or the network element, wherein the request includes... The method includes: receiving the instruction for a desired matching ratio between the output data and measurement data of a generative AI (GenAI) model; receiving a pre-trained generative AI (GenAI) model or the requested generative AI (GenAI) model output data associated with the desired matching ratio between the output data and measurement data; generating generative AI (GenAI) model output data associated with the desired matching ratio between the output data and measurement data, if the pre-trained generative AI (GenAI) model is received; performing performance monitoring of the ML model using at least a portion of the generated data or at least a portion of the received data; and transmitting the monitoring results to the network element.

[0149] Clause 2. The user equipment according to any of the preceding clauses, wherein the generative AI (GenAI) model is trained using real radio measurement data and dynamically adapted based on real-time network conditions.

[0150] Clause 3. The user equipment according to any of the preceding clauses shall also cause the user equipment to: receive from the network element a retraining request message having generalization information (GI) indication for retraining the ML model, and to perform the retraining at least in part using simulated monitoring data.

[0151] Clause 4. A user equipment according to any of the preceding clauses, wherein, upon determining that the required monitoring data is unavailable, the user equipment: initiates a request for the generation of simulated data to the UE provider server or the network element based on the generalization information (GI) level indicated in the monitoring request.

[0152] Clause 5. The user equipment according to any of the preceding clauses, wherein the GI level is used to control the generalization of the generated simulated monitoring data.

[0153] Clause 6. A method performed by a user equipment (UE) comprising: receiving a machine learning (ML) model monitoring request from a network element, the request including an indication of a desired matching ratio of generative AI (GenAI) model output data and measurement data, and a query for the availability of generative AI (GenAI) model output data for monitoring; determining whether the queried generative AI (GenAI) model output data is locally available at the UE; and if the queried data is unavailable, transmitting a request for the generative AI (GenAI) model output data to a UE provider server or the network element, wherein the request includes the generative AI (GenAI) model output data and measurement data. The indicated required matching ratio; receiving: a pre-trained generative AI (GenAI) model, or the requested generative AI (GenAI) model output data associated with the required matching ratio of the generative AI (GenAI) model output data and measurement data; if the pre-trained generative AI (GenAI) model is received, generating generative AI (GenAI) model output data associated with the required matching ratio of the generative AI (GenAI) model output data and measurement data; using at least a portion of the generated data or at least a portion of the received data to perform performance monitoring of the ML model; and transmitting the monitoring results to the network element (NW).

[0154] Clause 7. The method according to any of the preceding clauses, wherein the indication is used to control the generalization of the generated simulated monitoring data.

[0155] Clause 8. The method according to any of the preceding clauses, wherein if performance degradation is detected, the UE notifies the network element and requests model retraining.

[0156] Clause 9. A network element comprising: at least one processor; and at least one memory storing instructions that, when executed by the at least one processor, cause the network element to: send a machine learning (ML) performance monitoring request to a user device, the request including an indication of a desired match ratio between generative AI (GenAI) model output data and measurement data, and a query for the availability of generative AI (GenAI) model output data for monitoring; if the queried data is unavailable at the user device, receive from the user device a request for the generative AI (GenAI) model output data, wherein the request includes the indication of the desired match ratio between the generative AI (GenAI) model output data and measurement data; transmit a pre-trained generative AI (GenAI) model or the requested generative AI (GenAI) model output data associated with the desired match ratio between the generative AI (GenAI) model output data and measurement data; and receive monitoring results from the user device.

[0157] Clause 10. A network element according to any of the preceding clauses, wherein the request is generated based on historical ML model performance data and real-time network conditions.

[0158] Clause 11. A network element according to any of the preceding clauses, wherein the processor determines, based on UE capabilities, whether to transmit a pre-trained generative AI (GenAI) model or to transmit the requested generative AI (GenAI) model output data associated with the desired matching ratio of the generative AI (GenAI) model output data and measurement data, wherein the UE capabilities are received from the UE as part of a response to the requesting UE.

[0159] Clause 12. A network element according to any of the preceding clauses, wherein the data is simulated and includes an indication of fidelity matching that of real data.

[0160] Clause 13. A method performed by a network element, comprising: sending a machine learning (ML) performance monitoring request to a user device, the request including an indication of a desired match ratio between generative AI (GenAI) model output data and measurement data, and a query for the availability of generative AI (GenAI) model output data for monitoring; if the queried data is unavailable at the user device, receiving from the user device a request for the generative AI (GenAI) model output data, wherein the request includes the indication of the desired match ratio between the generative AI (GenAI) model output data and measurement data; transmitting a pre-trained generative AI (GenAI) model or the requested generative AI (GenAI) model output data associated with the desired match ratio between the generative AI (GenAI) model output data and measurement data; and receiving monitoring results from the user device.

[0161] Clause 14. The method described in any of the preceding clauses, wherein the GI level is dynamically adjusted based on real-time network conditions and past model performance.

[0162] Clause 15. The method according to any of the preceding clauses, wherein the network element determines whether the UE is able to generate simulated monitoring data before transmitting the GenAI model.

[0163] Clause 16. The method according to any of the preceding clauses, wherein the simulated monitoring data includes indications matching the fidelity of real data to ensure the consistency of ML performance evaluation.

[0164] Clause 17. The method according to any of the preceding clauses, wherein if the UE reports a performance degradation of the model, the network element initiates model retraining.

[0165] Clause 18. The method according to any of the preceding clauses, wherein the monitoring results received from the UE are stored and analyzed to improve future AI / ML model performance evaluation.

[0166] Clause 19. The method according to any of the preceding clauses, wherein the network element dynamically adjusts the frequency of ML monitoring requests based on UE performance trends.

Claims

1. A user equipment (UE) for communication, comprising: At least one processor; and at least one memory storing instructions that, when executed by the at least one processor, cause the UE to: Receive machine learning (ML) model monitoring requests from network elements (NW), the requests including an indication of the desired matching ratio of generative AI (GenAI) model output data and measurement data, and a query for the availability of generative AI (GenAI) model output data for monitoring; Determine whether the output data of the queried generative AI (GenAI) model is locally available at the UE; If the queried data is unavailable, a request for the generative AI (GenAI) model output data is transmitted to the UE provider server or the network element, wherein the request includes the indication of the desired matching ratio between the generative AI (GenAI) model output data and the measurement data. Receive a pre-trained generative AI (GenAI) model or the requested generative AI (GenAI) model output data associated with the desired matching ratio of the generative AI (GenAI) model output data and measurement data. When the pre-trained generative AI (GenAI) model is received, generative AI (GenAI) model output data is generated that is associated with the desired matching ratio of the generative AI (GenAI) model output data and the measurement data. The performance monitoring of the ML model is performed using at least a portion of the generated data or at least a portion of the received data. as well as The monitoring results are transmitted to the network elements.

2. The user equipment of claim 1, wherein the generative AI (GenAI) model is trained using real radio measurement data and dynamically adapted based on real-time network conditions.

3. The user equipment according to claim 1 or 2, further comprising: Receive from the network element a retraining request message with generalization information (GI) for retraining the ML model, and The retraining is implemented using simulated monitoring data, at least in part.

4. The user equipment according to claim 1 or 2, wherein after determining that the required monitoring data is unavailable, the user equipment: based on the generalization information (GI) level indicated in the monitoring request, initiates a request for the generation of simulated data to the UE provider server or the network element.

5. The user equipment according to claim 1 or 2, wherein the GI level is used to control the generalization of the generated simulated monitoring data.

6. A method performed by a user equipment (UE), comprising: Receive machine learning (ML) model monitoring requests from network elements, the requests including an indication of the desired matching ratio between generative AI (GenAI) model output data and measurement data, and a query for the availability of generative AI (GenAI) model output data for monitoring; Determine whether the output data of the queried generative AI (GenAI) model is locally available at the UE; If the queried data is unavailable, a request for the generative AI (GenAI) model output data is transmitted to the UE provider server or the network element, wherein the request includes the indication of the desired matching ratio between the generative AI (GenAI) model output data and the measurement data. take over: Pre-trained generative AI (GenAI) models, or The requested GenAI model output data associated with the desired matching ratio of the GenAI model output data and the measurement data; When the pre-trained generative AI (GenAI) model is received, generative AI (GenAI) model output data is generated that is associated with the desired matching ratio of the generative AI (GenAI) model output data and the measurement data. The performance monitoring of the ML model is performed using at least a portion of the generated data or at least a portion of the received data. as well as The monitoring results are transmitted to the network element (NW).

7. The method of claim 6, wherein the indication is used to control the generalization of the generated simulated monitoring data.

8. The method of claim 6 or 7, wherein if performance degradation is detected, the UE notifies the network element and requests model retraining.

9. A network element for communication, comprising: At least one processor; and at least one memory storing instructions that, when executed by at least one processor, cause the network element to: Send a machine learning (ML) performance monitoring request to the user device, the request including an indication of the required matching ratio of the generative AI (GenAI) model output data and measurement data, and a query for the availability of the generative AI (GenAI) model output data used for monitoring; If the queried data is not available at the user equipment, a request for the output data of the generative AI (GenAI) model is received from the user equipment, wherein the request includes the indication of the desired matching ratio between the output data of the generative AI (GenAI) model and the measurement data; Transmit the pre-trained generative AI (GenAI) model or the requested generative AI (GenAI) model output data associated with the desired matching ratio of the generative AI (GenAI) model output data and measurement data. Receive monitoring results from the user equipment.

10. The network element of claim 9, wherein the request is generated based on historical ML model performance data and real-time network conditions.

Images