Model management method and wireless communication equipment

Abstract

The invention provides a model management method for artificial intelligence (AI) / machine learning (ML). The wireless communication device performs the method to perform AI / ML model monitoring and / or perform lifecycle management (LCM) on one or more AI / ML models according to an AI / ML model monitoring configuration.

Description

Technical Field

[0001] This invention relates to the field of communication systems, and more particularly to a model management method and a wireless communication device. Background Technology

[0002] Artificial intelligence (AI) and machine learning (ML) are two related fields in computer science that aim to create systems capable of performing tasks that typically require human intelligence and learning abilities. Machine learning can be used to solve a variety of problems in fields such as natural language processing, computer vision, robotics, and bioinformatics. In recent years, AI / ML has been increasingly applied to telecommunications networks.

[0003] A general framework has been proposed for studying AI / ML based on open-source reasoning, which illustrates how data collection can facilitate certain AI behaviors, such as model training and model inference. Technical issues

[0004] The performance of wireless communication systems can be improved through AI / ML-assisted services. However, it is foreseeable that system performance may also degrade due to AI / ML model failures. Therefore, the relevant standards for enabling AI / ML model management operations should be clearly defined.

[0005] AI / ML-assisted services can improve the performance of wireless communication systems, but AI / ML models can also malfunction and degrade system performance. Therefore, establishing management standards for AI / ML models throughout their entire lifecycle is crucial.

[0006] Therefore, there is a need for an AI / ML model monitoring method to enhance current wireless communication systems. Technical solution

[0007] The purpose of this invention is to provide a wireless communication device, such as a user equipment (UE) or a base station, and a model management method based on machine learning.

[0008] According to a first aspect of the present invention, a model management method for machine learning (ML) is provided, which can be executed in at least one wireless communication device, the method comprising: Based on the AI / ML model monitoring configuration, perform AI / ML model monitoring and / or lifecycle management (LCM) on one or more AI / ML models.

[0009] According to a second aspect of the present invention, a wireless communication device is provided, the wireless communication device comprising: a processor configured to invoke and run a computer program stored in a memory to cause a device on which the processor is installed to perform the disclosed method.

[0010] According to a third aspect of the present invention, a wireless communication system or an artificial intelligence (AI) / machine learning (ML) system is provided, the system comprising: Base stations; and User Equipment (UE); The base station is configured to monitor one or more AI / ML models. The base station or the UE performs model monitoring and / or lifecycle management (LCM) on one or more AI / ML models according to the AI / ML model monitoring configuration; and The base station or the UE makes a decision related to at least one of the one or more AI / ML models.

[0011] The method of the present invention can be implemented in a chip. The chip may include a processor configured to call and run a computer program stored in memory to cause a device on which the chip is mounted to perform the disclosed method.

[0012] The method of the present invention can be programmed as computer-executable instructions stored in a non-transitory computer-readable medium. When loaded into a computer, the non-transitory computer-readable medium instructs the computer's processor to execute the method of the present invention.

[0013] Non-transitory computer-readable media may include at least one of the following groups: hard disk, CD-ROM, optical storage device, magnetic storage device, read-only memory, programmable read-only memory, erasable programmable read-only memory, EPROM, electrically erasable programmable read-only memory, and flash memory.

[0014] The present invention provides a computer program product, including a computer program, wherein the computer program causes a computer to perform the above-described method.

[0015] The present invention provides a computer program that causes a computer to perform the above-described method. Beneficial effects

[0016] This invention provides a method for monitoring AI / ML models between a user equipment (UE) and a network (gNB, LMF, etc.), which reduces signaling overhead and enhances system performance. The method involves sending and receiving model monitoring requests and responses between the UE and the network, and updating the AI / ML model accordingly.

[0017] This invention describes a complete method for performance and quality monitoring throughout the AI / ML model lifecycle management (LCM), an aspect that has not yet been addressed in existing research.

[0018] This invention enhances AI / ML models for wireless communication systems. Attached Figure Description

[0019] To more clearly illustrate the technical solutions in the embodiments of the present invention, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0020] Figure 1 A schematic diagram illustrating the AI / ML system within the AI / ML framework of this invention is shown.

[0021] Figure 2 A schematic diagram of a wireless communication system including user equipment (UE), base stations, and network entities is shown.

[0022] Figure 3 A system diagram is shown with an AI / ML functional framework for executing model management methods based on ML models.

[0023] Figure 4 A schematic diagram illustrating an embodiment of the method of the present invention is shown.

[0024] Figure 5 A schematic diagram illustrating the first configuration method of the system.

[0025] Figure 6 A schematic diagram illustrating the second configuration method of the drawing system.

[0026] Figure 7 A schematic diagram illustrating the third configuration method of the drawing system.

[0027] Figure 8 A schematic diagram of the fourth configuration method of the drawing system.

[0028] Figure 9 A schematic diagram illustrating a first example of signaling in the method of the present invention is shown.

[0029] Figure 10 A schematic diagram illustrating a second example of signaling in the method of the present invention is shown.

[0030] Figure 11 A schematic diagram illustrating a third example of signaling in the method of the present invention is shown.

[0031] Figure 12 A schematic diagram illustrating a fourth example of signaling in the method of the present invention is shown.

[0032] Figure 13 The diagram illustrates examples of triggering different lifecycle management (LCM) operations.

[0033] Figure 14A schematic diagram of a wireless communication system according to an embodiment of the present invention is shown. Detailed Implementation

[0034] The embodiments of the present invention are described in detail with reference to the accompanying drawings, including technical aspects, structural features, objectives, and effects. Specifically, the terminology used in the embodiments of the present invention is only used to describe the purpose of a particular embodiment and is not intended to limit the disclosure. The abbreviations used in this specification are listed below:

[0035] Embodiments of this invention relate to artificial intelligence (AI) and machine learning (ML) for wireless communication systems (e.g., LTE or New Radio (NR)) and address the problem of AI / ML model identification (ID).

[0036] For simplicity, the terms AI / ML model, AI model, ML model, and model are used interchangeably in this manual. Similarly, AI / ML model monitoring and model monitoring are also used interchangeably. In this manual, lifecycle management (LCM) may include, at least for single-sided and two-sided models, model selection, activation, deactivation, switching, rollback, model training, model monitoring, model registration, model deployment, model migration, and retraining / fine-tuning.

[0037] See Figure 1 , Figure 1 The system architecture of an AI / ML framework in a telecommunications network is illustrated. In this architecture, system 100 includes a monitoring agent 44 and a decision agent 46. The monitoring agent 44 receives monitoring input, performs calculations on the monitoring input, and generates monitoring output as the calculation results. The monitoring agent 44 provides the monitoring output to the decision agent 46. The decision agent 46 makes lifecycle management (LCM) decisions based on the monitoring output.

[0038] AI / ML model monitoring is the process of tracking and evaluating the performance of one or more AI / ML models and / or lifecycle management (LCM) over time. Monitoring agent 44 collects data as monitoring input, which may include at least inference accuracy, system throughput, spectral efficiency, NACK ratio, ACK ratio, SINR, RSRP, RSRQ, beam failure, link failure, positioning performance, intra-cell interference, inter-cell interference, BLER, MSER and FAR, performance loss, performance improvement, channel information (e.g., real-time channel matrix, long-term channel matrix, CSI, eigenvectors, etc.), input / output data distribution, and / or applicability conditions. Various metrics and calculations are applied to the monitoring input to evaluate the model's accuracy, reliability, fairness, and robustness. Based on the monitoring output, decision agent 46 can make decisions on maintaining the AI / ML model and / or take corrective actions for LCM (e.g., selecting, switching, activating, deactivating, or reverting to a different AI / ML model or non-AI module) to address any problems or anomalies detected by monitoring agent 44.

[0039] AI / ML models can be installed and executed on one or more UEs and networks (e.g., base stations, LMFs, etc.). In this specification, one or more AI / ML models can be installed and executed on UE 10, and / or installed and executed on network 20, wherein the AI / ML models are used for different features and / or functions, considering single-sided models and / or dual-sided models.

[0040] System 100 enables operators to train customized AI / ML models and integrate third-party AI / ML models under their control. This system architecture allows network-based AI / ML services to support a wider range of partners and provide services that meet market demands. The system and method of this invention can improve service quality and enhance market competitiveness.

[0041] See Figure 2 According to one embodiment of the present invention, a telecommunications system including UE 10a, UE 10b, base station (BS) 20a and network entity equipment 30 performs the disclosed method. Figure 2 For illustrative purposes only and not as a limitation, this system may include more UE, BS, and CN entities. Connections between devices and between device components are indicated by lines and arrows in the accompanying drawings.

[0042] Base station 20a can operate as a gNB in ​​a 5G NR network, an eNB in ​​an LTE network, or as a base station for future mobile communication systems beyond 5G. A gNB is a 5G wireless network node that connects to the core network via an NG interface; an eNB is a 4G wireless network node that connects to the evolved packet core network via an S1 interface; a base station for beyond 5G can be a Smart Virtual eNB (SVeNB), capable of performing the functions of EPS components and reducing end-to-end latency.

[0043] UE 10a may include processor 11a, memory 12a, and transceiver 13a. UE 10b may include processor 11b, memory 12b, and transceiver 13b. Base station 20a may include processor 21a, memory 22a, and transceiver 23a. Network entity device 30 may include processor 31, memory 32, and transceiver 33. Processors 11a, 11b, 21a, and 31 may be configured to implement the functions, processes, and / or methods set forth in this specification. The various layers of the wireless interface protocol may be implemented in processors 11a, 11b, 21a, and 31. Memory 12a, 12b, 22a, and 32 may operatively store various programs and information used to drive the respective processors. Transceivers 13a, 13b, 23a, and 33 may all be operatively coupled to the respective processors and used to transmit and / or receive wireless or wired signals. UE 10a may communicate with UE 10b via a sidelink. Base station 20a can be an eNB, gNB or other type of radio node, and can configure radio resources for UE 10a and UE 10b.

[0044] Processors 11a, 11b, 21a, and 31 may include application-specific integrated circuits (ASICs), other chipsets, logic circuits, and / or data processing devices. Memory 12a, 12b, 22a, and 32 may include read-only memory (ROM), random access memory (RAM), flash memory, memory cards, storage media, and / or other storage devices. Transceivers 13a, 13b, 23a, and 33 may include baseband circuitry and radio frequency (RF) circuitry for processing radio frequency signals. When this embodiment is implemented in software, the techniques described herein can be implemented in the form of modules, processes, functions, entities, etc., and perform related functions. Modules may be stored in memory and executed by the processor. Memory may be implemented inside or outside the processor and can communicate with the processor in various ways known in the art.

[0045] Network entity device 30 may be a node in a CN. The CN may include an LTE CN or a 5G core network (5GC), which includes User Plane Functions (UPF), Session Management Functions (SMF), Access and Mobility Management Functions (AMF), Unified Data Management (UDM), Policy Control Functions (PCF), Control Plane (CP) / User Plane (UP) Separation (CUPS), Authentication Server (AUSF), Network Slice Selection Function (NSSF), and Network Open Functions (NEF). It may also include LMF and AF.

[0046] The UE example in this specification can be either UE 10a or UE 10b. The base station example in this specification can be base station 20a. NW 20 can be a gNB or other type of base station, such as an eNB or a base station for use beyond 5G. The transmission of uplink (UL) control signals or data signals can be a transmission operation from the UE to the base station. The transmission of downlink (DL) control signals or data signals can be a transmission operation from the base station to the UE. DL control signals may include downlink control information (DCI) or radio resource control (RRC) signals from the base station to the UE.

[0047] AI / ML functional framework:

[0048] To demonstrate the logical relationships between lifecycle management (LCM) actions, a general AI / ML functional framework is used. Figure 3 Draw a possible AI / ML functional framework.

[0049] See Figure 3 A general AI / ML functional framework should consist of the following functional modules: • Data Collection: The data collection unit 1101 is a functional module used to collect measurement data / data required for various actions in lifecycle management (LCM), such as the model training unit 1102, the model inference unit 1104, and the model management / performance monitoring unit 1107. The data collection unit 1101 can be used for single-sided models and / or two-sided models. • Model Training: The model training unit 1102 is a functional module used to perform model training. The model training unit 1102 can perform model training, validation and testing, and finally generate a trained model. • Model Management / Performance Monitoring: The model management / performance monitoring unit 1107 is a functional module for performing model management, including at least one of the following: model monitoring, model selection, model activation, model deactivation, model switching, and rollback. Furthermore, this functional module can provide control signaling to the model inference unit 1104, the model training unit 1102, or the model storage unit 1106. • Model Inference: Model Inference Unit 1104 is a functional module for performing model inference. This module generates inference outputs from the AI / ML model. Model Inference Unit 1104 uses data from the data collection unit as input and the trained AI / ML model provided by Model Training Unit 1102 to generate a set of inference outputs. Model monitoring results can trigger a set of AI / ML model actions and can also serve as input for model monitoring. • Model Storage: Model storage unit 1106 is a functional module used to store models after training is completed / updated / retrained / fine-tuned.

[0050] Figure 4 An embodiment of the method of the present invention is illustrated. At least one wireless communication device can be a network entity that performs a machine learning-based model management method. In one embodiment, the at least one wireless communication device may include a user equipment (UE). In another embodiment, the at least one wireless communication device may include a network (NW). In yet another embodiment, the at least one wireless communication device may include a combination of a UE and an NW.

[0051] The at least one wireless communication device performs AI / ML model monitoring and / or lifecycle management (LCM) on one or more AI / ML models according to the AI / ML model monitoring configuration (step S001). Model monitoring can be performed by user equipment (UE) or network (NW).

[0052] Prior to monitoring, the method may include sending or receiving AI / ML model monitoring activation and deactivation signaling. The AI / ML model monitoring activation and deactivation signaling may include one or more of the following: AI / ML model monitoring threshold, AI / ML model monitoring object, activation time, and deactivation time.

[0053] In one embodiment, the activation of one or more AI / ML models can be controlled by AI / ML model monitoring configuration or activation message indication, or according to predefined rules.

[0054] The activation message includes AI / ML model activation / deactivation information for AI / ML model activation, and the AI / ML model activation / deactivation information includes one or more of the following: • AI / ML model identifier (AI / ML model ID); • Activation / deactivation information; • Activation time; • Activation time has expired; • Time window.

[0055] The AI / ML model monitoring configuration may include one or more of the following: AI / ML monitoring configuration identifier; AI / ML model identification; AI / ML model monitoring objects; AI / ML model monitoring metrics; AI / ML model monitoring report configuration; AI / ML model monitoring report configuration identifier; AI / ML model monitoring report types; AI / ML model monitoring thresholds; AI / ML model monitoring reference signal; AI / ML model monitoring time window; Trigger time; Number of model monitoring reports; Monitoring and reporting cycle; Hysteresis parameter (Hys); Activate time offset; UE identifier; Scene identification; Feature identifier; Community signage; Site identification; Area identification; Data quality indicator.

[0056] In some embodiments, AI / ML model monitoring configuration can be sent via downlink messages, which may include Radio Resource Control (RRC) messages, system information, Media Access Control (MAC) Control Unit (CE), Downlink Control Information (DCI), PDSCH, PDCCH, or LPP messages.

[0057] In some embodiments, AI / ML model monitoring configuration can be sent via MAC CE, UCI, Radio Resource Control (RRC) messages, Physical Uplink Control Channel (PUCCH), Physical Uplink Shared Channel (PUSCH), or LTE Location Protocol (LPP) messages.

[0058] The AI / ML model monitoring configuration can be configured by the network (NW) or the UE. In one embodiment, when both AI / ML model monitoring and configuration are performed by the network, the method further includes: configuring monitoring configuration for the one or more AI / ML models and / or configuring lifecycle management (LCM) for the one or more AI / ML models.

[0059] In one embodiment, when AI / ML model monitoring and decisions regarding monitoring results are performed by different devices, the method further includes: reporting the results of model monitoring.

[0060] When the AI / ML model monitoring results are sent via uplink messages, the AI / ML model monitoring results can be carried in MAC CE, UCI, Radio Resource Control (RRC) messages, Physical Uplink Control Channel (PUCCH), Physical Uplink Shared Channel (PUSCH), or LTE Location Protocol (LPP) messages.

[0061] When the AI / ML model monitoring results are sent via downlink messages, the AI / ML model monitoring results can be carried in Radio Resource Control (RRC) messages, system information, Media Access Control (MAC) control unit (CE), Downlink Control Information (DCI), PDSCH, PDCCH, or LPP messages.

[0062] The AI / ML model monitoring result report may include one or more of the following: AI / ML model monitoring report configuration identifier, AI / ML model identifier, AI / ML model monitoring result, and model monitoring report trigger event.

[0063] At least one wireless communication device makes a decision related to at least one AI / ML model, wherein the decision includes at least a lifecycle management action. The lifecycle management action may include AI / ML model activation, deactivation, model selection, model switching, or rollback. The decision may be triggered based on a threshold method, a rule-based method, an ML-based method, or an expert system-based method. The decision may be made by a UE or a network (NW).

[0064] The rule-based method includes at least one of the following: • Decision rules are based on thresholds; • Decision rules are based on candidate models; • Decision rules are based on AI / ML model monitoring methods; and / or • Decision rules are predefined or preconfigured; and / or • Decision rules include default rules.

[0065] In rule-based approaches, threshold-based decision rules can be based on different thresholds defined for different lifecycle management actions, and / or on different thresholds defined for different AI / ML model monitoring methods.

[0066] In one embodiment, model monitoring may be triggered when the performance of at least one AI / ML model deteriorates and falls below an AI / ML model performance threshold. In another embodiment, model monitoring may be triggered when the performance of at least one AI / ML model improves and exceeds an AI / ML model performance threshold.

[0068] definition: The terminology is defined as follows: 1) Global AI / ML model ID: The global AI / ML model identifier is a unique identifier for an AI / ML model and can be assigned at different granularity levels, such as global level, PLMN-specific level, operator-specific level, or AI / ML management platform-specific level. 2) Logical AI / ML model ID: A logical AI / ML model identifier is a unique identifier for AI / ML models within a network and maps to the global AI / ML model identifier. Logical identifiers can be assigned at different granularity levels within the network, such as cell-specific, link-specific, TA-specific, CU-specific, DU-specific, UPF-specific, AMF-specific, RRC-specific, or network slice-specific. "Cell-specific" means that each AI / ML model has a unique identifier within a specific cell; similarly, "link-specific," "TA-specific," "CU-specific," "DU-specific," "UPF-specific," "AMF-specific," and "network slice-specific" all indicate that each AI / ML model has a unique identifier within a specific network context. 3) Hardware capability: The hardware capabilities of a CPU and / or GPU depend on a variety of parameters, such as manufacturer, version, computing performance, and number of threads. Computing performance can be measured by metrics such as FLOPS, IPC, memory bandwidth, cache size, and clock speed. 4) Software capability: The system's software capabilities include components such as manufacturer, version, operating system (OS), application programming interface (API), software development kit (SDK), runtime environment, firmware, and compiler. 5) Supported AI capabilities: The system's AI support capabilities refer to the types of AI / ML models the system can execute, the computing resources and power consumption required for each model, and the file formats used to store or exchange models and related data. AI / ML model types include Convolutional Neural Networks (CNN), Recurrent Neural Networks (RNN), Deep Belief Networks (DBN), Support Vector Machines (SVM), Random Forests, K-Nearest Neighbors (KNN), Naive Bayes, etc. 6) Real-time performance: The real-time performance of a system involves multiple metrics, such as CPU / GPU / memory / storage / network utilization, temperature, fan speed, power consumption, voltage, hardware component error rate, response time, and thread utilization. 7) Network entity identifier: Network entities should be identified by at least one of the following network entity identifiers: • UE identifier: IMEI (International Mobile Equipment Identity) or UUID; • RAN Identifier: Cell Identifier, Physical Identifier, PLMN, UUID or other identifiers. 8) AI / ML model related information: AI / ML model-related information may include one or more of the following: • Global AI / ML model identification; • Provider; •Scene; ·feature; ·Function; ·Version; • Accuracy; • RRC descriptor; AI / ML model descriptors; • Calculate the descriptor; • Storage descriptor; Power consumption. The following is an explanation of some information related to AI / ML models: • Scenario: The scenario attribute of an AI / ML model specifies the scenario setting in which the AI / ML model is applicable. For example, in a given scenario, the AI / ML model is used for a specific feature in an FDD massive MIMO system. Furthermore, the scenario attribute can be set to different values ​​to indicate the type of environment the AI / ML model is optimizing for. For example, scenario values ​​could be indoor, outdoor, flight, water, movement speed, etc., each corresponding to an environment with different characteristics. • Feature: Feature attributes can be configured to functions such as CSI compression, CSI prediction, beam management, positioning, handover, and radio resource management. • Function: Functional attributes can be configured as CSI compression, CSI prediction, beam management, positioning, handover, radio resource management, etc. Furthermore, functional attributes can also be used to represent how AI / ML models are used. For example, the functional attributes of an AI / ML model can be configured as monitoring, inference, and / or training. • RRC descriptor: The RRC descriptor is used to define the reference signal configuration and CSI measurement parameters of the AI / ML model. This parameter includes one or more of the following: ■ Reference signal time domain configuration; ■ Reference signal frequency domain configuration; ■ Number of reference signal antenna ports; ■CSI Content Configuration. The reference signal time-domain configuration can include multiple parameters, such as the duration of each cycle, the time step interval, and the hopping pattern between different time slots. The reference signal frequency-domain configuration can specify different aspects of the frequency domain, such as frequency resource allocation and frequency hopping schemes for different line bands. The CSI content configuration determines the type of information conveyed by the CSI, such as a channel matrix describing the channel state and an eigenvector representing the dominant direction of the channel. • AI / ML model descriptor: The AI / ML model descriptor is used to describe the model attributes of the AI / ML model, including the following information: ■AI / ML type; ■Accuracy; ■ Input data requirements; ■ Output data; ■ Monitoring methods; ■ Input data distribution; ■ Output data distribution. • Compute descriptor: The computation descriptor specifies the computational resources required by the AI / ML model and the type of CPU or GPU needed, including the following information: ■ Number of CPU cores or GPU units; ■ Clock frequency; ■ Memory bandwidth; ■ Cache size. • Storage descriptor: The storage descriptor specifies the storage resources and memory types required by the AI / ML model, including the following information: ■Memory size; ■ Memory type (e.g., RAM, ROM, flash memory, etc.); ■Memory access speed; ■ Memory latency. • Power consumption: The power consumption attribute describes the power consumed by AI / ML model computation, including the following information: ■ Voltage; ■ Current; ■Average power consumption; ■Peak power consumption; ■ Power efficiency.

[0069] 9) Data collection descriptor: The data collection descriptor is used to specify one or both of the following: training data descriptor or monitoring descriptor. • Training data descriptor: The training data descriptor is used to specify the data collected for training the AI / ML model, including one or more of the following: channel information (e.g., real-time channel matrix, long-term channel matrix, CSI, feature vectors, etc.); system log information (e.g., computation, storage, memory, etc.). • Monitoring descriptor: The monitoring descriptor describes the input data used for monitoring AI / ML models or lifecycle management (LCM), including one or more of the following: inference accuracy; system throughput; spectral efficiency; NACK ratio; ACK ratio; SINR; RSRP; RSRQ; beam failure; link failure; positioning performance; intra-cell interference; inter-cell interference; BLER; MSER and FAR; performance loss; performance improvement; channel information (such as real-time channel matrix, long-term channel matrix, CSI, eigenvectors, etc.); input / output data distribution; applicable conditions. • Decision descriptor: The decision descriptor is used to describe the input data for decision-making, including model monitoring results, etc.

[0070] 10) Definitions related to model monitoring: Model monitoring is a process used to perform certain lifecycle management (LCM) actions, such as selecting, switching, activating, deactivating, or rolling back to different AI / ML models or non-AI modules. This process includes three steps: collecting relevant monitoring inputs, performing monitoring calculations to produce outputs or results, and making decisions based on the outputs or results. These decisions are used to maintain the current AI / ML model. Model monitoring and decision-making can be performed by the same entity or by different entities (UE, NW). Figure 1 The basic process of drawing the model monitoring process. This specification describes four different model monitoring methods for measuring AI / ML models, each with its own criteria and metrics. A summary of each method follows: M-Method A: Model-based inference accuracy monitoring of AI / ML models. The M-method measures the model's ability to predict correct results. • M-Method B: Model-based AI / ML models for system performance monitoring. System performance metrics include throughput, spectral efficiency, NACK ratio, ACK ratio, SINR, RSRP, RSRQ, beam failure, link failure, positioning performance, intra-cell interference, inter-cell interference, BLER, MSER, and FAR. M-method C: Model-based monitoring of input / output data distribution for AI / ML models. The M-method compares the statistical properties of the data used for training and testing the models. M-method D: Model-based applicability monitoring of AI / ML models. The M-method D specifies the scenarios and parameter ranges in which the model is effective and reliable. There are several ways to evaluate changes in system performance. One approach is model selection, which involves choosing from models with similar functions or features based on certain criteria. Another is model switching, which involves switching to a model with different features when the current model is unsuitable. A third approach is fallback, which involves replacing AI-based methods with traditional non-AI methods when they fail. A fourth approach is model updating, which involves retraining or fine-tuning the model to adapt to new data or conditions. These methods can be applied based on the degree of performance degradation of the monitored AI / ML model. For example: model selection is used when performance drops slightly; model switching is used when performance drops moderately; fallback is used when performance drops significantly; and model updating is used when it is necessary to improve model capabilities.

[0071] Example 1: Model Monitoring Process

[0072] To monitor AI / ML models on both the UE and NW sides, certain information needs to be transmitted over the air interface. This information may include: • AI / ML model monitoring configuration, which specifies the parameters and criteria for monitoring the model; • AI / ML model monitoring activation / deactivation signaling, which is used to indicate when to start or stop the monitoring process; • AI / ML model monitoring results reports / feedback, which provide data and metrics collected during the monitoring process; • AI / ML model monitoring and decision-making results reports / feedback, which are used to convey actions and recommendations based on the monitoring results.

[0073] The detailed descriptions of the above information types are as follows: • AI / ML Model Monitoring Configuration: The configuration may include one or more of the following information: AI / ML monitoring configuration identifier, AI / ML model identifier, AI / ML model monitoring object, AI / ML model monitoring metric, AI / ML model monitoring report configuration, AI / ML model monitoring report configuration identifier, AI / ML model monitoring report type, AI / ML model monitoring threshold, AI / ML model monitoring reference signal, AI / ML model monitoring time window, trigger time, number of model monitoring reports, monitoring report period, hysteresis parameter (Hys), activation time offset, UE identifier, scene identifier, feature identifier, cell identifier, site identifier, area identifier, and data quality indicator.

[0074] AI / ML Model Monitoring Configuration Instructions: AI / ML model monitoring configuration is a set of parameters used to specify how to monitor the performance and behavior of AI / ML models in actual deployments. AI / ML model monitoring configuration may include one or more of the following: AI / ML monitoring configuration identifier; AI / ML model identifier; AI / ML model monitoring object; AI / ML model monitoring metrics; AI / ML model monitoring report configuration; AI / ML model monitoring report configuration identifier; AI / ML model monitoring report type; AI / ML model monitoring threshold; AI / ML model monitoring reference signal; AI / ML model monitoring time window; trigger time; number of model monitoring reports; monitoring report cycle; hysteresis parameter (Hys); activation time offset; UE identifier; scene identifier; feature identifier; cell identifier; site identifier; region identifier; data quality indicator. ■AI / ML monitoring configuration ID: A unique identifier used to associate an AI / ML monitored object with an AI / ML model monitoring report configuration. This identifier specifies the monitoring configuration for a particular AI / ML model. ■AI / ML Model ID: Can be one or more of the following: logical model ID, global model ID, and / or AI / ML model information ID related to the AI / ML model. The AI / ML model ID is included in the AI / ML model activation / deactivation information, and / or in the model selection, model switching, and model identification information. ■AI / ML model monitoring object: refers to the monitoring object corresponding to a specific AI / ML model and / or a specific LCM action (such as deactivation / activation, selection, switching, rollback). ■AI / ML model monitoring metrics: Indicators that show different aspects of a model's performance and behavior, such as system performance, inference accuracy, input data distribution, output data distribution, and applicable conditions. ■AI / ML model monitoring report configuration: This specifies what information the model monitoring report will report, such as ground truth CSI / decoding / measurement information, performance loss or improvement, inference accuracy loss or improvement, input / output data distribution loss or improvement, applicability conditions loss or improvement, or the model measurement values ​​themselves (e.g., system performance, inference accuracy, etc.). ■AI / ML model monitoring report configuration ID: A unique identifier used to identify the configuration of the AI / ML model monitoring report. ■AI / ML model monitoring report type: Used to specify the type of model monitoring report, such as periodic or event-triggered. ■AI / ML model monitoring threshold: A numerical value or range of values ​​that defines the acceptable performance range of an AI / ML model, as described in Example 2 and / or Example 3. ■ AI / ML model monitoring reference signal: Contains information related to the AI / ML model, such as AI / ML model identifier, time / frequency configuration (periodic, non-periodic, semi-static), etc. ■AI / ML model monitoring time window: A parameter that indicates the duration of model monitoring, including at least one of the following: start time, duration, or time offset. ■ Trigger time: The time an event condition needs to be maintained before it triggers a model monitoring report. ■ Number of Model Monitoring Reports: The number of reports generated for event-triggered and / or periodic model monitoring report types. ■Monitoring report interval: The time interval between event triggers and / or periodic monitoring reports. ■ Hysteresis (Hys): A hysteresis parameter used to control the sensitivity of the model to monitoring events. ■ Activation time offset: Specifies the activation time offset of reference signaling, AI / ML models, or LCM functions in model monitoring. This offset can be applied to the message reception time (e.g., Figures 9 to 12 Message 1). ■UE Identifier (UE ID): Indicates the user equipment. ■Scenario ID ■Feature ID ■ Cell ID ■ Site ID ■Zone ID ■ Data quality indicator: Used to indicate data quality, which can be based on factors such as latency, data size, scenario, characteristics, cell, site and / or region. ●AI / ML model monitoring activation and deactivation signaling: This signaling includes one or more of the following: AI / ML model monitoring threshold, AI / ML model monitoring object, activation time, and deactivation time. This signaling is further detailed in Example 4. AI / ML model monitoring events are further described in Example 2. ■ AI / ML model monitoring thresholds: used to determine the acceptable range or limits of model performance or behavior (see Example 2 and / or Example 3). ■AI / ML Model Monitoring Objects: Used to specify the models and / or LCM actions to be monitored (e.g., activation, selection, switching, rollback). ■Activation Time: The time when the signaling, model, or LCM action begins to activate. ■ Deactivation time: The time during which signaling, modeling, or LCM actions cease. ●AI / ML Model Monitoring Result Report: Used to synchronize model monitoring results between network entities, it can be carried via uplink or downlink channels. The AI / ML model monitoring result report may include: AI / ML model monitoring report configuration identifier, AI / ML model identifier, AI / ML model monitoring result, and model monitoring report trigger event. Detailed explanation follows: ■AI / ML Model Monitoring Report Configuration Identifier: An identifier used to identify the report configuration and to report model monitoring results. ■AI / ML Model Identifier: Can be a logical model identifier, a global model identifier, and / or an identifier related to AI / ML models. AI / ML model identifiers are carried in model activation / deactivation information. ■AI / ML model monitoring results: may include one or more of the following: ◆Option A: The AI / ML model with the highest performance score (defined according to the monitoring configuration) is represented by a logical model identifier, model-related information, or an RRC-specific indicator. ◆Option B: Top N AI / ML models in terms of performance, which may or may not include their performance metrics, such as performance loss / improvement, inference accuracy loss / improvement, changes in input / output data distribution, changes in applicable conditions, etc. ◆Option C: The true value of the AI / ML model, such as based on non-AI CSI information or PDSCH decoded information. ■ Model monitoring report triggering event: This refers to the event that triggers the model monitoring result report. An example of the event is described in Example 2.

[0075] To facilitate monitoring of the decision results of AI / ML models, the decision results include the following information or signaling: AI / ML model monitoring activation / deactivation signaling as defined above, model identifier, and lifecycle management (LCM) actions as defined above.

[0076] In this specification, the AI / ML model being monitored may be referred to as the "monitored AI / ML model." The module performing model monitoring may be referred to as the "monitoring agent." The module that makes recommendations or triggers LCM actions based on the model monitoring results may be referred to as the "decision agent." The monitored AI / ML model, monitoring agent, and decision agent may all be deployed on the UE (e.g., UE10) or base station (e.g., NW20). The fact that a model or agent is deployed on a device indicates that the model or agent is installed on that device and executed by that device.

[0077] See Figure 5 , Figure 6 , Figure 7 and Figure 8 The monitored AI / ML model, monitoring agent 44, and decision agent 46 can be deployed on a UE (e.g., UE 10) or a base station (e.g., NW 20), respectively. A network entity with a monitored AI / ML model (e.g., UE 10 or NW 20) can be referred to as a "monitored network entity." A network entity with a monitoring agent (e.g., UE 10 or NW 20) can be referred to as a "monitoring network entity." A network entity with a decision agent (e.g., UE 10 or NW 20) can be referred to as a "decision network entity." It is understood that a UE (e.g., UE 10) can be a single monitored network entity, or a combination of a monitored network entity, a monitoring network entity, and a decision network entity. Similarly, a base station (e.g., NW 20) can also be a single monitored network entity, or a combination of a monitored network entity, a monitoring network entity, and a decision network entity.

[0078] To avoid redundancy, in this specification, AI / ML model-assisted operations can be interpreted as being performed by the monitored AI / ML model and / or the monitored network entity. Recommended or triggered LCM actions can be interpreted as being performed on the monitored AI / ML model and / or the monitored network entity.

[0079] In this specification, model monitoring operations can be interpreted as being performed by a monitoring agent (e.g., 44) and / or a monitoring network entity.

[0080] In this specification, the decision-making operations used to recommend or trigger LCM actions can be interpreted as being performed by decision agents (e.g., 46) and / or decision network entities.

[0081] The following sections, in conjunction with the above information or signaling, exemplify several scenarios and their corresponding procedures, and describe the expected behavior of the UE and the network (BS and / or CN) in each scenario.

[0082] Scenario 1: Model monitoring is performed on the UE side, and decision-making is performed on the NW side.

[0083] See Figure 5 Monitoring agent 44 is deployed on UE (e.g., UE 10), and decision agent 46 is deployed on NW (e.g., NW 20).

[0084] Figure 9 An example is illustrated where model monitoring is performed on the UE side, while decision-making is performed on the NW side. This model monitoring process can be applied to UE-side models, NW-side models, and / or both-side models, as detailed below:

[0085] Step 1: The network sends Message 1 to the UE for AI / ML model monitoring configuration. Message 1 may include one or more of the following information: AI / ML monitoring configuration identifier, AI / ML model identifier, AI / ML model monitoring object, AI / ML model monitoring metric, AI / ML model monitoring report configuration, AI / ML model monitoring report configuration identifier, AI / ML model monitoring report type, AI / ML model monitoring threshold, AI / ML model monitoring reference signal, AI / ML model monitoring time window, trigger time, number of model monitoring reports, monitoring report period, hysteresis parameter (Hys), activation time offset, UE identifier, scene identifier, feature identifier, cell identifier, site identifier, area identifier, and data quality indicator. Message 1 can be sent via dedicated RRC message and / or MAC CE and / or system information.

[0086] Step 2: Optionally, the NW sends message 2 to the UE to activate / deactivate the AI / ML model monitoring reference signal or AI / ML model-related measurements (e.g., AI-specific measurements and / or traditional measurements, which may include L1 measurements, L3 measurements, or newly defined measurements). Message 2 contains one or more of the following information: AI / ML model identifier, AI / ML model monitoring object, activation time, and deactivation time. Message 2 can be carried via MAC CE, DCI, RRC messages, PDCCH, PDSCH, etc.

[0087] Step 2': Alternatively, if the inference function is deployed on the NW side, the NW can send inference output to the UE via MAC CE, DCI, RRC messages, PDCCH or PDSCH, etc.

[0088] Step 3: The UE sends message 3 to the NW to report the model monitoring calculation results. Message 3 contains one or more of the following information: AI / ML model monitoring report configuration identifier, AI / ML model identifier, AI / ML model monitoring result, and the event that triggered the model monitoring report. Message 3 can be carried in RRC messages, PUSCH, PUCCH, UCI, MAC CE, etc.

[0089] Step 4: NW sends message 4 to UE to notify or recommend decision results, such as activation, deactivation, model selection, model switching, or rollback. The decision is based on the model monitoring calculation results and decision rules in step 3. Message 4 must include at least one of the following: model identifier, LCM action, AI / ML model monitoring activation / deactivation signaling, activation time, and deactivation time. Message 4 can be carried in RRC messages, PDSCH, PDCCH, DCI, MAC CE, etc.

[0090] Scenario 2: AI / ML model monitoring calculations are performed on the NW side, while decision-making is performed on the UE side.

[0091] See Figure 6 Decision agent 46 is deployed on UE (e.g., UE 10), and monitoring agent 44 is deployed on NW (e.g., NW 20).

[0092] Figure 10 An example is illustrated where AI / ML model monitoring computation is performed on the NW side, while decision-making is performed on the UE side. This approach is equally applicable to UE-side models, NW-side models, and / or dual-side models. An example of the process is shown below:

[0093] Step 1: The NW sends message 1 to the UE for AI / ML model monitoring configuration, and the steps are similar to Step 1 in Scenario 1. In addition, message 1 may also include decision rules. Message 1 can be carried in a dedicated RRC message, MAC CE, or system information.

[0094] Step 2: Optionally, the NW sends message 2 to the UE to activate / deactivate AI / ML model monitoring signaling or AI / ML model related measurements, and the steps are similar to step 2 in scenario 1. Message 2 can be carried in MAC CE, DCI, RRC messages, etc.

[0095] Step 3: The UE sends message 3 to the NW as input for model monitoring, such as reporting actual values, traditional measurements, or AI-specific measurements. Message 3 can be carried in RRC messages, UCI, PUSCH, PUCCH, etc.

[0096] Step 4: NW sends message 4 to UE. Message 4 is the output of model monitoring, such as the best performing AI / ML model, or the N best performing AI / ML models (which may or may not contain relevant measurement results). Message 4 can be carried in RRC messages, DCI, PDSCH, PDCCH, etc.

[0097] Step 5: The UE sends message 5 to the NW to notify or recommend the network-side decision result, such as activation, deactivation, model selection, model switching, or fallback. This decision result is based on the model monitoring calculation results and decision rules received in Step 4. Message 5 is similar to Step 4 in Scenario 1 and can be carried in RRC messages, PUSCH, PUCCH, UCI, MAC CE, etc. Alternatively, message 5 may simply be an ACK / NACK sent by the UE to the NW to indicate whether the UE has successfully received the monitoring result.

[0098] Scenario 3: AI / ML model monitoring calculations and decisions are all performed on the UE side.

[0099] See Figure 7 Both the monitoring agent 44 and the decision agent 46 are deployed on the UE (e.g., UE 10).

[0100] Figure 11 An example is illustrated where AI / ML model monitoring and decision-making are both performed on the UE side, and this can be applied to UE-side models, NW-side models, and / or dual-side models. An example flow is as follows:

[0101] Step 1: The NW sends message 1 to the UE for AI / ML model monitoring configuration, and the steps are similar to Step 1 in Scenario 1. In addition, message 1 may also contain decision rules. Message 1 can be carried in a dedicated RRC message, MAC CE, or system information.

[0102] Step 2: Optionally, the NW sends message 2 to the UE to activate / deactivate AI / ML model monitoring signaling or AI / ML model related measurements, and the steps are similar to step 2 in scenario 1. Message 2 can be carried in MAC CE, DCI, RRC messages, PDCCH, PDSCH, etc.

[0103] Step 3: Optionally, the UE sends message 3 to the NW to notify or recommend the network-side decision result, such as activation, deactivation, model selection, model switching, or fallback. The decision result is based on the model monitoring results and decision rules received in Step 2. This step is similar to Step 4 in Scenario 1. Message 3 can be carried as an RRC message, PUSCH, PUCCH, UCI, MACCE, etc. Alternatively, message 3 can simply be an ACK / NACK sent by the UE to the NW to indicate whether the UE has successfully received the activation / deactivation signaling.

[0104] Scenario 4: AI / ML model monitoring computation and decision-making are all performed on the NW side.

[0105] See Figure 8 Both monitoring agent 44 and decision agent 46 are deployed on NW (e.g., NW 20).

[0106] Figure 12 An example is illustrated where AI / ML model monitoring and decision-making are both performed on the NW side, applicable to UE-side models, NW-side models, and / or dual-side models. A possible workflow is as follows:

[0107] Step 1: The NW sends message 1 to the UE for AI / ML model monitoring configuration. This step is similar to step 1 in scenario 1. Message 1 can be carried in a dedicated RRC message, MAC CE, or system information.

[0108] Step 2: Optionally, the NW sends message 2 to the UE to activate / deactivate AI / ML model monitoring signaling or AI / ML model related measurements, and the steps are similar to Step 2 in Scenario 1. Message 2 can be carried in MAC CE, DCI, RRC messages, PDCCH, PDSCH, etc.

[0109] Step 3: The UE sends message 3 to the NW as input for model monitoring, such as reporting actual values, traditional measurements, and / or AI-specific measurements. Message 3 can be carried in UCI, RRC messages, PUCCH, PUSCH, etc.

[0110] Step 4: Optionally, the NW sends message 4 to the UE to notify or recommend decision results, such as activation, deactivation, model selection, model switching, or rollback. The decision is based on the model monitoring calculation results and decision rules received in step 3. This step is similar to step 4 in scenario 1. Message 4 can be carried in RRC messages, PUSCH, PUCCH, UCI, MAC CE, etc.

[0111] In some examples, to achieve more efficient LCM (Leveled Model Monitoring) and improve system performance, model monitoring may need to monitor one or more AI / ML models simultaneously. However, determining which AI / ML models need to be monitored is not always clear. Relevant solutions are described below.

[0112] When model monitoring is deployed on the UE side:

[0113] Solution 1 (Sol#1): The AI / ML model used for model monitoring can be determined by network configuration. Specifically, the network sends a message to the UE to assist in AI / ML model selection. This message may include one or more of the following information: model identifier, features, scenario, function, activation / deactivation signaling, activation time, deactivation time, and time window. The message can be a dedicated RRC message, MAC CE, DCI, PDSCH, PDCCH, or LPP message.

[0114] Solution 2 (Sol#2): The AI / ML model used for model monitoring can be determined based on the UE implementation method.

[0115] When model monitoring is deployed on the NW side:

[0116] Solution 1 (Sol#1): The AI / ML model used for model monitoring can be determined by UE request / UE capability reporting. Specifically, the UE sends a message to the NW to assist in selecting the AI / ML model. This message may include one or more of the following information: model identifier, features, scenario, function, activation / deactivation signaling, activation time, deactivation time, time window, UE storage capacity, and UE hardware capabilities. The message may be UE auxiliary information, a dedicated RRC message, MAC CE, UCI, PUSCH, PUCCH, or LPP message.

[0117] Solution 2 (Sol#2): The AI / ML model used for model monitoring can be determined by the network-side implementation method.

[0118] In some examples, among multiple AI / ML models used for model monitoring, some models are active while others are inactive / unused. However, a challenge in model monitoring is identifying and activating these inactive / unused AI / ML models. The options for identifying and activating inactive / unused AI / ML models are as follows:

[0119] Solution 1 (Sol#1): Identify and activate inactive / unused AI / ML models via activation messages. The message may include one or more of the following information: model identifier, features, scenario, function, activation time, deactivation time, time window, site, and region. The message can be carried via MAC CE, DCI, UCI, system information, PUCCH, PUSCH, PDCCH, or PDSCH.

[0120] Solution 2 (Sol#2): Identify and activate inactive / unused AI / ML models by monitoring configuration information messages. When the UE receives this message, it can activate the corresponding AI / ML model based on the information in the message. The monitoring configuration information message may include one or more of the following information: model identifier, features, scenario, function, activation time, deactivation time, time window, site, and region. The message may be carried in MAC CE, DCI, system information, PDCCH, or PDSCH.

[0121] Solution 3 (Sol#3): Inactive / unused AI / ML models can be identified and activated through predefined, fixed, or random methods. The identification and activation can be based on certain rules, such as activation time or the number of activated AI / ML models.

[0122] Example 2: Event-based model monitoring

[0123] To minimize over-the-air (OTA) overhead and / or air interface overhead, this invention provides an event-triggered AI / ML model monitoring method. The monitored content includes the AI / ML model and AI-specific LCM actions. AI / ML model monitoring includes reporting monitoring information. The method is triggered by a predefined specific event.

[0124] To trigger monitoring, the triggering conditions should be defined first. Below are some possible event examples. This specification defines at least two types of AI / ML model monitoring events. AI / ML model performance can be measured using M-Method A, M-Method B, M-Method C, M-Method D, or other methods. AI / ML model performance thresholds can be set based on inference accuracy, system performance, data distribution, or applicable conditions.

[0125] Event A1: (AI / ML model performance deteriorates to below the AI / ML model performance threshold)

[0126] Event A1 indicates that the system detects a performance degradation and triggers one of the following actions: deactivate the AI / ML model, select another AI / ML model, switch to another AI / ML model, or fall back to a non-AI module (or a non-AI-based solution).

[0127] The event entry condition is considered met when the following condition A1-1 is satisfied: Inequality A1-1 (entry condition): Ms+Hys <Thresh

[0128] The event exit condition is considered met when the following condition A1-2 is satisfied: Inequality A1-2 (leaving the condition): Ms-Hys>Thresh

[0129] The variables in the above formula are defined as follows: Ms: Represents the performance loss of the AI / ML model. Ms can be calculated in two ways (Solution A and Solution B). ●Option A: Ms is calculated based on the output of the AI / ML model (the model currently in use / a new candidate model) and the ground truth, without considering any offset. The ground truth refers to the non-AI / ML model used here. For example, Ms can be calculated via monitoring functions or an AI / ML model specifically designed for AI / ML model monitoring. Measurement results obtained from the monitored AI / ML model can be adjusted using the monitored object-related offset to form the output of the monitored AI / ML model. The monitored object-related offset is set for the AI / ML model and / or specific LCM actions (activation, selection, switching, rollback). ●Option B: Ms is calculated based on the input of the AI / ML model (the model currently in use / a new candidate model) and the ground truth. The ground truth also refers to the non-AI / ML model used here. For example, Ms can be calculated through monitoring functions or AI / ML models specifically designed for AI / ML model monitoring. Measurement results obtained from the monitored AI / ML model can also be adjusted by an offset related to the monitored object to form the output of the monitored AI / ML model, the offset being related to the AI / ML model and / or a specific LCM action. Thresh: The threshold parameter corresponding to event A1. The threshold may depend on the model monitoring method and may be a parameter specific to the AI / ML model. Hys (hysteresis parameter): The hysteresis parameter for event A1. The hysteresis parameter (Hys) can be configured via RRC signaling, including reporting configuration, AI / ML model monitoring configuration, or other signaling.

[0130] Event A2: (AI / ML model performance improves to above the AI / ML model performance threshold)

[0131] Event A2 indicates that the system detects a performance improvement and triggers one of the following actions: deactivate the AI / ML model, select another AI / ML model, switch to another AI / ML model, or fall back to a non-AI module (or a non-AI-based solution).

[0132] The event entry condition is considered met when the following condition A2-1 is satisfied: Inequality A2-1 (entry condition): Ms-Hys <Thresh

[0133] The event exit condition is considered met when the following condition A2-2 is satisfied: Inequality A2-2 (leaving the condition): Ms+Hys>Thresh

[0134] The variables in the above formula are defined as follows: Ms: Represents the performance gain of the AI / ML model. Ms can be calculated in two ways (Solution A and Solution B). ●Option A: Ms is calculated based on the output of the AI / ML model (the model currently in use / a new candidate model) and the ground truth, without considering any offset. The ground truth refers to the non-AI / ML model used here. The measurement results obtained from the monitored AI / ML model can be adjusted using the monitored object-related offset to form the output of the monitored AI / ML model. The monitored object-related offset is set specifically for the AI / ML model. ●Option B: Ms is calculated based on the input of the AI / ML model (the model currently in use / a new candidate model) and the ground truth, without considering any offset. The ground truth also refers to the non-AI / ML model used here. The measurement results obtained from the monitored AI / ML model can be adjusted by the offset related to the monitored object to form the output of the monitored AI / ML model, wherein the offset is set for the AI / ML model. Thresh: The threshold parameter corresponding to event A2. This threshold parameter depends on the model monitoring method and can be a parameter specific to the AI / ML model. Hys: The hysteresis parameter for event A2. The hysteresis parameter (Hys) can be configured via RRC signaling, including reporting configuration, AI / ML model monitoring configuration, or other signaling.

[0135] Example 3: LCM Decision Making for AI / ML Models Based on Model Monitoring

[0136] To improve the efficiency of Lifecycle Management (LCM), model monitoring can trigger various LCM actions, such as activation, deactivation, selection, switching, and rollback. However, since these LCM actions are applicable to different scenarios, decision rules need to be defined for each entity to select the appropriate LCM action based on the model monitoring results. The following are some feasible decision rule design schemes: Threshold-based methods: By setting specific thresholds for model monitoring output, relevant LCM actions can be triggered accordingly. For example, when the monitoring output of a monitored model exceeds a certain threshold, it may be necessary to perform a deactivation operation on the model. • Rule-based approach: This involves defining a set of rules based on the model's monitoring output and executing corresponding actions according to these rules. For example, when the monitoring output indicates a high level of interference, it may be necessary to perform a model switch on the monitored model. • Machine learning-based methods: By training a machine learning model, appropriate LCM actions can be predicted based on the monitoring output of the monitored model. This method is suitable for scenarios with complex decision-making processes that rely on large amounts of data. • Expert system-based methods: This method uses an expert system to determine appropriate LCM actions based on rules and a knowledge base. It is suitable for systems with a large number of... Scenarios involving domain knowledge.

[0137] Alternative Solution 1: Decision rules based on defined thresholds:

[0138] Based on any model monitoring method, when the monitoring output indicates a deterioration in system performance, a specific LCM action can be triggered, such as model selection, model switching, rollback, or model update. To determine which LCM action should be activated, different thresholds Φ can be defined for different LCM actions and different monitoring methods. Therefore, each threshold Φ can be a threshold specific to the LCM action and / or a threshold specific to the monitoring method. Table 2 and... Figure 13 It demonstrates specific threshold examples for each LCM action based on different model monitoring methods. Table 2: Thresholds Φ corresponding to LCM actions and model monitoring methods M-method A M-method B M-method C M-method D Model selection <![CDATA[Φ 1-1 ]]> <![CDATA[Φ 2-1 ]]> <![CDATA[Φ 3-1 ]]> null Model switching <![CDATA[Φ 2-1 ]]> <![CDATA[Φ 2-2 ]]> <![CDATA[Φ 2-3 ]]> <![CDATA[Φ 2-4 ]]> rollback <![CDATA[Φ 3-1 ]]> <![CDATA[Φ 3-2 ]]> <![CDATA[Φ 3-3 ]]> <![CDATA[Φ 4-4 ]]>

[0139] Alternative Solution 2: Decision rules based on candidate models:

[0140] The decision rule can be adjusted based on the available candidate models. When one or more candidate models are available, the decision rule can prioritize either model selection or model switching LCM actions when recommending or triggering a model.

[0141] If a candidate model has the same AI / ML features as the currently deployed AI / ML model, the decision rule will prioritize recommending that model.

[0142] If a candidate model has different AI / ML features than the currently deployed AI / ML model, the decision rule will prioritize recommending model switching.

[0143] If a candidate model supports multiple AI / ML features, where at least one feature is the same as the currently deployed model and at least one feature is different from the currently deployed model, then the decision rule will randomly recommend model selection or model switching.

[0144] Otherwise, when no candidate model exists, the decision rule will prioritize recommending a fallback.

[0145] Alternative Solution 3: Decision rules based on model monitoring methods:

[0146] The decision rules can depend on the specific model monitoring method employed. For example, when a monitoring network entity performs AI / ML model monitoring using any of the M-methods A, B, C, or D, the decision network entity will recommend or trigger a certain LCM action according to the decision rule of alternative solution 3 corresponding to the monitoring method. The decision rules can be defined for M-methods A, B, C, or D respectively. For example:

[0147] For M-methods A, B, or C, the decision network entity randomly recommends or triggers one of the following actions according to the decision rules: model selection, model switching, or rollback.

[0148] For the M-method D, the decision network entity prioritizes model switching and rollback in model recommendation or triggering operations, but the specific LCM action can still be randomly selected. Furthermore, when candidate AI / ML models exist, the decision rule will prioritize model switching over other LCM actions.

[0149] Alternative Option 4: Predefined or pre-configured decision rules:

[0150] The decision rules are predefined or preconfigured rules. These predefined or preconfigured decision rules assign higher priority to a given LCM action than to other actions, according to a predefined or preconfigured priority order. The following are some exemplary preconfigured orders (Options 1 to 6): ●Option 1: Model selection, model switching, rollback ●Option 2: Model selection, rollback, model switching ●Option 3: Model Switching, Model Selection, Undo ●Option 4: Model switching, rollback, model selection ●Option 5: Go back, model selection, model switching ●Option 6: Go back, model switch, model selection

[0151] Alternative 5: Default rules for AI / ML models:

[0152] The method can also be executed based on default rules. Default rules specify the triggering of a particular LCM action under specific circumstances. For example, in the following scenarios, the AI / ML system will switch (i.e., fall back) to a non-AI / ML working model:

[0153] —Link failure: According to the default rules, when the system (such as the monitoring network entity or the model deployment network entity) detects a link failure, the model deployment network entity stops using the AI-based processing flow and falls back to a non-AI / ML model.

[0154] —Handover: According to the default rules, when the system detects that a UE is performing a handover, the model deployment network entity uses a non-AI / ML model for all processing before the handover is completed.

[0155] —Random Access: According to the default rules, when the system detects that the UE is performing random access, the model deployment network entity will use a non-AI / ML model or disable AI-based processing procedures before the UE completes random access.

[0156] Example 4: Activation and Deactivation of AI / ML Models:

[0157] This example illustrates how the UE and network activate or deactivate AI / ML models for specific purposes (such as CSI compression, CSI prediction, beam management, or positioning). The following paragraphs show the AI / ML model activation / deactivation signaling and configuration information.

[0158] 1) AI / ML Model Activation / Deactivation Information: This information is used to activate or deactivate a specific AI / ML model and is carried by downlink or uplink signaling. The signaling may include UE capability reporting, PUCCH, PUSCH, PDCCH, PDSCH, MACCE, or RRC signaling. AI / ML model activation / deactivation information may include one or more of the following elements: AI / ML model identifier, activation / deactivation information, activation time, deactivation time, and time window. ●AI / ML model ID: A unique identifier that can refer to one or more of the following: - Logical AI / ML model identifier; -Global AI / ML model identifier; - Identification of information related to AI / ML models. ● Activate / Deactivate information: Indicates whether the corresponding AI / ML model is active. Its value can be a Boolean (true / false), a string (e.g., "on" or "off"), or a bitmap (bit sequence). The specific implementation is as follows: ■Solution A: Direct Activate / Deactivate: Each bit corresponds to a specific "feature, function, or model ID". When a bit is set to 1, the corresponding item is activated; when a bit is set to 0, the corresponding item is deactivated. For example, if the bit for a scene (such as scene A) is 1, then scene A is activated; otherwise, it is deactivated. The same applies to other objects such as "features, functions, and model IDs". ■Scheme B: Hierarchical Activate / Deactivate: Each bit corresponds to a group of features, functions, or model IDs with common attributes, and these are grouped according to hierarchical relationships. If a bit is set to 1, all items in the group are activated; if a bit is set to 0, all items in the group are deactivated. This hierarchical scheme is applicable to both activation and deactivation. When any hierarchical relationship exists between multiple items, activation based on this hierarchical scheme can reduce indication overhead. For example: When scene #1 contains feature #1 and feature #2; feature #1 contains function #1 and function #2; feature #2 contains function #3 and function #4; if scene #1 is activated / deactivated, all its features, functions, and model IDs are activated / deactivated. If scene #1 and feature #1 are activated / deactivated simultaneously, all their functions and model IDs are activated / deactivated. ● Activation time: This indicates the time after which a network entity (e.g., a UE) will start running the AI / ML model. ●Deactivation time: This indicates the time after which a network entity (e.g., a UE) will stop running its AI / ML model. ●Time window: Represents the fixed duration for which the AI / ML model is executed.

[0159] 2) When “AI / ML model activation / deactivation information” is sent from the UE to the network side via the uplink channel or from the UE to the network entity, the network side will provide the UE with configuration information containing one or more of the following information, and the UE can perform corresponding operations accordingly. • AI / ML model identifier: As a unique identifier, it can refer to one or more of the following: - Logical AI / ML model identifier; -Global AI / ML model identifier; - Identification of information related to AI / ML models. The AI / ML model identifier is carried by the AI / ML model activation / deactivation information. • Reference signal information configuration: This indicates the time-frequency resource allocation, duration, and activation mode of the reference signal corresponding to the AI / ML model identifier. Reference signals may include, but are not limited to, CSI-RS, SRS, DMRS, PTRS, etc. • CSI reporting information configuration: Used to configure the required CSI measurement information for the AI / ML model identifier, including measurement events, channel state information (CSI) reporting period, and related parameters. • Activation time: Network entities (such as UEs) will start and run the corresponding AI / ML model after the configured activation time. • Deactivation time: Network entities (such as UEs) will stop running the corresponding AI / ML models after the configured deactivation time. • Time window: Used to indicate the fixed duration of execution of the AI / ML model.

[0160] 3) When the "AI / ML model activation / deactivation information" is sent from the network side to the UE via the downlink channel or transmitted from the network side to the UE, the UE can perform corresponding operations accordingly. The following are exemplary operations: When an activation signaling is received, the UE initiates AI / ML model-related processes (such as AI / ML models for CSI measurement, AI / ML models for CSI reporting, etc.) according to the RRC configuration corresponding to the AI / ML model identifier. When a deactivation signaling is received, the UE terminates the AI / ML model-related processes (e.g., AI / ML models used for CSI measurement, AI / ML models used for CSI reporting, etc.) according to the RRC configuration corresponding to the AI / ML model identifier.

[0161] Example 5: Related procedures for specific reference signals in AI / ML models:

[0162] For reference signal configuration, different types of reference signals are used for different purposes. For example, CSI-RS is used for downlink channel state information (CSI) acquisition and reporting, and the UE needs to configure CSI-RS resources and the corresponding CSI reporting configuration; SRS, used for uplink CSI measurement, requires the configuration of corresponding SRS resources. Other reference signals include DMRS, PTRS, SSB, etc. If an AI / ML model is applied to the system, the reference signal related procedures can be further enhanced. The configuration of the reference signal related procedures in the AI / ML model may include one or more of the following information: • AI / ML model identifier: can be a logical AI / ML model identifier, a global AI / ML model identifier, and / or an identifier related to AI / ML models. • Reference signal information configuration: used to indicate the time-frequency resource allocation, duration, and activation method of the reference signal corresponding to the AI / ML model identifier. • CSI reporting information configuration: Used to configure the required CSI measurement information for the AI / ML model, including measurement events, channel state information (CSI), reporting period and other relevant information.

[0163] Figure 14 A block diagram illustrating an example system 700 for wireless communication according to an embodiment of the present invention is shown. The embodiments described herein can be implemented by any suitably configured hardware and / or software. Figure 14 The system 700 shown includes a radio frequency (RF) circuit 710, a baseband circuit 720, a processing unit 730, a memory / storage device 740, a display 750, a camera 760, a sensor 770, and an input / output (I / O) interface 780, and the components are interconnected as shown in the figure.

[0164] The processing unit 730 may include circuitry, such as, but not limited to, one or more single-core or multi-core processors. The processor may include any combination of general-purpose and special-purpose processors, such as a graphics processing unit (GPU) and an application processor (AP). The processor may be connected to a memory / storage device and configured to execute instructions stored therein to implement various applications and / or operating systems running on the system.

[0165] Radio frequency control functions may include, but are not limited to, signal modulation, encoding, decoding, and radio frequency conversion. In some embodiments, the baseband circuitry may support communication compatible with one or more radio technologies. For example, in some embodiments, the baseband circuitry may support 5G NR, LTE, Evolved Universal Terrestrial Radio Access Network (E-UTRAN) and / or other Wireless Metropolitan Area Network (WMAN), Wireless Local Area Network (WLAN), and Wireless Personal Area Network (WPAN) communication. In some embodiments, the baseband circuit is configured to support communication using multiple wireless protocols and may be referred to as a multimode baseband circuit. In various embodiments, the baseband circuit 720 may also include circuitry for processing signals that are not strictly within the baseband frequency range. For example, in some embodiments, the baseband circuit may include circuitry for processing intermediate frequency (IF) signals (between the baseband frequency and the radio frequency).

[0166] In various embodiments, system 700 may be a mobile computing device, such as, but not limited to, a laptop, tablet, netbook, ultrabook, smartphone, etc. In different embodiments, the system may contain more or fewer components and / or employ different architectures. Where appropriate, the methods described herein may be implemented by a computer program that may be stored on a storage medium, such as a non-transitory computer-readable storage medium.

[0167] An embodiment of the present invention is a combination of technologies / processes that can be used in 3GPP specifications to form a final product.

[0168] If software functional units are implemented independently and used and sold as products, they can be stored in computer-readable storage media. Based on this understanding, the technical solution proposed in this invention can be implemented in whole or in part as a software product; or some improvements required by conventional technologies can be implemented as software products. The software product is stored in a computer's storage medium and includes multiple instructions for causing a computing device (such as a personal computer, server, or network device) to execute all or part of the steps disclosed in the embodiments of this invention. The storage medium may be a USB flash drive, external hard drive, read-only memory (ROM), random access memory (RAM), floppy disk, or other media capable of storing program code.

[0169] This invention proposes a method for monitoring AI / ML models between a user equipment (UE) and a network (gNB, LMF, etc.), which can reduce signaling overhead and improve system performance. The method involves sending and receiving model monitoring requests and responses between the UE and the network, and updating the AI / ML model accordingly.

[0170] Embodiments of this invention describe a comprehensive monitoring method covering the entire AI / ML model lifecycle management (LCM) for monitoring model performance and quality. Existing research has not yet covered this aspect.

[0171] This invention enhances the AI / ML model in wireless communication systems.

[0172] While the invention has been described in conjunction with what are considered to be the most practical and preferred embodiments, it is to be understood that the invention is not limited to the disclosed embodiments, but is intended to cover various arrangements without departing from the broadest interpretation of the appended claims.

Claims

1. A model management method for artificial intelligence (AI) / machine learning (ML), executable in at least one wireless communication device, characterized in that, include: Based on the AI / ML model monitoring configuration, perform AI / ML model monitoring and / or lifecycle management (LCM) on one or more AI / ML models.

2. The model management method according to claim 1, characterized in that, It also includes: Report the model monitoring results.

3. The model management method according to claim 2, characterized in that, When the model monitoring results are sent as uplink messages, the model monitoring results are carried in the Medium Access Control (MAC) Control Unit (CE), Uplink Control Information (UCI), Radio Resource Control (RRC) messages, Physical Uplink Control Channel (PUCCH), Physical Uplink Shared Channel (PUSCH), or LTE Location Protocol (LPP) messages. Alternatively, when the model monitoring results are sent as downlink messages, the model monitoring results are carried in Radio Resource Control (RRC) messages, system information, Media Access Control (MAC) control unit (CE), Downlink Control Information (DCI), PDSCH, PDCCH, or LPP messages.

4. The model management method according to claim 2, characterized in that, The model monitoring results report includes one or more of the following: AI / ML model monitoring report configuration identifier, AI / ML model identifier, AI / ML model monitoring results, and triggering events for the model monitoring report.

5. The model management method according to claim 1, characterized in that, The AI / ML model monitoring configuration includes one or more of the following: AI / ML monitoring configuration identifier; AI / ML model identification; AI / ML model monitoring objects; AI / ML model monitoring metrics; AI / ML model monitoring report configuration; AI / ML model monitoring report configuration identifier; AI / ML model monitoring report types; AI / ML model monitoring thresholds; AI / ML model monitoring reference signal; AI / ML model monitoring time window; Trigger time; Number of model monitoring reports; Monitoring report interval; Hys, the hysteresis parameter; Activate time offset; UE identifier; Scene identification; Feature identifier; Community signage; Site identification; Area identification; and Data quality indicators.

6. The model management method according to claim 1, characterized in that, It also includes: Send or receive AI / ML model monitoring activation and deactivation signaling; The AI / ML model monitoring activation and deactivation signaling includes one or more of the following: AI / ML model monitoring threshold, AI / ML model monitoring object, activation time, and deactivation time.

7. The model management method according to claim 1, characterized in that, The AI / ML model monitoring configuration is sent in downlink messages including Radio Resource Control (RRC) messages, system information, Media Access Control (MAC) control unit (CE), Downlink Control Information (DCI), PDSCH, PDCCH, or LPP messages. Alternatively, the AI / ML model monitoring configuration is sent in an uplink message including MAC CE, UCI, Radio Resource Control (RRC) messages, Physical Uplink Control Channel (PUCCH), Physical Uplink Shared Channel (PUSCH), or LTE Location Protocol (LPP) messages.

8. The model management method according to claim 1, characterized in that, It also includes: Make decisions related to at least one AI / ML model, wherein the AI / ML model-related decisions include lifecycle management actions.

9. The model management method according to claim 8, characterized in that, The lifecycle management actions include AI / ML model activation, deactivation, model selection, model switching, or rollback.

10. The model management method according to claim 8, characterized in that, The AI / ML model's related decisions are triggered based on threshold methods, rule-based methods, ML methods, or expert system methods.

11. The model management method according to claim 8, characterized in that, The AI / ML model-related decisions are made by the user equipment (UE) or the network-side device (NW).

12. The model management method according to claim 10, characterized in that, The rule-based approach includes at least one of the following: Threshold-based decision rules; Decision rules based on candidate models; Decision rules based on AI / ML model monitoring methods; Predefined or pre-configured decision rules; and / or Default rules.

13. The model management method according to claim 12, characterized in that, In the aforementioned rule-based approach, different lifecycle management actions and / or different AI / ML model monitoring methods correspond to different thresholds.

14. The model management method according to claim 1, characterized in that, Model monitoring is triggered when the performance of at least one AI / ML model deteriorates and falls below the AI / ML model performance threshold; or when the performance of at least one AI / ML model improves and exceeds the AI / ML model performance threshold.

15. The model management method according to claim 1, characterized in that, The model monitoring is performed by the user equipment (UE) or the network-side device (NW).

16. The method according to claim 1, characterized in that, AI / ML model activation of the at least one AI / ML model is indicated by AI / ML model monitoring configuration or activation message, or controlled according to predefined rules.

17. The method according to claim 16, characterized in that, The activation message includes AI / ML model activation / deactivation information for AI / ML model activation, and the AI / ML model activation / deactivation information includes one or more of the following: AI / ML model identification, Activation / deactivation information Activation sequence, Deactivation timing, and Time window.

18. A user equipment, comprising: A processor is configured to invoke and run a computer program stored in memory to cause a device on which the processor is installed to perform the method of any one of claims 1 to 17.

19. A chip, comprising: The processor is configured to invoke and run a computer program stored in memory to cause a device on which the chip is mounted to perform the method of any one of claims 1-17.

20. A computer-readable storage medium storing a computer program for causing a computer to perform the method of any one of claims 1-17.

21. A computer program product comprising a computer program for causing a computer to perform the method of any one of claims 1-17.

22. A computer program for causing a computer to perform the method according to any one of claims 1-17.

23. A wireless communication system or an artificial intelligence (AI) / machine learning (ML) system, comprising: Base station; and User Equipment (UE); The base station is configured with AI / ML model monitoring configuration for at least one AI / ML model. One of the base stations or the UE performs model monitoring and / or lifecycle management (LCM) on the one or more AI / ML models according to the AI / ML model monitoring configuration; and The base station or one of the UEs makes a decision related to at least one of the one or more AI / ML models.

Images