Method, device and medium for communication

Abstract

Embodiments of the present disclosure provide a solution for an artificial intelligence / machine learning (AI / ML) model. If the first AI / ML model fails, the first entity determines at least one of: a first decision or a first action related to management of the AI / ML model. The first decision and / or the first action is determined based on first information related to a cause of a fault of the first AI / ML model and second information related to a state of the second AI / ML model. In this manner, an appropriate action may be selected to handle the fault. In addition, the gain of the AI / ML operation can be quickly recovered.

Description

Technical Field

[0001] Example embodiments of the present disclosure generally relate to the field of communication technologies, and more particularly, to methods, devices, and media for applying artificial intelligence / machine learning (AI / ML) models to air interfaces. Background Art

[0002] Several technologies have been proposed to improve communication performance. For example, communication devices can adopt artificial intelligence / machine learning (AI / ML) models to improve communication quality. The AI / ML models can be applied to different scenarios to achieve better performance. Therefore, in order to ensure meeting communication performance, how to ensure the accuracy of the output of the AI / ML models is worthy of research. Summary of the Invention

[0003] Generally, embodiments of the present disclosure provide methods, devices, and computer storage media for applying AI / ML models to air interfaces.

[0004] In a first aspect, a communication method is provided. The method includes: determining, at a first entity, at least one of the following based on a determination of a failure of a first artificial intelligence / machine learning (AI / ML) model: a first decision or a first action related to AI / ML model management, and wherein at least one of the first decision or the first action is determined based on: first information related to a cause of the failure of the first AI / ML model, and second information related to a state of a second AI / ML model.

[0005] In a second aspect, a communication method is provided. The method includes: determining, at a second entity, a model monitoring method for a first AI / ML model; monitoring the first AI / ML model based on the model monitoring method; and transmitting, to a first device, one of the following based on a determination of a failure of the first AI / ML model: first information related to a cause of the failure of the first AI / ML model, or fourth information about the model monitoring method for the first AI / ML model.

[0006] In a third aspect, a communication device is provided. The communication device includes: at least one processor; and at least one memory coupled to the at least one processor and storing instructions thereon, which when executed by the at least one processor, cause the communication device to perform the method according to the first aspect.

[0007] In a fourth aspect, a communication device is provided. The communication device includes: at least one processor; and at least one memory coupled to the at least one processor and storing instructions thereon, which when executed by the at least one processor, cause the communication device to perform the method according to the second aspect.

[0008] In a fifth aspect, there is provided a computer-readable medium having instructions stored thereon that, when executed on at least one processor, cause the at least one processor to perform the method according to the first aspect or the second aspect.

[0009] Other features of the present disclosure will become readily apparent from the following description. BRIEF DESCRIPTION OF THE DRAWINGS

[0010] The above and other objects, features and advantages of the present disclosure will become more apparent by describing some example embodiments of the present disclosure in more detail in the accompanying drawings, in which:

[0011] Figure 1 illustrates a framework of an AI / ML model according to some solutions;

[0012] Figure 2 illustrates an example communication environment in which example embodiments of the present disclosure may be implemented;

[0013] Figure 3 shows a schematic diagram of entities associated with an AI / ML model;

[0014] Figure 4 illustrates a signaling flow of AI / ML management according to some embodiments of the present disclosure;

[0015] Figure 5 shows an example of the state and corresponding actions of a second AI / ML model;

[0016] Figure 6A and Figure 6B respectively illustrate schematic diagrams of ways in which multiple first actions are taken;

[0017] Figure 7 illustrates a flowchart of a method implemented at a first entity according to some example embodiments of the present disclosure;

[0018] Figure 8 illustrates a flowchart of a method implemented at a second entity according to some example embodiments of the present disclosure; and

[0019] Figure 9 illustrates a simplified block diagram of an apparatus suitable for implementing example embodiments of the present disclosure.

[0020] In all the drawings, the same or similar reference numerals denote the same or similar elements. DETAILED DESCRIPTION

[0021] The principles of the present disclosure will now be described with reference to some example embodiments. It should be understood that these embodiments are described for illustrative purposes only and help those skilled in the art to understand and implement the present disclosure, without implying any limitation on the scope of the present disclosure. The embodiments described herein can be implemented in various ways other than those described below.

[0022] In the following description and claims, unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this disclosure pertains.

[0023] As used herein, the term "terminal device" refers to any device having wireless or wired communication capabilities. Examples of terminal devices include, but are not limited to, user equipment (UE), personal computers, desktop computers, mobile phones, cellular phones, smart phones, personal digital assistants (PDA), portable computers, tablets, wearable devices, Internet of Things (IoT) devices, ultra-reliable and low-latency communication (URLLC) devices, Internet of Everything (IoE) devices, machine type communication (MTC) devices, devices on vehicles for V2X communication (where X represents pedestrians, vehicles, or infrastructure / networks), devices for integrated access and backhaul (IAB), spaceborne or airborne vehicles in non-terrestrial networks (NTN) including satellites and high-altitude platforms (HAP) containing unmanned aerial vehicle systems (UAS), extended reality (XR) devices including different types of realities such as augmented reality (AR), mixed reality (MR), and virtual reality (VR), unmanned aerial vehicles (UAV), commonly known as drones (which are aircraft without any human pilot), devices on high-speed trains (HST), or image capture devices such as digital cameras, sensors, gaming devices, music storage and playback appliances, or Internet appliances that enable wireless or wired Internet access and browsing, etc. The "terminal device" may also have "multicast / broadcast" features to support public safety and critical missions, V2X applications, transparent IPv4 / IPv6 multicast delivery, IPTV, smart TV, radio services, software delivery over the air, group communication, and IoT applications. It may also incorporate one or more subscriber identity modules (SIM), referred to as multi-SIM. The term "terminal device" may be used interchangeably with UE, mobile station, subscriber station, mobile terminal, user terminal, or wireless device.

[0024] The term "network device" refers to a device that can provide or host a cell or coverage area where terminal devices can communicate. Examples of network devices include, but are not limited to, Node B (Node B or NB), evolved Node B (eNode B or eNB), next-generation Node B (gNB), transmission and reception point (TRP), remote radio unit (RRU), radio head (RH), remote radio head (RRH), IAB node, low-power node (such as femto node), pico node, reconfigurable intelligent surface (RIS), etc.

[0025] The terminal device or network device may have artificial intelligence (AI) or machine learning capabilities. It typically includes a model that has been trained from a large amount of collected data for a specific function and can be used to predict some information.

[0026] The terminal or network device can operate in several frequency ranges, for example, FR1 (e.g., 450 MHz to 6000 MHz), FR2 (e.g., 24.25 GHz to 52.6 GHz), frequency bands greater than 100 GHz, and terahertz (THz). It can also operate on licensed / unlicensed / shared spectrum. The terminal device can have more than one connection with a network device in a multi-radio dual connectivity (MR-DC) application scenario. The terminal device or network device can operate in full-duplex, flexible-duplex, and cross-duplex modes.

[0027] Embodiments of the present disclosure may be implemented in test equipment, such as signal generators, signal analyzers, spectrum analyzers, network analyzers, test terminal devices, test network devices, channel simulators. In some embodiments, a terminal device may be connected to a first network device and a second network device. One of the first network device and the second network device may be a master node, and the other may be a slave node. The first network device and the second network device may use different radio access technologies (RATs). In some embodiments, the first network device may be a first RAT device, and the second network device may be a second RAT device. In some embodiments, the first RAT device is an eNB, and the second RAT device is a gNB. Information related to different RATs may be transmitted to the terminal device from at least one of the first network device or the second network device. In some embodiments, a first piece of information may be sent from the first network device to the terminal device, and a second piece of information may be transmitted to the terminal device from the second network device directly or via the first network device. In some embodiments, information related to a configuration for the terminal device configured by the second network device may be transmitted from the second network device via the first network device. Information related to a reconfiguration for the terminal device configured by the second network device may be sent from the second network device to the terminal device directly or via the first network device.

[0028] As used herein, the singular forms "a", "an", and "the" are also intended to include the plural forms unless the context clearly dictates otherwise. The term "comprising" and its variants are to be understood as open terms meaning "including but not limited to". The term "based on" will be understood to mean "at least partially based on". The terms "one embodiment" and "an embodiment" will be understood to mean "at least one embodiment". The term "another embodiment" will be understood to mean "at least one other embodiment". The terms "first", "second", etc. may refer to different or the same objects. Other definitions, whether explicit or implicit, may be included below.

[0029] In some examples, a value, process, or device is referred to as "optimal", "lowest", "highest", "minimum", "maximum", etc. It should be understood that such descriptions are intended to indicate a selection among many functional alternatives in use, and such a selection does not necessarily need to be better, smaller, higher, or otherwise superior to other alternatives.

[0030] As used herein, the terms "resource", "transmission resource", "uplink resource", or "downlink resource" may refer to any resource used to perform communication, such as a resource in the time domain, a resource in the frequency domain, a resource in the spatial domain, a resource in the code domain, or any other resource enabling communication, etc. Hereinafter, unless explicitly stated, resources in the frequency domain and the time domain will be used as examples of transmission resources for describing some example embodiments of the present disclosure. Note that the example embodiments of the present disclosure are equally applicable to other resources in other domains.

[0031] As described above, AI / ML technologies have been proposed. Figure 1 A schematic diagram of the AL / ML framework 100 is shown. As Figure 1 shown, the AI / ML framework 100 may include a data collection module 110, a model training module 120, a model inference model 130, and an actuator 140. The data collection module 110 may implement the function of providing input data to the model training and model inference modules. No AI / ML algorithm-specific data preparation (e.g., data preprocessing and cleaning, formatting, and transformation) is performed in the data collection module 110. Examples of input data may include measurements from a UE or different network entities, feedback from the actuator, outputs from the AI / ML model. Training data may refer to the data required as input to the AI / ML model training module 120. Inference data may refer to the data input for the AI / ML model inference module 130.

[0032] The model training module 120 may implement the functions of performing AI / ML model training, validation, and testing, which may generate model performance metrics as part of the model testing procedure. If needed, the model training module is also responsible for data preparation (e.g., data preprocessing and cleaning, formatting, and transformation) based on the training data delivered by the data collection function. Model deployment / update may be used to initially deploy the trained, validated, and tested AI / ML model to the model inference module 130, or deliver the updated model to the model inference module 130.

[0033] The model inference module 130 may implement the function of providing AI / ML model inference output (e.g., prediction or decision). The model inference model 130 may provide model performance feedback to the model training module 120 when applicable. If needed, the model inference module 130 is also responsible for data preparation (e.g., data preprocessing and cleaning, formatting, and transformation) based on the inference data delivered by the data collection module 110. The output of the model inference module 130 may refer to the inference output of the AI / ML model generated by the model inference function. Model performance feedback may be used to monitor the performance of the AI / ML model when available.

[0034] The actuator 140 can implement the function of receiving the output from the model inference module 130 and triggering or executing the corresponding actions. The actuator 140 can trigger actions for other entities or itself.

[0035] In addition, the AI / ML model can be applied to different scenarios to achieve better performance. For example, the terminal device can perform beam management on the AI / ML model. In this case, the terminal device can measure a part of the candidate beam pairs and use AI or ML to estimate the quality for all candidate beam pairs. Additionally, the terminal device can perform CSI feedback based on the AI / ML model. In this case, the original CSI information can be compressed by an AI encoder located in the terminal device and restored by an AI decoder located in the network device. The AI / ML model can also be used for reference signal (RS) overhead reduction. For example, the terminal device can use a new RS pattern, e.g., a lower density of DMRS, fewer CSI-RS ports. Lifecycle management (LCM) is one of the core parts of AI / ML related research. Within LCM, model monitoring is the process of monitoring the inference performance of the AI / ML model.

[0036] In some cases, more than one factor may cause model failure. Therefore, different actions may be required based on the factor. However, this is not related signaling, and model monitoring can be based only on the implementation. Additionally, in some cases, the AI / ML model to be switched to may have different states and cannot be easily applied. In this case, different actions may also be required.

[0037] To solve at least a part of the above problems, embodiments of the present disclosure provide a solution for the AI / ML model. If the first AI / ML model fails, the first entity determines at least one of a first decision or a first action related to artificial intelligence / machine learning (AI / ML) model management. The first decision and / or the first action is determined based on: first information related to the cause of the failure of the first AI / ML model, and second information related to the state of the second AI / ML model. In other words, information about why the performance of an AI / ML model is not good and information about what the state of the AI / ML model to be used is can be exchanged between entities. In this case, corresponding actions based on the exchanged information can be taken to solve the AI / ML model failure. In this way, appropriate actions can be selected to handle the failure. Additionally, the gain of AI / ML operations can be quickly restored.

[0038] In the context of the present application, the term "AI / ML model" may be interchangeable with the term "model". The terms "AI / ML model failure", "model failure", "failure", and "incorrect inference" may be used interchangeably. The terms "AI / ML model success", "model success", "success", and "correct inference" may be used interchangeably.

[0039] In the context of the present application, the term "data collection" may refer to the process of collecting data by a network node, a management entity, or a UE for the purposes of AI / ML model training, data analysis, and inference. The term "AI / ML model" as used herein may refer to a data-driven algorithm that applies AI / ML techniques to generate a set of outputs based on a set of inputs. The term "AI / ML model training" as used herein may refer to the process of training an AI / ML model in a data-driven manner (by learning the input / output relationship) and obtaining a trained AI / ML model for inference. The term "AI / ML model inference" as used herein may refer to the process of using a trained AI / ML model to generate a set of outputs based on a set of inputs.

[0040] The term "AI / ML model validation" as used herein may refer to a sub-process of training to evaluate the quality of an AI / ML model using a dataset different from the dataset used for model training, which helps to select model parameters that generalize beyond the dataset used for model training. The term "AI / ML model testing" as used herein may refer to a sub-process of training to evaluate the performance of the final AI / ML model using a dataset different from the datasets used for model training and validation. Different from AI / ML model validation, testing does not assume subsequent tuning of the model.

[0041] The term "UE-side (AI / ML) model" as used herein may refer to an AI / ML model that performs its inference entirely at the UE. The term "network-side (AI / ML) model" as used herein may refer to an AI / ML model that performs its inference entirely at the network. The term "single-side (AI / ML) model" as used herein may refer to a UE-side (AI / ML) model or a network-side (AI / ML) model. The term "two-side (AI / ML) model" as used herein may refer to a paired (multiple) AI / ML model on which joint inference is performed, where joint inference includes AI / ML inference in which the inference is jointly performed across the UE and the network, i.e., the first part of the inference is first performed by the UE and then the remaining part is performed by the gNB, and vice versa.

[0042] The term "AI / ML model transfer" as used herein may refer to the delivery of an AI / ML model over the air interface, whether it is the parameters of a model structure known at the receiving end or a new model with parameters. The delivery may include the full model or a partial model. The term "model download" as used herein may refer to the transfer of a model from the network to the UE. The term "model upload" as used herein may refer to the transfer of a model from the UE to the network. The term "federated learning / federated training" as used herein may refer to a machine learning technique for training an AI / ML model across multiple decentralized edge nodes (e.g., UEs, gNBs), where each decentralized edge node performs local model training using local data samples. This technique requires multiple interactions of the model but does not require the exchange of local data samples.

[0043] The term "offline field data" as used herein may refer to data collected from the field and used for offline training of an AI / ML model. The term "online field data" as used herein may refer to data collected from the field and used for online training of an AI / ML model. The term "model monitoring" as used herein may refer to the process of monitoring the inference performance of an AI / ML model.

[0044] The term "supervised learning" as used herein may refer to the process of training a model from inputs and their corresponding labels. The term "unsupervised learning" as used herein may refer to the process of training a model without labeled data. The term "semi-supervised learning" as used herein may refer to the process of training a model using a mixture of labeled and unlabeled data. The term "reinforcement learning (RL)" as used herein may refer to the process of training an AI / ML model in an environment where the model interacts with inputs (also called states) and feedback signals (also called rewards) generated by the outputs of the model (also called actions).

[0045] The term "model activation" as used herein may refer to enabling an AI / ML model for a specific function. The term "model deactivation" as used herein may refer to disabling an AI / ML model for a specific function. The term "model switching" as used herein may refer to deactivating the currently active AI / ML model for a specific function and activating a different AI / ML model.

[0046] The term "model failure" as used herein may refer to a negative result of model monitoring. The term "model failure instance" as used herein may refer to an incorrect inference. The term "model failure instance indication" as used herein may refer to the indication of a model failure instance from a lower layer to a higher layer.

[0047] As used herein, the term "model success" may refer to a positive result of model monitoring. As used herein, the term "model success instance" may refer to a correct inference. As used herein, the term "model recovery" may refer to achieving model success again in response to a model failure. As used herein, the term "model management" may refer to a more general term that includes one or more of the following functions / processes: model activation, deactivation, selection, switching, fallback, and update (including retraining). As used herein, the term "model registration" may refer to the process of using an identifier to notify a network or a UE of the existence of an AI / ML model and the model description information of the AI / ML model so that the network can perform LCM. The model description information may include model format, model function, model applicability scenario, configuration, information about model input, information about model output, information about auxiliary information, etc. The model identifier may include model type, use case, vendor ID, version number, etc.

[0048] As used herein, the term "entity" may refer to a device (e.g., a network device or a terminal device). Different functions may be located on the same device, or different processes may be executed on the same device. As used herein, the term "entity" may refer to a model, block, or logical component responsible for a specific function / process.

[0049] The principles and implementation manners of the present disclosure will be described in detail below with reference to the accompanying drawings.

[0050] Figure 2 FIG. is a schematic diagram illustrating an example communication environment 200 in which example embodiments of the present disclosure may be implemented. In the communication environment 200, multiple communication devices including a terminal device 210 and a network device 220 may communicate with each other.

[0051] In Figure 2 example, the terminal device 210 may be a UE, and the network device 220 may be a base station serving the UE. The service area of the network device 220 may be referred to as a cell 202.

[0052] It should be understood that Figure 2 the number of devices and their connections shown in

[0053] In the following, for illustrative purposes, some example embodiments are described, where the terminal device 210 operates as a UE and the network device 220 operates as a base station. However, in some example embodiments, the operations described in connection with the terminal device may be implemented at the network device or other devices, and the operations described in connection with the network device may be implemented at the terminal device or other devices.

[0054] In some example embodiments, the link from the network device 220 to the terminal device 210 is referred to as the downlink (DL), while the link from the terminal device 210 to the network device 220 is referred to as the uplink (UL). In the DL, the network device 220 is the transmitting (TX) device (or transmitter), and the terminal device 210 is the receiving (RX) device (or receiver). In the UL, the terminal device 210 is the TX device (or transmitter), and the network device 220 is the RX device (or receiver).

[0055] The communication in the communication environment 200 may conform to any suitable standard, including but not limited to Global System for Mobile Communications (GSM), Long Term Evolution (LTE), LTE Evolution, Advanced LTE (LTE-A), New Radio (NR), Wideband Code Division Multiple Access (WCDMA), Code Division Multiple Access (CDMA), GSM EDGE Radio Access Network (GERAN), Machine Type Communication (MTC), etc. Embodiments of the present disclosure may be implemented according to any generation of communication protocols known currently or developed in the future. Examples of communication protocols include but are not limited to the first generation (1G), second generation (2G), 2.5G, 2.75G, third generation (3G), fourth generation (4G), 4.5G, fifth generation (5G) communication protocols, 5.5G, advanced 5G networks, or sixth generation (6G) networks.

[0056] Figure 3 A schematic diagram 300 of entities associated with an AI / ML model is shown. Information can be exchanged between entity 310, entity 320, and entity 330. In some embodiments, entity 310 may be responsible for taking actions other than model inference based on the performance model monitoring results. For example, entity 310 may be responsible for actions in response to an AI / ML model failure. In some embodiments, entity 310 may be responsible for model inference. Alternatively, or additionally, entity 310 may be responsible for one or more of the following: data collection, model training, measurement, configuration, model management. In some embodiments, entity 310 may be responsible for one or more of the following: collecting model monitoring results, and / or making decisions based on the model monitoring results.

[0057] In some embodiments, entity 320 may be responsible for model monitoring. Alternatively, or additionally, entity 320 may be responsible for making decisions based on the model monitoring results and providing the decisions / results to other entities.

[0058] In some embodiments, entity 330 may be responsible for model inference. Alternatively, or additionally, entity 330 may be responsible for at least one of the following: data collection, model training, measurement, configuration, or model management. In some other embodiments, entity 330 may be responsible for collecting model monitoring results, and / or for making decisions based on the model monitoring results.

[0059] In some embodiments, entity 310 may be implemented on a terminal device, or may be a terminal device, such as Figure 2 the terminal device 210 shown. In some other embodiments, entity 310 may be implemented on a network device, or may be a network device, such as Figure 2 the network device 220 shown. Entity 320 may be implemented on a terminal device, or may be a terminal device, such as Figure 2 the terminal device 210 shown. Alternatively, entity 320 may be implemented on a network device, or may be a network device, such as Figure 2 the network device 220 shown. In some embodiments, entity 330 may be implemented on a terminal device, or may be a terminal device, such as Figure 2 the terminal device 210 shown. Alternatively, entity 330 may be implemented on a network device, or may be a network device, such as Figure 2 the network device 220 shown. In some other embodiments, one or more of entity 310, entity 320, or entity 330 may be implemented on a third-party device. For example, the third-party device may refer to a server that performs AI / ML operations or provides AI / ML capabilities to network nodes including NW devices and terminal devices. Alternatively, the third-party device may refer to a location server in AI / ML for location use cases.

[0060] In some embodiments, entity 310, entity 320, and entity 330 may be located on different devices. Alternatively, entity 310, entity 320, and entity 330 may be located on the same device. Figure 3 Other entities that are omitted for clarity may also be included.

[0061] Referring to Figure 4 which illustrates a signaling flow 400 for applying an AI / ML model according to some embodiments of the present disclosure. For the purpose of discussion, the signaling flow 400 will be discussed with reference to Figure 3 (e.g., by using entity 310, entity 320, entity 330). Note that the signaling flow 400 is merely an example and not a limitation. Figure 4 The operations shown may be performed in a different order.

[0062] Entity 310 determines (4020) a model monitoring method for a first AI / ML model. In some embodiments, the first AI / ML model may be implemented on the terminal device 110. Alternatively, the first AI / ML model may be implemented on the network device 120. In other embodiments, the first AI / ML model may be implemented on a third-party device.

[0063] In some embodiments, entity 330 may transmit (4010) a configuration associated with the model monitoring method to entity 320. In this case, entity 320 may determine the model monitoring method based on the configuration. In this way, the (multiple) model monitoring methods may be aligned at least at the entity performing model inference and the entity performing model monitoring. In addition, the complexity caused by exhaustive monitoring through too many methods may be avoided.

[0064] Signaling may be required to configure model monitoring based on at least one of the following: different AI / ML model structures, whether the monitoring entity is the same as the inference entity, or the NW-UE cooperation level, etc. For example, if model inference and model monitoring are performed at the same device, information / configuration related to the model monitoring method may be exchanged between different entities without signaling.

[0065] In some embodiments, auxiliary information may be required. For example, if entity 320 and entity 330 are at the network device 220, which means the network device 220 performs both model inference and model monitoring, the network device 220 may require the terminal device 210 to provide measurement results to the network device 220. Alternatively, if entity 320 and entity 330 are at the terminal device 210, which means the terminal device 210 performs both model inference and model monitoring, the terminal device 210 may require the network device 220 to provide measurement resources to the terminal device 210.

[0066] In some embodiments, if entity 320 is at the terminal device 210 and entity 330 is at the network device 220, the information / configuration related to the model monitoring method may be transmitted via one of the following: radio resource control (RRC) configuration, medium access control (MAC) control element (CE), or downlink control information (DCI). Alternatively, if entity 320 is at the network device 220 and entity 330 is at the terminal device 210, the information / configuration related to the model monitoring method may be transmitted via one of the following: RRC signaling, MAC CE, or uplink control information (UCI). In some other embodiments, if one of entity 320 and entity 330 is a third-party device, the information / configuration related to the model monitoring method may be transmitted via 3GPP signaling or without 3GPP signaling. Table 1 below shows an example of signaling exchange related to the information / configuration related to the model monitoring method. Table 1

[0067] In some embodiments, the information / configuration related to the model monitoring method may be the first configuration of the model monitoring method. For example, the first configuration may include a first set of monitoring metrics related to inference accuracy. The first configuration may be configured for or determined by entity 320. Inference accuracy may refer to the extent to which a given monitoring metric / method reflects the performance of the model and the system. As an example, the first set of monitoring metrics may include the metric(s) related to the intermediate key performance indicator (KPI).

[0068] In some embodiments, the inference accuracy may be the accuracy as a result of comparison with the ground truth. The comparison result or the configuration used to obtain the ground truth may additionally be provided to entity 320. For example, the comparison result may vary for each use case. As an example, for the channel state information (CSI) compression scenario, the comparison result may be at least one of the following: generalized cosine similarity (CGS), square of GCS (SCGS), minimum mean square error (MMSE), etc. In some other embodiments, for the beam prediction scenario, the beam prediction accuracy including Top-1 (%) may represent the percentage that "the Top-1 ideal auxiliary beam is the Top-1 predicted beam", Top-K / 1 (%) may represent the percentage that "the Top-1 ideal auxiliary beam is one of the Top-K predicted beams", and Top-1 / K (%) may represent the percentage that "the Top-1 predicted beam is one of the Top-K ideal auxiliary beams". In some other embodiments, for the positioning scenario, the accuracy may be at least one of the following: line-of-sight (LOS) classification accuracy, timing estimation accuracy, angle estimation accuracy, etc. In some embodiments, at least one of the following may be provided to entity 320: a comparison threshold for the first AI / ML model, a timer, or a counter.

[0069] In some embodiments, the inference accuracy may be the confidence level of the prediction. The confidence level may be part of the output of the AI / ML model and may additionally be provided to entity 320. For example, the confidence level may be a value between 0 and 1. In some embodiments, at least one of the following may be provided to entity 320: a confidence level threshold for the first AI / ML model, a timer, a counter.

[0070] Alternatively, or additionally, the first configuration may include a second set of monitoring metrics related to system performance. For example, (multiple) monitoring KPIs of system performance may be additionally provided to the second entity. The second set of monitoring metrics may include one or more of the following: throughput, spectral efficiency, assumed block error rate (BLER), BLER, reference signal received power (RSRP), signal-to-noise ratio (SNR), signal-to-interference plus noise ratio (SINR), modulation and coding scheme (MCS), data rate, or overhead. In some embodiments, at least one of the following may be provided to entity 320: a system performance KPI threshold of the first AI / ML model, a timer, a counter.

[0071] In some embodiments, the first configuration may include a third set of monitoring metrics related to application conditions. The application conditions may include one or more of the following: scenario conditions, configuration conditions, regional conditions, sector conditions, site conditions, or cell conditions. The scenario conditions may include at least one of the following: LOS / NLOS, indoor / outdoor, high / medium / low mobility, frequency range 1 (FR1) frequency range, FR2 frequency range. In some embodiments, the configuration conditions may include one or more of the following: configuration identifier, number of beams, or number of resources. The regional conditions may include a regional identifier, the sector conditions may include a sector identifier, the site conditions may include a site identifier, and the cell conditions may include a cell identifier. In some embodiments, at least one of the following may be provided to entity 320: a suitable application condition of the first AI / ML model, a timer, a counter. Entity 320 may determine whether the current application condition is the same as the notified suitable application condition based on a threshold. Alternatively, whether the current application condition is the same as the notified suitable application condition may not be based on a threshold.

[0072] In some embodiments, the first configuration may include a fourth set of monitoring metrics related to power consumption. For example, the power consumption may be inference power consumption. In this case, the fourth set of monitoring metrics may be related to the inference power consumption. In some embodiments, at least one of the following may be provided to entity 320: a power consumption threshold of the first AI / ML model, a timer, a counter.

[0073] Alternatively, the first configuration may include a fifth set of monitoring metrics related to complexity. In some embodiments, complexity may refer to the computational cost for model monitoring. In this case, the fifth set of monitoring metrics may relate to the computational cost for model monitoring. Alternatively, or additionally, complexity may refer to the memory cost for model monitoring. For example, the fifth set of monitoring metrics may be related to the memory cost for model monitoring. In some other embodiments, complexity may refer to inference complexity. For example, the fifth set of monitoring metrics may be related to inference complexity. In some embodiments, the fifth set of monitoring metrics may be measured in one of the following: ML TOP, ML FLOP, or MAC. Alternatively, the fifth set of monitoring metrics may be measured in a logic unit (e.g., an AI processing unit). In some embodiments, the following may be provided to entity 320: at least one of a complexity threshold of the first AI / ML model, a timer, and a counter.

[0074] In some embodiments, the first configuration may include a sixth set of monitoring metrics related to storage. Storage may refer to the storage of the first AI / ML model. In some embodiments, the sixth set of monitoring metrics may be related to the storage of the first AI / ML model. Alternatively, or additionally, storage may refer to the storage of input / output data. In this case, the sixth set of monitoring metrics may be related to the storage of input / output data. In some embodiments, the following may be provided to entity 320: at least one of a storage threshold of the first AI / ML model, a timer, and a counter.

[0075] In some other embodiments, the first configuration is a seventh set of monitoring metrics related to latency. In some embodiments, latency may refer to inference latency. Given the purpose of model monitoring, latency may refer to the timeliness of the monitoring result from model failure to action.

[0076] Alternatively, the information / configuration related to the model monitoring method may include a second configuration for monitoring the first AI / ML model. In other words, the second configuration may refer to the general configuration for model monitoring.

[0077] In some embodiments, the second configuration may include a set of thresholds related to the monitoring metrics for the first AI / ML model. For example, the set of thresholds may include (a) threshold(s) for determining that the model is performing poorly. Alternatively, or additionally, the set of thresholds may include (a) threshold(s) for evaluating model failure.

[0078] For different model monitoring methods, the set of thresholds can be different. For example, depending on the model monitoring method, the set of thresholds can be one or more of the following: a first set of monitoring metrics related to inference accuracy, a second set of monitoring metrics related to system performance, a third set of monitoring metrics related to application conditions, a fourth set of monitoring metrics related to power consumption, a fifth set of monitoring metrics related to complexity, a sixth set of monitoring metrics related to storage, or a seventh set of monitoring metrics related to latency.

[0079] In some embodiments, the set of thresholds can be different depending on different actions for improving the performance of the model. For example, in some embodiments, the set of thresholds can include a threshold for falling back to non-AI operations. Alternatively, or additionally, the set of thresholds can include a threshold for falling back to a default AI / ML model. The set of thresholds can include one or more of the following: a threshold for enhancing data processing of the current AI / ML model, a threshold for switching to a different AI / ML model (e.g., switching to a second AI / ML model), a threshold for training a new AI / ML model (e.g., training a second AI / ML model). Alternatively, the set of thresholds can be an offset determined by comparing a first AI / ML model with a different AI / ML model.

[0080] In some other embodiments, the second configuration can include a timer. For example, the timer can start based on the occurrence of an AI / ML model failure instance. The timer can be configured to evaluate the performance of the AI / ML model within a specific duration. For example, if a failure instance of the first AI / ML model occurs, the timer can start or restart. In some embodiments, the failure instance of the first AI / ML model can be indicated from a lower layer.

[0081] Alternatively, the second configuration can include a counter. For example, the counter can be used to count the number of AI / ML model failure instances. The counter can be configured to count the number of AI / ML model failure instances before declaring the AI / ML model as failed. Alternatively, the counter can count the consecutive number of AI / ML model failure instances before declaring the AI / ML model as failed. In some other embodiments, the counter can count the ratio of AI model failure instances to AI / ML model successful instances. For example, if a failure instance of the first AI / ML model occurs, the counter can start or restart. For each subsequent AI / ML model failure instance, the value of the counter can be incremented by 1. If the value of the counter exceeds a maximum value, the AI / ML model can be declared as failed, and a corresponding action can be requested.

[0082] In some embodiments, the second configuration may indicate the start of monitoring the first AI / ML model. Alternatively, or additionally, the second configuration may indicate the suspension of monitoring the first AI / ML model. The second configuration may also indicate the end of monitoring the first AI / ML model. In some embodiments, the second configuration may indicate the control of monitoring the first AI / ML model.

[0083] In some embodiments, the second configuration may indicate the periodicity of monitoring the first AI / ML model. Alternatively, or additionally, the second configuration may indicate the duty cycle of monitoring the first AI / ML model. In some other embodiments, the second configuration may indicate the duration of monitoring the first AI / ML model. In some embodiments, the second configuration may indicate the time offset configuration of monitoring the first AI / ML model.

[0084] In some embodiments, the second configuration may indicate the capabilities for the model monitoring method. For example, the capabilities may indicate the (multiple) supported model monitoring methods that the entity 320 is capable of performing.

[0085] In some embodiments, multiple first AI / ML models may be monitored. In such a case, in some embodiments, the information / configuration related to the model monitoring method may include the model identifiers of the multiple first AI / ML models. Alternatively, the information / configuration related to the model monitoring method may include the model group identifiers of the multiple first AI / ML models. In some embodiments, the above-mentioned first configuration may be configured for each of the multiple first AI / ML models. Alternatively, the above-mentioned second configuration may be configured for each of the multiple first AI / ML models. In other words, for each AI / ML model or each group of AI / ML models, at least one of the following may be configured: the configuration of the model monitoring method, conditions, thresholds, timers, counters, start / suspension / end / control signaling for model monitoring, periodicity / duty cycle / duration / time offset configuration of model monitoring, and the capability signaling for indicating whether the entity 320 is capable of performing the model monitoring method. In some embodiments, at least one of the following capabilities may also be indicated: the number of supported AI / ML models that can be monitored, whether the entity 320 can monitor more than one AI / ML model, or whether a default AI / ML model is supported.

[0086] The entity 320 monitors (4030) the first AI / ML model based on the model monitoring method. For example, the entity 320 may monitor the metrics of the first AI / ML model associated with the model monitoring method. In some embodiments, if the entity 330 is at the network device 220 and the entity 320 is at the terminal device 210, the entity 320 may perform the AI / ML model monitoring of the NW-side AI / ML mode based on the information / configuration related to the model monitoring method.

[0087] If a failure of the first AI / ML model occurs, entity 320 transmits (4040) first information related to the cause of the failure of the first AI / ML model to entity 310. Alternatively, if a failure of the first AI / ML model occurs, entity 320 transmits (4040) fourth information regarding a model monitoring method for the first AI / ML model to entity 310. In this way, since the reason for the poor performance of the first AI / ML model is indicated, the corresponding entity can select the correct action to quickly recover the gain of AI / ML operations.

[0088] In some embodiments, as described above, the first information related to the cause of the failure of the first AI / ML model may be transmitted to entity 310. In this case, entity 310 may determine (4050) the cause of the failure of the first AI / ML model based on the first information.

[0089] In some embodiments, the first information may indicate a failure of the first AI / ML model caused by an inference accuracy problem. For example, a failure of the first AI / ML model caused by an inference accuracy problem may include an inference accuracy below an inference accuracy threshold. Alternatively, a failure of the first AI / ML model caused by an inference accuracy problem may include a confidence level below a confidence level threshold. In some embodiments, the first information may explicitly indicate a low inference accuracy or a low confidence level. Alternatively, the first information may explicitly indicate the value of the accuracy. For example, the value of the accuracy may include one or more of the following: a comparison result with the ground truth, a value of the confidence level, or a value of other accuracy KPIs described above. In some other embodiments, the first information may implicitly indicate a failure of the first AI / ML model caused by an inference accuracy problem. For example, the implicit indication may include one or more of the following: no report, missing configuration, special value, reserved point, or out-of-range value.

[0090] Alternatively, or additionally, the first information may indicate a failure of the first AI / ML model caused by a system performance problem. In some embodiments, a failure of the first AI / ML model caused by a system performance problem may include a degradation of the system performance. For example, the first information may explicitly indicate a degradation of the system performance. Alternatively, the first information may explicitly indicate the value of the system performance KPI described above. In some other embodiments, the first information may implicitly indicate a failure of the first AI / ML model caused by a system performance problem. For example, the implicit indication may include at least one of the following: a non-acknowledgment (NACK) for data transmission or a configuration loss.

[0091] In some other embodiments, the first information may indicate a failure of the first AI / ML model caused by a data distribution problem. For example, a failure of the first AI / ML model caused by a data distribution problem may include out-of-distribution of input data and / or out-of-distribution of output data. Alternatively, or additionally, a failure of the first AI / ML model caused by a data distribution problem may include input data distribution drift and / or output data distribution drift. In some embodiments, the first information may explicitly indicate out-of-distribution of input data and / or out-of-distribution of output data. Alternatively, the first information may explicitly indicate input data distribution drift and / or output data distribution drift. Alternatively, or additionally, the first information may explicitly indicate one or more of the following: the distribution of input data and / or output data, data sharing for delivery parts (i.e., abnormal data), all input / output data, or the likelihood between the distribution of input data and / or output data for model inference and the distribution of input data and / or output data for model training. In some other embodiments, the first information may implicitly indicate a failure of the first AI / ML model caused by a data distribution problem. For example, the implicit indication may include a request for enhanced data processing. Alternatively, the implicit indication may include a configuration for enhanced data processing.

[0092] In some embodiments, the first information may indicate a failure of the first AI / ML model caused by an application condition problem. For example, a failure of the first AI / ML model caused by an application condition problem may include mismatched application conditions. For example, the first information may explicitly indicate mismatched application conditions for the first AI / ML model. Alternatively, the first information may explicitly indicate the current application conditions for the first AI / ML model. The first information may explicitly indicate a change in the application conditions for the first AI / ML model. In some other embodiments, the first information may implicitly indicate a failure of the first AI / ML model caused by an application condition problem. For example, the implicit indication may include a model switching configuration or a model switching request.

[0093] Alternatively, or additionally, the first information may indicate a failure of the first AI / ML model caused by one or more of the following associated with the first AI / ML model: power issues, storage issues, complexity issues, or latency. For example, the first information may explicitly indicate a power below a power threshold. Alternatively, or additionally, the first information may explicitly indicate a storage resource below a storage resource threshold. In some embodiments, the first information may explicitly indicate a computing resource below a computing resource threshold. In some other embodiments, the first information may explicitly indicate a complexity exceeding a complexity threshold. Alternatively, the first information may explicitly indicate a latency exceeding a latency threshold. In some embodiments, the first information may explicitly indicate one or more of the following: the value of the current power, the value of the remaining power, the value of the current storage, the value of the remaining storage, the value of the current computing resource, the value of the remaining computing resource, the value of the complexity, or the value of the latency. In some other embodiments, the first information may implicitly indicate the cause of the failure. For example, the implicit indication may include a fallback request. Alternatively, the implicit indication may include no inference output within a given time or given power or given resource.

[0094] Alternatively, as described above, the fourth information regarding the model monitoring method for the first AI / ML model is transmitted to the entity 310. In this case, the entity 310 may determine (4050) the cause of the failure of the first AI / ML model based on the fourth information. In some embodiments, the fourth information may include a first configuration of the model monitoring method for the first AI / ML model. Alternatively, the fourth information may include a second configuration for monitoring the first AI / ML model. The details of the first configuration and the second configuration have been described above and are omitted to avoid redundancy.

[0095] For example, entity 310 can determine the cause of a failure of a first AI / ML model based on monitoring metrics associated with a model monitoring method. For example, if a first set of monitoring metrics related to inference accuracy is monitored based on the model monitoring method, entity 310 can determine a failure of the first AI / ML model caused by an inference accuracy problem. In some embodiments, if a second set of monitoring metrics related to system performance is monitored based on the model monitoring method, entity 310 can determine a failure of the first AI / ML model caused by a system performance problem. Alternatively, or in addition, if a third set of monitoring metrics related to application conditions is monitored based on the model monitoring method, entity 310 can determine a failure of the first AI / ML model caused by an application condition problem. In some other embodiments, if a fourth set of monitoring metrics related to power consumption is monitored based on the model monitoring method, entity 310 can determine a failure of the first AI / ML model caused by a power consumption problem. In some embodiments, if a fifth set of monitoring metrics related to complexity is monitored based on the model monitoring method, entity 310 can determine a failure of the first AI / ML model caused by a complexity problem. Alternatively, or in addition, if a sixth set of monitoring metrics related to storage is monitored based on the model monitoring method, entity 310 can determine a failure of the first AI / ML model caused by a storage problem. In some embodiments, if a seventh set of monitoring metrics related to latency is monitored based on the model monitoring method, entity 310 can determine a failure of the first AI / ML model caused by a latency problem.

[0096] In some embodiments, if entity 310 is at network device 220 and entity 320 is at terminal device 210, terminal device 210 can report a failure of the NW-side AI / ML model and the cause of the failure to network device 220. In this case, network device 220 can perform an action in response to the report from terminal device 210. Alternatively, if entity 310 is at terminal device 210 and entity 320 is at network device 220, network device 220 can indicate a failure of the UE-side AI / ML model and the cause of the failure to terminal device 210. In this case, terminal device 210 can perform an action in response to the indicated failure and cause.

[0097] In some embodiments, the signaling requirement for the first information or the fourth indication can be based on: different AI / ML model structures and whether the model monitoring entity is the same as the model inference entity and / or whether the entity needs to take an action. Table 2 below shows an example of signaling exchange related to the first information or the fourth information. Table 2

[0098] In some embodiments, as described above, multiple AI / ML models can be monitored. In such cases, in some embodiments, multiple first information related to the causes of failures of multiple first AI / ML models can be obtained. The multiple first information can include the model identities of the multiple first AI / ML models that are experiencing model failures. Alternatively, the multiple first information can include the model group identities of the multiple first AI / ML models that are experiencing model failures. In some embodiments, the above first information can be configured for each of the multiple first AI / ML models. In other words, for each AI / ML model or each group of AI / ML models, the cause of the model failure can be transmitted or determined. In some other embodiments, the fourth information can indicate the identity of the first AI / ML model among the multiple first AI / ML models.

[0099] Entity 310 determines (4060) one or more of the following based on the first information related to the cause of the failure of the first AI / ML model and the second information related to the state of the second AI / ML model: a first decision or a first action related to AI / ML management. In some embodiments, the second information can indicate that the second AI / ML model is unavailable. Alternatively, the second information can indicate that the second AI / ML model is not registered. In some other embodiments, the second information can indicate that the second AI / ML model has not been received by the first entity. The second information can indicate that the second AI / ML model has not been deployed by the first entity. In some embodiments, the second information can indicate that the second AI / ML model has been deployed. Alternatively, the second information can indicate that the second AI / ML model is activated. The second information can also indicate that the second AI / ML model is known or selected. In some embodiments, the second information can also indicate that the second AI / ML model is not activated. Alternatively or additionally, the second information can also indicate that the second AI / ML model is unknown or not selected. In this way, the correct action can be selected in response to the model failure, and the gain of quickly restoring AI / ML operations can be achieved.

[0100] In some embodiments, entity 310 can receive the second information from other entities. For example, network device 220 can provide the state of the NW-side AI / ML model to entity 310. Alternatively, or additionally, terminal device 210 can provide the state of the UE-side AI / ML model to entity 310.

[0101] In some embodiments, entity 310 may be capable of performing a first action. Alternatively, the first action may be performed by other entities, such as entity 410. In some embodiments, entity 310 or entity 410 may indicate ability information for actions in response to an AI / ML model failure. Alternatively, the ability information for actions in response to an AI / ML model failure may be provided to entity 310 or entity 410. In such cases, in some embodiments, entity 310 may determine (4060) one or more of the following based on first information related to the cause of the failure of the first AI / ML model, second information related to the state of the second AI / ML model, and the ability information of entity 310 / 410: a first decision or a first action related to AI / ML management. The ability information may indicate which first action(s) entity 310 or entity 410 is / are capable of supporting and / or which first action(s) entity 310 or entity 410 is / are not capable of supporting. In some other embodiments, the ability information may be exchanged among the terminal device 210, the network device 220, or a third-party device.

[0102] In some embodiments, entity 310 may determine (4060) one or more of the following based on first information related to the cause of the failure of the first AI / ML model, second information related to the state of the second AI / ML model, and the requirements of the first decision or the first action: a first decision or a first action related to AI / ML management. Examples and details of the requirements will be described later.

[0103] In some embodiments, the first decision may include one or more of the following: applying the first AI / ML mode, switching to the second AI / ML model, training the second AI / ML model, falling back to the default AI / ML model, or stopping using AI / ML. In some embodiments, the first action may include at least one of the following: registration of the second AI / ML model, transfer of the second AI / ML model, deployment of the second AI / ML model, activation of the second AI / ML model, deactivation of the second AI / ML model, update of the second AI / ML model, training of the second AI / ML model, or enhanced data processing of the first AI / ML model. Table 3 provides an overview of different decisions and actions that may be taken if a model failure is detected. Details of the first decision and the first action will be described later. Table 3

[0104] The first action may vary based on the state of the second AI / ML model. Figure 5Shows an example of the state of a second AI / ML model and corresponding actions. In some embodiments, if the state indicates that the second AI / ML model is unavailable, the first action can be model training (510) or tuning of the second AI / ML model. In some embodiments, the state can indicate that the second AI / ML model is registered. For example, the second AI / ML model can be available (e.g., the training of the second AI / ML model is completed), but not yet registered. In this case, the first action can be model registration (520). Alternatively, the state can indicate that the second AI / ML model has not been received. For example, the second AI / ML model can be registered (i.e., has been assigned a model ID) but not received. In this case, the first action can be model transfer (530) or delivery. In some other embodiments, the state can indicate that the second AI / ML model has not been deployed. In this case, the first action can be model deployment (540). In some embodiments, the second AI / ML model can be deployed, e.g., the runtime environment for the second AI / ML model is ready. In this case, the first action can be model activation. Alternatively, the state can indicate that the second AI / ML model is activated. In this case, the first action can be model switching. In some other embodiments, the second AI / ML model can be known. In other words, it is a more general state for an available model. In this case, at least the entity used for model inference knows the existence of such a model. Additionally, the state can indicate that the second AI / ML model is selected. The first action can also include one or more of the following: model selection or model deactivation. Note that the first action can also include Figure 5 other actions not shown in

[0105] In some embodiments, the first decision can be to fallback to a non-AI / ML mode or non-AI / ML method. For example, if the second AI / ML model is unavailable, then the fallback to the non-AI / ML mode can be applied. In some embodiments, if the reason for the first AI / ML model is one of the following: deterioration of system performance, low prediction accuracy, low power, or low storage, then the fallback to the non-AI / ML mode can be applied. In such cases, in some embodiments, the configuration for performing the non-AI / ML method may be available at the entity (e.g., entity 330) used for model inference. The configuration can be a normal measurement report. For example, the configuration can be related to CSI measurement and reporting. Alternatively, the configuration can be related to measurement and reporting for beam management. The configuration can be related to measurement and reporting for positioning information. In some embodiments, entity 310 may transmit at least one of the following for non-AI / ML operations: a request, a notification, or a suggestion. Alternatively, signaling related to one or more of the following may be exchanged: configuration, activation, or triggering of non-AI / ML operations. In some other embodiments, the implicit application of non-AI / ML operations can be based on a specific reason for the failure.

[0106] Alternatively, the first decision can be to fallback to a default AI / ML model. For example, if the default AI / ML model is known at the model inference entity (i.e., entity 330), then the fallback to the default AI / ML model can be applied. In some embodiments, if the reason for the first AI / ML model is one of the following: low prediction accuracy or incorrect application conditions, then the fallback to the default AI / ML model can be applied. In some embodiments, the default AI / ML model can be an AI / ML model with better generalization performance. Alternatively, the default AI / ML model can be a previously working AI / ML model. In some other embodiments, the default AI / ML model can be an AI / ML model with a specific model ID. In some embodiments, entity 310 may transmit at least one of the following for the default AI / ML model: a request, a notification, or a suggestion. Alternatively, signaling related to one or more of the configuration, activation, or triggering of the default AI / ML model can be exchanged. In some other embodiments, the implicit application of the default AI / ML model can be based on a specific reason for the failure.

[0107] In some other embodiments, the first decision may be to apply the first AI / ML model. For example, if the second AI / ML model is the same as the first AI / ML model, the first AI / ML model can still be used. In some embodiments, if the reason for the first AI / ML model is one of the following: inference input distribution or inference output data distribution, the first AI / ML model can still be applied. In this case, the first action may be enhanced data processing of the first AI / ML model. For example, preprocessing of the input data can be improved. In some embodiments, the input data may be clean. For example, abnormal input data (e.g., out-of-distribution data) can be removed. Alternatively, the threshold for the input data for the first AI / ML model can be updated. For example, the threshold can be increased to have high-quality input data. In some embodiments, the input data can be enhanced. For example, more data samples (e.g., historical data) can be included in the input data. In some embodiments, the existing data samples can be randomized to expand the dataset of the input data. Alternatively, additional data can be generated, for example, using the generative adversarial network (GAN) method.

[0108] In some embodiments, postprocessing of the output data can be improved. In some embodiments, the output data may be clean. For example, abnormal output data (e.g., out-of-distribution data) can be removed. Alternatively, output data with a low confidence level can be removed.

[0109] Alternatively, processing of the intermediate data can be improved. In some embodiments, the intermediate data resolution can be enhanced. For example, more bits can be allocated to represent the intermediate data.

[0110] In some embodiments, entity 310 may transmit at least one of the following to enhance data processing: requests, notifications, or suggestions. Alternatively, signaling related to one or more of the following may be exchanged to enhance data processing: configuration, activation, or triggering. In some other embodiments, the implicit application of enhanced data processing may be based on a specific reason for the failure.

[0111] Alternatively or additionally, still applying the first AI / ML model may include fine-tuning of the first AI / ML model. Alternatively, still applying the first AI / ML model may include minor updates to the first AI / ML model.

[0112] In some embodiments, the first decision may be to switch to a second AI / ML model. For example, if the state of the second AI / ML model is one of the following: known, available, registered, received, deployed, or activated, then the switch to the second AI / ML model may be made. In some embodiments, if the reason for the first AI / ML model is incorrect application conditions, then the switch to the second AI / ML model may be made. In some embodiments, the switch may be a switch based on the model ID. In some embodiments, compared to the first AI / ML model, the second AI / ML model may include one or more of the following attributes: requires less time to perform inference, requires fewer resources to perform inference, requires less power to perform inference, is suitable for different application conditions, is suitable for different input / output data distributions, or has better performance (e.g., accuracy or generalization performance).

[0113] In some embodiments, if the second AI / ML model has not been registered, the first action may include: model registration, model transfer, model deployment, and model activation. Alternatively, if the second AI / ML model has not been received, the first action may include: model transfer, model deployment, and model activation. In some other embodiments, if the second AI / ML model has not been deployed, the first action may include: model deployment and model activation. In some embodiments, if the second AI / ML model has been deployed, the first action may include model activation. Alternatively, if the second AI / ML model has been activated, the first action may include model switching. In some embodiments, entity 310 may transmit at least one of the following for the first action of the second AI / ML model: a request, a notification, or a suggestion. Alternatively, signaling related to one or more of the following may be exchanged: configuration, activation, or triggering of the first action for the second AI / ML model. In some other embodiments, the implicit application of the first action for the second AI / ML model may be based on a specific reason for the failure.

[0114] In some embodiments, the first decision may be to find a new AI / ML model different from the first AI / ML model. For example, if the second AI / ML model is in an unavailable state, a new AI / ML model may be needed. In other words, if the current second AI / ML model is unavailable, the new AI / ML model may be regarded as another second AI / ML model. In some embodiments, if the reason for the first AI / ML model is one of the following: deterioration of system performance, low inference accuracy, no suitable application, or no matching data distribution, a new AI / ML model may be needed. In some embodiments, finding a new AI / ML model may include model training or model update using new data. In this case, the first action may include: data collection, model training, model registration, model transfer, model deployment, and model activation. Alternatively, finding a new AI / ML model may include model training or model update without using new data. In this case, the first action may include: model training, model registration, model transfer, model deployment, and model activation. In some embodiments, model training or model update may result in a completely new AI / ML model. Alternatively, model training or model update may result in a change in a part of the first AI / ML model. In some embodiments, entity 310 may transmit at least one of the following to train the new AI / ML model: a request, a notification, or a suggestion. Alternatively, signaling related to one or more of the following may be exchanged: configuration, activation, or triggering of the training of the new AI / ML model. In some other embodiments, the implicit application of the training of the new AI / ML model may be based on the specific cause of the failure. In some embodiments, if new data is involved in model training or model update, the new data may be shared. For example, the new data may be provided to the entity responsible for model training.

[0115] Alternatively, the first action may include deactivating the first AI / ML model. For example, if the failure of the first AI / ML model is announced and the first decision is not to keep the same AI / ML model, the deactivation of the first AI / ML model may be performed. For example, if one of the following decisions is made, the deactivation of the first AI / ML model may be applied: fallback to a non-AI / ML method, fallback to the default AI / ML mode, switch to the second AI / ML model, train a new AI / ML model.

[0116] In some other embodiments, the first action may include model selection. For example, model selection may be different from model switching based on the assumption that the failed first AI / ML model may not be activated. Model selection may refer to selecting a second AI / ML model to be applied from the activated AI / ML models.

[0117] In some embodiments, as described above, multiple AI / ML models can be monitored. In such cases, in some embodiments, multiple second information related to the states of the multiple second AI / ML models can be obtained. The multiple second information can include the model identities of the multiple second AI / ML models. Alternatively, the multiple second information can include the model group identities of the multiple second AI / ML models. In some embodiments, the above-mentioned second information can be configured for each of the multiple second AI / ML models. In other words, for each second AI / ML model or each group of second AI / ML models, the state of the model can be transmitted or determined. In such cases, entity 310 can determine multiple first decisions and / or multiple first actions based on the multiple first information and the multiple second information. In other words, for each first AI / ML model or each group of first AI / ML models, a first decision and / or an action can be determined.

[0118] Entity 310 can transmit (4070) a request for the first decision and / or the first action to entity 410 responsible for taking the first action. Entity 410 can be at the terminal device 210. Alternatively, entity 410 can be at the network device 220. In some other embodiments, entity 410 can be at a third-party device. Alternatively, entity 310 itself can take the first action.

[0119] In some embodiments, the need for signaling to exchange the first decision and / or enable the first action in response to a model failure can be based on different AI / ML model structures and whether the entity that needs to take an action and / or the entity that needs to make a decision is the same as the model inference entity and / or the model monitoring entity. Table 4 below shows an example of signaling exchange related to the first decision and / or the first action. Table 4

[0120] In some embodiments, multiple first actions can be taken jointly. In some embodiments, the multiple first actions can include possible combinations of actions that can be configured / reported / selected. In such cases, multiple timers can be configured. When the model monitoring gives a positive result (e.g., successful inference), all timers can be stopped or set to 0. Figure 6A and Figure 6B respectively illustrate schematic diagrams of methods in which multiple first actions are taken. Note that Figure 6A and FIG. 6b are merely examples and not limitations.

[0121] Reference Figure 6A, the timer 611 can be started first. During the running time of the timer 611, data processing can be enhanced (602). Then the performance of the first AI / ML model can be monitored. If the performance is good enough, the first AI / ML model using the enhanced data processing can be used (603). After the timer 611 expires, the timer 612 can be started. If the performance is still not good, during the running time of the timer 612, a different AI / ML model can be switched to (604). Then the performance of the first AI / ML model can be monitored. If the performance is good enough, the different AI / ML model can be used (605). After the timer 612 expires, the timer 613 can be started. If the performance is still not good, a new AI / ML model can be trained (606). Then the performance of the first AI / ML model can be monitored. If the performance is good enough, the newly trained AI / ML model can be used (607). After the timer 613 expires, the timer 614 can be started. If the performance is still not good, a non-AI / ML method can be fallback (608).

[0122] Reference Figure 6B , if a model failure is declared (601) for the first AI / ML model, all the timers 621, 622, 623, and 624 can be started. During the running time of the timer 621, data processing can be enhanced (602). Then the performance of the first AI / ML model can be monitored. If the performance is good enough, the first AI / ML model using the enhanced data processing can be used (603). When the timer 622 is still running, the timer 621 expires. In other words, the time length of the timer 622 is longer than that of the timer 621. If the performance is still not good, during the running time of the timer 622, a different AI / ML model can be switched to (604). Then the performance of the first AI / ML model can be monitored. If the performance is good enough, the different AI / ML model can be used (605). When the timer 623 is still running, the timer 622 expires. In other words, the time length of the timer 623 is longer than that of the timer 622. If the performance is still not good, a new AI / ML model can be trained (606). Then the performance of the first AI / ML model can be monitored. If the performance is good enough, the newly trained AI / ML model can be used (607). When the timer 624 is still running, the timer 623 expires. In other words, the time length of the timer 624 is longer than that of the timer 623. If the performance is still not good, a non-AI / ML method can be fallback (608).

[0123] As described above, the first decision and / or the first action may also be determined based on the requirements of the first decision or the first action. In some embodiments, the first action may be associated with latency requirements for AI / ML management. For example, different actions in response to a model failure may be associated with different requirements. The latency requirement may refer to the duration from the model failure to the model success. For example, T start may be the following time points: when the model failure is announced, when reporting / receiving model failure information to / from other entities, making a decision, or taking an action, etc. T action may be the length of time required to complete the required action for the model failure. In some embodiments, before T start + T action the first AI / ML model may be used. Alternatively, the first AI / ML model may be deactivated. For example, non-AI methods, default AI / ML models, or still-good AI / ML models may be used. In some embodiments, after T start + T action a second AI / ML model (including a switched / selected / indicated / updated / trained model, or using enhanced data processing) may be used.

[0124] In some embodiments, different actions may require different lengths of time, for example, in the following order: "fall back to non-AI" < "enhance data processing for the same AI / ML model" < "switch to an existing AI / ML model" < "train a new AI / ML model". In some embodiments, within "switch to an existing AI / ML model", the lengths of time may be in the following order: "not available" > "available but not yet registered" > "registered but not yet received" > "received but not yet deployed" > "deployed but not yet activated" > "activated". Alternatively, within "train a new AI / ML model", the lengths of time may be in the following order: "not training the new AI / ML model with new data" < "training the new AI / ML model with new data", or "updating the model using a part of the parameters" < "model training for a complete new AI / ML model".

[0125] Alternatively, or in addition, the latency requirement may also include one or more of the following: the requirement for the latency from the model failure to the action, from the action to the recovery, or from the model failure to the decision, from the decision to the action.

[0126] In some embodiments, it may be expected that entity 320 gives a positive result before T start + T action In addition, it may be desirable for the terminal device 210 or the network device 220 to receive signaling that the second AI / ML model is performing well. If at T start + T actionAfter that, the entity 320 can still give a negative result. In this case, the model recovery can be considered a failure. This can be achieved by defining a model recovery timer to control the overall time that can be spent on model recovery.

[0127] Additionally, or alternatively, the following at least one can be defined for AI / ML management: an overhead requirement, a storage requirement, a complexity requirement, or a power consumption requirement. In some embodiments, if at least one of the following is exhausted, the model monitoring can still give a negative result: the overhead, storage, complexity, or power consumption for model recovery. In this case, the model recovery can be considered a failure. In some embodiments, the above requirements can be predefined, for example, predefined in a protocol. Alternatively, the above requirements can be configured by the network device 220. In some other embodiments, the above requirements can be reported by the terminal device 210. In some embodiments, the above requirements can be notified by a third-party device.

[0128] Figure 7 The flowchart of a communication method 700 implemented at a first entity (e.g., entity 310) according to some embodiments of the present disclosure is illustrated. The first entity can be implemented at one of the following: the terminal device 210, the network device 220, or a third-party device.

[0129] In some embodiments, at block 710, the first entity can receive first information related to the cause of a failure of the first AI / ML model or fourth information about a model monitoring method for the first AI / ML model from the entity 320.

[0130] In some embodiments, the first information indicates at least one of the following: a failure of the first AI / ML model caused by an inference accuracy problem, a failure of the first AI / ML model caused by a system performance problem, a failure of the first AI / ML model caused by a data distribution problem, a failure of the first AI / ML model caused by an application condition problem, or a failure of the first AI / ML model caused by one or more of the following associated with the first AI / ML model: a power problem, a storage problem, a complexity problem, or a latency. In some embodiments, a failure of the first AI / ML model caused by an inference accuracy problem includes at least one of the following: an inference accuracy below an inference accuracy threshold, or a confidence level below a confidence level threshold. In some embodiments, a failure of the first AI / ML model caused by a system performance problem includes: a degradation of system performance. In some embodiments, a failure of the first AI / ML model caused by a data distribution problem includes at least one of the following: outside the input data distribution, outside the output data distribution, input data distribution drift, output data distribution drift. In some embodiments, a failure of the first AI / ML model caused by an application condition problem includes: a mismatched application condition. In some embodiments, a failure of the first AI / ML model is also caused by at least one of the following: a power below a power threshold, a storage resource below a storage resource threshold, a computing resource below a computing resource threshold, a complexity exceeding a complexity threshold, or a latency exceeding a latency threshold.

[0131] In some embodiments, the first entity may receive first information from the second entity indicating a cause of a failure of the first AI / ML model. The first entity may determine the cause of the failure of the first AI / ML model based on the first information.

[0132] In some embodiments, the first entity may receive fourth information from the second entity regarding a model monitoring method for the first AI / ML model. The first entity may determine the cause of the failure of the first AI / ML model based on the fourth information.

[0133] In some embodiments, the fourth information regarding the model monitoring method includes a first configuration of the model monitoring method for the first AI / ML model. In some embodiments, the first configuration includes at least one of the following: a first set of monitoring metrics related to inference accuracy, a second set of monitoring metrics related to system performance, a third set of monitoring metrics related to application conditions, a fourth set of monitoring metrics related to power consumption, a fifth set of monitoring metrics related to complexity, a sixth set of monitoring metrics related to storage, or a seventh set of monitoring metrics related to latency.

[0134] In some embodiments, the fourth information regarding the model monitoring method includes a second configuration for monitoring a first AI / ML model. In some implementations, the second configuration includes at least one of the following: a set of thresholds related to monitoring metrics for the first AI / ML model, a timer initiated based on the occurrence of an AI / ML model failure instance, a counter for the number of AI / ML model failure instances, monitoring the start of the first AI / ML model, monitoring the pause of the first AI / ML model, monitoring the end of the first AI / ML model, monitoring the control of the first AI / ML model, monitoring the periodicity of the first AI / ML model, monitoring the duty cycle of the first AI / ML model, monitoring the duration of the first AI / ML model, monitoring the time offset configuration of the first AI / ML model, or the capabilities of the model monitoring method.

[0135] In some embodiments, if multiple first AI / ML models are monitored, the fourth information may indicate the identity of the first AI / ML model among the multiple first AI / ML models.

[0136] At block 720, if a failure of the first AI / ML model occurs, the first entity determines at least one of the following: a first decision or a first action related to AI / ML model management. At least one of the first decision or the first action is determined based on: first information related to the cause of the failure of the first AI / ML model, and second information related to the state of the second AI / ML model.

[0137] In some embodiments, the first decision includes at least one of the following: applying the first AI / ML mode, switching to the second AI / ML model, training a new AI / ML model different from the second AI / ML model, falling back to the default AI / ML model, or stopping using AI / ML.

[0138] In some embodiments, the first action includes at least one of the following: falling back to a non-AI / ML method, registration of the second AI / ML model, transfer of the second AI / ML model, deployment of the second AI / ML model, activation of the second AI / ML model, deactivation of the second AI / ML model, update of the second AI / ML model, training of the second AI / ML model, or enhanced data processing of the first AI / ML model.

[0139] In some embodiments, the enhanced data processing of the first AI / ML model includes at least one of the following: an updated threshold for the input data of the first AI / ML model, augmentation of the input data, or removal of the output data of the first AI / ML model, where the confidence level of the output data is lower than the confidence level threshold.

[0140] In some embodiments, the second information indicates at least one of the following: the second AI / ML model is unavailable, the second AI / ML model is not registered, the second AI / ML model is not received by the first entity, the second AI / ML model is not deployed by the first entity, the second AI / ML model is deployed, the second AI / ML model is activated, the second AI / ML model is known, or the second AI / ML model is selected.

[0141] In some embodiments, the first configuration is configured for each of the plurality of first AI / ML models. In some embodiments, the second configuration is configured for each of the plurality of first AI / ML models. In some embodiments, the first action is associated with a latency requirement for AI / ML management.

[0142] In some embodiments, the first entity may receive second information indicating the state of the second AI / ML model from a third entity. In some embodiments, the first entity may transmit a request to a fourth entity for at least one of the following: a first decision or a first action to be performed by the fourth entity.

[0143] In some embodiments, if a plurality of first AI / ML models fail, the first entity may obtain a plurality of first information related to the cause of the failure of the plurality of first AI / ML models. The first entity may obtain a plurality of second information related to the state of the plurality of second AI / ML models. The first entity may determine at least one of the following based on the plurality of first information and the plurality of second information: a plurality of first decisions and a plurality of first actions.

[0144] Figure 8 A flowchart of a communication method 800 implemented at a second entity (e.g., entity 320) according to some embodiments of the present disclosure is illustrated. The second entity may be implemented at one of the following: the terminal device 210, the network device 220, or the third-party device.

[0145] At block 810, the second entity determines a model monitoring method for the first AI / ML model.

[0146] At block 820, the second entity monitors the first AI / ML model based on the model monitoring method.

[0147] At block 830, according to the determination of the failure of the first AI / ML model, the second entity sends one of the following to the first device: the first information related to the cause of the failure of the first AI / ML model, or the fourth information about the model monitoring method for the first AI / ML model.

[0148] In some embodiments, the first information indicates at least one of the following: a failure of the first AI / ML model caused by an inference accuracy problem, a failure of the first AI / ML model caused by a system performance problem, a failure of the first AI / ML model caused by a data distribution problem, a failure of the first AI / ML model caused by an application condition problem, or a failure of the first AI / ML model caused by one or more of the following associated with the first AI / ML model: a power problem, a storage problem, a complexity problem, or a latency.

[0149] In some embodiments, a failure of the first AI / ML model caused by an inference accuracy problem includes at least one of the following: an inference accuracy below an inference accuracy threshold, or a confidence level below a confidence level threshold. In some embodiments, a failure of the first AI / ML model caused by a system performance problem includes: a degradation of system performance. In some embodiments, a failure of the first AI / ML model caused by a data distribution problem includes at least one of the following: outside of the input data distribution, outside of the output data distribution, input data distribution drift, output data distribution drift. In some embodiments, a failure of the first AI / ML model caused by an application condition problem includes: a mismatched application condition. In some embodiments, a failure of the first AI / ML model is further caused by at least one of the following: a power below a power threshold, a storage resource below a storage resource threshold, a computing resource below a computing resource threshold, a complexity exceeding a complexity threshold, or a latency exceeding a latency threshold.

[0150] In some embodiments, determining a model monitoring method includes: receiving, from a third device, a first configuration of the model monitoring method for the first AI / ML model; and determining the model monitoring method based on the first configuration.

[0151] In some embodiments, the first configuration includes at least one of the following: a first set of monitoring metrics related to inference accuracy, a second set of monitoring metrics related to system performance, a third set of monitoring metrics related to application conditions, a fourth set of monitoring metrics related to power consumption, a fifth set of monitoring metrics related to complexity, a sixth set of monitoring metrics related to storage, or a seventh set of monitoring metrics related to latency.

[0152] In some embodiments, fourth information regarding the model monitoring method includes a first configuration of the model monitoring method for the first AI / ML model.

[0153] In some embodiments, determining a model monitoring method includes: receiving, from a third device, a second configuration for monitoring the first AI / ML model; and determining the model monitoring method based on the first configuration.

[0154] In some embodiments, the second configuration includes at least one of the following: a set of thresholds related to monitoring metrics for the first AI / ML model, a timer initiated upon the occurrence of an AI / ML model failure instance, a counter for the number of AI / ML model failure instances, monitoring the start of the first AI / ML model, monitoring the pause of the first AI / ML model, monitoring the end of the first AI / ML model, monitoring the control of the first AI / ML model, monitoring the periodicity of the first AI / ML model, monitoring the duty cycle of the first AI / ML model, monitoring the duration of the first AI / ML model, monitoring the time offset configuration of the first AI / ML model, or the capabilities of the model monitoring method.

[0155] In some embodiments, the fourth information regarding the model monitoring method includes a second configuration for monitoring the first AI / ML model.

[0156] In some embodiments, if multiple first AI / ML models are monitored, the fourth information also indicates the identity of the first AI / ML model among the multiple first AI / ML models.

[0157] In some embodiments, the first configuration is configured for each of the multiple first AI / ML models, or the second configuration is configured for each of the multiple first AI / ML models.

[0158] In some embodiments, if multiple first AI / ML models are monitored, the first information is related to the cause of the failures of the multiple first AI / ML models.

[0159] In some embodiments, the first entity is implemented on a terminal device and the second entity is implemented on a network device, or the first entity is implemented on a network device and the second entity is implemented on a terminal device.

[0160] Figure 9 is a simplified block diagram of a device 900 suitable for implementing embodiments of the present disclosure. The device 900 can be considered a further exemplary implementation of any device as Figure 1 shown. Accordingly, the device 900 can be implemented at or as at least a part of the terminal device 210 or the network device 220.

[0161] As shown, device 900 includes a processor 910, a memory 920 coupled to the processor 910, a suitable transmitter (TX) / receiver (RX) 940 coupled to the processor 910, and a communication interface coupled to the TX / RX 940. The memory 910 stores at least a portion of a program 930. The TX / RX 940 is used for two-way communication. The TX / RX 940 has at least one antenna to facilitate communication, but in practice, the access nodes mentioned in this application may have several antennas. The communication interface may represent any interface required to communicate with other network elements, such as the X2 / Xn interface for two-way communication between eNB / gNB, the S1 / NG interface for communication between a mobility management entity (MME) / access and mobility management function (AMF) / SGW / UPF and an eNB / gNB, the Un interface for communication between an eNB / gNB and a relay node (RN), or the Uu interface for communication between an eNB / gNB and a terminal device.

[0162] It is assumed that the program 930 is assumed to include program instructions that, when executed by the associated processor 910, cause the device 900 to operate in accordance with embodiments of the present disclosure, as referred to herein Figures 1 to 8 discussed. The embodiments herein can be implemented by computer software executable by the processor 910 of the device 900, or by hardware, or by a combination of software and hardware. The processor 910 can be used to implement various embodiments of the present application. In addition, the combination of the processor 910 and the memory 920 can form a processing component 950 suitable for implementing various embodiments of the present disclosure.

[0163] As a non-limiting example, the memory 920 can be of any type suitable for the local technology network and can be implemented using any suitable data storage technology, such as non-transitory computer-readable storage media, semiconductor-based memory devices, magnetic memory devices and systems, optical memory devices and systems, fixed memory, and removable memory. Although only one memory 920 is shown in the device 900, there may be several physically distinct memory modules in the device 900. As a non-limiting example, the processor 910 can be of any type suitable for the local technology network and can include one or more of a general-purpose computer, a special-purpose computer, a microprocessor, a digital signal processor (DSP), and a processor based on a multi-core processor architecture. The device 900 can have multiple processors, such as an application-specific integrated circuit chip that is clocked subordinate to a synchronous main processor in time.

[0164] In some embodiments, a communication device includes circuitry configured to: determine at least one of a first decision or a first action related to artificial intelligence / machine learning (AI / ML) model management based on a determination of a failure of a first AI / ML model, and wherein the at least one of the first decision or the first action is determined based on: first information related to a cause of the failure of the first AI / ML model, and second information related to a state of a second AI / ML model.

[0165] According to an embodiment of the present disclosure, the circuitry may be configured to perform any method implemented by the first entity as described above.

[0166] In some embodiments, a communication device includes circuitry configured to: determine a model monitoring method for a first AI / ML model; monitor the first AI / ML model based on the model monitoring method; and transmit, based on a determination of a failure of the first AI / ML model, to a first device one of: first information related to a cause of the failure of the first AI / ML model, or fourth information regarding the model monitoring method for the first AI / ML model.

[0167] According to an embodiment of the present disclosure, the circuitry may be configured to perform any method implemented by the second entity as described above.

[0168] As used herein, the term "circuitry" may refer to hardware circuitry and / or a combination of hardware circuitry and software. For example, the circuitry may be a combination of analog and / or digital hardware circuitry and software / firmware. As another example, the circuitry may be any portion of a hardware processor, software, and (one or more) memories with software (including (one or more) digital signal processors) that work together to enable a device, such as a communication device or a base station, to perform various functions. In yet another example, the circuitry may be a hardware circuit and / or a processor, such as a microprocessor or a portion of a microprocessor, that requires software / firmware to operate but may not have the software when not required to operate. As used herein, the term circuitry also encompasses implementations that are only hardware circuitry or (one or more) processors or a portion of a hardware circuit or (one or more) processors and their (or their) accompanying software and / or firmware.

[0169] In one aspect, a communication device includes: at least one processor; and at least one memory coupled to the at least one processor and storing instructions that, when executed by the at least one processor, cause the communication device to perform the methods implemented by the communication device as discussed above.

[0170] In one aspect, a computer-readable medium having instructions stored thereon that, when executed on at least one processor, cause the at least one processor to perform the methods implemented by the communication device as discussed above.

[0171] In one aspect, a computer program comprising instructions that, when executed on at least one processor, cause the at least one processor to perform the methods implemented by the communication device as discussed above.

[0172] Generally, the various embodiments of the present disclosure may be implemented in hardware or specific circuits, software, logic, or any combination thereof. Some aspects may be implemented in hardware, while other aspects may be implemented in firmware or software executable by a controller, microprocessor, or other computing device. Although the various aspects of the embodiments of the present disclosure are shown and described as block diagrams, flowcharts, or using some other graphical representation, it should be understood that, by way of non-limiting example, the blocks, devices, systems, techniques, or methods described herein may be implemented in hardware, software, firmware, specific circuits or logic, general hardware or a controller, or other computing devices, or some combination thereof.

[0173] The present disclosure also provides at least one computer program product tangibly stored on a non-transitory computer-readable storage medium. The computer program product includes computer-executable instructions, such as those included in program modules executed in a device on a target real or virtual processor, to perform the processes or methods as referred to above Figures 1 to 8 described. Generally, program modules include routines, programs, libraries, objects, classes, components, data structures, etc. that perform specific tasks or implement specific abstract data types. In various embodiments, the functions of the program modules may be combined or split as needed among the program modules. The machine-executable instructions for the program modules may be executed within a local or distributed device. In a distributed device, the program modules may be located in both local and remote storage media.

[0174] The program code for performing the methods of the present disclosure may be written in any combination of one or more programming languages. The program code may be provided to a processor or controller of a general-purpose computer, a special-purpose computer, or other programmable data processing device such that the program code, when executed by the processor or controller, causes the functions / operations specified in the flowchart and / or block diagram to be implemented. The program code may be executed entirely on the machine, partially on the machine, as a stand-alone software package, partially on the machine and partially on a remote machine, or entirely on a remote machine or server.

[0175] The above program code can be embodied on a machine-readable medium, which can be any tangible medium that can contain or store a program used by or in conjunction with an instruction execution system, apparatus, or device. The machine-readable medium can be a machine-readable signal medium or a machine-readable storage medium. The machine-readable medium can include, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, apparatus, or devices, or any suitable combination of the foregoing. More specific examples of the machine-readable storage medium will include electrical connections with one or more wires, portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fibers, portable compact disc read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination of the foregoing.

[0176] Moreover, although the operations are depicted in a particular order, this should not be construed as requiring that the operations be performed in the particular order shown or in sequential order, or that all of the illustrated operations be performed to achieve the desired result. In some cases, multitasking and parallel processing may be advantageous. Similarly, although several specific implementation details are included in the above discussion, these should not be construed as limitations on the scope of the present disclosure, but rather as descriptions of features that may be specific to particular embodiments. Certain features described in the context of separate embodiments may also be implemented in combination in a single embodiment. Conversely, the various features described in the context of a single embodiment may also be implemented separately or in any suitable sub-combination in multiple embodiments.

[0177] Although the present disclosure has been described in language specific to structural features and / or methodological acts, it is to be understood that the disclosure defined in the appended claims is not necessarily limited to the specific features or acts described above. Rather, the above specific features and acts are disclosed as example forms of implementing the claims.

Claims

1. A communication method, comprising: Based on the determination of a fault in a first artificial intelligence / machine learning (AI / ML) model, at a first entity, determine at least one of the following: a first decision or a first action related to AI / ML model management, and wherein at least one of the first decision or the first action is determined based on: first information related to a cause of the fault of the first AI / ML model, and second information related to a state of a second AI / ML model.

2. The method according to claim 1, wherein the first decision includes at least one of the following: Applying the first AI / ML pattern, Switching to the second AI / ML model, Training a new AI / ML model different from the second AI / ML model, Falling back to the default AI / ML model, or Stopping using AI / ML.

3. The method according to claim 1 or 2, wherein the first action includes at least one of the following: Registration of the second AI / ML model, Transfer of the second AI / ML model, Deployment of the second AI / ML model, Activation of the second AI / ML model, Deactivation of the second AI / ML model, Update of the second AI / ML model, Training of the second AI / ML model, or Enhanced data processing of the first AI / ML model.

4. The method according to claim 3, wherein the enhanced data processing of the first AI / ML model includes at least one of the following: An updated threshold for the input data for the first AI / ML model, Augmentation of the input data, or Removal of the output data of the first AI / ML model, the confidence level of the output data being lower than the confidence level threshold.

5. The method according to any one of claims 1 to 4, wherein the second information indicates at least one of the following: The second AI / ML model is unavailable, The second AI / ML model is not registered, The second AI / ML model is not received by the first entity, The second AI / ML model is not deployed by the first entity, The second AI / ML model is deployed, The second AI / ML model is activated, The second AI / ML model is known, or The second AI / ML model is selected.

6. The method according to any one of claims 1 to 5, wherein the first information indicates at least one of the following: the failure of the first AI / ML model caused by an inference accuracy problem, the failure of the first AI / ML model caused by a system performance problem, the failure of the first AI / ML model caused by a data distribution problem, the failure of the first AI / ML model caused by an application condition problem, or the failure of the first AI / ML model caused by one or more of the following associated with the first AI / ML model: power problem, storage problem, complexity problem, or latency.

7. The method according to claim 6, wherein the failure of the first AI / ML model caused by the inference accuracy problem includes at least one of the following: the inference accuracy lower than an inference accuracy threshold, or the confidence level lower than a confidence level threshold; or wherein the failure of the first AI / ML model caused by the system performance problem includes: Degradation of the system performance; Or wherein the fault of the first AI / ML model caused by the data distribution problem includes at least one of the following: Out-of-distribution input data, Out-of-distribution output data, Input data distribution drift, Output data distribution drift, or wherein the fault of the first AI / ML model caused by the application condition problem includes: Mismatched application conditions, wherein the fault of the first AI / ML model is also caused by at least one of the following: Power below a power threshold, Storage resources below a storage resource threshold, Computing resources below a computing resource threshold, Complexity exceeding a complexity threshold, or Latency exceeding a latency threshold.

8. The method according to any one of claims 1 to 7, further comprising: Receive, from a second entity, the first information indicating the cause of the fault of the first AI / ML model; And Based on the first information, determine the cause of the fault of the first AI / ML model.

9. The method according to any one of claims 1 to 7, further comprising: Receive, from a second entity, fourth information regarding a model monitoring method for the first AI / ML model; And Based on the fourth information, determine the cause of the fault of the first AI / ML model.

10. The method according to claim 9, wherein the fourth information regarding the model monitoring method includes a first configuration of the model monitoring method of the first AI / ML model, and the first configuration includes at least one of the following: a first set of monitoring metrics related to inference accuracy, a second set of monitoring metrics related to system performance, a third set of monitoring metrics related to application conditions, a fourth set of monitoring metrics related to power consumption, a fifth set of monitoring metrics related to complexity, a sixth set of monitoring metrics related to storage, or a seventh set of monitoring metrics related to latency.

11. The method according to claim 9, wherein the fourth information regarding the model monitoring method includes a second configuration for monitoring the first AI / ML model, and wherein the second configuration includes at least one of the following: a set of thresholds related to the monitoring metrics for the first AI / ML model, a timer started based on the occurrence of an AI / ML model failure instance, a counter for the number of AI / ML model failure instances, the start of monitoring the first AI / ML model, the pause of monitoring the first AI / ML model, the end of monitoring the first AI / ML model, the control of monitoring the first AI / ML model, the periodicity of monitoring the first AI / ML model, the duty cycle of monitoring the first AI / ML model, the duration of monitoring the first AI / ML model, the time offset configuration of monitoring the first AI / ML model, or the capabilities for the model monitoring method.

12. The method according to any one of claims 9 to 11, wherein if multiple first AI / ML models are monitored, the fourth information further indicates the identity of the first AI / ML model among the multiple first AI / ML models.

13. The method according to claim 12, wherein the first configuration is configured for each of the plurality of first AI / ML models, or wherein the second configuration is configured for each of the plurality of first AI / ML models.

14. The method according to any one of claims 1 to 13, further comprising: Receive, from a third entity, the second information indicating the state of the second AI / ML model.

15. The method according to any one of claims 1 to 14, further comprising: Transmit, to a fourth entity, a request for at least one of the following: the first decision or the first action to be performed by the fourth entity.

16. The method according to any one of claims 1 to 15, wherein the first action is associated with at least one of the following: a latency requirement for the AI / ML management, an overhead requirement for the AI / ML management, a storage requirement for the AI / ML management, a complexity requirement for the AI / ML management, or a power consumption requirement for the AI / ML management.

17. The method according to any one of claims 1 to 16, wherein if a plurality of first AI / ML models fail, the method further comprises at least one of the following: obtaining a plurality of first information related to the cause of the failure of the plurality of first AI / ML models; obtaining a plurality of second information related to the state of the plurality of second AI / ML models; and determining at least one of the following based on the plurality of first information and the plurality of second information: a plurality of first decisions and a plurality of first actions.

18. The method according to any one of claims 1 to 17, wherein the first entity is implemented at a terminal device, or wherein the first entity is implemented at a network device.

19. A communication method, comprising: Determine, at a second entity, a model monitoring method for a first AI / ML model; Based on the model monitoring method, monitor the first AI / ML model; and According to the determination of a fault in the first AI / ML model, transmit to the first entity one of the following: First information related to the cause of the fault of the first AI / ML model, or Fourth information regarding a model monitoring method for the first AI / ML model.

20. A communication device, comprising: At least one processor; And At least one memory coupled to the at least one processor and storing instructions thereon, the instructions, when executed by the at least one processor, cause the communication device to perform the method according to any one of claims 1 to 18 or claim 19.

Images