Monitoring data events for updated model

Abstract

Example embodiments of the present disclosure relate to model updating. A device obtains a configuration indicating a data event for data associated with a model to be monitored. The data event indicates a parameter to be monitored and a threshold associated with the parameter. The equipment triggers monitoring of parameters; and when the parameter is monitored, detecting whether the value of the parameter meets the condition indicated in the configuration or not. In this manner, training of the model is enhanced to process data events.

Description

Technical Field

[0001] Various example embodiments of the present disclosure relate generally to the field of telecommunications, and in particular to methods, devices, apparatuses, and computer-readable storage media for updating models employed in telecommunications systems. Background Art

[0002] In the telecommunication industry, techniques for improving the performance of telecommunication systems have been proposed. For example, artificial intelligence / machine learning (AI / ML) models have been adopted in telecommunication systems to improve the performance of telecommunication systems. Therefore, it is worthwhile to study how and when to update the AI / ML models adopted in telecommunication systems. Summary of the invention

[0003] In a first aspect of the present disclosure, a device is provided. The device includes: at least one processor; and at least one memory storing instructions, wherein when the instructions are executed by the at least one processor, the device at least performs: obtaining a configuration indicating a data event for data associated with a model to be monitored, the data event indicating: a parameter to be monitored and a threshold value associated with the parameter; triggering monitoring of the parameter based on the configuration indicating the data event; and while monitoring the parameter, detecting whether the value of the parameter satisfies a condition indicated in the configuration.

[0004] In a second aspect of the present disclosure, a device is provided. The device includes at least one processor; and at least one memory storing instructions, which, when executed by the at least one processor, cause the device to at least perform: sending a configuration indicating a data event for data associated with a model to be monitored to another device, the data event indicating: a parameter to be monitored and a threshold value associated with the parameter.

[0005] In a third aspect of the present disclosure, a method is provided. The method includes: obtaining, at a device, a configuration indicating a data event for data associated with a model to be monitored, the data event indicating: a parameter to be monitored and a threshold associated with the parameter; triggering monitoring of the parameter based on the configuration indicating the data event; and while monitoring the parameter, detecting whether a value of the parameter satisfies a condition indicated in the configuration.

[0006] In a fourth aspect of the present disclosure, a method is provided, comprising: sending, at a device and to another device, a configuration indicating a data event for data associated with a model to be monitored, the data event indicating: a parameter to be monitored and a threshold associated with the parameter.

[0007] In a fifth aspect of the present disclosure, a device is provided. The device includes: a component for obtaining a configuration indicating a data event for data associated with a model to be monitored, the data event indicating: a parameter to be monitored and a threshold associated with the parameter; a component for triggering monitoring of the parameter based on the configuration indicating the data event; and a component for detecting whether a value of the parameter satisfies a condition indicated in the configuration while monitoring the parameter.

[0008] In a sixth aspect of the present disclosure, a device is provided, comprising a component for sending a configuration indicating a data event for data associated with a model to be monitored to another device, the data event indicating: a parameter to be monitored and a threshold value associated with the parameter.

[0009] In a seventh aspect of the present disclosure, a computer-readable medium is provided, wherein the computer-readable medium includes instructions stored thereon, and is used to cause a device to at least execute the method according to the third aspect.

[0010] In an eighth aspect of the present disclosure, a computer-readable medium is provided, wherein the computer-readable medium includes instructions stored thereon, and is used to cause a device to at least execute the method according to the fourth aspect.

[0011] It should be understood that this summary of the invention is not intended to identify the key or essential features of the embodiments of the present disclosure, nor is it intended to be used to limit the scope of the present disclosure. Other features of the present disclosure will become readily understood through the following description. BRIEF DESCRIPTION OF THE DRAWINGS

[0012] Some example embodiments will now be described with reference to the accompanying drawings, in which:

[0013] Figure 1A and Figure 1B An example communication environment is shown in which example embodiments of the present disclosure may be implemented;

[0014] Figure 2A and Figure 2B shows a signaling diagram for communication according to some example embodiments of the present disclosure;

[0015] Figure 3 shows a signaling diagram for communication according to some example embodiments of the present disclosure;

[0016] Figure 4 shows a signaling diagram for communication according to some example embodiments of the present disclosure;

[0017] Figure 5 shows a signaling diagram for communication according to some example embodiments of the present disclosure;

[0018] FIG. 6A to FIG. 6DA schematic diagram showing modeling according to some example embodiments of the present disclosure;

[0019] Figure 7 A flowchart showing a method according to some example embodiments of the present disclosure is shown;

[0020] Figure 8 A flowchart showing a method according to some example embodiments of the present disclosure is shown;

[0021] Fig. 9 A flowchart showing a method according to some example embodiments of the present disclosure is shown;

[0022] Fig.10 shows a simplified block diagram of a device suitable for implementing an example embodiment of the present disclosure; and

[0023] Fig.11 A block diagram of an example computer-readable medium is shown according to some example embodiments of the present disclosure.

[0024] Throughout the drawings, the same or similar reference numerals refer to the same or similar elements. DETAILED DESCRIPTION

[0025] The principles of the present disclosure will now be described with reference to some example embodiments. It should be understood that these embodiments are only for illustrative purposes and are helpful for those skilled in the art to understand and implement the present disclosure without implying any limitation on the scope of the present disclosure. The embodiments described herein can be implemented in various ways different from the ways described below.

[0026] In the following description and claims, unless defined otherwise, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this disclosure belongs.

[0027] References in this disclosure to "one embodiment," "an embodiment," "an example embodiment," etc. indicate that the described embodiment may include a particular feature, structure, or characteristic, but not every embodiment necessarily includes the particular feature, structure, or characteristic. Furthermore, such phrases do not necessarily refer to the same embodiment. Furthermore, when a particular feature, structure, or characteristic is described in conjunction with an embodiment, it is considered to be within the knowledge of those skilled in the art to affect such feature, structure, or characteristic in conjunction with other embodiments (whether or not explicitly described).

[0028] It should be understood that although the terms "first", "second", etc. may be used herein to describe various elements, these elements should not be limited by these terms. These terms are only used to distinguish one element from another element. For example, without departing from the scope of the exemplary embodiments, a first element may be referred to as a second element, and similarly, a second element may be referred to as a first element. As used herein, the term "and / or" includes any and all combinations of one or more of the listed terms.

[0029] As used herein, “at least one of: ” and “at least one of ” and similar expressions (wherein a list of two or more elements is connected by “and” or “or”) refer to at least any one of the elements, or at least any two or more of the elements, or at least all of the elements.

[0030] As used herein, unless explicitly stated, performing a step “in response to A” does not mean performing the step immediately after “A” occurs, but may include one or more intermediate steps.

[0031] The terms used herein are only used for the purpose of describing specific embodiments and are not intended to limit the exemplary embodiments. As used herein, unless the context clearly indicates otherwise, the singular forms "a", "an", and "the" are intended to include the plural forms as well. It should also be understood that when used herein, the terms "comprises", "comprising", "has", "having", "includes", and / or "including" specify the presence of features, elements, and / or parts, etc., but do not exclude the presence or addition of one or more other features, elements, parts, and / or combinations thereof.

[0032] As used in this application, the term "circuitry" may refer to one or more or all of the following: (a) hardware circuit implementation only (e.g., implementation only in analog and / or digital circuitry), and (b) a combination of hardware circuitry and software, such as (where applicable): (i) a combination of analog and / or digital circuitry and software / firmware, and (ii) any portion of a hardware processor with software (including a digital signal processor, software and memory that work together to enable a device such as a mobile phone to perform various functions), and (c) A hardware circuit and / or processor that requires software (eg, firmware) for operation, such as a microprocessor or portion of a microprocessor, but the software may not be present when not required for operation.

[0033] This definition of circuitry applies to all uses of the term in this application, including in any claims. As another example, as used in this application, the term circuitry also covers an implementation of only a hardware circuit or processor (or multiple processors) or a portion of a hardware circuit or processor and its accompanying software and / or firmware. The term circuitry also covers, for example and if applicable to a particular claim element, a baseband integrated circuit or processor integrated circuit for a mobile device or a similar integrated circuit in a server, cellular network device, or other computing or network device.

[0034] As used herein, the term "communication network" refers to a network that complies with any suitable communication standard, such as New Radio (NR), Long Term Evolution (LTE), Advanced LTE (LTE-A), Wideband Code Division Multiple Access (WCDMA), High Speed ​​Packet Access (HSPA), Narrowband Internet of Things (NB-IoT), etc. In addition, the communication between the terminal device and the network device in the communication network can be performed according to any suitable generation communication protocol, including but not limited to the first generation (1G), second generation (2G), 2.5G, 2.75G, third generation (3G), fourth generation (4G), 4.5G, fifth generation (5G), sixth generation (6G) communication protocol and / or any other protocol currently known or developed in the future. The embodiments of the present disclosure can be applied to various communication systems. In view of the rapid development of communication, there are of course future types of communication technologies and systems that can implement the present disclosure. It should not be considered that the scope of the present disclosure is limited to the above-mentioned system.

[0035] As used herein, the term "network device" refers to a node in a communication network via which a terminal device accesses a network and receives services from it. A network device may refer to a base station (BS) or an access point (AP), such as a Node B (NodeB or NB), an evolved Node B (eNodeB or eNB), a NR NB (also referred to as a gNB), a remote radio unit (RRU), a radio head (RH), a remote radio head (RRH), a repeater, an integrated access and backhaul (IAB) node, a low-power node (e.g., a femto, a micro), a non-terrestrial network (NTN) or a non-terrestrial network device (such as a satellite network device, a low earth orbit (LEO) satellite and a geosynchronous orbit (GEO) satellite, an aircraft network device, etc.), depending on the terminology and technology applied. In some example embodiments, a radio access network (RAN) split architecture includes a centralized unit (CU) and a distributed unit (DU) at an IAB donor node. An IAB node includes a mobile terminal (IAB-MT) portion that behaves like a UE toward a parent node, and a DU portion of an IAB node behaves like a base station toward a next-hop IAB node.

[0036] The term "terminal device" refers to any terminal device capable of wireless communication. As an example and not limitation, the terminal device may also be referred to as a communication device, a user equipment (UE), a subscriber station (SS), a portable subscriber station, a mobile station (MS) or an access terminal (AT). The terminal device may include, but is not limited to, a mobile phone, a cellular phone, a smart phone, a voice over IP (VoIP) phone, a wireless local loop phone, a tablet computer, a wearable terminal device, a personal digital assistant (PDA), a portable computer, a desktop computer, an image capture terminal device such as a digital camera, a game terminal device, a music storage and playback facility, a vehicle-mounted wireless terminal device, a wireless endpoint, a mobile station, a laptop embedded device (LEE), a laptop mounted device (LME), a USB dongle, a smart device, a wireless user equipment (CPE), an Internet of Things (IoT) device, a watch or other wearable device, a head mounted display (HMD), a vehicle, a drone, medical equipment and applications (e.g., remote surgery), industrial equipment and applications (e.g., robots and / or other wireless devices operating in industrial and / or automatic processing chain environments), consumer electronic devices, equipment operating on commercial and / or industrial wireless networks, etc. The terminal device may also correspond to the mobile terminal (MT) part of an IAB node (eg, a relay node).In the following description, the terms "terminal device", "communication device", "terminal", "user equipment" and "UE" may be used interchangeably.

[0037] As used herein, the terms "resource", "transmission resource", "resource block", "physical resource block" (PRB), "uplink resource" or "downlink resource" may refer to any resource used to perform communication (e.g., communication between a terminal device and a network device), such as a resource in the time domain, a resource in the frequency domain, a resource in the spatial domain, a resource in the code domain, or any other combination of time domain, frequency domain, spatial domain and / or code domain resources to achieve communication. In the following, unless explicitly stated, resources in the frequency domain and the time domain will be used as examples of transmission resources for describing some example embodiments of the present disclosure. It should be noted that the example embodiments of the present disclosure are also applicable to other resources in other domains.

[0038] As described above, how and when to update the AI / ML models of network functions of a wireless communication system, such as a 5G wireless communication system (5GS), is worth studying. In some solutions, AI / ML management capabilities and services of the 5GS in which the AI / ML models are used are specified. An AI / ML entity is an AI / ML model or contains an AI / ML model and can be managed as a single composite entity. AI / ML training-related capabilities and services are specified as part of the current scope. The AI / ML training function, which plays the role of an AI / ML training MnS generator, can consume various data for AI / ML model training purposes to train or retrain the AI / ML model.

[0039] AI / ML Training (AIMLT) MnS Producer triggers the job of AI / ML training. According to some solutions, the job of AI / ML training can be triggered based on a request from an AIMLT MnS consumer. However, it is beneficial that the job of AI / ML training can be triggered by the AIMLT MnS producer itself without an explicit request for the AIMLT MnS consumer. In particular, given changes in the network, the AIMLT MnS producer can trigger AI / ML training based on new network events. In order to trigger AI / ML training for a given use case, the AIMLT MnS producer needs to monitor applicable network events and their changes. AI / ML training can then be triggered based on some specified characteristics or based on conditions related to the observed network events. However, there is no component for enabling the AIMLT MnS producer to know the type of network event or monitor the required data. In addition, there is no method for the AIMLT MnS provider to perform and determine the need for triggering new training or retraining of the AI / ML model. Example Environment

[0040] Figure 1A and Figure 1BAn example of a device of a wireless communication system 100 (e.g., a 5G wireless communication system (5GS)) in which an example embodiment of the present disclosure can be implemented is shown. The device includes a device 110, a device 120, and a device 130 that can communicate with each other via a wired and / or wireless connection. In some example embodiments, the device 110 may include an AIML data monitoring module or entity. The AIML data monitoring module or entity described herein may be an independent information object class (IOC) that performs the intended function. In some example embodiments, the device 120 may include a producer, such as an AIML management service (MnS) producer. For example, the device 120 may be a network management node of a wireless communication system, a base station of a radio access network of the wireless communication system 100, or a core network device of the wireless communication system 100 (e.g., a network device of a core network). In some example embodiments, the device 120 may include other producers. In some example embodiments, the device 130 may include a consumer, such as an AIML MnS consumer. For example, an AIML MnS consumer may be an analysis consumer or an operator. In some examples, the devices 120 and 130 may include one or more network functions. Alternatively, devices 120 and 130 may include management functions or operations, administration, and maintenance (OAM) functions. The term "function" used herein may refer to a physical module or a virtual module.

[0041] In some example embodiments, device 110, device 120, and device 130 may be implemented on one device. Alternatively, device 110, device 120, and device 130 may be implemented on different devices. Figure 1A As shown, device 110, device 120, and device 130 may be implemented on three different devices. In some example embodiments, device 120 and device 130 may be implemented on one device, and device 110 may be implemented on another device. Alternatively, device 110 and device 120 may be implemented on one device, and device 130 may be implemented on another device. For example, Figure 1B As shown, device 110 may be a part of device 120. In some example embodiments, device 110 and device 130 may be implemented on one device, while device 120 may be implemented on another device.

[0042] It should be understood that the number of devices and their connections shown in Figure 1 is for illustration purposes only and does not imply any limitation. Wireless communication system 100 may include any suitable number of devices configured to implement example embodiments of the present disclosure.

[0043] In some example embodiments, the link from a network device (e.g., a base station or a core network device) to a terminal device is referred to as a downlink (DL), and the link from a terminal device to a network device (e.g., a base station or a core network device) is referred to as an uplink (UL). In DL, the network device is a transmitting (TX) device (or transmitter), and the terminal device is a receiving (RX) device (or receiver). In UL, the terminal device is a TX device (or transmitter), and the network device is an RX device (or receiver).

[0044] The communication in the wireless communication system 100 can be implemented according to any communication protocol, including but not limited to the cellular communication protocol of the first generation (1G), second generation (2G), third generation (3G), fourth generation (4G), fifth generation (5G), sixth generation (6G) radio access technology and / or any other protocol currently known or developed in the future. In addition, the communication between the base station and the terminal can utilize any suitable technology, including but not limited to: code division multiple access (CDMA), frequency division multiple access (FDMA), time division multiple access (TDMA), frequency division duplex (FDD), time division duplex (TDD), multiple input multiple output (MIMO), orthogonal frequency division multiple access (OFDM), discrete Fourier transform spread OFDM (DFT-s-OFDM) and / or any other technology currently known or developed in the future. How it works and example signaling used for communication

[0045] According to some example embodiments of the present disclosure, a solution for updating an AI / ML model is provided. In the solution, a producer of a device obtains a configuration from a consumer, wherein the configuration indicates a data event for data associated with an AI / ML model to be monitored. The data event indicates: a parameter to be monitored and a threshold associated with the parameter. The device includes a producer that triggers an AI / ML data monitoring module or entity to perform monitoring of the parameter, and detects whether the value of the parameter satisfies a condition indicated in the configuration while monitoring the parameter. In this way, the update of the model is enhanced to handle data events.

[0046] The embodiments of the present disclosure are described in detail below with reference to the accompanying drawings.

[0047] Reference now Figure 2A , which shows a signaling diagram 200 for communication according to some example embodiments of the present disclosure. Figure 2A As shown, signaling diagram 200 shows signaling between device 110, device 120, and device 130. For discussion purposes, reference is made to Figure 1A The signaling diagram 200 is described in which the device 110 is separated from the device 120 or is external to the device 120. Figure 2AOne device 110 and one device 130 are shown in FIG. 1 , but it should be understood that there may be multiple devices that perform similar operations as described below with respect to device 110 and multiple third devices that perform similar operations as described below with respect to device 130 .

[0048] Device 130 sends 2005 to device 120 a configuration indicating a data event for data associated with an AI / ML model ("model"). In some example embodiments, a model may be referred to as an AIML entity. An AIML entity may have a list of data events to be monitored for (re)training. Data events are to be monitored. Data events indicate one or more parameters to be monitored. For example, one or more parameters may include a performance measurement (PM) to be monitored. Alternatively or additionally, one or more parameters may include a key performance indicator (KPI) to be monitored. In some example embodiments, one or more parameters may include a network control parameter. Alternatively, one or more parameters may include a network performance metric. In some other example embodiments, one or more parameters may include a communication environment parameter. The data event also indicates a threshold value associated with the parameter. For example, the data event may indicate a threshold value. In some example embodiments, the data event may indicate a threshold direction. For example, the threshold direction may indicate above a threshold value or below a threshold value. Alternatively or additionally, the data event may include a hysteresis value. In this way, the updating of the model is enhanced to handle data events.

[0049] In some example embodiments, a data event may include a threshold periodicity for notifications associated with satisfying a condition indicated in a configuration. For example, a data event may include a periodicity for (re)training based on notifications of crossing a threshold. The periodicity may imply the number of threshold notifications to be received for considering retraining. For example, a particular PM may be configured to be monitored for values ​​above 100, and a notification will be sent whenever the value is above 100. A data event may be configured to configure the number of such notifications that may be received before a decision to train or retrain is triggered. By way of example only, a parameter indicated in a data event may be the number of terminal devices connected to the network. The threshold indicated in a data event may be 95% of the maximum number of terminal devices. The threshold direction may be above a threshold.

[0050] In some example embodiments, the configuration may be sent in a request for AIML training. In some example embodiments, the ALIMLTrainingRequest IOC may be extended to have a new information element (IE) to accommodate the data events to be monitored. For example, Table 1 below shows an example of a request for AIML training. It should be noted that Table 1 is only an example and not a limitation. Table 1 Property Name Support qualifiers isReadable isWritable isInvariant isNotifyable aIMLEntityId M T T F T candidataTraningDataSource O T T F T traingDataQualityScore O T T F T traningRequestSource M T T F T requestStatus M T T F T expectedRuntimeContext O T T F T performanceRequirements M T T F T cancelRequest O T T F T suspendRequest O T T F T monitoredDataEvents O T T F T

[0051] Property "MonitoredDataEvents< <datatype>>" can represent data events that need to be monitored to determine the need for retraining by the AIMLTMnS producer. This includes the PMs and KPIs to be monitored, their granularity periods, and the ranges used to generate the PMs and KPIs. This also includes threshold-related configuration directions, values, and hysteresis values. Use this < <datatype>>Notifications specified for the IOC may be applicable. Table 2 below shows an example of MonitoredDataEvents. It should be noted that Table 2 is only an example and not a limitation. Table 2

[0052] As described above, the model may be referred to as an AIML entity. In some example embodiments, an AIML entity may be modeled as an IOC, and the following new IEs (such as data events and their details) may be stored in the AIML entity. The IOC "AIMLEntity" may represent an AI / ML entity, which may be an AI / ML model or a supporting AI / ML function containing an AI / ML model. AIML training may be requested for an AI / ML model or a supporting AI / ML function. The AI / ML model or the algorithm supporting the AI / ML function is not standardized. For each AIMLEntity under training, one or more ALIMLLiningProcesses may be instantiated. AIMLEntity may contain 3 types of contexts—TrainingContext, which is a context in which the AIMLEntity has been trained; ExpectedRunTimeContext, which is a context in which the AIMLEntity is expected to be applied; or / and RunTimeContext, which is a context in which the model is applied. AIML entities may also contain data events that are monitored to update the corresponding AIML entity. IOC "AIMLEntity" may include attributes inherited from the top IOC and the following attributes (shown in Table 3). Public notifications may be valid for this IOC without exceptions or additions. Table 4 shows an example of attribute constraints. It should be noted that Tables 3 and 4 are only examples and not limitations. Table 3 Property Name Support qualifiers isReadable isWritable isInvariant isNotifyable inferenceType M T F F T aIMLEntityVersion M T F F T expectedRunTimeContext O T T F T trainingContext CM T F F T runTimeContext O T F F T monitoredDataevents O T F F T Role-related attributes trainingProcessRef M T F F T trainingRequestRef CM T F F T trainingReportRef M T F F T Table 4

[0053] Device 120 sends 2010 to device 110 a configuration indicating a data event. As described above, the data event indicates a parameter to be monitored. For example, the parameter may include a PM to be monitored. Alternatively or additionally, the parameter may include a KPI to be monitored. The data event also indicates a threshold associated with the parameter. For example, the data event may indicate a threshold. In some example embodiments, the data event may indicate a threshold direction. For example, the threshold direction may indicate above a threshold or below a threshold. Alternatively or additionally, the data event may include a hysteresis value. In this way, the training of the model is enhanced to handle data events.

[0054] In some example embodiments, device 110 may refer to AIML data monitoring. For example, AIML data monitoring may be a stand-alone function. Alternatively, AILML data monitoring may be a subroutine within a function. In the case of a stand-alone function, the following information may detail the definition of the AIML data monitoring IOC. The IOC "AIMLDataMonitoring" may represent data events to be monitored for an AIML entity. An "AIMLDataMonitoring" management object instance (MOI) may be contained under an ALIMLTinningFunction MOI. Public notifications may be valid for this IOC without exceptions or additions. Table 5 shows an example of attributes. Table 5

[0055] Device 110 triggers monitoring of the 2015 parameter. In some example embodiments, based on the parameter to be monitored, device 110 may trigger a fifth device for monitoring the parameter. For example, device 110 may instantiate a "PerMetricJob" (performance metric job) for monitoring the parameter. In some example embodiments, if the data event includes a KPI to be monitored, device 110 may determine one or more PMs associated with the KPI. In this case, device 110 may be triggered to monitor the determined one or more PMs.

[0056] Device 110 may trigger 2020 to monitor for conditions indicated in the configuration being met. In some example embodiments, device 110 may trigger a sixth device to monitor for crossing of a threshold. For example, device 110 may instantiate or trigger a "ThresholdMonitor" job for monitoring crossing of a threshold. The term "threshold crossing" may refer to reaching a threshold. The sixth device may monitor one or more of the following: a threshold, a threshold direction, and a hysteresis value for a data event. In this case, if the threshold is crossed in the configured direction taking into account the hysteresis value (if configured), the sixth device may send a notification of the crossing of the threshold.

[0057] While monitoring the parameter, the device 110 detects 2025 whether the value of the parameter satisfies the condition indicated in the configuration. For example, in some example embodiments, the condition may be a value of a particular PM that is above 100. In this case, the device 110 may detect whether the value of PM is above 100. If the value of PM is above 100, the device 110 may detect that the condition is satisfied. Alternatively, if the value of PM is not above 100, the device 110 may detect that the condition is not satisfied.

[0058] In some example embodiments, the device 110 may determine the periodicity of notifications associated with the crossing of a threshold. Alternatively or additionally, the device 110 may determine the threshold direction in which the threshold is crossed. In this case, the device 110 may determine whether the value of the parameter satisfies the condition based on the periodicity of the notification and the threshold direction. For example, the device 110 may determine whether the periodicity reaches the threshold periodicity. The device 110 may determine whether the threshold direction satisfies the configured direction. In this case, if the periodicity of the notification reaches the threshold periodicity and the threshold direction satisfies the configured direction, the device may determine that the value of the parameter satisfies the condition. In some example embodiments, if the periodicity of the notification does not reach the threshold periodicity, the device 110 may determine that the model does not need to be updated. Alternatively or additionally, if the threshold direction does not satisfy the configured direction, the device 110 may determine that the model does not need to be updated. In some example embodiments, the monitoring information may include the monitored PM. In this case, the device 110 may determine the KPI based on the monitored PM.

[0059] By way of example only, device 110 may consume notifications sent from a "ThresholdMonitor" job. In this case, these notifications may be compared to an acceptable periodicity configured in device 110 for a given data event. Based on this, device 110 may determine whether to initiate (re)training. For example, device 110 may receive multiple notifications of threshold crossings. In this case, if the number of notifications reaches a threshold number indicated in the data event, device 110 may determine that the model needs to be updated.

[0060] In some example embodiments, device 110 may trigger 2030 an update of a model. For example, device 110 may directly instantiate an AIML training process corresponding to a model that needs to be trained or retrained. In this case, device 110 may receive the updated result of the model. Device 110 may send 2035 the updated result of the model to device 120. In some example embodiments, the update of the model may include one or more of: training the model, retraining the model, updating a hyperparameter of the model, updating a configuration indicating a data event associated with the model, or updating an operating context of the model. The term "hyperparameter" as used herein may refer to a tuning parameter in a machine learning algorithm set by a user.

[0061] Alternatively, device 110 may send 2040 to device 120 an indication of whether the model needs to be updated. In this case, in some example embodiments, device 120 may evaluate notifications and events and decide on the need for (re)training. In this case, if necessary, device 120 may perform retraining. For example, device 120 may trigger an update of the model. For example, if retraining or training of the model is required, device 120 may instantiate an AIML training process for the model, and an updated result of the updated training may be provided by device 120 to device 130. In some example embodiments, device 120 may send 2045 the updated result of the model to device 130. In this case, in some example embodiments, the updated result of the model may include one or more of the following: a newly trained model, a retrained model, an updated hyperparameter of the model, an updated configuration indicating a data event associated with the model, or an updated running context of the model.

[0062] In some example embodiments, device 130 may reconfigure the data events to be monitored for the model. In this case, device 130 may update the model with the reconfigured values ​​for the data events. The reconfigured values ​​may then be indicated to device 110 to begin monitoring for the updated events.

[0063] Reference now Figure 2B , which shows a signaling diagram 201 for communication according to some example embodiments of the present disclosure. Figure 2B As shown, signaling diagram 201 involves device 110, device 120, and device 130. For the purpose of discussion, reference is made to Figure 1B A signaling diagram 200 is described, in which device 110 is a part of device 120 .

[0064] Device 130 sends 2105 to device 120 a configuration indicating a data event for data associated with the model. In some example embodiments, the model may refer to an AIML entity. The AIML entity may have a list of data events to be monitored for (re) training. The data event will be monitored. The data event indicates one or more parameters to be monitored. For example, one or more parameters may include a performance measurement (PM) to be monitored. Alternatively or additionally, one or more parameters may include a key performance indicator (KPI) to be monitored. In some example embodiments, one or more parameters may include a network control parameter. Alternatively, one or more parameters may include a network performance metric. In some other example embodiments, one or more parameters may include a communication environment parameter. The data event also indicates a threshold value associated with the parameter. For example, the data event may indicate a threshold value. In some example embodiments, the data event may indicate a threshold direction. For example, the threshold direction may indicate above a threshold value or below a threshold value. Alternatively or additionally, the data event may include a hysteresis value. In this way, the training of the model is enhanced to handle data events.

[0065] In some example embodiments, a data event may include a threshold periodicity for notifications associated with satisfying a condition indicated in a configuration. For example, a data event may include a periodicity for (re)training based on notifications of threshold crossings. The periodicity may imply the number of threshold notifications to be received for considering retraining. For example, a particular PM may be configured to be monitored for values ​​above 100, and a notification will be sent whenever the value is above 100. A data event may be configured to configure the number of such notifications that may be received before a decision to (re)train is triggered. By way of example only, a parameter indicated in a data event may be the number of terminal devices connected to the network. The threshold indicated in the data event may be 95% of the maximum number of terminal devices. The threshold direction may be above the threshold.

[0066] In some example embodiments, the configuration may be sent in a request for AIML training. In some example embodiments, the ALIMLTrainingRequest IOC may be extended to have a new information element (IE) to accommodate the data events to be monitored. For example, Table 1 below shows an example of a request for AIML training.

[0067] Device 120 (e.g., device 110 within device 120) triggers 2110 monitoring of a parameter. In some example embodiments, based on the parameter to be monitored, device 120 may trigger a fifth device for monitoring the parameter. For example, device 120 may instantiate a "PerMetricJob" (performance metric job) for monitoring the parameter. In some example embodiments, if the data event includes a KPI to be monitored, device 120 may determine one or more PMs associated with the KPI. In this case, device 120 may be triggered to monitor the determined one or more PMs.

[0068] Device 120 (e.g., device 110 within device 120) may trigger monitoring for conditions indicated in the configuration to be met. In some example embodiments, device 120 may trigger a sixth device to monitor for crossing of a threshold. For example, device 120 may instantiate or trigger a "ThresholdMonitor" job for monitoring for crossing of a threshold. The term "crossing of a threshold" may refer to reaching a threshold. The sixth device may monitor one or more of: a threshold, a threshold direction, and a hysteresis value for a data event. In this case, if the threshold is crossed in the configured direction taking into account the hysteresis value (if configured), the sixth device may send a notification of the threshold crossing.

[0069] Device 120 (e.g., device 110 within device 120) detects 2115 whether the value of the parameter satisfies a condition indicated in the configuration while monitoring the parameter. For example, in some example embodiments, the condition may be a value of a particular PM that is higher than 100. In this case, device 120 may detect whether the value of PM is higher than 100. If the value of PM is higher than 100, device 120 may detect that the condition is satisfied. Alternatively, if the value of PM is not higher than 100, device 120 may detect that the condition is not satisfied.

[0070] In some example embodiments, the device 120 may determine the periodicity of notifications associated with the crossing of a threshold. Alternatively or additionally, the device 120 may determine the threshold direction in which the threshold is crossed. In this case, the device 120 may determine whether the value of the parameter satisfies the condition based on the periodicity of the notification and the threshold direction. For example, the device 120 may determine whether the periodicity reaches the threshold periodicity. The device 120 may determine whether the threshold direction satisfies the configured direction. In this case, if the periodicity of the notification reaches the threshold periodicity and the threshold direction satisfies the configured direction, the device may determine that the value of the parameter satisfies the condition. In some example embodiments, if the periodicity of the notification does not reach the threshold periodicity, the device 110 may determine that the model does not need to be updated. Alternatively or additionally, if the threshold direction does not satisfy the configured direction, the device 110 may determine that the model does not need to be updated. In some example embodiments, the monitoring information may include the monitored PM. In this case, the device 120 may determine the KPI based on the monitored PM.

[0071] By way of example only, device 120 may consume notifications sent from a "ThresholdMonitor" job. In this case, these notifications may be compared to the acceptable periodicity configured in device 110 for a given data event. Based on this, device 110 may determine whether to initiate (re)training. For example, device 110 may receive multiple notifications of threshold crossings. In this case, if the number of notifications reaches the threshold number indicated in the data event, device 110 may determine that the model needs to be updated.

[0072] Device 120 may update 2120 the model. For example, if retraining or training of the model is required, device 120 may instantiate an AIML training process for the model, and an updated result of the updated training may be provided by device 120 to device 130. In some example embodiments, device 120 may send 2125 the updated result of the model to device 130. In this case, in some example embodiments, the updated result of the model may include one or more of: a newly trained model, a retrained model, updated hyperparameters of the model, an updated configuration indicating data events associated with the model, or an updated running context of the model.

[0073] According to an example embodiment of the present disclosure, a component for a 3GPP management system that enables an authorized consumer to configure an AIMLT MnS producer to trigger training for a given AI / ML entity is proposed, as well as a component that enables the AIMLT MnS producer to independently trigger training based on network events or changes thereto and is configured by the AIMLT MnS consumer. Specifically, it proposes: capabilities and metadata for an AI / ML entity that enables the AIMLT MnS producer to initiate training for that specific AI / ML entity. The following features are proposed in the AI / ML entity: the AI / ML entity is enabled with a list of events to be monitored for (re)training; data events represent information to be monitored, such as a PM or KPI; each data event can be associated with a threshold monitor; and each data event also defines the need for retraining upon indication from the threshold monitor.

[0074] In addition, according to an exemplary embodiment of the present disclosure, capabilities and features of AIMLT MnS consumers are proposed, which enable consumers to configure AIMLT MnS producers to trigger training for a given AI / ML entity. The following features are proposed in AIMLTMnS producers and consumers: AIMLT consumers can configure / request AIML training for producers based on certain data events; the configuration will contain data events, which can be PMs or KPIs to be monitored by the producer; these data events are monitored for configured thresholds; and the producer decides to (re)train based on the violation of the configured thresholds.

[0075] Furthermore, according to an exemplary embodiment of the present disclosure, capabilities and features of an AIMLT MnS producer are proposed, which enable consumers to configure the AIMLT MnS producer and enable the AIMLT MnS producer to trigger (re)training for a given AI / ML entity. The following features are proposed in the AIMLT MnS producer and consumer: the producer or its ML App is configured; the producer then monitors the configured events and conditions; the producer triggers training as needed; the producer can then report training instances.

[0076] According to an exemplary embodiment of the present disclosure, it enables an AIML producer to receive information about a data source that needs to be monitored. In addition, a possible solution is proposed to perform analysis of the data and decide whether to perform new training or retraining.

[0077] Figure 3 A signaling diagram showing an interaction 300 between an AIMLT MnS consumer 310 and an AIMLT MnS producer 320. For example, the AIMLT MnS consumer 310 may be implemented on the device 130, while the AIMLT MnS producer 320 may be implemented at the device 120.

[0078] The AIMLT MnS consumer 310 may send 3010 an AIML training request with a configuration of data events to be monitored. The data events may include one or more of: information to be monitored (PM and / or KPI), threshold, threshold direction, hysteresis value, or acceptable periodicity of threshold notifications (which implies the number of threshold notifications to be received for consideration of retraining).

[0079] The AIMLT MnS producer 320 may instantiate 3020 a data monitor. The data monitor may then instantiate other jobs on PMs and / or KPIs based on configured data events.

[0080] The AIMLT MnS producer 320 may receive 3030 a notification about a monitored data event from the data monitoring or from an instantiated job via data monitoring. The notification may indicate a need for (re)training.

[0081] The AIMLT MnS producer 320 may evaluate 3040 the notifications and / or events. The AIMLT MnS producer 320 may decide the need for (re)training. If necessary, the AIMLT MnS producer 320 may perform 3050 retraining. The AIMLT MnS producer 320 may provide the results of the updated training to the AIMLT MnS consumer 310.

[0082] Figure 4 A signaling diagram showing the interaction 400 between an AIMLT MnS consumer and an AIMLT entity is shown. Figure 4 As shown, interaction 400 may involve AIMLT MnS consumer 410, AIMLT MnS producer 420, and ALML data monitoring function 430. For example, AIMLT MnS consumer 410 may be implemented on device 130, while ALML data monitoring function 430 may be implemented at device 110. Alternatively, ALML data monitoring function 430 may be implemented at device 120.

[0083] In some example embodiments, after performing initial training based on a training request, the AIMLT MnS producer may generate a new object of an AIML entity (e.g., a newly trained AIML model, or a new version of an AIML application with a newly trained model). The AIML entity instance may contain information defined in the AIML entity IOC. The information may also contain data events monitored for a particular AIML entity 420.

[0084] The AIMLT MnS consumer 410 can change or reconfigure 4010 a data event, which includes the data event itself (PM / KPI) or a granularity period or an object instance that needs to be scoped for data event generation. Then, the AIMLTMnS consumer 410 can directly update the AIML entity 420 and reconfigure the data event for the required changes. The AIML entity 420 can reconfigure 4020 the data event to the AIML data monitoring function 430. The AIML data monitoring function 430 can perform 4030 the required actions for perfmetric and threshold monitoring jobs. For example, the perfmetric job and the threshold measurement job can be appropriately updated. The AIML data monitoring function 430 can respond 4040 to the AIML entity 420 with the success or failure of the reconfiguration of the data event. The AIML entity 420 can respond to the AIMLT MnS consumer 410 with the success or failure of the reconfiguration event of the data event.

[0085] Figure 5 A signaling diagram of the interaction 500 between devices is shown. Figure 5 As shown, interaction 500 may involve AIMLT MnS consumer 510, AIMLT MnS producer 520, ALML data monitoring function 530, MnS producer PerMetric job 540, MnS producer ThresholdMonitor job 550, AIML training entity 560, AIML training entity 570. For example, AIMLTMnS consumer 510 may be implemented on device 130, ALML data monitoring function 530 may be implemented at device 110, and AIMLT MnS producer 520 may be implemented at device 120.

[0086] The AIMLT MnS consumer 510 may send 5001 a training request to the AIMLT MnS producer 520. The training request may contain the configuration of the data events to be monitored by the producer. The data events may include one or more of the following information: information to be monitored (PM and / or KPI), threshold, threshold direction, hysteresis value, or acceptable periodicity of threshold notification.

[0087] The AIMLT MnS producer 520 may instantiate 5002 data monitoring based on the configured data events to be monitored. If the configured data events contain KPIs, the ALML data monitoring function 530 may infer 5003 the appropriate PMs for those KPIs.

[0088] The ALML data monitoring function 530 may instantiate 5004 a MnS producer PerMetric job 540 for a configured / derived PM. The ALML data monitoring function 530 may instantiate 5005 a MnS producer ThresholdMonitor job 550 for configured and derived PMs. The MnS producer ThresholdMonitor job 550 may monitor 5006 the configured / derived PM for threshold crossings. The MnS producer ThresholdMonitor job 550 may send 5007 a notification if a threshold of a configured / derived PM is crossed.

[0089] The ALML data monitoring function 530 may check 5008 whether an acceptable periodicity has been reached for a given data event. For example, notifications received at the ALML data monitoring may be checked against a relatively acceptable number of occasions for the data event configuration.

[0090] The ALML data monitoring function 530 may infer 5009 KPIs from the appropriate PM. The ALML data monitoring function 530 may determine 5010 the need for (re)training.

[0091] The AIML (re)training may be initiated by the ALML data monitoring function 530 itself. Alternatively, the ALML data monitoring function 530 may instruct the AIMLT MnS producer 520, and the AIMLT MnS producer 520 may then trigger the (re)training.

[0092] In case the AIMLT MnS producer 520 triggers retraining, the ALML data monitoring function 530 may indicate 5011 the need for (re)training to the AIML producer. The AIMLT MnS producer 520 may instantiate 5012 a training process and configure it with the required inputs. The AIML training entity 560 may perform 5013 training or retraining of the AIML entity. The AIML training entity 560 may provide 5014 updated training results to the AIMLT MnS producer 520.

[0093] In the case where the ALML data monitoring function 530 instantiates the data monitoring process itself, the ALML data monitoring function 530 can directly instantiate 5015 the AIML training process. The AIML training entity 570 can perform 5016 the training or retraining of the AIML entity. The AIML training entity 570 can provide 5017 the updated training results to the ALML data monitoring function 530. The ALML data monitoring function 530 can provide 5018 the updated training results to the AIMLT MnS producer 520. The AIMLT MnS producer 520 can then update or create 5019 the AIML entity based on whether the training performed is retraining or new training, respectively. The AIMLT MnS producer 520 can forward 5020 the results to the AIMLT MnS consumer 510.

[0094] Fig. 6A A schematic diagram showing the class relationships associated with modeling of AIML entities is shown. Fig. 6A As shown, there may be a managed entity 611 belonging to ProxyClass, an AIML training function 612 belonging to InformationObjectClass, an AIML entity 613 belonging to InformationObjectClass, an AIML training request 614 belonging to InformationObjectClass, an AIML training process 615 belonging to InformationObjectClass, and an AIML training report 616 belonging to InformationObjectClass. Figure 6B A schematic diagram of class inheritance associated with modeling of AIML entities is shown. Figure 6B As shown, there may be a top 621 belonging to InformationObjectClass, an AIML training request 622 belonging to InformationObjectClass, an AIML training process 623 belonging to InformationObjectClass, an AIML training report 624 belonging to InformationObjectClass, an AIML entity 625 belonging to InformationObjectClass, an AIML training function 626 belonging to InformationObjectClass, and a managed function 627 belonging to InformationObjectClass.

[0095] Figure 6C A schematic diagram showing the class relationships associated with modeling of data monitoring. Figure 6C As shown, there may be a managed entity 631 belonging to ProxyClass, an AIML entity 632 belonging to InformationObjectClass, an AIML training function 633 belonging to InformationObjectClass, an AIML training request 634 belonging to InformationObjectClass, an AIML training process 635 belonging to InformationObjectClass, an AIML training report 636 belonging to InformationObjectClass, and an AIML data monitoring function 637 belonging to InformationObjectClass. Fig.6D A schematic diagram of class inheritance associated with modeling data monitoring is shown. Fig.6D As shown, there may be a top 641 belonging to InformationObjectClass, an AIML training request 642 belonging to InformationObjectClass, an AIML training process 643 belonging to InformationObjectClass, an AIML training report 644 belonging to InformationObjectClass, an AIML data monitoring function 645 belonging to InformationObjectClass, an AIML training function 646 belonging to InformationObjectClass, and a managed function 647 belonging to InformationObjectClass. In some example embodiments, due to the complexity and time-varying nature of the network, the previously deployed AI / ML entity may no longer be applicable to the network after running for a period of time. The performance of the trained model may degrade over time. The AI / ML entity needs to be updated in a timely manner to ensure the performance of reasoning and analysis. The performance of the AIML entity may depend on the commonality of the data distribution used for training and the data distribution used for reasoning. Model performance may be very good soon after deployment. This is because the data distribution used for training and the chance of the sample distribution selected for reasoning are the same. Over time, the distribution of network data may change compared to the distribution of training data. In such a scenario, the performance of the AIML entity degrades over time. Therefore, it is necessary to monitor network events such as PM, KPI, alarm, etc. and use this information in the producer to decide on retraining. Table 6 shows an example of retraining of AIML entities initiated by the producer based on network events. It should be noted that Table 6 is only an example and not a limitation. Table 6 Example Method

[0096] Figure 7 A flowchart of an example method 700 implemented at or performed by a first device according to some example embodiments of the present disclosure is shown. For the purpose of discussion, Figure 1A and Figure 1B Method 700 is described from the perspective of device 110 in FIG. 1 . In some embodiments, device 110 may be separate from device 120 . Alternatively, device 110 may be internal to device 120 .

[0097] At block 710, device 110 receives a configuration from a device indicating data events associated with a model to be monitored. In some example embodiments, device 110 may receive the configuration from device 120. Alternatively, device 110 within device 120 may receive the configuration from device 130.

[0098] The data event indicates: a parameter to be monitored and a threshold associated with the parameter. The data event may include one or more of the following: a threshold direction, a hysteresis value, or a threshold periodicity of notifications associated with a threshold crossing. In some example embodiments, the parameter may include a performance measurement. Alternatively or additionally, the parameter may include a key performance indicator. In some example embodiments, the parameter may include one or more of the following: a network control parameter, a network performance metric, or a communication environment parameter.

[0099] At block 720, device 110 triggers monitoring of the parameter based on the configuration indicating the data event.For example, device 110 may trigger a fifth device to monitor the parameter.

[0100] At block 730 , the device 110 detects whether the value of the parameter satisfies a condition in the configuration while monitoring the parameter.

[0101] In some example embodiments, device 110 may determine the model to be updated based on the monitoring information. In this case, based on determining the model to be updated, device 110 may send an indication of the model to be updated to device 120.

[0102] In some example embodiments, device 110 may determine the periodicity of notifications associated with a threshold crossing. Device 110 may determine a threshold direction of the threshold crossing. In this case, in some example embodiments, device 110 may determine whether a model needs to be updated based on the periodicity of the notifications and the threshold direction.

[0103] In some example embodiments, device 110 may determine whether the periodicity reaches a threshold periodicity. Device 110 may determine whether the threshold direction satisfies a configured direction. In this case, in some example embodiments, based on determining that the periodicity of the notification reaches the threshold periodicity and the threshold direction satisfies the configured direction, device 110 may determine that the model needs to be updated.

[0104] In some example embodiments, the parameter may include a key performance indicator. In this case, the device 110 may determine a performance measurement to be monitored for the key performance indicator.

[0105] In some example embodiments, the monitoring information may include monitored performance measurements. In this case, device 110 may determine key performance indicators based on the monitored performance measurements.

[0106] Figure 8 FIG. 8 is a flowchart showing an example method 800 implemented at a second device according to some example embodiments of the present disclosure. Figure 1A Method 800 is described from the perspective of device 120 in FIG.

[0107] At block 810, device 120 receives a configuration from device 130, from a third device, indicating a data event associated with a model to be monitored. The data event indicates: a parameter to be monitored and a threshold associated with the parameter. The data event may include one or more of: a threshold direction, a hysteresis value, or a threshold periodicity of notifications associated with the crossing of a threshold. In some example embodiments, the parameter may include a performance measure. Alternatively or additionally, the parameter may include a key performance indicator. In some example embodiments, the parameter includes at least one of a performance measure or a key performance indicator.

[0108] At block 820, device 120 sends a configuration indicating a data event to device 110. In some example embodiments, device 120 may receive an indication of a model to be updated from device 110. In this case, device 120 may initiate retraining of the model. Device 120 may receive updated training results of the model from a fourth device. Alternatively, device 120 may receive updated training results of the model from device 110.

[0109] Fig. 9 A flowchart of an example method 900 implemented at a third device according to some example embodiments of the present disclosure is shown. For the purpose of discussion, Figure 1A and Figure 1B Method 900 is described from the perspective of device 130 in FIG.

[0110] At block 910, device 130 sends a configuration indicating a data event associated with a model to be monitored to device 120, the data event indicating: a parameter to be monitored and a threshold associated with the parameter. The data event indicates: a parameter to be monitored and a threshold associated with the parameter. The data event may include one or more of: a threshold direction, a hysteresis value, or a threshold periodicity of notifications associated with the crossing of a threshold. In some example embodiments, the parameter may include a performance measure. Alternatively or additionally, the parameter may include a key performance indicator. In some example embodiments, the parameter includes at least one of a performance measure or a key performance indicator.

[0111] In some example embodiments, at block 920, device 130 may receive the results of the retraining of the model. In some example embodiments, device 130 may reconfigure the data event to device 120. Example devices, equipment, and media

[0112] In some example embodiments, a first device (eg, Figure 1A and Figure 1B The device 110 in the embodiment of the present invention may include components for performing the corresponding operations of the method 700. The components may be implemented in any suitable form. For example, the components may be implemented in a circuit or a software module. In some example embodiments, the first device may be implemented as or included in the device 110. Alternatively, the first device may be implemented as or included in the device 120.

[0113] In some example embodiments, the first device includes: a component for obtaining, at a device, a configuration indicating a data event for data associated with a model to be monitored, the data event indicating: a parameter to be monitored and a threshold associated with the parameter; a component for triggering monitoring of the parameter based on the configuration indicating the data event; and a component for detecting whether the value of the parameter satisfies a condition indicated in the configuration while monitoring the parameter.

[0114] In some example embodiments, means for obtaining a configuration indicative of a data event comprises means for receiving a configuration indicative of a data event from another device.

[0115] In some example embodiments, the first apparatus comprises means for updating the model based on determining that a value of the parameter satisfies a condition.

[0116] In some example embodiments, the first apparatus comprises means for triggering updating the model based on determining that a value of a parameter satisfies a condition.

[0117] In some example embodiments, the component for updating the model includes at least one of: a component for training the model; a component for retraining the model; a component for updating a hyperparameter of the model; a component for updating a configuration indicating data events associated with the model; or a component for updating an operating context of the model.

[0118] In some example embodiments, the parameter includes at least one of: a network control parameter, a network performance metric, a communication environment parameter, a performance measurement, or a key performance indicator.

[0119] In some example embodiments, the data event further includes at least one of: a threshold direction, a hysteresis value, or a threshold periodicity of notification associated with satisfying the condition.

[0120] In some example embodiments, the first apparatus comprises: means for sending updated training results or retraining results of the model to the second device.

[0121] In some example embodiments, components for detecting whether a value of a parameter satisfies a condition indicated in a configuration include: components for determining a periodicity of notifications associated with a threshold crossing; components for determining a threshold direction of the threshold crossing; and components for determining whether the value of the parameter satisfies the condition based on the notification periodicity and the threshold direction.

[0122] In some example embodiments, the first device includes: a component for determining whether the periodicity reaches a threshold periodicity; a component for determining whether the threshold direction satisfies a configuration direction indicated in the configuration; and a component for determining whether the value of the parameter satisfies a condition based on determining that the periodicity of the notification reaches the threshold periodicity and the threshold direction satisfies the configured direction.

[0123] In some example embodiments, the device includes an artificial intelligence machine learning (AIML) monitoring function, or the device includes an artificial intelligence machine learning producer having an artificial intelligence machine learning monitoring function.

[0124] In some example embodiments, a second device (eg, Figure 1A The device 120 in FIG. 1 may include components for performing the corresponding operations of method 800. The components may be implemented in any suitable form. For example, the components may be implemented in a circuit or a software module. The second device may be implemented as or included in the device 120 in FIG. 1 .

[0125] In some example embodiments, the second apparatus comprises: a component for receiving a configuration indicating a data event associated with a model to be monitored from a third device, the data event indicating: a parameter to be monitored and a threshold value associated with the parameter; and a component for sending a configuration indicating the data event to the third device.

[0126] In some example embodiments, the data event further includes at least one of: a threshold direction, a hysteresis value, or a threshold periodicity of notifications associated with the crossing of the threshold.

[0127] In some example embodiments, the second apparatus further comprises: a component for receiving an indication regarding updating the model from the first device; a component for initiating retraining of the model; and a component for receiving updated training results of the model from a fourth device.

[0128] In some example embodiments, the second apparatus further comprises: means for receiving updated training results of the model from the first device.

[0129] In some example embodiments, the parameter comprises at least one of a performance measure or a key performance indicator.

[0130] In some example embodiments, the second apparatus further comprises: means for sending updated training results of the model to a third device.

[0131] In some example embodiments, the second apparatus further comprises: means for performing other operations in some example embodiments of method 800 or device 120. In some example embodiments, the means comprises: at least one processor; and at least one memory storing instructions that, when executed by the at least one processor, cause execution of the second apparatus.

[0132] In some example embodiments, a third device (eg, Figure 1A and Figure 1B The device 130 in the second apparatus may include components for performing the corresponding operations of the method 900. The components may be implemented in any suitable form. For example, the components may be implemented in a circuit or a software module. The second device may be implemented as or included in Figure 1A and Figure 1B In the device 130.

[0133] In some example embodiments, the third apparatus comprises: means for sending, at the device and to another device, a configuration indicating a data event for data associated with the model to be monitored, the data event indicating: a parameter to be monitored and a threshold associated with the parameter.

[0134] In some example embodiments, the parameter comprises at least one of: a network control parameter, a network performance metric, a communication environment parameter, a performance measurement, or a key performance indicator.

[0135] In some example embodiments, the data event further includes at least one of: a threshold direction, a hysteresis value, or a threshold periodicity of notification associated with satisfying the condition.

[0136] In some example embodiments, the third apparatus comprises: means for receiving an updated training result or a retraining result of the model from another device.

[0137] In some example embodiments, the device includes an artificial intelligence machine learning consumer.

[0138] Fig.10 1 is a simplified block diagram of a device 1000 suitable for implementing an example embodiment of the present disclosure. The device 1000 may be provided to implement a communication device, such as Figure 1A and Figure 1B The device 110 or the device 120 shown. As shown, the device 1000 includes one or more processors 1010, one or more memories 1020 coupled to the processor 1010, and one or more communication modules 1040 coupled to the processor 1010.

[0139] The communication module 1040 is used for two-way communication. The communication module 1040 has one or more communication interfaces to facilitate communication with one or more other modules or devices. The communication interface may represent any interface necessary for communicating with other network elements. In some example embodiments, the communication module 1040 may include at least one antenna.

[0140] As non-limiting examples, processor 1010 may be of any type suitable for a local technology network, and may include one or more of the following: a general purpose computer, a special purpose computer, a microprocessor, a digital signal processor (DSP), and a processor based on a multi-core processor architecture. Device 1000 may have multiple processors, such as application specific integrated circuit chips, which are time-slaved to a clock that synchronizes a master processor.

[0141] The memory 1020 may include one or more non-volatile memories and one or more volatile memories. Examples of non-volatile memories include, but are not limited to, read-only memory (ROM) 1024, electrically programmable read-only memory (EPROM), flash memory, hard disk, compact disk (CD), digital video disk (DVD), optical disk, laser disk, and other magnetic storage and / or optical storage. Examples of volatile memories include, but are not limited to, random access memory (RAM) 1022 and other volatile memories that do not persist during a power outage.

[0142] The computer program 1030 includes computer executable instructions executed by the associated processor 1010. The instructions of the program 1030 may include instructions for performing the operations / actions of some example embodiments of the present disclosure. The program 1030 may be stored in a memory, such as ROM 1024. The processor 1010 may perform any suitable actions and processes by loading the program 1030 into the RAM 1022.

[0143] The exemplary embodiments of the present disclosure may be implemented by the program 1030 so that the device 1000 may execute the Figures 2A to 9 Any process of the present disclosure discussed. The exemplary embodiments of the present disclosure may also be implemented by hardware or a combination of software and hardware.

[0144] In some example embodiments, the program 1030 may be tangibly embodied in a computer-readable medium that may be included in the device 1000 (such as in the memory 1020) or other storage device accessible by the device 1000. The device 1000 may load the program 1030 from the computer-readable medium to the RAM 1022 for execution. In some example embodiments, the computer-readable medium may include any type of non-transitory storage medium, such as ROM, EPROM, flash memory, hard disk, CD, DVD, etc. As used herein, the term "non-transitory" is a limitation of the medium itself (i.e., tangible rather than a signal), not a limitation on data storage persistence (e.g., RAM vs. ROM).

[0145] Fig.11 An example of a computer readable medium 1100 is shown, which may be in the form of a CD, DVD, or other optical storage disk. The computer readable medium 1100 has a program 1030 stored thereon.

[0146] Generally, various embodiments of the present disclosure may be implemented in hardware or dedicated circuits, software, logic, or any combination thereof. Some aspects may be implemented in hardware, and other aspects may be implemented in firmware or software that may be executed by a controller, microprocessor, or other computing device. Although various aspects of the embodiments of the present disclosure are shown and described as block diagrams, flow charts, or using some other graphical representations, it should be understood that the blocks, devices, systems, techniques, or methods described herein may be implemented in hardware, software, firmware, dedicated circuits or logic, general hardware or controllers or other computing devices, or some combination thereof in non-limiting examples.

[0147] Some example embodiments of the present disclosure also provide at least one computer program product tangibly stored on a computer-readable medium such as a non-transitory computer-readable medium. The computer program product includes computer-executable instructions, such as those included in a program module, executed in a device on a target physical or virtual processor to perform any of the methods described above. Typically, a program module includes routines, programs, libraries, objects, classes, components, data structures, etc. that perform specific tasks or implement specific abstract data types. The functions of the program modules can be combined or split between program modules as needed in various embodiments. The machine executable instructions for the program modules can be executed in local or distributed devices. In distributed devices, program modules can be located in local and remote storage media.

[0148] The program code for executing the method of the present disclosure can be written in any combination of one or more programming languages. The program code can be provided to a processor or controller of a general-purpose computer, a special-purpose computer or other programmable data processing device so that the program code, when executed by the processor or controller, enables the functions / operations specified in the flow chart and / or block diagram to be implemented. The program code can be executed entirely on the machine, partially on the machine, as an independent software package, partially on the machine and partially on a remote machine, or entirely on a remote machine or server.

[0149] In the context of the present disclosure, computer program codes or related data may be carried by any suitable carrier to enable a device, apparatus or processor to perform various processes and operations as described above. Examples of carriers include signals, computer readable media, etc.

[0150] The computer readable medium may be a computer readable signal medium or a computer readable storage medium. The computer readable medium may include, but is not limited to, an electronic, magnetic, optical, electromagnetic, infrared or semiconductor system, device or apparatus, or any suitable combination of the foregoing. More specific examples of computer readable storage media would include an electrical connection with one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

[0151] In addition, although operations are depicted in a particular order, this should not be understood as requiring such operations to be performed in the particular order shown or in a sequential order, or performing all of the operations shown to achieve the desired result. In some cases, multitasking and parallel processing may be advantageous. Similarly, although several specific implementation details are included in the above discussion, these details should not be interpreted as limitations on the scope of the present disclosure, but should be interpreted as descriptions of features that may be specific to a particular embodiment. Unless explicitly stated, certain features described in the context of a separate embodiment may also be implemented in combination in a single embodiment. On the contrary, unless explicitly stated, the various features described in the context of a single embodiment may also be implemented in multiple embodiments individually or in any suitable sub-combination.

[0152] Although the disclosure has been described in language specific to structural features and / or methodological acts, it should be understood that the disclosure defined in the appended claims is not necessarily limited to the specific features or acts described above. Rather, the specific features and acts described above are disclosed as example forms of implementing the claims.< / datatype> < / datatype>

Claims

1. A device comprising: at least one processor; as well as at least one memory storing instructions that, when executed by the at least one processor, cause the apparatus to at least perform: Obtaining a configuration indicating a data event for data associated with a model to be monitored, the data event indicating: a parameter to be monitored and a threshold value associated with the parameter; triggering monitoring of the parameter based on the configuration indicating the data event; as well as While monitoring the parameter, it is detected whether the value of the parameter meets the condition indicated in the configuration.

2. The apparatus of claim 1 , wherein obtaining the configuration indicating the data event comprises: The configuration indicative of the data event is received from another device.

3. The apparatus of claim 1 , wherein the apparatus is configured to perform: Based on determining that the value of the parameter satisfies the condition, the model is updated.

4. The apparatus of claim 1 , wherein the apparatus is configured to perform: Based on determining that the value of the parameter satisfies the condition, an update of the model is triggered.

5. The apparatus according to claim 3 or 4, wherein updating the model comprises at least one of the following: training the model; retraining the model; updating hyperparameters of the model; updating the configuration indicating the data events associated with the model; or Updates the execution context of the model.

6. The device according to any one of claims 1 to 5, wherein the parameter comprises at least one of the following: Network control parameters, Network performance metrics, Communication environment parameters, Performance measurement, or Key performance indicators.

7. The device according to any one of claims 1 to 6, wherein the data event further comprises at least one of the following: Threshold direction, Hysteresis value, or A threshold periodicity for notifications associated with satisfying the condition.

8. The apparatus according to any one of claims 1 to 7, wherein the apparatus is further configured to perform: Send updated training or retraining results of a model to another device.

9. The device according to any one of claims 1 to 8, wherein detecting whether the value of the parameter satisfies the condition indicated in the configuration comprises: determining a periodicity of notifications associated with threshold crossings; determining a threshold direction in which the threshold is crossed; as well as Based on the notification periodicity and the threshold direction, it is determined whether the value of the parameter satisfies the condition.

10. The apparatus of claim 9, wherein the apparatus is further configured to perform: determining whether the periodicity reaches the threshold periodicity; determining whether the threshold direction satisfies a configuration direction indicated in the configuration; and Based on determining that the periodicity of the notifications reaches the threshold periodicity and the threshold direction satisfies a configured direction, determining whether the value of the parameter satisfies the condition.

11. The device according to any one of claims 1 to 10, wherein the device comprises an artificial intelligence machine learning (AIML) monitoring function, or The device includes an artificial intelligence machine learning producer with artificial intelligence machine learning monitoring function.

12. A device comprising: at least one processor; as well as at least one memory storing instructions that, when executed by the at least one processor, cause the apparatus to at least: selecting a configuration indicating a data event for data associated with a model to be monitored, the data event indicating: a parameter to be monitored and a threshold value associated with the parameter; as well as The configuration is sent to another device.

13. The apparatus according to claim 12, wherein the parameter comprises at least one of the following: Network control parameters, Network performance metrics, Communication environment parameters, Performance measurement, or Key performance indicators.

14. The apparatus according to claim 12 or 13, wherein: The data event also includes at least one of the following: Threshold direction, Hysteresis value, or A threshold periodicity for notifications associated with satisfying the condition.

15. The apparatus according to any one of claims 12 to 14, wherein the apparatus is further configured to perform: An updated training result or a retraining result of the model is received from the other device.

16. A device according to any one of claims 12 to 15, wherein the device includes an artificial intelligence machine learning consumer.

17. A method comprising: Obtaining, at a device, a configuration indicating a data event for data associated with a model to be monitored, the data event indicating: a parameter to be monitored and a threshold value associated with the parameter; triggering monitoring of the parameter based on the configuration indicating the data event; as well as While monitoring the parameter, it is detected whether the value of the parameter meets the condition indicated in the configuration.

18. The method of claim 17, wherein obtaining the configuration indicative of the data event comprises: The configuration indicative of the data event is received from another device.

19. The method according to claim 17, further comprising: Based on determining that the value of the parameter satisfies the condition, the model is updated.

20. The method of claim 17, further comprising: Based on determining that the value of the parameter satisfies the condition, updating the model is triggered.

21. The method of claim 19 or 20, wherein updating the model comprises at least one of: training the model; retraining the model; updating hyperparameters of the model; updating the configuration indicating the data events associated with the model; or Updates the execution context of the model.

22. The method according to any one of claims 17 to 21, wherein the parameters include at least one of the following: Network control parameters, Network performance metrics, Communication environment parameters, Performance measurement, or Key performance indicators.

23. The method according to any one of claims 17 to 22, wherein the data event further comprises at least one of the following: Threshold direction, Hysteresis value, or A threshold periodicity for notifications associated with satisfying the condition.

24. The method according to any one of claims 17 to 23, further comprising: Send updated training or retraining results of a model to another device.

25. The method according to any one of claims 17 to 24, wherein detecting whether the value of the parameter satisfies the condition indicated in the configuration comprises: determining a periodicity of notifications associated with threshold crossings; determining a threshold direction in which the threshold is crossed; as well as Based on the notification periodicity and the threshold direction, it is determined whether the value of the parameter satisfies the condition.

26. The method according to claim 25, further comprising: determining whether the periodicity reaches the threshold periodicity; determining whether the threshold direction satisfies a configuration direction indicated in the configuration; as well as Based on determining that the periodicity of the notifications reaches the threshold periodicity and the threshold direction satisfies a configured direction, determining whether the value of the parameter satisfies the condition.

27. A method according to any one of claims 17 to 26, wherein the device comprises an artificial intelligence machine learning (AIML) monitoring function, or The device includes an artificial intelligence machine learning producer with artificial intelligence machine learning monitoring function.

28. A method comprising: A configuration indicating a data event for data associated with a model to be monitored is sent at a device to another device, the data event indicating: a parameter to be monitored and a threshold value associated with the parameter.

29. The method of claim 28, wherein the parameters include at least one of the following: Network control parameters, Network performance metrics, Communication environment parameters, Performance measurement, or Key performance indicators.

30. The method according to claim 28 or 29, wherein the data event further comprises at least one of the following: Threshold direction, Hysteresis value, or A threshold periodicity for notifications associated with satisfying the condition.

31. The method according to any one of claims 28 to 30, further comprising: An updated training result or a retraining result of the model is received from the other device.

32. A method according to any one of claims 28 to 31, wherein the device includes an artificial intelligence machine learning consumer.

33. An apparatus comprising: means for obtaining a configuration indicating a data event for data associated with a model to be monitored, the data event indicating: a parameter to be monitored and a threshold value associated with the parameter; means for triggering monitoring of said parameter based on said configuration indicative of said data event; as well as A means for detecting, while monitoring the parameter, whether the value of the parameter satisfies the condition indicated in the configuration.

34. An apparatus comprising: Means for sending to another device a configuration indicating a data event for data associated with a model to be monitored, the data event indicating: a parameter to be monitored and a threshold value associated with the parameter.

35. A computer readable medium comprising instructions stored thereon, the instructions being used to cause an apparatus to at least perform the method of any one of claims 17 to 27 or the method of any one of claims 28 to 32.

Images