Enhanced support for multiple machine learning models in machine learning model service providers and consumers

Abstract

A Network Data Analytics function, NWDAF, may comprise a Model Training Logical Function, MTLF, and act as an ML model service provider for other network functions, NFs, acting as ML model service consumers. A model service consumer (210) may transmit a machine learning, ML, model subscription request (230) to a model service provider (220). The model service provider may determine a plurality of ML models to use in fulfilling the ML model subscription request (230) and transmit a notification (235, 240) identifying the plurality of ML models as corresponding to the ML model subscription request (230) to the model service consumer.

Description

[0001] ENHANCED SUPPORT FOR MULTIPLE MACHINE LEARNING MODELS IN MACHINE LEARNING MODEL SERVICE PROVIDERS AND CONSUMERS

[0002] TECHNICAL FIELD

[0003] The present disclosure generally relates to the technical field of wireless communication networks and, more particularly, to enhancing Machine Learning (ML) service providers within such networks to support multiple ML models.

[0004] BACKGROUND

[0005] In wireless communication networks, a Network Data Analytics function (NWDAF) may comprise a Model Training Logical Function (MTLF) or similar function. The MTLF provides ML model provisioning and ML model training services to one or more Network Function (NF) consumers. The NWDAF supports a set of standard messages by which an NF consumer can request such services.

[0006] 3GPP Technical Specification TS 23.288 V18.5.0 provides procedure and high-level service description for ML Model Provisioning and ML Model Training Services. Therein, the ML Model Provisioning and ML Model Training Services are provided by an NWDAF (comprising MTLF) for providing / training ML Model based on request from NF consumer.

[0007] SUMMARY

[0008] Although traditional networks allow an NF consumer to request ML model services from an NWDAF, a problem arises when more than one model may bear relevance to a request. For example, it is unclear how to map the parameters of a consumer request to more than one model. It is also unclear how a service provider (e.g., an NWDAF or MTLF) would decide to provide and / or train more than one model based on a single request from a consumer.

[0009] Embodiments of the present disclosure propose enhancements over traditional services in order to overcome one or more of the problems discussed herein. As will be explained in greater detail below, one or more such embodiments enhance any of the Nnwdaf_MLModelProvision, Nnwdaf_MLModellnfo, Nnwdaf_MLModelTraining, and Nnwdaf_MLModelTraininglnfo services or any combination thereof.

[0010] Among other things, particular embodiments include enhancements to the parameters of a request from an NF consumer for clearer mapping of the requirements of multiple ML Models, whether the request comprises a single Analytics ID or plurality of Analytics IDs.

[0011] Particular embodiments additionally or alternatively enhance operations at the model service provider, like an NWDAF and / or MTLF, establishing a way to decide whether to involve multiple ML Models based on a single consumer request.

[0012] According to the invention there is provided a method, implemented by a network node acting as a model service provider in a wireless communication network. The method comprises receiving a machine learning, ML, model subscription request from a model service consumer and determining a plurality of ML models to use in fulfilling the ML model subscription request.

[0013] There is also provided a network node operable to act as a model service provider of a wireless communication network. The network node is configured to receive a machine learning, ML, model subscription request from a model service consumer and determine a plurality of ML models to use in fulfilling the ML model subscription request.

[0014] Further, there is provided a method implemented by a network node acting as a model service consumer in a wireless communication network. The method comprises transmitting a machine learning, ML, model subscription request to a model service provider and receiving, from the model service provider, a model provisioning notification identifying more than one ML model as corresponding to the ML model subscription request.

[0015] There is also provided a network node operable to act as a model service consumer of a wireless communication network. The network node is configured to transmit a machine learning, ML, model subscription request to a model service provider and receive from the model service provider, a notification identifying more than one ML model as corresponding to the ML model subscription request.

[0016] The above network nodes may comprise interface circuitry and processing circuitry communicatively connected to the interface circuitry, wherein the processing circuitry is configured to perform the aforementioned methods.

[0017] There are further provided corresponding computer programs which, when executed on processing circuitry of a network node, cause the processing circuitry to carry out the respective above methods.

[0018] The model service provider may be an NWDAF, particularly an NWDAF comprising an MTLF or similar function, or any other network function or network node implementing or comprising an MTLF or similar function.

[0019] The model service consumer may be an NWDAF, particularly an NWDAF comprising an Analytics Logical Function, AnLF, or similar function, or any other network function or network node implementing or comprising an AnLF or similar function.

[0020] BRIEF DESCRIPTION OF THE DRAWINGS

[0021] Aspects of the present disclosure are illustrated by way of example and are not limited by the accompanying figures with like references indicating like elements.

[0022] FIG. 1 is a schematic block diagram illustrating an example wireless communication network, according to one or more embodiments of the present disclosure.

[0023] FIG. 2 is a schematic block diagram illustrating an example NWDAF, according to one or more embodiments of the present disclosure. FIG. 3 is a flow diagram illustrating an example method implemented by a network node acting as a model service provider, according to one or more embodiments of the present disclosure.

[0024] FIG. 4 is a flow diagram illustrating an example method implemented by a network node acting as a model service consumer, according to one or more embodiments of the present disclosure.

[0025] FIGS. 5-8 are signaling diagrams illustrating different examples of signaling exchanged between a model service provider and a model service consumer, according to various embodiments of the present disclosure.

[0026] FIG. 9 is a schematic block diagram illustrating an example network node, according to one or more embodiments of the present disclosure.

[0027] DETAILED DESCRIPTION

[0028] In the examples provided herein, examples may be given that are commonly associated with a particular generation of Third Generation Partnership Project (3GPP) wireless networks. For example, the terms NWDAF and / or MTLF are 5G examples of an ML model service provider. These specific examples are given solely for purposes of illustration and should not be interpreted to exclude other network elements, whether presently known or to be developed in the future. Indeed, the embodiments described herein may be applied to any generation of 3GPP network, or other types of networks unless otherwise specified.

[0029] FIG. 1 illustrates an example reference architecture of such a wireless communication network 10. The communication network 10 comprises a Radio Access Network (RAN) 20 and a core network 30 employing a service-based architecture. The RAN 20 and the core network 30, when operated by the same operator, are sometimes collectively referred to as a Public Land Mobile Network (PLMN). In other embodiments, the RAN 20 and the core network 30 may be a Standalone Non-Public Network (SNPN) or Public Mobile Network (PMN).

[0030] The RAN 20 comprises one or more network nodes that are configured to provide radio access to one or more UEs 110 operating within a coverage area of the PLMN. To do so, the RAN supports one or more Radio Access Technologies (RATs). In the context of certain 5G networks, a network node that provides radio access to a UE 110 may be referred to as a gNodeB (gNB) 25. The RAT used by 5G networks is typically referred to as New Radio (NR).

[0031] The core network 30 provides a connection between the RAN 20 and one or more data networks (DNs) 90 (e.g., the Internet). A breakout to the DN 90 may occur at a Home PLMN (HPLMN) or a Visited PLMN (VPLMN). Whether the HPLMN or a VPLMN provides the breakout may depend respectively on whether or not the UE 110 is roaming. The UEs 110 receive one or more network services from the core network 30, e.g., so that the UE 110 may exchange data with the DN 90. The core network 30 comprises a plurality of network functions (NFs). These NFs may be in either the user plane 33 or the control plane 37 of the core network 30. The user plane 33 (sometimes referred to as the data plane) typically carries user data traffic. The control plane 37 typically carries signaling traffic (e.g., control packets).

[0032] In this example, the NFs of the user plane 33 comprise a User Plane Function (UPF) 35. There may be more than one UPF in the user plane of a given network, wherein the UPFs may be interconnected. The NFs of the control plane 37 comprise an Access and Mobility Management Function (AMF) 40, a Session Communication Proxy (SCP) 42, a Session Management Function (SMF) 45, a Network Slice Specific Authentication and Authorization Function (NSSAAF) 47, a Policy Control Function (PCF) 50, a Unified Data Management (UDM) function 55, a Unified Data Repository (UDR) function 57, an Authentication Server function (AUSF) 60, a Network Data Analytics Function (NWDAF) 65, a Network Exposure Function (NEF) 70, a Network Repository Function (NRF) 75, and a Network Slice Selection Function (NSSF) 80. The control plane 37 of the core network 30 also includes an Application Function (AF) 85.

[0033] Each NF comprises a logical entity that resides on one or more core network nodes. Each core network node may be implemented using computing hardware, such as one or more processors, memory, network interfaces, or a combination thereof. The NFs communicate with one another using predefined interfaces. Some of the interfaces are referred to by standardized reference points within the network, whereas other interfaces are simply named.

[0034] N1 is a reference point between a UE 110 and the AMF 40. N2 is a reference point between the RAN 20 and the AMF 40. The N3 is a reference point between the RAN 20 and the UPF 35. N4 is a reference point between the SMF 45 and the UPF 35. N6 is a reference point between the UPF 35 and the DN 90. N9 is a reference point between UPFs 35. Several of the NFs expose a service-based interface named after them in the format Nxxx, wherein xxx is the name of the NF. For example, the NEF 70 provides an Nnef interface, the NRF 75 provides an Nnrf interface, and so on.

[0035] Generally speaking, the AMF 40 typically manages registration, connection, and mobility management between the network 10 and the UE 110. The AMF 40 also typically mediates communication between the UE 110 and the SMF 45.

[0036] The SMF 45 is typically responsible for session establishment, modification, and release, including selection and control of UPF entities, maintaining the topology of the involved Packet Data Unit (PDU) Session Anchor (PSAs), and establishing and releasing the tunnel between the RAN 20 and the UPF 35 and between UPFs 35. In this context, a PSA is a UPF 35 that terminates the N6 interface of a PDU session within the core network 30. The SMF 45 typically also configures traffic forwarding at the UPF 35. The SMF 45 also typically interacts with the UPF 35 using Packet Forwarding Control Protocol (PFCP) procedures. The NRF 75 typically acts as a central registry, holding information about other NFs in the network 10 and sharing information with other NFs. In this regard, an NF may register its supported services in a profile with the NRF 75 and may discover other NFs in the network 10, and their supported services, by interfacing with the NRF 75.

[0037] The NWDAF 65 typically analyzes data collected from NFs and UEs in the network and publishes the results to subscribing data analytics consumers. NFs supporting data analytics may implement an event exposure service-based Application Programming Interface (API). The NWDAF 65 typically subscribes to uniquely identified events published by NFs in order to acquire data that it then analyzes and exposes to its subscribing consumers. These consumers may then use the analyzed data to modify the operation and / or configuration of the network 10, possibly even in real time.

[0038] As shown in FIG. 2, the NWDAF 65 may comprise an Analytics Logical Function (AnLF) 66 and / or an MTLF 67. The AnLF 66 is traditionally responsible for collecting analytics requests and sending responses to service consumers. The MTLF 67 is traditionally responsible for training and deploying models. Thus, an NWDAF 65 may be either a model service consumer, a model service provider, or both, depending on the embodiment. Indeed, as will be explained further below, embodiments disclosed herein include examples in which a first NWDAF acting as a model service consumer may communicate with a second NWDAF acting as a model service provider. In many such embodiments, the first NWDAF may comprise an AnLF 66 and the second NWDAF may comprise an MTLF 67.

[0039] Generally, an NWDAF or any other network function or network node comprising an MTLF or providing MTLF functionality or service can serve as example of a model service provider, and an NWDAF or any other network function or network node comprising an AnLF or providing AnLF functionality or service can serve as example of a model service consumer.

[0040] A network function or network node implementing or comprising an MTLF and / or AnLF and / or similar functionality, e.g. an NWDAF, may register itself to the NRF including, in its NF profile, a (list of) Analytics ID(s).

[0041] In case of MTLF or similar functionality, it may further include the ML model provisioning services (e.g. Nnwdaf_MLModel Provision, Nnwdaf_MLModellnfo) as one of the supported services during the registration in NRF when trained ML models are available for one or more Analytics ID(s). It may provide to the NRF a (list of) Analytics ID(s) corresponding to the trained ML models and possibly the ML Model Filter Information for the trained ML model per Analytics I D(s), if available. It may be foreseen that only the S-NSSAI(s) and Area(s) of Interest from the ML Model Filter Information for the trained ML model per Analytics ID(s) may be registered into the NRF. For each Analytics ID, if ML Model interoperability is supported, the registration to the NRF may also include an ML Model Interoperability indicator. Thereby the services of an NWDAF or other network function or node providing MTLF or similar services become discoverable for other network functions or network nodes, like a network function or network node implementing or comprising an AnLF (e.g. an NWDAF).

[0042] As discussed above, embodiments generally relate to situations in which more than one model may bear relevance to a given subscription request. Accordingly, and as will be explained in greater detail below, particular embodiments of the present disclosure include a method 300 performed by a network node, e.g., as illustrated in FIG. 3. The method 300 comprises receiving an ML model subscription request from a model service consumer 210 (block 310). The method 300 further comprises determining a number of machine learning models to use in fulfilling the ML model subscription request (block 320). The network node performing this method 300 may also be denoted a model service provider.

[0043] Correspondingly, embodiments of the present disclosure include a method 400 performed by a network node, e.g., as illustrated in FIG. 4. The method 400 comprises transmitting an ML model subscription request to a model service provider (block 410). The method 400 further comprises receiving, from the model service provider, a notification identifying more than one machine learning model as corresponding to the ML model subscription request (block 420). The network node performing this method 400 may also be denoted a model service consumer.

[0044] FIG. 5 is a signaling diagram illustrating example signaling exchanged between a model service consumer 210 and model service provider 220, either or both of which may be an NWDAF 65. For example, the model service consumer 210 may be a first NWDAF comprising an AnLF 66. The first NWDAF may, in some embodiments, seek to be notified when ML model information becomes available at the model service provider 220. The model service provider 220 may be a second NWDAF comprising an MTLF 67. Alternatively, the AnLF 66 and MTLF 67 may be comprised in the same NWDAF 65, depending on the embodiment.

[0045] The signaling of FIG. 5 supports an ML model analytics subscription procedure. The procedure may be used either to subscribe or unsubscribe to analytics produced by an ML model. The model service consumer 210 may send a subscription request 230 to the model service provider 220. Correspondingly, the model service provider 220 may send a subscription response 235 to the model service consumer 210 in response to the subscription request 210.

[0046] The subscription request 230 may be a subscribe message (e.g., Nnwdaf_MLModelProvision_Subscribe) for subscribing to analytics. Alternatively, the subscription message 210 may be an unsubscribe message (e.g., Nnwdaf_MLModelProvision_Unsubscribe) for unsubscribing to analytics. For example, the model service consumer 210 may be an NWDAF 65 comprising an AnLF 66. The NWDAF 65 subscribes to, modifies, or cancels subscription for one or more trained ML Models associated with an Analytics ID by invoking the Nnwdaf_MLModelProvision_Subscribe and / or Nnwdaf_MLModelProvision_Unsubscribe service operations. The model service consumer 210 may, in some embodiments, indicate its support for multiple ML models if available in the subscription request 230. The model service consumer 210 may additionally or alternatively include, in the subscription request 230, one or more Analytics IDs. For each of the AnalyticslD(s) provided by the model service consumer 210, the model service provider 220 (e.g., another NWDAF comprising an MTLF 67) may make one or more determinations.

[0047] In some embodiments, the model service provider 220 may determine whether one or more existing trained ML Models can be provided to the model service consumer 210. Additionally or alternatively, the model service provider 220 may determine whether triggering further training for one or more existing trained ML models is needed for the subscription action. Additionally or alternatively, the model service provider 220 may determine whether multiple ML models are needed based on the request and / or based on a local configuration.

[0048] If the model service provider 220 determines that further training is needed, the model service provider 220 may initiate data collection from one or more NFs, (e.g., an AMF, a Data Collection Coordination Function (DCCF), an Analytics Data Repository Function (ADRF)), UE Applications (e.g., via AFs) or Operations, Administration and Maintenance (OAM) to generate the ML model. It will be understood that selection of the NF(s) from which training data collection is initiated may depend on the parameters of the subscription request, e.g. the AnalyticslD(s), based on which it can be determined what types of data are required.

[0049] If the subscription request is for a subscription modification or subscription cancelation, the model service consumer 210 may include an identifier (e.g., a Subscription Correlation ID) of a subscription to be modified, e.g., in the Nnwdaf_MLModelProvision_Subscribe message.

[0050] If the model service consumer 210 subscribes to a combination (or multiple combinations) of trained ML models associated with an AnalyticsID, the model service provider 220 notifies the model service consumer 210 of how model provisioning has been performed using an ML model provisioning notification 240. The ML model provisioning notification 240 may comprise one or more tuples of unique ML Model identifiers, ML Model Information associated with each Analytics ID requested by the model service consumer 210, and a subgroup ID (e.g., an identifier of a subset of UEs) for each of the ML Models per Analytics ID.

[0051] The ML model provisioning notification 240 may invoke a Nnwdaf_MLModelProvision_Notify service operation executing on the model service consumer 210, for example. The model service provider 220 may additionally or alternatively invoke the Nnwdaf_MLModelProvision_Notify service operation to notify an available retrained ML model when the model service provider 220 determines that the previously provided trained ML Model required retraining.

[0052] When the subscription request 230 is for a subscription modification, the model service provider 220 may provide either a new trained ML model different from the previously provided one, or a retrained ML model (e.g., by invoking the Nnwdaf_MLModelProvision_Notify service operation).

[0053] The subscription request 230 may invoke a Nnwdaf_MLModelProvision_Subscribe or Nnwdaf_MLModelProvision_Unsubscribe service operation of the model service provider 220. In particular, the subscription request 230 may, e.g., subscribe to an ML model of the model service provider 220. In this regard, the subscription request 230 may comprise one or more Analytics IDs, a Notification Target Address, and / or a Notification Correlation ID.

[0054] In some embodiments, the subscription request 230 may further comprise a Subscription Correlation ID (when requesting modification of an ML model subscription), ML Model Filter Information to indicate the conditions for which ML model for the analytics is requested per Analytics ID and / or a Target of ML Model Reporting to indicate one or more objects for which the ML model is requested. The target of ML reporting may indicate, for each ML model, one or more specific UEs or one or more groups of UEs. A group of UEs may, for example, be a subset of a UE group. In some embodiments, a UE group may be specified by a sub-group ID. Indeed, any one or more of the UEs 110 (e.g., all UEs) may be indicated, in some embodiments.

[0055] Other parameters included in the subscription request 230 may include one or more of the following:

[0056] - NF consumer information;

[0057] - requested representative ratio, indicating a minimum percentage of analytics objects, like UEs, in a group whose data is a non-empty set and can be used in the model training;

[0058] - ML Model Reporting Information (including, e.g., an ML Model Target Period indicating a time interval for which an ML model is requested);

[0059] - Expiry time, indicating when / whether the subscription can expire, e.g. based on the operator's policy;

[0060] - Use case context, indicating the context of use of the analytics;

[0061] - Inference Input Data information, containing information about settings that are expected to be used during inferences, e.g. input data that are expected to be used;

[0062] - indication of support for multiple ML models;

[0063] - multiple ML models Filter Information, to indicate the conditions for which multiple ML models are requested for each of the Analytics IDs;

[0064] - ML Model Interoperability Information, relating to model-specific properties like model file format, (execution) environment etc.;

[0065] - time when model is needed;

[0066] - ML Model Monitoring Information (including e.g. ML Model metric, ML model monitoring reporting mode, ML Model Accuracy Threshold, DataSetTag and ADRF ID, ML Model Identifier). When the subscription is accepted, a Subscription Correlation ID useful for management of the subscription may be included in the subscription response 235. The subscription response 235 may additionally or alternatively include an expiry time (e.g., if the subscription can expire based on the operator's policy).

[0067] The ML model provisioning notification 240 may invoke an Nnwdaf_MLModelProvision_Notify service operation of the model service consumer 210. The ML model provisioning notification 240 may be sent by the model service provider 220 to notify the model service consumer 210 that subscribed of ML model information. The ML model provisioning notification 240 may comprise Notification Correlation Information, an Analytics ID, one or more unique ML Model identifiers, ML Model Information, and / or a sub-group ID identifying a subset of UEs for each of the ML Models per Analytics ID. In some embodiments, the ML model provisioning notification 240 may further include ML Model Accuracy Information. Although not shown, in some embodiments, the model service consumer 210 may respond to the ML model provisioning notification by indicating a result of the operation (e.g., success, failure).

[0068] In some embodiments, the model service consumer 210 may request ML model information from the model service provider 220, e.g., by sending an ML model information request 290 as shown in FIG. 6. The ML model information request 290 may invoke a Nnwdaf_MLModellnfo_Request service operation of the model service provider. The ML model information request may comprise one or more Analytics IDs. In some embodiments, the ML model information request further comprises ML Model Filter Information to indicate one or more conditions for which ML model for the analytics is requested (e.g., per Analytics ID). In some embodiments, the ML model information request further comprises a Target of ML Model Reporting to indicate one or more objects for which the ML model is requested (e.g. specific UEs, a group of UE(s), subsets of UEs, etc., as described above). In some embodiments, the ML model information request further comprises NF consumer information, a requested representative ratio, ML Model Reporting Information (including, e.g., an ML Model Target Period), Use case context, Inference Input Data Information, indication of support for multiple ML models, multiple ML models Filter Information to indicate the conditions for which multiple ML models are requested for each of the Analytics IDs, ML Model Interoperability Information, and / or ML Model Accuracy Monitoring Information.

[0069] In response to the ML model information request, the model service consumer 210 may receive an ML model information response 295 from the model service provider 220 including one or more Analytics IDs, one or more tuples of unique ML Model identifier and ML Model Information, and a sub-group ID (e.g., identifying a subset of UEs) for each of the ML Models per Analytics ID. In some embodiments, the ML model information response 295 also includes ML Model Accuracy Information. As noted above, in some embodiments, the model service provider 220 may decide to train an ML model in order to fulfill a subscription request 230. FIG. 7 is a signaling diagram illustrating an example of such an embodiment.

[0070] The signaling of FIG. 7 supports an ML model analytics subscription procedure that triggers ML model training. In this example, the model service consumer 210 may send a subscription request 230 as discussed above to the model service provider 220. Correspondingly, the model service provider 220 may send a subscription response 235 (see above) to the model service consumer 210 in response to the subscription request 210. The model service provider 220 begins training an ML model to fulfill the subscription request 230 (block 250) and sends an ML model training notification 260 to the model service consumer 210 accordingly.

[0071] For example, in order to enable Federated Learning (FL), the model service consumer 210 may act as an FL Server NWDAF and subscribe to multiple NWDAFs comprising MTLFs. In such an example, the model service providers 220 act as FL Client NWDAFs, which are selected by the FL Server NWDAF.

[0072] The FL server NWDAF may use the subscription request 230 to check whether an NWDAF 65 can meet a ML model training requirement (e.g. ML Model Interoperability information, Analytics ID, Serving Area and / or availability of data and time). In such case, the FL server NWDAF may include an ML Preparation Flag. When the ML Preparation Flag is present in the subscription request 210, the model service provider 220 checks whether it can meet the ML model training requirement (e.g., ML Model Interoperability information, Analytics ID, Serving Area and / or availability of data and time) and I or can successfully download the model if the model information is provided.

[0073] The FL server NWDAF may use the subscription request 230 to get the Model Accuracy of the global ML Model calculated by the FL Client NWDAFs. In such cases, the service consumer NWDAF includes a Model Accuracy Check Flag. When the Model Accuracy Check Flag is present in the request, the service provider NWDAF uses the local training data as the testing dataset to calculate the Model Accuracy of the ML model provided by the service consumer NWDAF.

[0074] When the model service consumer 210 determines to further update the ML model, the model service consumer 210 may modify the subscription by invoking Nnwdaf_MLModelTraining_Subscribe service operation including Subscription Correlation ID with ML Model Information. Such determination may be made if the Model Accuracy is determined to be not sufficient for the analytics to be performed, e.g. when classification or analytics results of the testing dataset yield too many errors, wherein e.g. a tolerable percentage of errors may be set depending on the use case resp. analytics to be performed.

[0075] The model service provider 220 trains the ML model at step 250 by collecting new data or reusing already obtained data. If a ML model file is not provided in the subscription request 230, the model service provider 220 may retrieve the ML model using the information indicated in the subscription request 230.

[0076] The model service provider 220 may decide to train multiple ML models for the same AnalyticsID based on the indication from the model service consumer 210 that it supports multiple ML Models for the same AnalyticsID (or based on a local configuration).

[0077] When the model service provider 220 completes ML model training, the model service provider 220 notifies the model service consumer 210 by sending the ML model training notification 260. The ML model training notification may comprise ML Model Information of the updated ML Model.

[0078] To notify the model service consumer 210, the ML model training notification 260 may invoke a Nnwdaf_MLModelTraining_Notify service operation of the model service consumer 210.

[0079] If the model service provider 220 determines to terminate the ML model training (e.g., the model service provider 220 determines that it will not provide further notifications related to the subscription request 230), then the model service provider may notify the model service consumer 210 by sending a termination request (not shown). The termination request may indicate a reason for the termination, e.g., by including a cause code in the termination request. For example, the reason may include, e.g., an overload condition, unavailability of a FL process, or other reasons. The termination request may invoke a Nnwdaf_MLModelTraining_Notify service operation of the model service consumer 210.

[0080] If the model service provider determines that multiple ML Models will be trained for the same AnalyticsID, a tuple comprising an Analytics ID and one or more ML Model identifiers may be provided to the consumer. For each ML Model, ML model Information, ML Training Information, and / or Training Filter Information may be included in the notification to model service consumer 210.

[0081] For example, in order to enable FL, the model service provider 220 (acting as an FL Client NWDAF) may notify the model service consumer 210 (acting as FL Server NWDAF) of the local ML model information and provide a status report of FL training. The status report may include accuracy of local model and Training Input Data Information (e.g. areas covered by the data set, sampling ratio, maximum / minimum of value of each dimension, etc.). If a Model Accuracy Check Flag is present in the subscription request 230, the model service provider 220 may notify the model service consumer 210 (e.g., acting as FL Server NWDAF) of the Model Accuracy of the global ML Model.

[0082] In view of the above, the subscription request 230 may invoke a Nnwdaf_MLModelTraining_Subscribe service operation of the model service provider. In this regard, the subscription request 230 may subscribe to a ML model training service provided by the model service provider 220. The subscription request 230 may include an analytics identifier, ML Model Interoperability information, a Notification Target Address, and / or a Notification Correlation ID. In some embodiments, the subscribe request 230 further comprises one or more of the following:

[0083] Per Analytics ID, one or more ML Model ID(s) that identify the provided ML model(s);

[0084] ML Model Information per ML Model;

[0085] ML model file(s);

[0086] Subscription Correlation ID (in the case of modification of the ML Model Training subscription);

[0087] ML Training Information per ML Model (e.g., a data availability requirement, a time availability requirement);

[0088] ML Preparation Flag;

[0089] ML Model Accuracy Check Flag;

[0090] ML Correlation ID;

[0091] Training Filter Information per ML Model;

[0092] Target of Training Reporting;

[0093] Training Reporting Information;

[0094] Use case context;

[0095] Iteration round ID;

[0096] Expiry time.

[0097] When the subscription request 230 is accepted, the model service provider 220 may send a subscription response 235 to the model service consumer 210. The subscription response 235 may include a Subscription Correlation ID. The subscription correlation identifier may be used to readily manage a corresponding subscription. When the request is not accepted, the subscription response 235 may comprise an error response (e.g., a cause code indicating a failure). For example, an error response indicating that the model service provider 220 does not meet the ML training requirements, that ML training is not complete, that the model service provider 220 is experiencing an overload, that the model service provider 220 is not available for the FL process anymore, or other causes for the error may be indicated.

[0098] In some embodiments, the subscription response 235 further comprises an ML Correlation ID (e.g., confirming the subscription for this FL process).

[0099] The ML model training notification 260 may invoke a Nnwdaf_MLModelTraining_Notify service operation of the model service consumer 210. In this regard, the ML model training notification may serve to notify the model service consumer 210 of one or more trained ML models. The model service provider 220 may also use the ML model training notification to indicate to the model service consumer of ML model training termination.

[0100] The ML model training notification 260 may comprise notification correlation information indicating the Notification Correlation ID that has been assigned by the consumer during ML model training. In some embodiments, the ML model training notification 260 further comprises one or more of the following: an Analytics ID;

[0101] ML model Information per ML Model;

[0102] ML Training Information and / or Training Filter Information for each of the ML Models);

[0103] ML Correlation ID (e.g., in FL-related embodiments);

[0104] Corresponding Use case context;

[0105] Termination Request indicator indicating that the model service provider 220 requests to terminate the ML model training and / or that the model service provider 220 will not provide further notifications related to the subscription request 230. A cause code may be provided with this indicator indicating any of the cause for the termination (e.g. NWDAF overload, not available for the FL process anymore, etc.);

[0106] ML Model ID identifying the provisioned ML model;

[0107] Global ML Model Accuracy indicating the model accuracy of the global ML model. This may be calculated, e.g., by the node acting as FL Client NWDAF using the local training data as the testing dataset;

[0108] Status report of FL training. The status report may comprise a local ML Model metric and / or Training Input Data Information (e.g. areas covered by the data set, sampling ratio, maximum / minimum of value of each dimension, etc.). This information may be generated, e.g., by node acting as the FL Client NWDAF during FL procedure;

[0109] Delay Event Notification

[0110] Iteration round ID.

[0111] Although not shown, the model service consumer 210 may respond to the ML model training notification 260 with a result indication (e.g., success, failure).

[0112] As shown in FIG. 8, model training information may be obtained (e.g., by the model service consumer 210) by sending a model training information request 297 (e.g., to the model service provider 220). The model training information request 297 may invoke an Nnwdaf_MLModelTraininglnfo_Request service operation of the model service provider. The model training information request 297 may request information about ML model training. The model training information request 297 may comprise an Analytics ID and / or ML Model Interoperability information.

[0113] In some embodiments, the model training information request 297 may further comprise any one or more of the following:

[0114] ML Model ID(s) identifying the provided ML model(s);

[0115] ML Model Information per ML Model;

[0116] ML Model file(s); ML Training Information (e.g., a data availability requirement, a time availability requirement) per ML Model;

[0117] Training Reporting Information; an ML Preparation Flag; an ML Model Accuracy Check Flag; an ML Correlation ID; a Termination Request, e.g., when terminating the FL identified by the ML Correlation ID. An indication of a reason for the termination may additionally be provided, e.g. when an FL Client NWDAF is unselected by the FL Server NWDAF for the FL process, the FL process is suspended, etc.

[0118] Training Filter Information per ML Model.

[0119] Target of Training Reporting.

[0120] Use case context.

[0121] As shown at step 298, the model service provider 220 obtains the relevant information about the model training. In some embodiments, the model service provider 220 obtains the information about the model training based on an ML model file or other ML model information.

[0122] The model service provider 220 may respond to the model training information request with a model training information response 299. When the request is accepted, the model training information response may comprise with a result indicator relating to the request. When the request is not accepted, the model service provider 220 may respond with an error response. The error response may comprise a cause code (e.g. NWDAF does not meet the ML training requirements, ML training is not complete, NWDAF overload, not available for the FL process anymore, etc.).

[0123] The model training information response 299 may, in some embodiments, additionally include one or more of the following:

[0124] ML Model ID(s).

[0125] Analytics ID

[0126] ML model Information per ML Model

[0127] ML Training Information

[0128] Training Filter Information for each of the ML Models ML Correlation ID (e.g., for Federated Learning). Corresponding Use case context.

[0129] Global ML Model Accuracy indicating the model accuracy of the global ML model. The accuracy may be calculated, e.g., by the network node acting as FL Client NWDAF using the local training data as the testing dataset.

[0130] Status report of FL training. The status report may comprise a local ML model metric and Training Input Data Information (e.g., areas covered by the data set, sampling ratio, maximum / minimum of value of each dimension of data, etc.) generated by the network node acting as FL Client NWDAF during FL procedure.

[0131] Delay Event Notification; global ML model metric.

[0132] The model service consumer 210 and / or the model service provider 220 may be implemented by a network node 700 as schematically illustrated in the example of FIG. 9. The network node 700 of FIG. 9 comprises processing circuitry 710, memory circuitry 720, and interface circuitry 730. The processing circuitry 710 is communicatively coupled to the memory circuitry 720 and the interface circuitry 730, e.g., via a bus 704. The processing circuitry 710 may comprise one or more microprocessors, microcontrollers, hardware circuits, discrete logic circuits, hardware registers, digital signal processors (DSPs), field-programmable gate arrays (FPGAs), application-specific integrated circuits (ASICs), or a combination thereof. For example, the processing circuitry 710 may be programmable hardware capable of executing software instructions stored, e.g., as a machine-readable computer program 740 in the memory circuitry 720. The memory circuitry 720 of the various embodiments may comprise any non-transitory machine-readable media known in the art or that may be developed, whether volatile or nonvolatile, including but not limited to solid state media (e.g., SRAM, DRAM, DDRAM, ROM, PROM, EPROM, flash memory, solid state drive, etc.), removable storage devices (e.g., Secure Digital (SD) card, miniSD card, microSD card, memory stick, thumb-drive, USB flash drive, ROM cartridge, Universal Media Disc), fixed drive (e.g., magnetic hard disk drive), or the like, wholly or in any combination.

[0133] The interface circuitry 730 may be a controller hub configured to control the input and output (I / O) data paths of the network node 700. Such I / O data paths may include data paths for exchanging signals over a network. The interface circuitry 730 may be implemented as a unitary physical component, or as a plurality of physical components that are contiguously or separately arranged, any of which may be communicatively coupled to any other or may communicate with any other via the processing circuitry 710. For example, the interface circuitry 730 may comprise a transmitter 732 configured to send wireless communication signals and a receiver 734 configured to receive wireless communication signals.

[0134] The network node 700 may be configured (e.g., by the processing circuitry 710) to perform the method 300 and / or the method 400 described above.

[0135] Still other embodiments include a computer program 740 comprising instructions that, when executed on processing circuitry 710 of a network node 700, cause the network node 700 to carry out the method 300 and / or method 400 described above.

[0136] Yet other embodiments include a carrier containing the computer program 740. The carrier is one of an electronic signal, optical signal, radio signal, or computer readable storage medium. Additional service and signaling variations will now be described. Any of the embodiments above may be modified, supported, or enhanced by the services and signaling described below. The services and signaling below may, e.g., serve to enhance compatibility and / or integration of the embodiments described above with traditional networks. Additionally or alternatively, the services and signaling below may enable a wider variety of practical uses cases in which the embodiments above can be used.

[0137] An alternative ML Model provisioning procedure may include an NWDAF 65 containing AnLF 66 that is locally configured with (a set of) IDs of NWDAFs 65 containing MTLF(s) 67 and the Analytics ID(s) supported by each NWDAF 65 containing MTLF 67 to retrieve trained ML models or that may use the NWDAF discovery procedure for discovering NWDAFs 65 containing MTLF 67. An NWDAF 65 containing MTLF 67 may determine that further training for an existing ML model is needed when it receives the ML model subscription or the ML model request, e.g. based on parameters included in the request, as described above.

[0138] An alternative ML Model Subscribe / Unsubscribe procedure may have a similar signaling flow to that depicted in FIG. 5 and may be used by a model service consumer 210, e.g., an NWDAF 65 containing AnLF 66 to subscribe / unsubscribe at another NWDAF 65, e.g., an NWDAF 65 containing MTLF 67, to be notified when ML Model Information on the related Analytics becomes available, using Nnwdaf_MLModel Provision services. The ML Model Information is used by an NWDAF 65 containing AnLF 66 to derive analytics. The service is also used by an NWDAF 65 to modify existing ML Model Subscription(s). An NWDAF 65 can be at the same time a consumer of this service provided by other NWDAF(s) and a provider of this service to other NWDAF(s).

[0139] With respect to the subscription request 230, the model service consumer 210 subscribes to, modifies, or cancels subscription for a (set of) trained ML Model(s) associated with a / an (set of) Analytics ID(s) by invoking the Nnwdaf_MLModelProvision_Subscribe I Nnwdaf_MLModelProvision_Unsubscribe service operation. The model service consumer 210 optionally indicates its support for multiple ML models if available.

[0140] When a subscription for a trained ML model associated with an Analytics ID is received, the model service provider 220 (e.g. an NWDAF containing MTLF) may determine whether: existing trained ML Model(s) can be used for the subscription; or triggering further training for the existing trained ML models is needed for the subscription.

[0141] If the NWDAF 65 containing MTLF 67 determines that further training is needed, this NWDAF 65 may initiate data collection from NFs, (e.g. AMF / DCCF / ADRF), UE Application (via AF) or OAM to generate the ML model.

[0142] If the service invocation is for a subscription modification or subscription cancelation, the model service consumer 210 may include an identifier (Subscription Correlation ID) to be modified in the invocation of Nnwdaf_MLModelProvision_Subscribe. With respect to the ML model training notification 260, if the model service consumer 210 subscribes to a (set of) trained ML model(s) associated to a (set of) Analytics I D(s), the NWDAF 65 containing MTLF 67 notifies the model service consumer 210 with a set of pair(s) of unique ML Model Identifier and ML Model Information associated with each Analytics ID requested by the service consumer.

[0143] The Nnwdaf_MLModelProvision_Notify service operation is invoked. The NWDAF 65 containing MTLF 67 also invokes the Nnwdaf_MLModelProvision_Notify service operation to notify an available retrained ML model when the model service provider 220 determines that the previously provided trained ML Model required retraining.

[0144] When the subscription request 230 is for a subscription modification (e.g., including Subscription Correlation ID), the NWDAF 65 containing MTLF 67 may provide either a new trained ML model different to the previously provided one, or retrained ML model by invoking Nnwdaf_MLModelProvision_Notify service operation.

[0145] A model service consumer of ML model provisioning services (e.g., an NWDAF 65 containing AnLF 66) may provide the input parameters as listed below:

[0146] • Information of the analytics for which the requested ML model is to be used, including: o A list of Analytics ID(s): identifies the analytics for which the ML model is used. o [OPTIONAL] NF consumer information: identifies the vendor of NWDAF containing AnLF. o [OPTIONAL] Use case context: indicates the context of use of the analytics to select the most relevant ML model. The model service provider 220 can use the parameter "Use case context" to select the most relevant ML model, when several ML models are available for the requested Analytics ID(s). o [OPTIONAL] ML Model Interoperability Information. This is vendor-specific information that conveys, e.g., requested model file format, model execution environment, etc. The encoding, format, and value of ML Model Interoperable Information is not specified since it is vendor specific information, and is agreed between vendors, if necessary for sharing purposes. o [OPTIONAL] ML Model Filter Information: enables the model service provider 220 to select which ML model for the analytics is requested, e.g. S-NSSAI, Area of Interest. Parameter types in the ML Model Filter Information are the same as parameter types in the Analytics Filter Information which are defined in procedures. o [OPTIONAL] Target of ML Model Reporting: indicates the object(s) for which ML model is requested, e.g. specific UEs, a group of UE(s) or any UE (i.e. all UEs). o [OPTIONAL] Requested representative ratio: a minimum percentage of UEs in the group whose data is a non-empty set and can be used in the model training when the Target of ML Model Reporting is a group of UEs. o ML Model Reporting Information. When used with the Nnwdaf_MLModelProvision_Subscribe service, this may include an Event Reporting Information Parameter. o [OPTIONAL] ML Model Target Period: indicates time interval (e.g., start, end) for which ML model for the Analytics is requested. The time interval may be expressed with actual start time and actual end time (e.g. via UTC time). o [OPTIONAL] Inference Input Data information: contains information about various settings that are expected to be used by AnLF 66 during inferences such as the "Input Data" that are expected be used, each of them optionally accompanied by metrics that show the granularity with which this data will be used (e.g., a sampling ratio, the maximum number of input values, and / or a maximum time interval between the samples of this input data). This can be a subset of the possible Input Data specified for a certain analytics type. o the data sources that are expected to be used as a list of NF instance (or NF set) identifiers.

[0147] • A Notification Target Address (+ Notification Correlation ID) enabling correlation of notifications received from the NWDAF 65 containing MTLF 67 with this subscription.

[0148] • [OPTIONAL] An indication of supporting multiple ML models.

[0149] • [OPTIONAL] Accuracy level(s) of Interest.

[0150] • [OPTIONAL] Number of ML model(s), indicating the maximum number of ML models that the NWDAF 65 containing MTLF 67 could provide to the NWDAF 65 containing

[0151] • [OPTIONAL] Time when model is needed: indicates the latest time when the consumer expects to receive the ML model(s).

[0152] • [OPTIONAL] ML Model Monitoring Information, e.g.: o [OPTIONAL] ML Model metric (e.g., ML Model Accuracy) o [OPTIONAL] ML model monitoring reporting mode, such as Accuracy reporting interval or pre-determined status. Depending on the reporting mode, the NWDAF 65 containing MTLF 67 reports the model accuracy to NWDAF 65 containing AnLF 66 either periodically or when the ML model accuracy is crossing an ML Model Accuracy threshold, i.e. the accuracy either becomes higher or lower than the ML Model Accuracy threshold. o [OPTIONAL] ML Model Accuracy Threshold indicating the accuracy threshold of the ML Model requested by the consumer (as a kind of pre-determined status). It also can be used as an indication that the MTLF is triggered to execute the accuracy monitoring operations for the ML Model provisioned to AnLF. o [OPTIONAL] DataSetTag and ADRF ID if available. This may indicate the inference data (including input data, prediction and the ground truth data at the time which the prediction refers to) stored in ADRF which can be used by MTLF to retrain or reprovision of the ML model. o [OPTIONAL] ML Model Identifier: indicates the Model that the data corresponding to the DataSetTag is related to (in the case of subscription modification).

[0153] The NWDAF 65 containing MTLF 67 may provide to the consumer of the ML model provisioning service operations the output information as listed below:

[0154] • The Notification Correlation Information (used with the Nnwdaf_MLModelProvision_Notify service)

[0155] • For each Analytics ID requested by the model service consumer 210, a set of pair(s) of unique ML Model identifier and the following information: o ML Model Information, which includes the ML model file address (e.g. Uniform Resource Locator (URL) or Fully Qualified Domain Name (FQDN) or ADRF (Set) ID. When ADRF (Set) ID is provisioned, a Storage Transaction ID may also be provisioned. o [OPTIONAL] ML model degradation indicator: indicates whether the provided ML model is degraded. o [OPTIONAL] Validity period indicating time period when the provided ML Model Information applies and / or spatial validity indicating an area where the provided ML Model Information applies. Spatial validity and validity period are determined by MTLF 67 internal logic and it is a subset of Aol if provided in ML Model Filter Information and of ML Model Target Period, respectively. o [OPTIONAL] ML model representative ratio: indicating the percentage of UEs in the group whose data is used in the ML model training when the Target of ML Model Reporting is a group of UEs. o [OPTIONAL] Training Input Data Information: contains information about various settings that have been used by MTLF during training, such as:

[0156] ■ the "Input Data" that have been used, each of them optionally accompanied by metrics that show the data characteristics and granularity with which this data has been used (e.g., a sampling ratio, the maximum number of input values and / or a maximum time interval between the samples of this input data, data range including maximum and minimum values, mean and standard deviation and data distribution when applicable) and the time, e.g., timestamp and duration, when this data was obtained. ■ the data sources related to the "Input Data" that were used for ML model training, which have been identified by a list of NF instance (or NF set) identifiers. This can be a subset of the possible Input Data specified for a certain analytics type. Note that data source information may enable ML Model selection when different models are available for an Analytics ID, or it enables a consumer to avoid selecting a ML model that used data from a specific data source at a particular time or used data characterized by specific data characteristics. o [OPTIONAL] ML Model Accuracy Information: indicates the accuracy of the ML model if ML Model accuracy threshold is requested, which includes:

[0157] ■ the accuracy value of the ML model.

[0158] ■ [OPTIONAL] ML model metric (i.e. ML Model Accuracy).

[0159] An alternative ML model information request procedure may include signaling similar to that depicted in FIG. 6. The procedure may be used by a model service consumer 210 (e.g., an NWDAF 65 containing AnLF 66) to request and get from another NWDAF 65 (e.g., an NWDAF 65 containing MTLF 67) ML Model Information using Nnwdaf_MLModellnfo services. The ML Model Information is used by an NWDAF 65 containing AnLF 66 to derive analytics. An NWDAF 65 can be at the same time a consumer of this service provided by other NWDAF(s) and a provider of this service to other NWDAF(s).

[0160] With respect to the ML model information request 290, the model service consumer 210 may request a (set of) ML Model(s) associated with a / an (set of) Analytics ID(s) by invoking Nnwdaf_MLModellnfo_Request service operation. The model service consumer 210 optionally indicates its support for multiple ML models if available.

[0161] When an ML Model Information request 290 for the Analytics is received, the NWDAF 65 containing MTLF 67 may:

[0162] • determine whether existing trained ML Model(s) can be used for the request; or

[0163] • determine whether triggering further training for the existing trained ML models is needed for the request.

[0164] If the NWDAF 65 containing MTLF 67 determines that further training is needed, this NWDAF 65 may initiate data collection from NFs, (e.g. AMF / DCCF / ADRF), UE Application (via AF) or OAM to generate the ML model.

[0165] With respect to the ML model information response 295, the NWDAF 65 containing MTLF 67 responds to the model service consumer 210 by invoking the Nnwdaf_MLModellnfo_Request response service operation including a set of pair(s) of unique ML Model identifier and the ML Model Information for each Analytics ID that the Model service consumer 210 requests. An alternative ML model training subscription procedure may include signaling similar to that depicted in FIG. 7. The procedure may be used by a model service consumer 210 (e.g., an NWDAF 65 containing MTLF 67) to subscribe to another NWDAF 65 (e.g., an NWDAF 65 containing MTLF 67) for a trained ML model based on the ML model file or ML Model information provided by the model service consumer 210. The service may be used by an NWDAF 65 containing MTLF 67 to enable, e.g., Federated Learning or to update ML model. The service may also used by a model service consumer 210 to request an NWDAF 65 containing MTLF 67 to prepare training ML model or modify existing ML Model training subscription.

[0166] With respect to the subscription request 230, the model service consumer 210 may subscribe or unsubscribe for training an ML model by invoking the Nnwdaf_MLModelTraining_Subscribe / Nnwdaf_MLModelTraining_Unsubscribe service operation. In order to enable Federated Learning, the model service consumer 210 may act as an FL Server NWDAF and can subscribe to multiple NWDAFs 65 containing MTLF 67 acting as FL Client NWDAFs, which are selected by the FL Server NWDAF.

[0167] The FL server NWDAF may use the request to check if an NWDAF can meet the ML model training requirement (e.g. ML Model Interoperability information, Analytics ID, Serving Area and / or availability of data and time). In such case, the FL server NWDAF includes an ML Preparation Flag. When the ML Preparation Flag presents in the request, the service provider NWDAF only checks if it can meet the ML model training requirement (e.g. ML Model Interoperability information, Analytics ID, Serving Area and / or availability of data and time) and I or can successfully download the model if the model information is provided.

[0168] The FL server NWDAF may use the request to get the Model Accuracy of the global ML Model calculated by the FL Client NWDAFs. In such cases, the service consumer NWDAF includes a Model Accuracy Check Flag. When the Model Accuracy Check Flag is present in the request, the service provider NWDAF uses the local training data as the testing dataset to calculate the Model Accuracy of the ML model provided by the service consumer NWDAF.

[0169] When Model service consumer 210 determine to further update the ML model, Model service consumer 210 modifies the subscription by invoking Nnwdaf_MLModelTraining_Subscribe service operation including Subscription Correlation ID with ML Model Information.

[0170] With respect to training the ML model at step 250, the model service provider 220 trains ML model indicated in the subscription request 210 by collecting new data or re-use the data that it owns. If the ML model file is not provided in the subscription request 230, the model service provider 220 shall first get the ML model using the information indicated in the subscription request 230.

[0171] With respect to the ML model training notification 260, when the model service provider 220 completes ML model training, the model service provider 220 notifies the NWDAF service consumer with ML Model Information of updated ML Model by invoking the Nnwdaf_MLModelTraining_Notify service operation.

[0172] If the model service provider 220 determines to terminate the ML model training or determines not to provide further notifications related to this request, then the model service provider 220 may notify the model service consumer 210 a Terminate Request indication with cause code (e.g. NWDAF overload, not available for the FL process anymore, etc.) by invoking the Nnwdaf_MLModelTraining_Notify service operation.

[0173] In order to enable Federated Learning, model service provider 220 acting as FL Client NWDAF can notify model service consumer 210 acting as FL Server NWDAF the local ML model information and status report of FL training including accuracy of local model and Training Input Data Information (e.g. areas covered by the data set, sampling ratio, maximum / minimum of value of each dimension, etc.).

[0174] If the Model Accuracy Check Flag is present in the Nnwdaf_MLModelTraining_Subscribe, the model service provider 220 acting as FL Client NWDAF may notify the model service consumer 210 acting as FL Server NWDAF the Model Accuracy of the global ML Model.

[0175] A model service consumer 210 of ML model training services (e.g., an NWDAF 65 comprising an MTLF 67) may provide the input parameters in Nnwdaf_MLModelTraining_Subscribe or Nnwdaf_MLModelTraininglnfo_Request service operations as listed below:

[0176] • Analytics ID: identifies the analytics for which the ML model is requested to be trained.

[0177] • ML Model Interoperability Information

[0178] • A Notification Target Address (+ Notification Correlation ID) allowing to correlate notifications received from the NWDAF containing MTLF with the subscription, e.g,. wihen used for Nnwdaf_MLModelTraining_Subscribe

[0179] • [OPTIONAL] ML Model Information

[0180] • [OPTIONAL] ML Model file

[0181] • [OPTIONAL] ML Model ID: identifies the provided ML model.

[0182] • [OPTIONAL] ML Preparation Flag: identifies whether the request is for preparing Federated Learning or executing Federated Learning.

[0183] • [OPTIONAL] ML Model Accuracy Check Flag: identifies that the request is for using the local training data as the testing dataset to calculate the Model Accuracy of the global ML model provided by the NWDAF service consumer acting as the FL Server NWDAF.

[0184] • [OPTIONAL] ML Correlation ID: identifies the Federated Learning procedure for training the ML model. This parameter is included when the service is used for Federated Learning. • [OPTIONAL] Available data requirement. This is for informing the requirement on available data for the ML model training, e.g. FL Server NWDAF sends the requirement in preparation request to a FL Client NWDAF for selecting the FL Client NWDAF which can meet the available data requirement. The following available data requirements can be included: o Event ID list to be collected for local model training. o Dataset statistical properties o Time window of the data samples. o Minimum number of data samples.

[0185] • [OPTIONAL] Availability time requirement. This is for informing the requirement on availability time for the ML model training, e.g. FL Server NWDAF sends the requirement in preparation request to FL Client NWDAF for selecting the FL Client NWDAF which is available in the required time for training ML model.

[0186] • [OPTIONAL] Training Filter Information: enables to select which data for the ML model training is requested, e.g. S-NSSAI, Area of Interest. Parameter types in the Training Filter Information are the same as or subset of parameter types in the ML Model Filter Information

[0187] • [OPTIONAL] Target of Training Reporting: indicates the object(s) for which data for ML model training is requested, i.e. a group of UEs or any UE (i.e. all UEs).

[0188] • [OPTIONAL] Use case context: indicates the context of use of ML model.

[0189] • [OPTIONAL] Training Reporting Information including a maximum response time indicating a maximum time for waiting notifications (e.g., model training results).

[0190] • [OPTIONAL] Iteration round ID: indicates the iteration round number of current ML model training.

[0191] • [OPTIONAL] Expiry time.

[0192] The model service provider 220 provides to the model service consumer 210 output information in a notification. The output may comprise:

[0193] • The Notification Correlation Information.

[0194] • [OPTIONAL] ML Model Information

[0195] • [OPTIONAL] ML Model ID: identifies the provisioned ML model.

[0196] • [OPTIONAL] Model Accuracy: The model accuracy of the global ML model, which is calculate by the FL Client NWDAF using the local training data as the testing dataset.

[0197] • [OPTIONAL] Status report of FL training: Accuracy of local model and Training Input Data Information (e.g. areas covered by the data set, sampling ratio, maximum / minimum of value of each dimension, etc.), which are generated by the FL Client NWDAF during FL procedure. • [OPTIONAL] ML Correlation ID. This parameter may be included when the service is used for Federated Learning.

[0198] • [OPTIONAL] Iteration round ID: indicates the iteration round number of ML model training indicated by the FL Server NWDAF.

[0199] • [OPTIONAL] Delay Event Notification with the following parameters: o Delay event indication: this parameter indicates that FL Client NWDAF is not able to complete the training of the interim local ML model within the maximum response time provided by the FL Server NWDAF. o [OPTIONAL] cause code (e.g. local ML model training failure, more time necessary for local ML model training, etc.). o [OPTIONAL] Expected time to complete the training: Indicates to the FL Server NWDAF that expected remaining training time and may be provided with Delay Event Notification.

[0200] An alternative ML model training information request procedure may include a similar signaling flow to that depicted in FIG. 8. The procedure may be used by a model service consumer 210 (e.g., an NWDAF 65) to request from a model service provider 220 (e.g., another NWDAF) information about ML model training based on the ML model file or ML Model information as provided by the model service consumer 210. The service may be used to enable, e.g., Federated Learning.

[0201] With regard to the model training information request 297, the model service consumer 210 may send the request to the model service provider 220 to get the information about the ML model training based on the ML model file or ML Model information provided by the model service consumer 210 by invoking the Nnwdaf_MLModelTraininglnfo_Request service operation.

[0202] To enable Federated Learning, the model service consumer 210 (e.g., acting as FL Server NWDAF) may request to get ML Model Training Information from the model service provider 220 (e.g., an NWDAF containing MTLF acting as FL Client NWDAF which is selected by the FL Server NWDAF).

[0203] The model service consumer 210 may use the request to check if an NWDAF 65 can meet the ML model training requirements (e.g. ML Model Interoperability information, Analytics ID, Service Area / DNAI and / or availability of data and time). In such cases, the NWDAF service consumer includes an ML Preparation Flag.

[0204] The model service consumer 210 may use the request to get the Model Accuracy of the ML Model provided by the service consumer using local training data in the NWDAF containing MTLF as the testing dataset. In such cases, the service consumer NWDAF includes a Model Accuracy Check Flag.

[0205] At step 298, when the ML Preparation Flag is present in the request 297, the model service provider 220 checks whether it can meet the ML model training requirement and / or can successfully download the model if the model information is provided. Based on the check result, the model service provider 220 gets a successful return code or failure cause code (e.g. NWDAF does not meet the ML training requirements) as the information about the ML model training.

[0206] When the Model Accuracy Check Flag is present in the request 297, the model service provider 220 uses the local training data as the testing dataset to calculate the Model Accuracy of the ML model provided by the service consumer. The model service provider 220 includes the Model Accuracy into the information about the ML model training.

[0207] When the model service provider 220 is ongoing ML model training based on the ML model file or ML Model information as described in clause 2.1.2.2 provided by the model service consumer 210, the model service provider 220 gets a failure cause code (e.g. ML training is not complete) as the information about the ML model training.

[0208] When the model service provider 220 completes ML model training based on the ML model file or ML Model information as described in clause 2.1.2.2 provided by the model service consumer 210, the model service provider 220 gets a successful return code and the ML Model Information of the trained ML model as the information about the ML model training.

[0209] With respect to the model training information response 299, the model service provider 220 replies to the model service consumer 210 with the information about the ML model training by invoking the Nnwdaf_MLModelTraininglnfo_Request response service operation.

[0210] The Nnwdaf_MLModelProvision service is also supported in some embodiments. The Nnwdaf_MLM odel Provision service enables the model service consumer 210 to receive a notification when an ML model matching the subscription parameters becomes available.

[0211] When the subscription is accepted by the model service provider 220, the model service consumer 210 receives an identifier (Subscription Correlation ID) allowing to further manage (modify, delete) this subscription. The modification of ML model subscription can be enforced, e.g., by an NWDAF 65 based on operator policy and configuration.

[0212] The Nnwdaf_MLModelProvision_Subscribe service operation may also be supported in some embodiments. The Nnwdaf_MLModelProvision_Subscribe service may be used to subscribe to NWDAF ML model provisioning. The service may, e.g., be invoked by a subscription request 230, which may provide one or more inputs to the service.

[0213] The inputs to the service may include one or more Analytics IDs, a Notification Target Address (+ Notification Correlation ID).

[0214] The inputs may optionally include a Subscription Correlation ID (in the case of modification of the ML model subscription), ML Model Filter Information to indicate the conditions for which ML model for the analytics is requested and Target of ML Model Reporting to indicate the object(s) for which ML model is requested (e.g. specific UEs, a group of UE(s) or any UE (i.e. all UEs)), NF consumer information, Requested representative ratio, ML Model Reporting Information (including e.g. ML Model Target Period), Expiry time, Use case context, Inference Input Data information, indication of support for multiple ML models, multiple ML models Filter Information to indicate the conditions for which multiple ML models are requested, ML Model Interoperability Information, Time when model is needed, ML Model Monitoring Information (including e.g. ML Model metric, ML model monitoring reporting mode, ML Model Accuracy Threshold, DataSetTag and ADRF ID, ML Model Identifier).

[0215] The service may produce a subscription response 235 comprising one or more outputs. The output(s) may include, when the subscription is accepted, a Subscription Correlation ID (which may be used for management of this subscription), Expiry time (which may be useful if the subscription can be expired based on the operator's policy).

[0216] The Nnwdaf_MLModelProvision_Unsubscribe service may be supported in one or more embodiments. The Nnwdaf_MLModelProvision_Unsubscribe service may be invoked by a subscription request 230 to unsubscribe to ML model provisioning notifications, e.g., by providing a subscription Correlation ID in the request. An operation execution result indication may be included in a corresponding subscription response 235.

[0217] The Nnwdaf_MLModelProvision_Notify service may be supported in some embodiments. The Nnwdaf_MLModelProvision_Notify service may be used to provide ML model information to a model service consumer 210 that has subscribed to a specific service.

[0218] Inputs may include Notification Correlation Information and a set including an Analytics ID, and one or more tuples of unique ML Model identifier and ML Model Information. In some embodiments, the inputs may also include ML Model Accuracy Information.

[0219] In response, an operation execution result indication may be provided.

[0220] The Nnwdaf_MLModellnfo_Request service may be supported in some embodiments. The Nnwdaf_MLModellnfo_Request service may enable the model service consumer 210 to request NWDAF ML Model Information. The service may take one or more Analytics IDs as input. In some embodiments, the service may additionally take, as input, ML Model Filter Information to indicate the conditions for which ML model for the analytics is requested and Target of ML Model Reporting to indicate the object(s) for which ML model is requested (e.g. specific UEs, a group of UE(s) or any UE (i.e. all UEs)), NF consumer information, Requested representative ratio, ML Model Reporting Information (including e.g. ML Model Target Period), Use case context, Inference Input Data Information, indication of support for multiple ML models, multiple ML models Filter Information to indicate the conditions for which multiple ML models are requested, ML Model Interoperability Information, ML Model Accuracy Monitoring Information.

[0221] In response to the request, the service may output one or more Analytics IDs. For each Analytics ID, one or more tuples of unique ML Model identifier and ML Model Information may be provided. In response, the service may output ML Model Accuracy Information. The Nnwdaf_MLModelTraining service may be supported in some embodiments. The service enables the model service consumer 210 to subscribe / unsubscribe / notify / modify for ML model training.

[0222] When used for Federated Learning, this service enables FL server NWDAF to enable Federated Learning while providing global ML model information to FL Client NWDAF and getting local ML model information and status report of FL training.

[0223] This service may also be used by the model service consumer 210 (e.g., FL Server NWDAF) to check if the model service provider 220 (e.g., FL Client NWDAF) can meet an ML model training requirement.

[0224] This service may also be used by the model service consumer 210 to request the service provider to calculate and provide Model Accuracy of the global ML Model.

[0225] The Nnwdaf_MLModelTraining_Subscribe service may be supported by some embodiments. The Nnwdaf_MLModelTraining_Subscribe service may be used to subscribe to ML model training. The service may take, as input:

[0226] Analytics ID;

[0227] ML Model Interoperability information;

[0228] Notification Target Address (+ Notification Correlation ID).

[0229] In some embodiments, input may further include one or more of: ML Model ID: identifies the provided ML model.

[0230] ML Model Information (as defined in clause 2.1.1.3);

[0231] ML model file;

[0232] Subscription Correlation ID (in the case of modification of the ML Model Training subscription);

[0233] ML Training Information, i.e. data availability requirement, time availability requirement.

[0234] ML Preparation Flag;

[0235] ML Model Accuracy Check Flag;

[0236] ML Correlation ID;

[0237] Training Filter Information;

[0238] Target of Training Reporting;

[0239] Training Reporting Information as defined in clause 2.1.2.2;

[0240] Use case context;

[0241] Iteration round ID;

[0242] Expiry time.

[0243] The service may produce a variety of outputs. When the request is accepted, the service may output a Subscription Correlation ID (useful for management of this subscription). When the request is not accepted, the service may output an error response with cause code (e.g. NWDAF 65 does not meet the ML training requirements, ML training is not complete, NWDAF overload, not available for the FL process anymore, etc.).

[0244] In some embodiments, the outputs further includes an ML Correlation ID (e.g., confirmation of the subscription for this FL process).

[0245] The Nnwdaf_MLModelTraining_Unsubscribe service may be supported by one or more embodiments. The Nnwdaf_MLModelTraining_Unsubscribe service may be used to Terminate NWDAF ML model training. The service may take, as input, a Subscription Correlation ID. As output, the service may provide an operation execution result indication. Optionally, the service may additionally provide a cause code (e.g. FL Client NWDAF is unselected by the FL Server NWDAF for the FL process, or the FL process is suspended or finished, etc.). Final aggregated ML model information (if FL has finished) or updated aggregated ML model information (if FL is suspended).

[0246] In some embodiments, the Nnwdaf_MLModelTraining_Notify service is supported. The Nnwdaf_MLModelTraining_Notify service may be used by a model service provider 220 to notify a model service consumer 210 of information about the trained ML model that has subscribed to the specific NWDAF service. The service can also be used to indicate to the model service consumer that the model service provider 220 will terminate the ML model training.

[0247] Inputs to the service may include Notification Correlation Information. This parameter indicates the Notification Correlation ID that has been assigned by the consumer during ML model training.

[0248] Inputs to the service may additionally include:

[0249] One or more tuples comprising an Analytics ID and ML model Information;

[0250] ML Correlation ID, when for Federated Learning;

[0251] Corresponding Use case context;

[0252] Termination Request: this parameter indicates that NWDAF requests to terminate the ML model training, i.e. NWDAF will not provide further notifications related to this request, with cause code (e.g. NWDAF overload, not available for the FL process anymore, etc.);

[0253] ML Model ID: this parameter identifies the provisioned ML model;

[0254] Global ML Model Accuracy: The model accuracy of the global ML model, which is calculate by the FL Client NWDAF using the local training data as the testing dataset;

[0255] Status report of FL training: local ML Model metric and Training Input Data Information (e.g. areas covered by the data set, sampling ratio, maximum / minimum of value of each dimension, etc.), which are generated by the FL Client NWDAF during FL procedure;

[0256] Delay Event Notification;

[0257] Iteration round ID.

[0258] As output, the service may provide an operation execution result indication. In some embodiments, the Nnwdaf_MLModelTraininglnfo Service is supported. This service enables the model service consumer 210 to request information about ML model training based on the ML Model file or ML Model information provided by the model service consumer 210.

[0259] When used for Federated Learning, this service enables FL server NWDAF to enable Federated Learning while providing global ML model information to FL Client NWDAF and getting local ML model information from the FL Client NWDAF.

[0260] The Nnwdaf_MLModelTraininglnfo_Request service may be supported in some embodiments. The Nnwdaf_MLModelTraininglnfo_Request may be used to request information about NWDAF ML model training with specific parameters.

[0261] Inputs to the service may include an Analytics ID and ML Model Interoperability information.

[0262] In some embodiments, inputs may additionally include one or more of:

[0263] ML Model ID: identifies the provided ML model.

[0264] ML Model Information

[0265] ML Model file.

[0266] ML Training Information (e.g., data availability requirement, time availability requirement).

[0267] Training Reporting Information

[0268] ML Preparation Flag.

[0269] ML Model Accuracy Check Flag.

[0270] ML Correlation ID.

[0271] Termination Request, when terminating the Federated Learning identified by the ML Correlation ID and optionally indicating the reason, e.g. FL Client NWDAF is unselected by the FL Server NWDAF for the FL process, or the FL process is suspended, etc.

[0272] Training Filter Information.

[0273] Target of Training Reporting.

[0274] Use case context.

[0275] The service may produce a variety of outputs depending on the embodiment. For example, when the request is accepted, the service may output an operation execution result indication. When the request is not accepted, the service may output an error response with cause code (e.g. NWDAF 65 does not meet the ML training requirements, ML training is not complete, NWDAF overload, not available for the FL process anymore, etc.).

[0276] In some embodiments, the output(s) may further include one or more of:

[0277] ML Model ID.

[0278] One or more tuples comprising an Analytics ID and ML model Information.

[0279] ML Correlation ID, when for Federated Learning.

[0280] Corresponding Use case context. Global ML Model Accuracy: The model accuracy of the global ML model, which is calculated by the FL Client NWDAF using the local training data as the testing dataset.

[0281] Status report of FL training: local ML model metric and Training Input Data Information (e.g. areas covered by the data set, sampling ratio, maximum / minimum of value of each dimension of data, etc.), which are generated by the FL Client NWDAF during FL procedure.

[0282] Delay Event Notification; global ML model metric.

[0283] Although the various communication devices described herein may include the illustrated combination of hardware components, other embodiments may comprise computing and / or communication hardware with different combinations of components. It is to be understood that these computing devices may comprise any suitable combination of hardware and / or software needed to perform the tasks, features, functions, and methods disclosed herein. Further, while components are depicted as single boxes located within a larger box, or nested within multiple boxes, in practice, the devices described herein may comprise multiple different physical components that make up a single illustrated component, and functionality may be partitioned between separate components.

Claims

CLAIMS1. A method (300), implemented by a network node (700) acting as a model service provider (220) in a wireless communication network (10), the method comprising: receiving (310) a machine learning, ML, model subscription request (230) from a model service consumer (210); and determining (320) a plurality of ML models to use in fulfilling the ML model subscription request (230).

2. The method of claim 1 , wherein determining the plurality of ML models to use in fulfilling the ML model subscription request (230) is based on one or more parameters contained in the ML model subscription request (230).

3. The method of claim 1 or 2, wherein the ML model subscription request (230) comprises one or more of the following parameters:- model service consumer (210) information- indication of support for multiple ML models- one or more analytics IDs- Notification Target Address- Notification Correlation ID- required time interval for which an ML model is requested- requested representative ratio- use case context- conditions for which multiple ML models are requested- ML model interoperability information.

4. The method of any of the preceding claims, further comprising transmitting, to the model service consumer (210), a model subscription response (235) indicating whether the model subscription request (230) can be fulfilled.

5. The method of claim 4, wherein the model subscription response (235) comprises a Subscription Correlation ID and / or an expiry time.

6. The method of any of claims 1 to 5, further comprising transmitting, to the model service consumer (210), a model provisioning notification (235, 240) identifying more than one ML model as corresponding to the ML model subscription request (230).

7. The method of claim 6, wherein the model provisioning notification comprises one or more tuples of ML Model identifiers, ML Model Information and a sub-group ID identifying analytics targets for each of the ML Models.

8. The method of claim 7, wherein the model subscription request (230) comprises one or more Analytics IDs, and the ML Model Information and sub-group ID are associated with at least one of these Analytics IDs.

9. The method of any of claims 6 to 8 when dependent on claim 4 or 5, wherein the model provisioning notification is transmitted together with or comprised in the model subscription response (235).

10. The method of any of the preceding claims, further comprising providing the model service consumer (210) with analytics produced by the plurality of the ML models in accordance with the ML model subscription request (230).

11. The method of any one of any of the preceding claims, further comprising determining whether the ML models to use in fulfilling the ML model subscription request already exist.

12. The method of any of the preceding claims, further comprising triggering further training of an existing one or more of the ML models in order to fulfill the ML model subscription request (230).

13. The method of claim 12, wherein triggering further training of an existing one or more of the ML models comprises triggering further training of a plurality of ML models in response to the ML model subscription request (230) indicating that the model service consumer (210) supports more than one ML model per analytics identifier.

14. The method of any of the preceding claims, wherein the ML model subscription request (230) requests that the network node (700) determine whether the network node is able to meet a ML training requirement and the method further comprises: determining whether the network node (700) is able to meet a ML training requirement based on the ML models; and sending, to the model service consumer, an analytics service subscription response (235) indicating whether the network node (700) is able to meet the ML training requirement.

15. The method of any one of claims 1 to 14, further comprising: receiving an ML model information request (290, 297) from the model service consumer (210); and transmitting information about a plurality of the ML models in a same ML model information response (295, 299) in response to the ML model information request (290, 297).

16. The method of claim 15, wherein the ML model information request (290, 297) is a request for information about ML model training.

17. The method of claim 4, wherein the model subscription response (235) comprises an error indication indicating that the request is not accepted.

18. A network node (700) operable to act as a model service provider (220) of a wireless communication network (10), the network node configured to: receive a machine learning, ML, model subscription request (230) from a model service consumer (210); and determine (320) a plurality of ML models to use in fulfilling the ML model subscription request (230).

19. The network node of claim 18, further configured to perform the method of any one of claims 2 to 17.

20. A network node (700) operable to act as a model service provider (220) of a wireless communication network (10), the network node comprising: interface circuitry (730) and processing circuitry (710) communicatively connected to the interface circuitry (730), wherein the processing circuitry (710) is configured to: receive a machine learning, ML, model subscription request (230) from a model service consumer (210) via the interface circuitry (730); and determine (320) a plurality of ML models to use in fulfilling the ML model subscription request (230).

21. The network node of claim 20, wherein the processing circuitry (710) is further configured to perform the method of any one of claims 2-17.

22. A computer program, comprising instructions which, when executed on processing circuitry (710) of a network node (700), cause the processing circuitry (710) to carry out the method according to any one of claims 1-17.

23. A carrier containing the computer program of claim 22, wherein the carrier is one of an electronic signal, optical signal, radio signal, or computer readable storage medium.

24. A method (400), implemented by a network node (700) acting as a model service consumer (210) in a wireless communication network (10), the method comprising: transmitting (410) a machine learning, ML, model subscription request (230) to a model service provider (220); and receiving (420), from the model service provider (220), a model provisioning notification (235, 240) identifying more than one ML model as corresponding to the ML model subscription request (230).

25. The method of claim 24, wherein the ML model subscription request (230) comprises one or more of the following parameters:- model service consumer (210) information- indication of support for multiple ML models- one or more analytics IDs- Notification Target Address- Notification Correlation ID- required time interval for which an ML model is requested- requested representative ratio- use case context- conditions for which multiple ML models are requested- ML model interoperability information.

26. The method of claim 24 or 25, wherein the model provisioning notification comprises one or more tuples of ML Model identifiers, ML Model Information and a sub-group ID identifying analytics targets for each of the ML Models.

27. The method of claim 26, wherein the model subscription request (230) comprises one or more Analytics IDs, and the ML Model Information and sub-group ID are associated with at least one of these Analytics IDs.

28. The method of any of claims 24 to 27, wherein the model provisioning notification is transmitted together with or comprised in a model subscription response (235) indicating whether the model subscription request (230) can be fulfilled.

29. The method of any of claims 24 to 28, further comprising receiving, from the model service provider (220), analytics produced by a plurality of the ML models in accordance with the ML model subscription request (230).

30. The method of any one of claims 24 to 29, wherein: the ML model subscription request (230) requests that the model service provider (220) determine whether the model service provider (220) is able to meet a ML training requirement; and the method further comprises receiving, from the model service provider (220), an analytics service subscription response (235) indicating whether the model service provider (220) is able to meet the ML training requirement.

31. The method of any one of claims 24 to 30, further comprising: transmitting an ML model information request (290, 297) to the model service provider (220); and receiving information about a plurality of the more than one ML model in a same ML model information response (295, 299) in response to the ML model information request (290, 297).

32. The method of claim 31, wherein the ML model information request (290, 297) is a request for information about ML model training.

33. A network node (700) operable to act as a model service consumer (210) of a wireless communication network (10), the network node configured to: transmit a machine learning, ML, model subscription request (230) to a model service provider (220); and receive, from the model service provider (220), a notification identifying more than one ML model as corresponding to the ML model subscription request (230).

34. The network node of claim 33, further configured to perform the method of any one of claims 25 to 32.

35. A network node (700) operable to act as a model service consumer (210) of a wireless communication network (10), the network node comprising: interface circuitry (730) and processing circuitry (710) communicatively connected to the interface circuitry (730), wherein the processing circuitry (710) is configured to:transmit a machine learning, ML, model subscription request (230) to a model service provider (220); and receive (420), from the model service provider (220), a notification identifying more than one ML model as corresponding to the ML model subscription request (230).

36. The network node of claim 35, wherein the processing circuitry (710) is further configured to perform the method of any one of claims 25 to 32.

37. A computer program, comprising instructions which, when executed on processing circuitry (710) of a network node (700), cause the processing circuitry (710) to carry out the method according to any one of claims 24 to 32.

38. A carrier containing the computer program of claim 37, wherein the carrier is one of an electronic signal, optical signal, radio signal, or computer readable storage medium.

Images