Enhanced support for multiple machine learning models in machine learning model service providers and consumers

Abstract

A network data analytics function, NWDAF, can comprise a model training logic function, MTLF, and act as an ML model service provider for other network functions, NFs, that act as ML model service consumers. A model service consumer (210) can send a machine learning, ML, model subscription request (230) to a model service provider (220). The model service provider can determine a plurality of ML models for satisfying the ML model subscription request (230) and send a notification (235, 240) to the model service consumer identifying the plurality of ML models as corresponding to the ML model subscription request (230).

Description

Technical Field

[0001] This disclosure generally relates to the field of wireless communication network technology, and more specifically, to enhancing machine learning (ML) service providers within such networks to support multiple ML models. Background Technology

[0002] In wireless communication networks, Network Data Analysis Function (NWDAF) may include Model Training Logic Function (MTLF) or similar functions. MTLF provides ML model provisioning and ML model training services to one or more Network Function (NF) consumers. NWDAF supports a set of standard messages that NF consumers can use to request such services.

[0003] 3GPP Technical Specification TS 23.288 V18.5.0 provides process and high-level service descriptions for ML model provisioning and ML model training services. These services are provided by the NWDAF (including MTLF) to provide / train ML models based on requests from NF consumers. Summary of the Invention

[0004] While traditional networks allow NF consumers to request ML model services from NWDAFs, problems arise when multiple models may be associated with a request. For example, it is unclear how the parameters of a consumer's request are mapped to multiple models. It is also unclear how a service provider (e.g., NWDAF or MTLF) will decide to provide and / or train multiple models based on a single request from a consumer.

[0005] The embodiments disclosed herein propose enhancements to traditional services to overcome one or more of the problems discussed herein. As will be explained in more detail below, one or more such embodiments enhance any one or any combination of the Nnwdaf_MLModelProvision, Nnwdaf_MLModelInfo, Nnwdaf_MLModelTraining, and Nnwdaf_MLModelTrainingInfo services.

[0006] Among other things, specific embodiments include enhancements to the parameters of requests from NF consumers to more clearly map the requirements of multiple ML models, whether the request includes a single analytics ID or multiple analytics IDs.

[0007] Additionally or alternatively, certain embodiments enhance operations at model service providers (such as NWDAF and / or MTLF) to establish a method for determining whether multiple ML models are involved based on a single consumer request.

[0008] According to the present invention, a method is provided implemented by a network node acting as a model service provider in a wireless communication network. The method includes: receiving a machine learning (ML) model subscription request from a model service consumer, and determining a plurality of ML models to satisfy the ML model subscription request.

[0009] A network node operable as a model service provider in a wireless communication network is also provided. The network node is configured to: receive machine learning (ML) model subscription requests from model service consumers, and determine a plurality of ML models to satisfy the ML model subscription requests.

[0010] Furthermore, a method is provided implemented by a network node acting as a model service consumer in a wireless communication network. The method includes: sending a machine learning (ML) model subscription request to a model service provider, and receiving from the model service provider a model provisioning notification identifying a plurality of ML models corresponding to the ML model subscription request.

[0011] A network node operable as a model service consumer in a wireless communication network is also provided. The network node is configured to: send machine learning (ML) model subscription requests to a model service provider, and receive model provisioning notifications from the model service provider identifying multiple ML models corresponding to the ML model subscription requests.

[0012] The aforementioned network node may include an interface circuit and a processing circuit communicatively connected to the interface circuit, wherein the processing circuit is configured to perform the aforementioned method.

[0013] A corresponding computer program is also provided, which, when executed on the processing circuitry of a network node, causes the processing circuitry to perform the aforementioned corresponding method.

[0014] The model service provider may be an NWDAF, particularly an NWDAF that includes MTLF or similar functionality, or any other network function or network node that implements or includes MTLF or similar functionality.

[0015] The model service consumer can be an NWDAF, specifically an NWDAF that includes the analysis logic function AnLF or similar functions, or any other network function or network node that implements or includes AnLF or similar functions. Attached Figure Description

[0016] Various aspects of this disclosure are shown by way of example and are not limited to the accompanying drawings, wherein like reference numerals indicate like elements.

[0017] Figure 1This is a schematic block diagram illustrating an example wireless communication network according to one or more embodiments of the present disclosure;

[0018] Figure 2 This is a schematic block diagram illustrating an example NWDAF according to one or more embodiments of the present disclosure;

[0019] Figure 3 This is a flowchart illustrating an example method implemented by a network node acting as a model service provider according to one or more embodiments of this disclosure;

[0020] Figure 4 This is a flowchart illustrating an example method implemented by a network node acting as a model service consumer according to one or more embodiments of the present disclosure;

[0021] Figure 5-8 This is a signaling diagram illustrating different examples of signaling exchanged between a model service provider and a model service consumer according to various embodiments of the present disclosure;

[0022] Figure 9 This is a schematic block diagram illustrating an example network node according to one or more embodiments of the present disclosure. Detailed Implementation

[0023] The examples provided herein are typically associated with a particular generation of 3GPP (3rd Generation Partnership Project) wireless networks. For instance, the terms NWDAF and / or MTLF are 5G examples for ML model service providers. These specific examples are for illustrative purposes only and should not be construed as excluding other network elements (whether currently known or developed in the future). In fact, unless otherwise specified, the embodiments described herein can be applied to any generation of 3GPP networks or other types of networks.

[0024] Figure 1 An example reference architecture for such a wireless communication network 10 is shown. The communication network 10 includes a radio access network (RAN) 20 and a core network 30 employing a service-based architecture. The RAN 20 and core network 30 are sometimes collectively referred to as a Public Land Mobile Network (PLMN) when operated by the same operator. In other embodiments, the RAN 20 and core network 30 may be a Standalone Non-Public Network (SNPN) or a Public Mobile Network (PMN).

[0025] RAN 20 includes one or more network nodes configured to provide radio access to one or more UEs 110 operating within the coverage area of ​​the PLMN. For this purpose, the RAN supports one or more Radio Access Technologies (RATs). In the context of a specific 5G network, the network node providing radio access to UEs 110 may be referred to as gNodeB (gNB) 25. RATs used by 5G networks are typically referred to as New Radios (NRs).

[0026] Core network 30 provides connectivity between RAN 20 and one or more data networks (DNs) 90 (e.g., the Internet). Offloading to DN 90 may occur at either the home PLMN (HPLMN) or the visited PLMN (VPLMN). Whether the HPPLMN or VPLMN provides offloading may depend on whether UE 110 is roaming. UE 110 receives one or more network services from core network 30, for example, to enable UE 110 to exchange data with DN 90.

[0027] The core network 30 includes multiple network functions (NFs). These NFs can reside in either the user plane 33 or the control plane 37 of the core network 30. The user plane 33 (sometimes referred to as the data plane) typically carries user data services. The control plane 37 typically carries signaling services (e.g., control packets).

[0028] In this example, the NF of user plane 33 includes User Plane Functions (UPF) 35. Multiple UPFs can exist in the user plane of a given network, and these UPFs can be interconnected. The NF of control plane 37 includes Access and Mobility Management Functions (AMF) 40, Session Communication Agent (SCP) 42, Session Management Functions (SMF) 45, Network Slice Specific Authentication and Authorization Functions (NSSAAF) 47, Policy Control Functions (PCF) 50, Unified Data Management (UDM) Functions 55, Unified Data Repository (UDR) Functions 57, Authentication Server Functions (AUSF) 60, Network Data Analysis Functions (NWDAF) 65, Network Openness Functions (NEF) 70, Network Repository Functions (NRF) 75, and Network Slice Selection Functions (NSSF) 80. The control plane 37 of core network 30 also includes Application Functions (AF) 85.

[0029] Each NF consists of a logical entity residing on one or more core network nodes. Each core network node can be implemented using computing hardware, such as one or more processors, memory, network interfaces, or combinations thereof. NFs communicate with each other using predefined interfaces. Some interfaces are identified by standardized reference points within the network, while others are simply identified by name.

[0030] N1 is the reference point between UE 110 and AMF 40. N2 is the reference point between RAN 20 and AMF 40. N3 is the reference point between RAN20 and UPF 35. N4 is the reference point between SMF 45 and UPF 35. N6 is the reference point between UPF 35 and DN 90. N9 is the reference point between UPF 35. Multiple NFs expose service-based interfaces named after them, in the format Nxxx, where xxx is the name of the NF. For example, NEF 70 provides the Nnef interface, NRF 75 provides the Nnrf interface, and so on.

[0031] Generally, AMF 40 typically manages registration, connectivity, and mobility management between network 10 and UE 110. AMF 40 also typically mediates communication between UE 110 and SMF 45.

[0032] SMF 45 is typically responsible for session establishment, modification, and release, including the selection and control of UPF entities, maintaining the topology of the involved Packet Data Unit (PDU) Session Anchor Points (PSAs), and establishing and releasing tunnels between RAN 20 and UPF 35, as well as between UPF 35s. In this context, the PSA is the UPF 35 on the N6 interface that terminates PDU sessions within core network 30. SMF 45 also typically configures traffic forwarding at UPF 35. SMF 45 also typically interacts with UPF 35 using Packet Forwarding Control Protocol (PFCP) procedures.

[0033] The NRF 75 typically acts as a central registry, storing information about other NFs in network 10 and sharing that information with them. At this point, NFs can register the services they support with the NRF 75 in their profiles and can discover other NFs in network 10 and the services they support by interfacing with the NRF 75.

[0034] NWDAF 65 typically analyzes data collected from NFs and UEs in the network and publishes the results to its subscribed data analysis consumers. The NFs supporting data analysis can implement an application programming interface (API) based on event-driven open services. NWDAF 65 typically subscribes to uniquely identified events published by the NF to obtain data, then analyzes the data and exposes it to its subscribed consumers. These consumers can then use the analyzed data to modify the operation and / or configuration of network 10, potentially even in real time.

[0035] like Figure 2As shown, NWDAF 65 may include an Analysis Logic Function (AnLF) 66 and / or an MTLF 67. AnLF 66 is traditionally responsible for collecting analysis requests and sending responses to service consumers. MTLF 67 is traditionally responsible for training and deploying models. Therefore, depending on the embodiment, NWDAF 65 may be a model service consumer, a model service provider, or both. In fact, as will be further explained below, the embodiments disclosed herein include examples in which a first NWDAF acting as a model service consumer can communicate with a second NWDAF acting as a model service provider. In many such embodiments, the first NWDAF may include AnLF 66 and the second NWDAF may include MTLF 67.

[0036] Typically, an NWDAF, or any other network function or network node that includes an MTLF or provides MTLF functionality or services, can be used as an example of a model service provider, while an NWDAF, or any other network function or network node that includes an AnLF or provides AnLF functionality or services, can be used as an example of a model service consumer.

[0037] Network functions or network nodes that implement or include MTLF and / or AnLF and / or similar functions (such as NWDAF) can register themselves with the NRF, thereby including an analysis ID (list) in their NF profile.

[0038] In the case of MTLF or similar functions, when a trained ML model is available for one or more analytics IDs, the MTLF or similar function may also include ML model provisioning services (e.g., Nnwdaf_MLModelProvision, Nnwdaf_MLModelInfo) as one of the supported services during registration with the NRF. The MTLF or similar function may provide the NRF with a list of analytics IDs corresponding to the trained ML models, and may provide ML model filter information for the trained ML model for each analytics ID (if available). It is expected that only the S-NSSAI and related regions in the ML model filter information for the trained ML model for each analytics ID can be registered with the NRF. For each analytics ID, if ML model interoperability is supported, the registration with the NRF may also include an ML model interoperability indicator. Thus, the services of NWDAF or other network functions or nodes that provide MTLF or similar services become discoverable by other network functions or network nodes (such as network functions or network nodes that implement or include AnLF, such as NWDAF).

[0039] As discussed above, embodiments typically involve situations where multiple models may be associated with a given subscription request. Therefore, and as will be explained in more detail below, specific embodiments of this disclosure include a method 300 performed by a network node, such as... Figure 3 As shown in the diagram. Method 300 includes: receiving an ML model subscription request from a model service consumer 210 (box 310). Method 300 also includes: determining a plurality of machine learning models to satisfy the ML model subscription request (box 320). A network node performing method 300 may also be represented as a model service provider.

[0040] Accordingly, embodiments of this disclosure include a method 400 performed by a network node, such as... Figure 4 As shown in the diagram. Method 400 includes: sending an ML model subscription request to a model service provider (box 410). Method 400 also includes: receiving from the model service provider a notification identifying a plurality of machine learning models as corresponding to the ML model subscription request (box 420). A network node performing method 400 may also be represented as a model service consumer.

[0041] Figure 5 This is a signaling diagram illustrating example signaling exchanged between a model service consumer 210 and a model service provider 220 (one or both of which may be an NWDAF 65). For example, the model service consumer 210 may be a first NWDAF including AnLF 66. In some embodiments, the first NWDAF may be notified when ML model information becomes available at the model service provider 220. The model service provider 220 may be a second NWDAF including MTLF 67. Alternatively, depending on the embodiment, AnLF 66 and MTLF 67 may be included in the same NWDAF 65.

[0042] Figure 5 The signaling supports the ML model analytics subscription process. This process can be used to subscribe to or unsubscribe from analytics generated by the ML model. The model service consumer 210 can send a subscription request 230 to the model service provider 220. Accordingly, the model service provider 220 can send a subscription response 235 to the model service consumer 210 in response to the subscription request 210.

[0043] Subscription request 230 can be a subscription message for subscribing to an analytics (e.g., Nnwdaf_MLModelProvision_Subscribe). Alternatively, subscription message 210 can be an unsubscribe message for unsubscribing from an analytics (e.g., Nnwdaf_MLModelProvision_Unsubscribe). For example, model service consumer 210 can be NWDAF 65, including AnLF 66. NWDAF 65 subscribes to, modifies, or unsubscribes from one or more trained ML models associated with an analytics ID by invoking the Nnwdaf_MLModelProvision_Subscribe and / or Nnwdaf_MLModelProvision_Unsubscribe service operations.

[0044] In some embodiments, model service consumer 210 may indicate its support for multiple ML models (if multiple ML models are available in subscription request 230). Additionally or alternatively, model service consumer 210 may include one or more Analytics IDs in subscription request 230. For each Analytics ID provided by model service consumer 210, model service provider 220 (e.g., another NWDAF including MTLF 67) may make one or more determinations.

[0045] In some embodiments, model service provider 220 may determine whether one or more existing trained ML models can be provided to model service consumer 210. Additionally or alternatively, model service provider 220 may determine whether a subscription action requires triggering further training on one or more existing trained ML models. Additionally or alternatively, model service provider 220 may determine whether multiple ML models are needed based on the request and / or based on local configuration.

[0046] If model service provider 220 determines that further training is needed, model service provider 230 may initiate data collection from one or more NFs (e.g., AMF, Data Collection Coordination Function (DCCF), Analytics Data Repository Function (ADRF)), UE applications (e.g., via AF), or Operations, Administration and Maintenance (OAM) to generate an ML model. It will be understood that the selection of one or more NFs from which to initiate training data collection may depend on parameters of the subscription request (e.g., AnalyticsID), based on which the type of data required can be determined.

[0047] If the subscription request is for subscription modification or subscription cancellation, the model service consumer 210 may include, for example, the identifier of the subscription to be modified (e.g., subscription association ID) in the Nnwdaf_MLModelProvision_Subscribe message.

[0048] If model service consumer 210 subscribes to a combination (or multiple combinations) of trained ML models associated with AnalyticsID, model service provider 220 uses ML model provisioning notification 240 to notify model service consumer 210 how model provisioning has been performed. ML model provisioning notification 240 may include one or more tuples of the following: a unique ML model identifier, ML model information associated with each AnalyticsID requested by model service consumer 210, and a subgroup ID (e.g., an identifier for a subset of UEs) for each of the multiple ML models for each AnalyticsID.

[0049] For example, ML model provisioning notification 240 may invoke the Nnwdaf_MLModelProvision_Notify service operation executed on model service consumer 210. Additionally or alternatively, when model service provider 220 determines that a previously provided trained ML model needs retraining, model service provider 200 may invoke the Nnwdaf_MLModelProvision_Notify service operation to notify of the availability of a retrained ML model.

[0050] When subscription request 230 is for subscription modification, model service provider 220 can provide a new trained ML model that is different from the previously provided model, or provide a retrained ML model (e.g., by invoking the Nnwdaf_MLModelProvision_Notify service operation).

[0051] Subscription request 230 may invoke the Nnwdaf_MLModelProvision_Subscribe or Nnwdaf_MLModelProvision_Unsubscribe service operation of model service provider 220. Specifically, subscription request 230 may, for example, subscribe to an ML model of model service provider 220. In this regard, subscription request 230 may include one or more analytics IDs, notification target addresses, and / or notification association IDs.

[0052] In some embodiments, the subscription request 230 may further include a subscription association ID (when a modification of the ML model subscription is requested), ML model filter information (to indicate the conditions for requesting an ML model for analysis based on the analysis ID), and / or the target of the ML model report (to indicate one or more objects for which the ML model is requested). For each ML model, the target of the ML report may indicate one or more specific UEs or a group or more groups of UEs. A group of UEs may, for example, be a subset of a UE group. In some embodiments, the UE group may be specified by a subgroup ID. In fact, in some embodiments, it may indicate any one or more of UEs 110 (e.g., all UEs).

[0053] Other parameters included in subscription request 230 may include one or more of the following:

[0054] - NF Consumer Information;

[0055] - The requested representativeness ratio, indicating the minimum percentage of the analyzed objects (such as UEs) in the group whose data is a non-empty set and can be used for model training;

[0056] - ML model reporting information (including, for example, the ML model target time period, which indicates the time interval for requesting the ML model);

[0057] - Expiration date, which, for example, indicates when / whether the subscription is likely to expire based on the operator's policy;

[0058] - Use case context, indicating the context in which the analysis is used;

[0059] - Inference input data information, including information about the settings expected to be used during inference, such as the input data expected to be used;

[0060] - Indicators of support for multiple ML models;

[0061] - Multiple ML model filter information to indicate the conditions for requesting multiple ML models for each analysis ID in the analysis ID;

[0062] - ML model interoperability information, which relates to model-specific characteristics such as model file format and (execution) environment;

[0063] - The time required for the model;

[0064] - ML model monitoring information (including, for example, ML model metrics, ML model monitoring report patterns, ML model accuracy thresholds, DataSetTag and ADRF ID, ML model identifier).

[0065] When a subscription is accepted, the subscription association ID, which can be used for subscription management, can be included in the subscription response 235. Additionally or alternatively, the subscription response 235 may include an expiration date (e.g., if the subscription may expire based on the operator's policy).

[0066] The ML model provisioning notification 240 may invoke the Nnwdaf_MLModelProvision_Notify service operation of the model service consumer 210. The ML model provisioning notification 240 may be sent by the model service provider 220 to notify the model service consumer 210 that has subscribed to ML model information. The ML model provisioning notification 240 may include notification association information, an analytics ID, one or more unique ML model identifiers, ML model information, and / or a subgroup ID (which identifies a subset of UEs for each of a plurality of ML models for each analytics ID). In some embodiments, the ML model provisioning notification 240 may also include ML model accuracy information. Although not shown, in some embodiments, the model service consumer 210 may respond to the ML model provisioning notification by indicating the result of the operation (e.g., success, failure).

[0067] In some embodiments, the model service consumer 210 may request ML model information from the model service provider 220, for example, by sending an ML model information request 290. Figure 6 As shown in the diagram. ML Model Information Request 290 can invoke the Nnwdaf_MLModelInfo_Request service operation of a model service provider. The ML Model Information Request may include one or more analytics IDs. In some embodiments, the ML Model Information Request may also include ML Model filter information to indicate (e.g., based on the analytics ID) one or more conditions for requesting an ML model for analysis. In some embodiments, the ML Model Information Request may also include the target of the ML Model Report to indicate one or more objects for which the ML Model is requested (e.g., a specific UE, a set of UEs, a subset of UEs, etc., as described above). In some embodiments, the ML Model Information Request may also include NF consumer information, the requested representativeness ratio, ML Model Report information (including, for example, the ML Model target time period), use case context, inference input data information, indication of support for multiple ML Models, multiple ML Model filter information (to indicate conditions for requesting multiple ML Models for each analytics ID), ML Model interoperability information, and / or ML Model accuracy monitoring information.

[0068] In response to an ML model information request, model service consumer 210 may receive an ML model information response 295 from model service provider 220. The ML model information response 295 includes one or more analysis IDs, one or more tuples of the following: a unique ML model identifier, ML model information, and a subgroup ID (e.g., identifying a subset of UEs) for each of a plurality of ML models for each analysis ID. In some embodiments, the ML model information response 295 may also include ML model accuracy information.

[0069] As described above, in some embodiments, the model service provider 220 may decide to train an ML model in order to satisfy the subscription request 230. Figure 7 This is a signaling diagram illustrating an example of such an embodiment.

[0070] Figure 7 The signaling supports the ML model analysis subscription process that triggers ML model training. In this example, model service consumer 210 can send a subscription request 230 to model service provider 220 as discussed above. Accordingly, model service provider 220 can send a subscription response 235 to model service consumer 210 in response to subscription request 230 (see above). Model service provider 220 begins training the ML model to satisfy subscription request 230 (box 250) and sends an ML model training notification 260 to model service consumer 210 accordingly.

[0071] For example, to enable federated learning (FL), model service consumer 210 can act as FL server NWDAF and subscribe to multiple NWDAFs, including MTLF. In such an example, model service provider 220 acts as FL client NWDAF, which is selected by the FL server NWDAF.

[0072] The FL server NWDAF can use subscription request 230 to check whether NWDAF 65 can meet the ML model training requirements (e.g., ML model interoperability information, analytics ID, service region, and / or data and time availability). In such a case, the FL server NWDAF may include an ML ready flag. When the ML ready flag is present in subscription request 230, model service provider 220 checks whether it can meet the ML model training requirements (e.g., ML model interoperability information, analytics ID, service region, and / or data and time availability) and / or whether the model can be successfully downloaded if the model information is provided.

[0073] The FL server NWDAF can use subscription request 230 to obtain the model accuracy of the global ML model computed by the FL client NWDAF. In this case, the service consumer NWDAF includes a model accuracy check flag. When the model accuracy check flag is present in the request, the service provider NWDAF uses its local training data as a test dataset to compute the model accuracy of the ML model provided by the service consumer NWDAF.

[0074] When model service consumer 210 determines that the ML model needs further updates, it can modify the subscription by calling the Nnwdaf_MLModelTraining_Subscribe service operation (including the subscription association ID with ML model information). This determination can be made if the model accuracy is determined to be insufficient for the analysis to be performed, for example, when the classification or analysis results of the test dataset produce too many errors. The permissible percentage of error can be set, for example, depending on the use case and the corresponding analysis to be performed.

[0075] In step 250, model service provider 220 trains the ML model by collecting new data or reusing already acquired data. If the ML model file is not provided in subscription request 230, model service provider 220 can retrieve the ML model using the information indicated in subscription request 230.

[0076] Model service provider 220 can decide to train multiple ML models for the same AnalyticsID based on an indication from model service consumer 210 that it supports multiple ML models for the same AnalyticsID (or based on local configuration).

[0077] When model service provider 220 completes ML model training, it notifies model service consumer 210 by sending an ML model training notification 260. The ML model training notification may include updated ML model information.

[0078] In order to notify the model service consumer 210, the ML model training notification 260 can call the Nnwdaf_MLModelTraining_Notify service operation of the model service consumer 110.

[0079] If model service provider 220 determines to terminate ML model training (e.g., model service provider 220 determines it will not provide further notifications related to subscription request 230), the model service provider can notify model service consumer 210 by sending a termination request (not shown). The termination request can indicate the reason for termination, for example by including a reason code in the termination request. For example, the reason can include, for example, an overload condition, unavailability of the FL process, or other reasons. The termination request can invoke the Nnwdaf_MLModelTraining_Notify service operation of model service consumer 210.

[0080] If the model service provider determines that it will train multiple ML models for the same AnalyticsID, it can provide the consumer with a tuple that includes the AnalyticsID and one or more ML model identifiers. For each ML model, ML model information, ML training information, and / or training filter information can be included in the notification sent to the model service consumer 210.

[0081] For example, to enable FL, model service provider 220 (acting as FL client NWDAF) can notify model service consumer 210 (acting as FL server NWDAF) of local ML model information and provide a status report of FL training. The status report may include the accuracy of the local model and training input data information (e.g., the region covered by the dataset, sampling rate, maximum / minimum values ​​for each dimension, etc.). If a model accuracy check flag is present in subscription request 230, model service provider 220 can notify model service consumer 210 (e.g., acting as FL server NWDAF) of the model accuracy of the global ML model.

[0082] Based on the above, subscription request 230 may invoke the Nnwdaf_MLModelTraining_Subscribe service operation of the model service provider. In this respect, subscription request 230 may subscribe to the ML model training service provided by model service provider 220. Subscription request 230 may include an analysis identifier, ML model interoperability information, a notification target address, and / or a notification association ID. In some embodiments, subscription request 230 may also include one or more of the following:

[0083] - Each analysis ID identifies one or more ML model IDs of the provided ML model(s);

[0084] - ML model information for each ML model;

[0085] - (one or more) ML model files;

[0086] - Subscribe to associated ID (in case the ML model training subscription is modified);

[0087] - ML training information for each ML model (e.g., data availability requirements, time availability requirements);

[0088] - ML preparation flag;

[0089] - ML model accuracy check flags;

[0090] - ML associated ID;

[0091] - Training filter information for each ML model;

[0092] - The goal of the training report;

[0093] - Training report information;

[0094] - Use case context;

[0095] - Iteration round ID;

[0096] - Expiry date.

[0097] When subscription request 230 is accepted, model service provider 220 may send subscription response 235 to model service consumer 210. Subscription response 235 may include a subscription association ID. The subscription association identifier can be used to easily manage the corresponding subscription. When the request is not accepted, subscription response 235 may include an error response (e.g., a reason code indicating failure). For example, an error response may indicate that model service provider 220 does not meet ML training requirements, ML training is incomplete, model service provider 220 is under load, model service provider 220 is no longer available for the FL process, or other error reasons.

[0098] In some embodiments, subscription response 235 may also include an ML association ID (e.g., confirmation of subscription to the FL process).

[0099] The ML model training notification 260 can invoke the Nnwdaf_MLModelTraining_Notify service operation of the model service consumer 210. In this respect, the ML model training notification can be used to notify the model service consumer 210 of one or more trained ML models. The model service provider 220 can also use the ML model training notification to instruct the model service consumer to terminate ML model training.

[0100] The ML model training notification 260 may include notification association information indicating a notification association ID that has been assigned by the consumer during ML model training. In some embodiments, the ML model training notification 260 may also include one or more of the following:

[0101] - Analyze ID;

[0102] - ML model information for each ML model;

[0103] - ML training information and / or training filter information for each ML model;

[0104] - ML associated ID (e.g., in FL associated embodiments);

[0105] - The corresponding use case context;

[0106] - A termination request indicator that indicates that model service provider 220 requests termination of ML model training and / or that model service provider 220 will not provide further notification related to subscription request 230. A reason code may be provided along with this indicator, indicating any reason for termination (e.g., NWDAF overload, no longer available for FL process, etc.).

[0107] - ML model ID, identifying the supplied ML model;

[0108] - Global ML model accuracy, indicating the accuracy of the global ML model. This can be calculated, for example, by a node acting as an FL client NWDAF using its local training data as the test dataset;

[0109] - FL training status report. The status report may include local ML model metrics and / or training input data information (e.g., the region covered by the dataset, sampling rate, maximum / minimum values ​​for each dimension, etc.). This information may be generated, for example, by the node acting as the FL client NWDAF during the FL process;

[0110] - Delayed event notification

[0111] - Iteration round ID.

[0112] Although not shown, the model service consumer 210 may use result indications (e.g., success, failure) to respond to the ML model training notification 260.

[0113] like Figure 8As shown, model training information can be obtained (e.g., by sending a model training information request 297 to model service provider 220) or by model service consumer 210. The model training information request 297 can invoke the Nnwdaf_MLModelTrainingInfo_Request service operation of the model service provider. The model training information request 297 can request information about ML model training. The model training information request 297 may include analytics ID and / or ML model interoperability information.

[0114] In some embodiments, the model training information request 297 may also include any one or more of the following:

[0115] - (one or more) ML model IDs, identifying (one or more) provided ML models;

[0116] - ML model information for each ML model;

[0117] - (one or more) ML model files;

[0118] - ML training information for each ML model (e.g., data availability requirements, time availability requirements);

[0119] - Training report information;

[0120] - ML preparation flag;

[0121] - ML model accuracy check flags;

[0122] - ML associated ID;

[0123] - Termination requests, such as when terminating an FL identified by an ML association ID. Additionally, an indication of the termination reason can be provided, such as when the FL server NWDAF deselects the FL client NWDAF for the FL process, or when the FL process is paused;

[0124] - Training filter information for each ML model;

[0125] - The goal of the training report;

[0126] - Use case context.

[0127] As shown in step 298, the model service provider 220 obtains relevant information about model training. In some embodiments, the model service provider 220 obtains information about model training based on ML model files or other ML model information.

[0128] Model service provider 220 may respond to a model training information request using model training information response 299. When the request is accepted, the model training information response may include a result indicator associated with the request. When the request is not accepted, model service provider 220 may respond using an error response. The error response may include a reason code (e.g., NWDAF does not meet ML training requirements, ML training is incomplete, NWDAF is overloaded, or it is no longer available for the FL process).

[0129] In some embodiments, the model training information response 299 may also include one or more of the following:

[0130] - (one or more) ML model IDs;

[0131] - Analyze ID;

[0132] - ML model information for each ML model;

[0133] - ML training information;

[0134] - Training filter information for each ML model;

[0135] - ML association ID (e.g., for federated learning);

[0136] - The corresponding use case context;

[0137] - Global ML model accuracy, indicating the accuracy of the global ML model. Accuracy can be calculated, for example, by a network node acting as an FL client NWDAF using its local training data as the test dataset;

[0138] - Status report for FL training. The status report may include local ML model metrics and training input data information generated by the network nodes acting as FL clients (NWDAF) during the FL process (e.g., the region covered by the dataset, sampling rate, maximum / minimum values ​​for each dimension of the data, etc.).

[0139] - Delayed event notification;

[0140] - Global ML model metrics.

[0141] Model service consumer 210 and / or model service provider 220 can be provided by, for example Figure 9 The example schematically illustrates network node 700. Figure 9Network node 700 includes processing circuitry 710, memory circuitry 720, and interface circuitry 730. Processing circuitry 710 is communicatively coupled to memory circuitry 720 and interface circuitry 730, for example, via bus 704. Processing circuitry 710 may include one or more microprocessors, microcontrollers, hardware circuitry, discrete logic circuitry, hardware registers, digital signal processors (DSPs), field-programmable gate arrays (FPGAs), application-specific integrated circuits (ASICs), or combinations thereof. For example, processing circuitry 710 may be programmable hardware capable of executing software instructions stored in memory circuitry 720, for example, as a machine-readable computer program 740. The memory circuitry 720 in various embodiments may include any non-transitory machine-readable medium (whether volatile or non-volatile) known in or developed in the art, including but not limited to solid-state media (e.g., SRAM, DRAM, DDRAM, ROM, PROM, EPROM, flash memory, solid-state drives, etc.), removable storage devices (e.g., Secure Digital (SD) cards, mini SD cards, micro SD cards, memory sticks, thumb drives, USB flash drives, ROM cartridges, universal media discs), fixed drives (e.g., magnetic hard disk drives), etc., including all of them or any combination thereof.

[0142] Interface circuitry 730 may be a controller hub configured to control the input and output (I / O) data paths of network node 700. Such I / O data paths may include data paths for exchanging signals over the network. Interface circuitry 730 may be implemented as a single physical component or as multiple physical components arranged consecutively or separately, any one of which may be communicatively coupled to any other component or may communicate with any other component via processing circuitry 710. For example, interface circuitry 730 may include a transmitter 732 configured to transmit wireless communication signals and a receiver 734 configured to receive wireless communication signals.

[0143] Network node 700 can (e.g., by processing circuitry 710) be configured to perform method 300 and / or method 400 as described above.

[0144] Other embodiments include a computer program 740 containing instructions that, when executed on the processing circuitry 710 of the network node 700, cause the network node 700 to perform the methods 300 and / or 400 described above.

[0145] Other embodiments include a carrier containing a computer program 740. This carrier is one of an electronic signal, an optical signal, a radio signal, or a computer-readable storage medium.

[0146] Additional services and signaling variations will now be described. Any of the embodiments described above can be modified, supported, or enhanced by the services and signaling described below. The services and signaling below can, for example, be used to enhance the compatibility and / or integration of the embodiments described above with legacy networks. Additionally or alternatively, the services and signaling below can implement a wider variety of practical use cases in which the embodiments described above can be used.

[0147] Alternative ML model provisioning processes may include an NWDAF 65 containing AnLF 66, which is locally configured with a set of IDs of an NWDAF 65 containing MTLF 67 and one or more analytics IDs supported by each NWDAF 65 containing MTLF 67 to retrieve a trained ML model. Alternatively, an NWDAF discovery process may be used to discover NWDAF 65s containing MTLF 67. An NWDAF 65 containing MTLF 67 can determine the need for further training of an existing ML model, as described above, upon receiving an ML model subscription or ML model request, for example, based on parameters included in the request.

[0148] The alternative ML model subscription / unsubscription process can have the same characteristics as... Figure 5 The signaling flow is similar to that shown, and can be used by model service consumer 210 (e.g., NWDAF 65 containing AnLF 66) to subscribe / unsubscribe at another NWDAF 65 (e.g., NWDAF 65 containing MTLF 67) to be notified using the Nnwdaf_MLModelProvision service when ML model information regarding the relevant analysis becomes available. This ML model information is used by the NWDAF 65 containing AnLF 66 to export the analysis. This service is also used by NWDAF 65 to modify one or more existing ML model subscriptions. NWDAF 65 can be both a consumer of this service provided by one or more other NWDAFs and a provider of this service to one or more other NWDAFs.

[0149] Regarding subscription request 230, model service consumer 210 subscribes to, modifies, or unsubscribes from a set of trained ML models associated with an analysis ID(s) by invoking the Nnwdaf_MLModelProvision_Subscribe / Nnwdaf_MLModelProvision_Unsubscribe service operations. Optionally, model service consumer 210 indicates its support for multiple ML models(s), if available.

[0150] When a subscription to a trained ML model associated with an analytics ID is received, the model service provider 220 (e.g., an NWDAF containing MTLF) can determine:

[0151] - Can an existing trained ML model(s) be used for this subscription; or

[0152] - Whether this subscription should trigger further training on the existing trained ML model.

[0153] If the NWDAF 65 containing MTLF 67 determines that further training is needed, the NWDAF 65 can initiate data collection from NF (e.g., AMF / DCCF / ADRF), UE applications (via AF), or OAM to generate an ML model.

[0154] If the service call is for subscription modification or subscription cancellation, the model service consumer 210 can include the identifier to be modified (subscription association ID) in the call to Nnwdaf_MLModelProvision_Subscribe.

[0155] Regarding ML model training notification 260, if model service consumer 210 subscribes to a trained ML model associated with an analytics ID (set of analytics IDs), then NWDAF 65, containing MTLF 67, notifies model service consumer 210 of a unique pair of ML model identifiers and ML model information associated with each analytics ID requested by the service consumer.

[0156] The Nnwdaf_MLModelProvision_Notify service operation is invoked. When the model service provider 220 determines that a previously provided trained ML model needs to be retrained, NWDAF 65, which includes MTLF 67, also invokes the Nnwdaf_MLModelProvision_Notify service operation to notify the available retrained ML model.

[0157] When subscription request 230 is for subscription modification (e.g., including subscription association ID), NWDAF65 containing MTLF 67 can provide a new trained ML model that is different from the previously provided ML model, or provide a retrained ML model by invoking the Nnwdaf_MLModelProvision_Notify service operation.

[0158] Model service consumers of ML model provisioning services (e.g., NWDAF 65 containing AnLF 66) can provide the following input parameters:

[0159] Information regarding the analysis that will use the requested ML model includes:

[0160] Analysis ID list: Identifies the analysis for which an ML model is used.

[0161] [Optional] NF Consumer Information: Identifies the supplier of NWDAF containing AnLF.

[0162] [Optional] Use Case Context: Indicates the context in which the analytics are used to select the most relevant ML model. When multiple ML models are available for the requested analytics(s), the model service provider 220 can use the "Use Case Context" parameter to select the most relevant ML model.

[0163] [Optional] ML Model Interoperability Information. This is vendor-specific information that transmits information such as the requested model file format, model execution environment, etc. The encoding, format, and values ​​of the ML model interoperability information are not specified because it is vendor-specific and will be agreed upon between vendors for sharing purposes if necessary.

[0164] [Optional] ML Model Filter Information: Enables the model service provider 220 to select which ML model to request for analysis, such as S-NSSAI or related regions. The parameter types in the ML model filter information are the same as those in the analysis filter information defined during the process.

[0165] [Optional] The target of the ML model report: indicates the object(s) for which it requests the ML model, such as a specific UE, a group of UEs, or any UE (i.e., all UEs).

[0166] [Optional] Requested representativeness ratio: The minimum percentage of UEs in a group whose data is a non-empty set and can be used for model training when the target of the ML model is a set of UEs.

[0167] ML Model Reporting Information: When used with the Nnwdaf_MLModelProvision_Subscribe service, this can include event reporting information parameters.

[0168] [Optional] ML Model Target Time Period: Indicates the time interval (e.g., start, end) of the ML model requested for analysis. The time interval can be expressed as the actual start time and actual end time (e.g., via UTC time).

[0169] [Optional] Inference Input Data Information: Contains information about various settings that AnLF 66 is expected to use during inference, such as the "input data" to be used. Each input data may optionally be accompanied by a metric showing the granularity at which the data will be used (e.g., sampling rate, maximum number of input values, and / or the maximum time interval between samples of that input data). This may be a subset of possible input data specified for a particular analysis type.

[0170] The data source is expected to be used as a list of NF instance (or NF group) identifiers.

[0171] The notification target address (+notification association ID) enables notifications received from NWDAF 65 containing MTLF 67 to be associated with this subscription.

[0172] [Optional] Instructions to support multiple ML models.

[0173] [Optional] (one or more) relevant accuracy levels.

[0174] [Optional] Number of ML models: Indicates the maximum number of ML models that NWDAF 65 containing MTLF 67 can provide to NWDAF 65.

[0175] [Optional] When the model is needed: Indicates the latest time when the consumer expects to receive (one or more) ML models.

[0176] [Optional] ML model monitoring information, such as:

[0177] [Optional] ML model metrics (e.g., ML model accuracy)

[0178] [Optional] ML model monitoring reporting mode, such as accuracy reporting interval or scheduled status. Depending on the reporting mode, NWDAF 65 containing MTLF 67 periodically reports model accuracy to NWDAF 65 containing AnLF 66, or when the ML model accuracy exceeds the ML model accuracy threshold (i.e., the accuracy becomes higher or lower than the ML model accuracy threshold).

[0179] [Optional] ML Model Accuracy Threshold, indicating the accuracy threshold of the ML model requested by the consumer (as a predefined state). It can also be used as an indication that MTLF is triggered to perform accuracy monitoring operations on the ML models supplied to AnLF.

[0180] [Optional] DataSetTag and ADRF ID (if available). This can indicate the inference data stored in ADRF (including input data, predictions, and baseline fact data at the time referenced in the prediction), which MTLF can use to retrain or refeed the ML model.

[0181] [Optional] ML Model Identifier: Indicates the model involved in the data corresponding to the DataSetTag (in the case of subscription modification).

[0182] The NWDAF 65, which includes MTLF 67, can provide the following output information to consumers of ML model-based service operations:

[0183] Notification related information (used with the Nnwdaf_MLModelProvision_Notify service)

[0184] For each analytics ID requested by model service consumer 210, a set of one or more of the following items: a unique ML model identifier, and the following information:

[0185] ML model information: This includes the ML model file address, such as a Uniform Resource Locator (URL), a Fully Qualified Domain Name (FQDN), or an ADRF (Group) ID. When the ADRF (Group) ID is supplied, the storage transaction ID can also be supplied.

[0186] [Optional] ML Model Degradation Indicator: Indicates whether the provided ML model is degraded.

[0187] [Optional] Validity Period: Indicates the period during which the ML model information is provided by the application, and / or Spatial Validity: Indicates the region of the ML model information provided by the application. Spatial validity and validity period are determined by the internal logic of MTLF 67 and are subsets of AoI (if provided in the ML model filter information) and the target period of the ML model, respectively.

[0188] [Optional] ML Model Representativeness Ratio: Indicates the percentage of UEs in the group whose data were used for training the ML model when the target of the ML model is a set of UEs.

[0189] [Optional] Training input data information: Contains information about various settings that have been used by MTLF during training, such as:

[0190] The “input data” that has been used, each input data may optionally be accompanied by a metric that shows the characteristics of the data and the granularity of the data’s use (e.g., sampling rate, maximum number of input values ​​and / or maximum time interval between samples of the input data, data range including maximum and minimum values, mean and standard deviation, and data distribution (where applicable)) and the time when the data was obtained (e.g., timestamp and duration).

[0191] The data sources associated with the "input data" used for training the ML model are identified by a list of NF instance (or NF group) identifiers. This can be a subset of possible input data specified for a particular analysis type. Note that data source information enables ML model selection when different models are available for the analysis ID, or it allows consumers to avoid selecting an ML model that uses data from a specific data source at a specific time or data characterized by specific data features.

[0192] [Optional] ML Model Accuracy Information: If an ML model accuracy threshold is requested, this indicates the accuracy of the ML model, including:

[0193] Accuracy value of the ML model.

[0194] [Optional] ML model metrics (i.e., ML model accuracy).

[0195] The alternative ML model information request process can include and Figure 6Similar signaling is shown in the diagram. Model service consumer 210 (e.g., NWDAF 65 containing AnLF 66) can use this procedure to request and obtain ML model information from another NWDAF 65 (e.g., NWDAF 65 containing MTLF 67) using the Nnwdaf_MLModelInfo service. The NWDAF 65 containing AnLF 66 uses this ML model information to derive an analysis. NWDAF 65 can be both a consumer of this service provided by one or more other NWDAFs and a provider of this service to one or more other NWDAFs.

[0196] Regarding ML model information request 290, model service consumer 210 can request a set of ML models associated with an analysis ID by invoking the Nnwdaf_MLModelInfo_Request service operation. Optionally, model service consumer 210 indicates its support for multiple ML models (if available).

[0197] When a request for ML model information 290 for analysis is received, NWDAF 65, which includes MTLF 67, can:

[0198] Determine whether an existing trained ML model(s) can be used for the request; or

[0199] Determine whether the request requires triggering further training on the existing trained ML model.

[0200] If the NWDAF 65 containing MTLF 67 determines that further training is needed, the NWDAF 65 can initiate data collection from NF (e.g., AMF / DCCF / ADRF), UE applications (via AF), or OAM to generate an ML model.

[0201] Regarding the ML model information response 295, the NWDAF 65 containing MTLF 67 responds to the model service consumer 210 by invoking the Nnwdaf_MLModelInfo_Request response service operation, which includes one or more pairs of the following: a unique ML model identifier, and ML model information for each analytics ID requested by the model service consumer 210.

[0202] The alternative ML model training subscription process can include with Figure 7Similar signaling is shown in the diagram. Model service consumer 210 (e.g., NWDAF 65 containing MTLF 67) can use this process to subscribe to another NWDAF 65 (e.g., NWDAF 65 containing MTLF 67) for a trained ML model based on the ML model file or ML model information provided by model service consumer 210. NWDAF 65 containing MTLF 67 can use this service to enable, for example, joint learning or update the ML model. Model service consumer 210 can also use this service to request NWDAF 65 containing MTLF 67 to prepare for training an ML model or modify an existing ML model training subscription.

[0203] Regarding subscription request 230, model service consumer 210 can subscribe to or unsubscribe from training an ML model by calling the Nnwdaf_MLModelTraining_Subscribe / Nnwdaf_MLModelTraining_Unsubscribe service operations. To enable joint learning, model service consumer 210 can act as an FL server NWDAF and can subscribe to multiple NWDAFs 65 containing MTLF 67 that act as FL client NWDAFs (which are selected by the FL server NWDAF).

[0204] The FL server NWDAF can use this request to check if the NWDAF can meet the ML model training requirements (e.g., ML model interoperability information, analytics ID, service region, and / or data and time availability). In such cases, the FL server NWDAF includes an ML ready flag. When this ML ready flag is present in the request, the service provider NWDAF only checks whether it can meet the ML model training requirements (e.g., ML model interoperability information, analytics ID, service region, and / or data and time availability) and / or whether it can successfully download the model if the model information is provided.

[0205] The FL server NWDAF can use this request to obtain the model accuracy of the global ML model computed by the FL client NWDAF. In this case, the service consumer NWDAF includes a model accuracy check flag. When this model accuracy check flag is present in the request, the service provider NWDAF uses its local training data as a test dataset to compute the model accuracy of the ML model provided by the service consumer NWDAF.

[0206] When the model service consumer 210 determines that the ML model needs to be updated further, the model service consumer 210 can modify the subscription by calling the Nnwdaf_MLModelTraining_Subscribe service operation (including the subscription association ID with ML model information).

[0207] Regarding training the ML model in step 250, model service provider 220 trains the ML model indicated in subscription request 230 by collecting new data or reusing data it possesses. If the ML model file is not provided in subscription request 230, model service provider 220 should first obtain the ML model using the information indicated in subscription request 230.

[0208] Regarding ML model training notification 260, when model service provider 220 completes ML model training, model service provider 220 notifies NWDAF service consumer of the updated ML model information by calling the Nnwdaf_MLModelTraining_Notify service operation.

[0209] If the model service provider 220 determines to terminate ML model training, or determines not to provide further notification related to the request, the model service provider 220 can notify the model service consumer 210 of the termination request indication with a reason code (e.g., NWDAF overload, no longer available for FL process, etc.) by invoking the Nnwdaf_MLModelTraining_Notify service operation.

[0210] To enable federated learning, the model service provider 220, acting as the FL client NWDAF, can notify the model service consumer 210, acting as the FL server NWDAF, of local ML model information and FL training status reports, including the accuracy of the local model and training input data information (e.g., the region covered by the dataset, sampling rate, maximum / minimum values ​​for each dimension, etc.).

[0211] If the model accuracy check flag is present in Nnwdaf_MLModelTraining_Subscribe, the model service provider 220, acting as an FL client NWDAF, can notify the model service consumer 210, acting as an FL server NWDAF, of the model accuracy of the global ML model.

[0212] The model service consumer 210 of the ML model training service (e.g., NWDAF 65 including MTLF 67) can provide the following input parameters in the Nnwdaf_MLModelTraining_Subscribe or Nnwdaf_MLModelTrainingInfo_Request service operation:

[0213] Analysis ID: Identifies the analysis for which a request to train an ML model was made.

[0214] ML model interoperability information.

[0215] The notification target address (+notification association ID) allows notifications received from an NWDAF containing MTLF to be associated with a subscription, such as when used with Nnwdaf_MLModelTraining_Subscribe.

[0216] [Optional] ML model information.

[0217] [Optional] ML model file.

[0218] [Optional] ML Model ID: Identifies the provided ML model.

[0219] [Optional] ML preparation flags: Indicate whether the request is for preparing for federated learning or for performing federated learning.

[0220] [Optional] ML Model Accuracy Check Flag: Identifies that the request is used to calculate the model accuracy of the global ML model provided by the NWDAF service consumer, which acts as the FL server NWDAF, using local training data as the test dataset.

[0221] [Optional] ML Association ID: Identifies the joint learning process used to train the ML model. This parameter is included when the service is used for joint learning.

[0222] [Optional] Available Data Requirements. This is used to inform about the required available data for training the ML model. For example, the FL server NWDAF sends this requirement to the FL client NWDAF in the preparation request to select the FL client NWDAF that can meet the available data requirements. Available data requirements may include the following:

[0223] A list of event IDs to be collected for local model training.

[0224] Statistical characteristics of the dataset.

[0225] The time window for the data sample.

[0226] Minimum number of data samples.

[0227] [Optional] Availability Time Requirement. This is used to notify about the availability time required for training the ML model. For example, the FL server NWDAF sends this requirement to the FL client NWDAF in a preparation request to select an FL client NWDAF that is available for the time required to train the ML model.

[0228] [Optional] Training Filter Information: Enables selection of which data to request for ML model training, such as S-NSSAI or related regions. The parameter types in the training filter information are the same as or a subset of the parameter types in the ML model filter information.

[0229] [Optional] The target of the training report: indicates one or more objects for which it requests data for training the ML model, i.e., a set of UEs or any UE (i.e., all UEs).

[0230] [Optional] Use case context: Indicates the context in which the ML model is used.

[0231] [Optional] Training report information, including maximum response time, which indicates the maximum time to wait for notification (e.g., model training results).

[0232] [Optional] Iteration Round ID: Indicates the iteration round number of the current ML model training.

[0233] [Optional] Expiry date.

[0234] The model service provider 220 provides output information to the model service consumer 210 in a notification. The output may include:

[0235] Notification related information.

[0236] [Optional] ML model information.

[0237] [Optional] ML Model ID: Identifies the supplied ML model.

[0238] [Optional] Model accuracy: The model accuracy of the global ML model, which is calculated by the FL client NWDAF using the local training data as the test dataset.

[0239] [Optional] FL training status report: the accuracy of the local model and training input data information generated by the FL client NWDAF during the FL process (e.g., the region covered by the dataset, sampling rate, maximum / minimum values ​​for each dimension, etc.).

[0240] [Optional] ML Association ID. This parameter can be included when the service is used for federated learning.

[0241] [Optional] Iteration Round ID: Indicates the iteration round number of the ML model training as instructed by the FL server NWDAF.

[0242] [Optional] Delayed event notification, with the following parameters:

[0243] Delay event indicator: This parameter indicates that the FL client NWDAF cannot complete the training of the temporary local ML model within the maximum response time provided by the FL server NWDAF.

[0244] [Optional] Reason code (e.g., local ML model training failed, local ML model training requires more time, etc.).

[0245] [Optional] Expected time to complete training: Indicate the expected remaining training time to the FL server NWDAF, and may be provided together with a delay event notification.

[0246] The alternative ML model training information request process can include... Figure 8 A similar signaling flow is shown in the diagram. Model service consumer 210 (e.g., NWDAF 65) can use this process to request information about ML model training from model service provider 220 (e.g., another NWDAF) based on ML model files or ML model information provided by model service consumer 210. This service can be used to enable, for example, joint learning.

[0247] Regarding model training information request 297, model service consumer 210 can send a request to model service provider 220 by calling the Nnwdaf_MLModelTrainingInfo_Request service operation to obtain information about ML model training based on ML model files or ML model information provided by model service consumer 210.

[0248] To enable federated learning, model service consumer 210 (e.g., acting as FL server NWDAF) can request ML model training information from model service provider 220 (e.g., acting as MTLF-containing NWDAF of FL client NWDAF selected by FL server NWDAF).

[0249] Model service consumer 210 can use this request to check whether NWDAF 65 can meet ML model training requirements (e.g., ML model interoperability information, analytics ID, service region / DNAI, and / or data and time availability). In such cases, the NWDAF service consumer includes ML readiness flags.

[0250] Model service consumer 210 can use this request to obtain the model accuracy of the ML model provided by the service consumer using local training data in an NWDAF containing MTLF as the test dataset. In such a case, the service consumer NWDAF includes model accuracy check flags.

[0251] In step 298, when the ML ready flag is present in request 297, model service provider 220 checks whether it meets the ML model training requirements and / or whether the model can be successfully downloaded (if model information is provided). Based on the check results, model service provider 220 obtains a success return code or a failure reason code (e.g., NWDAF does not meet ML training requirements) as information about ML model training.

[0252] When the model accuracy check flag is present in request 297, model service provider 220 uses local training data as a test dataset to calculate the model accuracy of the ML model provided by the service consumer. Model service provider 220 includes the model accuracy in the information about the ML model training.

[0253] When model service provider 220 is training an ML model based on ML model files or ML model information provided by model service consumer 210 (as described in Section 2.1.2.2), model service provider 210 obtains a failure reason code (e.g., ML training not completed) as information about the ML model training.

[0254] When the model service provider 220 completes ML model training based on ML model files or ML model information provided by the model service consumer 210 (as described in Section 2.1.2.2), the model service provider 220 obtains a success return code and ML model information of the trained ML model as information about the ML model training.

[0255] Regarding model training information response 299, model service provider 220 responds to model service consumer 210 with information about ML model training by calling Nnwdaf_MLModelTrainingInfo_Request response service operation.

[0256] In some embodiments, the Nnwdaf_MLModelProvision service is also supported. The Nnwdaf_MLModelProvision service enables the model service consumer 210 to receive a notification when an ML model matching the subscription parameters becomes available.

[0257] When a subscription is accepted by model service provider 220, model service consumer 210 receives an identifier (subscription association ID), allowing for further management (modification, deletion) of the subscription. Modifications to ML model subscriptions can be implemented, for example, by NWDAF 65 based on operator policies and configurations.

[0258] In some embodiments, the Nnwdaf_MLModelProvision_Subscribe service operation may also be supported. The Nnwdaf_MLModelProvision_Subscribe service can be used to subscribe to an NWDAF ML model provision. This service can be invoked, for example, by subscription request 230, which can provide one or more inputs to the service.

[0259] The input for this service may include one or more analytics IDs, notification target address (+notification associated ID).

[0260] These inputs may optionally include: subscription association ID (in the case of modifying the ML model subscription), ML model filter information (to indicate the conditions for requesting the ML model for analysis) and ML model reporting target (to indicate the object(s) for which the ML model(s) are requested (e.g., a specific UE, a group of UEs, or any UE(i.e., all UEs)), NF consumer information, requested representativeness ratio, ML model reporting information (including, for example, the ML model target time period), expiration time, use case context, inference input data information, indication of support for multiple ML models, multiple ML model filter information (to indicate the conditions for requesting multiple ML models), ML model interoperability information, the time required for the model, and ML model monitoring information (including, for example, ML model metrics, ML model monitoring reporting patterns, ML model accuracy thresholds, DataSetTag and ADRF ID, and ML model identifier).

[0261] The service can generate a subscription response 235 that includes one or more outputs. When the subscription is accepted, the output(s) may include a subscription association ID (which can be used to manage the subscription) and an expiration time (which can be useful if the subscription may expire based on the operator's policy).

[0262] In one or more embodiments, the Nnwdaf_MLModelProvision_Unsubscribe service may be supported. The Nnwdaf_MLModelProvision_Unsubscribe service can be invoked by subscription request 230 to unsubscribe from ML model provisioning notifications, for example, by providing the subscription association ID in the request. An indication of the operation's result can be included in the corresponding subscription response 235.

[0263] In some embodiments, the Nnwdaf_MLModelProvision_Notify service may be supported. The Nnwdaf_MLModelProvision_Notify service can be used to provide ML model information to model service consumers 210 that have subscribed to a specific service.

[0264] The input may include notification-related information and a set of items including: an analysis ID, and one or more tuples containing a unique ML model identifier and ML model information. In some embodiments, the input may also include ML model accuracy information.

[0265] In response, an indication of the operation execution result can be provided.

[0266] In some embodiments, the Nnwdaf_MLModelInfo_Request service may be supported. The Nnwdaf_MLModelInfo_Request service enables the model service consumer 210 to request NWDAF ML model information. The service can take one or more analysis IDs as input. In some embodiments, the service can also take the following as input: ML model filter information (indicating the conditions for the ML model requested for analysis) and ML model reporting target (indicating the object(s)(e.g., a specific UE, a group of UEs, or any UE(i.e., all UEs)) for the requested ML model), NF consumer information, the requested representativeness ratio, ML model reporting information (including, for example, the ML model target time period), use case context, inference input data information, indication of support for multiple ML models, multiple ML model filter information (indicating the conditions for the multiple ML models requested), ML model interoperability information, and ML model accuracy monitoring information.

[0267] In response to this request, the service can output one or more analysis IDs. For each analysis ID, one or more tuples containing a unique ML model identifier and ML model information can be provided. In response, the service can output ML model accuracy information.

[0268] In some embodiments, the Nnwdaf_MLModelTraining service may be supported. This service enables the model service consumer 210 to subscribe to / unsubscribe from / notify / modify ML model training.

[0269] When used for federated learning, this service enables the FL server NWDAF to enable federated learning, while providing global ML model information to the FL client NWDAF and obtaining local ML model information and FL training status reports.

[0270] This service can also be used by model service consumer 210 (e.g., FL server NWDAF) to check whether model service provider 220 (e.g., FL client NWDAF) can meet the training requirements of ML model.

[0271] This service can also be used by model service consumer 210 to request service provider to calculate and provide the model accuracy of the global ML model.

[0272] Some implementations can support the Nnwdaf_MLModelTraining_Subscribe service. The Nnwdaf_MLModelTraining_Subscribe service can be used to subscribe to ML model training. This service can take the following as input:

[0273] - Analyze ID;

[0274] - ML model interoperability information;

[0275] - Notify target address (+notification associated ID).

[0276] In some embodiments, the input may also include one or more of the following:

[0277] - ML Model ID: Identifies the provided ML model;

[0278] - ML model information (as defined in section 2.1.1.3);

[0279] - ML model files;

[0280] - Subscribe to associated ID (in case the ML model training subscription is modified);

[0281] - ML training information, namely, data availability requirements and time availability requirements;

[0282] - ML preparation flag;

[0283] - ML model accuracy check flags;

[0284] - ML associated ID;

[0285] - Training filter information;

[0286] - The goal of the training report;

[0287] - Training report information, as defined in section 2.1.2.2;

[0288] - Use case context;

[0289] - Iteration round ID;

[0290] - Expiry date.

[0291] This service can generate various outputs. When a request is accepted, the service can output the subscription association ID (which can be used to manage the subscription). When a request is not accepted, the service can output an error response with a reason code (e.g., NWDAF 65 does not meet ML training requirements, ML training is incomplete, NWDAF is overloaded, and it is no longer available for the FL process).

[0292] In some embodiments, the output may also include an ML association ID (e.g., confirmation of subscription to the FL process).

[0293] One or more embodiments may support the Nnwdaf_MLModelTraining_Unsubscribe service. The Nnwdaf_MLModelTraining_Unsubscribe service can be used to terminate NWDAF ML model training. This service can take the subscription association ID as input. As output, the service can provide an indication of the operation execution result. Optionally, the service can also provide a reason code (e.g., the FL server NWDAF deselected the FL client NWDAF for the FL process, or the FL process was paused or completed, etc.). Finally, it can provide aggregated ML model information (if FL has completed) or updated aggregated ML model information (if FL was paused).

[0294] In some embodiments, the Nnwdaf_MLModelTraining_Notify service is supported. Model service provider 220 can use the Nnwdaf_MLModelTraining_Notify service to notify model service consumer 210 about trained ML models that have subscribed to a specific NWDAF service. This service can also be used to instruct model service consumer that model service provider 220 will terminate ML model training.

[0295] The input to this service can include notification association information. This parameter indicates the notification association ID that has been assigned by the consumer during ML model training.

[0296] The input for this service may also include:

[0297] - Includes one or more tuples containing analysis ID and ML model information;

[0298] - ML associated ID, when used for federated learning;

[0299] - The corresponding use case context;

[0300] - Termination Request: This parameter indicates that NWDAF requests to terminate ML model training. That is, NWDAF will not provide any further notification related to this request and will have a reason code (e.g., NWDAF overload, no longer available for FL process, etc.).

[0301] - ML Model ID: This parameter identifies the supplied ML model;

[0302] - Global ML Model Accuracy: The model accuracy of the global ML model, which is calculated by the FL client NWDAF using the local training data as the test dataset;

[0303] - FL training status report: local ML model metrics and training input data information (e.g., regions covered by the dataset, sampling rate, maximum / minimum values ​​for each dimension, etc.), which are generated by the FL client NWDAF during the FL process;

[0304] - Delayed event notification;

[0305] - Iteration round ID.

[0306] As output, the service can provide an indication of the operation execution results.

[0307] In some embodiments, the Nnwdaf_MLModelTrainingInfo service is supported. This service enables the model service consumer 210 to request information about ML model training based on ML model files or ML model information provided by the model service consumer 210.

[0308] When used for federated learning, this service enables the FL server NWDAF to enable federated learning, while providing global ML model information to the FL client NWDAF and obtaining local ML model information from the FL client NWDAF.

[0309] In some embodiments, the Nnwdaf_MLModelTrainingInfo_Request service may be supported. Nnwdaf_MLModelTrainingInfo_Request can be used to request information about the training of an NWDAF ML model with specific parameters.

[0310] The inputs to this service can include analytics IDs and ML model interoperability information.

[0311] In some embodiments, the input may also include one or more of the following:

[0312] - ML Model ID: Identifies the provided ML model;

[0313] - ML model information;

[0314] - ML model files;

[0315] - ML training information (e.g., data availability requirements, time availability requirements);

[0316] - Training report information;

[0317] - ML preparation flag;

[0318] - ML model accuracy check flags;

[0319] - ML associated ID;

[0320] - Terminate request when terminating federated learning identified by the ML association ID and optionally indicating a reason (e.g., FL server NWDAF deselects FL client NWDAF for FL process, or FL process is paused, etc.);

[0321] - Training filter information;

[0322] - The goal of the training report;

[0323] - Use case context.

[0324] Depending on the implementation, the service may generate various outputs. For example, when a request is accepted, the service may output an indication of the operation execution result. When a request is not accepted, the service may output an error response with a reason code (e.g., NWDAF 65 does not meet ML training requirements, ML training is incomplete, NWDAF is overloaded, no longer available for FL process, etc.).

[0325] In some embodiments, the output(s) may also include one or more of the following:

[0326] - ML model ID;

[0327] - Includes one or more tuples containing analysis ID and ML model information;

[0328] - ML associated ID, when used for federated learning;

[0329] - The corresponding use case context;

[0330] - Global ML Model Accuracy: The model accuracy of the global ML model, which is calculated by the FL client NWDAF using the local training data as the test dataset;

[0331] - FL training status report: local ML model metrics and training input data information (e.g., region covered by the dataset, sampling rate, maximum / minimum values ​​for each dimension, etc.), which are generated by the FL client NWDAF during the FL process;

[0332] - Delayed event notification;

[0333] - Global ML model metrics.

[0334] While the various communication devices described herein may include the combinations of hardware components shown, other embodiments may include computing and / or communication hardware with different combinations of components. It will be understood that these computing devices may include any suitable combination of hardware and / or software required to perform the tasks, features, functions, and methods disclosed herein. Furthermore, although components are depicted as single boxes located within larger boxes or nested within multiple boxes, in practice, the devices described herein may include multiple different physical components constituting a single illustrated component, and functionality may be partitioned among the individual components.

Claims

1. A method (300) implemented by a network node (700) acting as a model service provider (220) in a wireless communication network (10), the method comprising: Receive (310) machine learning ML model subscription request (230) from model service consumer (210); as well as Determine (320) multiple ML models to satisfy the ML model subscription request (230).

2. The method according to claim 1, wherein, The plurality of ML models used to satisfy the ML model subscription request (230) are determined based on one or more parameters included in the ML model subscription request (230).

3. The method according to claim 1 or 2, wherein, The ML model subscription request (230) includes one or more of the following parameters: - Model service consumer (210) information, - Indicators of support for multiple ML models, - One or more analytics IDs, - Notify the target address, - Notify associated ID, - The time interval required to request the ML model. - The requested representativeness ratio, - Use case context, - Conditions for requesting multiple ML models - ML model interoperability information.

4. The method according to any one of the preceding claims, further comprising: Send a model subscription response (235) to the model service consumer (210) indicating whether the model subscription request (230) can be satisfied.

5. The method according to claim 4, wherein, The model subscription response (235) includes the subscription association ID and / or expiration time.

6. The method according to any one of claims 1 to 5, further comprising: Send a model supply notification (235, 240) to the model service consumer (210) that identifies multiple ML models as corresponding to the ML model subscription request (230).

7. The method according to claim 6, wherein, The model provisioning notification includes one or more tuples of the following: ML model identifier, ML model information, and a subgroup ID that identifies the analysis target of each ML model.

8. The method according to claim 7, wherein, The model subscription request (230) includes one or more analytics IDs, and the ML model information and subgroup ID are associated with at least one of these analytics IDs.

9. The method according to any one of claims 6 to 8 when dependent on claim 4 or 5, wherein, The model supply notification is sent together with or included in the model subscription response (235).

10. The method according to any one of the preceding claims, further comprising: The analysis generated by the plurality of ML models based on the ML model subscription request (230) is provided to the model service consumer (210).

11. The method according to any one of the preceding claims, further comprising: Determine whether the ML model used to satisfy the ML model subscription request already exists.

12. The method according to any one of the preceding claims, further comprising: Trigger further training of one or more existing ML models to satisfy the ML model subscription request (230).

13. The method according to claim 12, wherein, Triggering further training of one or more existing ML models includes: in response to the ML model subscription request (230), instructing the model service consumer (210) to support multiple ML models for each analysis identifier, thereby triggering further training of the multiple ML models.

14. The method according to any one of the preceding claims, wherein, The ML model subscription request (230) requests the network node (700) to determine whether the network node can meet the ML training requirements, and the method further includes: Based on the ML model, determine whether the network node (700) can meet the ML training requirements; and Send an analysis service subscription response (235) to the model service consumer, indicating whether the network node (700) can meet the ML training requirements.

15. The method according to any one of claims 1 to 14, further comprising: Receive ML model information requests (290, 297) from the model service consumer (210). as well as In response to the ML model information request (290, 297), information about multiple ML models is sent in the same ML model information response (295, 299).

16. The method according to claim 15, wherein, The ML model information request (290, 297) is a request for information about the training of the ML model.

17. The method according to claim 4, wherein, The model subscription response (235) includes an error indication that the request is not accepted.

18. A network node (700) operable to act as a model service provider (220) of a wireless communication network (10), the network node being configured to: Receive machine learning ML model subscription requests (230) from the model service consumer (210); and Determine (320) multiple ML models to satisfy the ML model subscription request (230).

19. The network node of claim 18 is further configured to perform the method of any one of claims 2 to 17.

20. A network node (700) operable to act as a model service provider (220) of a wireless communication network (10), the network node comprising: An interface circuit (730) and a processing circuit (710) communicatively connected to the interface circuit (730), wherein the processing circuit (710) is configured to: The interface circuit (730) receives a machine learning ML model subscription request (230) from the model service consumer (210); and Determine (320) multiple ML models to satisfy the ML model subscription request (230).

21. The network node according to claim 20, wherein, The processing circuit (710) is also configured to perform the method according to any one of claims 2-17.

22. A computer program comprising instructions that, when executed on a processing circuit (710) of a network node (700), cause the processing circuit (710) to perform the method according to any one of claims 1-17.

23. A carrier comprising the computer program according to claim 22, wherein, The carrier is one of electronic signals, optical signals, radio signals, or computer-readable storage media.

24. A method (400) implemented by a network node (700) acting as a model service consumer (210) in a wireless communication network (10), the method comprising: Send (410) machine learning ML model subscription request (230) to the model service provider (220); as well as Receive (420) a model provisioning notification (235, 240) from the model service provider (220) that identifies multiple ML models as corresponding to the ML model subscription request (230).

25. The method according to claim 24, wherein, The ML model subscription request (230) includes one or more of the following parameters: - Model service consumer (210) information, - Indicators of support for multiple ML models, - One or more analytics IDs, - Notify the target address, - Notify associated ID, - The time interval required to request the ML model. - The requested representativeness ratio, - Use case context, - Conditions for requesting multiple ML models - ML model interoperability information.

26. The method according to claim 24 or 25, wherein, The model provisioning notification includes one or more tuples of the following: ML model identifier, ML model information, and a subgroup ID that identifies the analysis target of each ML model.

27. The method according to claim 26, wherein, The model subscription request (230) includes one or more analytics IDs, and the ML model information and subgroup ID are associated with at least one of these analytics IDs.

28. The method according to any one of claims 24 to 27, wherein, The model supply notification is sent together with or included in the model subscription response (235) indicating whether the model subscription request (230) can be satisfied.

29. The method according to any one of claims 24 to 28, further comprising: Receives analysis generated by multiple ML models based on the ML model subscription request (230) from the model service provider (220).

30. The method according to any one of claims 24 to 29, wherein: The ML model subscription request (230) requests the model service provider (220) to determine whether the model service provider (220) can meet the ML training requirements; as well as The method further includes receiving an analysis service subscription response (235) from the model service provider (220) indicating whether the model service provider (220) can meet the ML training requirements.

31. The method according to any one of claims 24 to 30, further comprising: Send an ML model information request (290, 297) to the model service provider (220); as well as In response to the ML model information request (290, 297), information about multiple ML models is received in the same ML model information response (295, 299).

32. The method according to claim 31, wherein, The ML model information request (290, 297) is a request for information about the training of the ML model.

33. A network node (700) operable to act as a model service consumer (210) of a wireless communication network (10), the network node being configured to: Send a machine learning ML model subscription request (230) to the model service provider (220); and Receive a notification from the model service provider (220) that identifies multiple ML models as corresponding to the ML model subscription request (230).

34. The network node of claim 33 is further configured to perform the method of any one of claims 25 to 32.

35. A network node (700) operable to act as a model service consumer (210) of a wireless communication network (10), the network node comprising: An interface circuit (730) and a processing circuit (710) communicatively connected to the interface circuit (730), wherein the processing circuit (710) is configured to: Send a machine learning ML model subscription request (230) to the model service provider (220); and Receive a notification from the model service provider (220) that identifies multiple ML models as corresponding to the ML model subscription request (230).

36. The network node according to claim 35, wherein, The processing circuit (710) is also configured to perform the method according to any one of claims 25 to 32.

37. A computer program comprising instructions that, when executed on a processing circuit (710) of a network node (700), cause the processing circuit (710) to perform the method according to any one of claims 24 to 32.

38. A carrier comprising the computer program according to claim 37, wherein, The carrier is one of electronic signals, optical signals, radio signals, or computer-readable storage media.

Images