Configuration for machine learning training

Abstract

The present disclosure is related to artificial intelligence (AI) and machine learning (ML), and in particular, to technologies and techniques for ML training (MLT) configuration. A service producer receives a request to configure an MLT policy for controlling MLT of an ML model. The MLT policy defines various performance requirements, conditions, constraints, parameters, and / or other criteria under which the MLT is to take place. The service producer collects data based on the configured policy, and controls the MLT of the ML model based on the collected data and according to the configured MLT policy. Other embodiments may be described and / or claimed.

Description

[0001] CONFIGURATION FOR MACHINE LEARNING TRAINING

[0002] CROSS REFERENCE TO RELATED APPLICATIONS

[0003] The present application claims priority to U.S. Provisional App. No. 63 / 484,720 filed February 13, 2023, the contents of which is hereby incorporated by reference in its entirety.

[0004] BACKGROUND

[0005] Artificial intelligence (Al) and machine learning (ML) techniques and relevant applications are being increasingly adopted by the wider industries and proved to be successful. These are now being applied to the telecommunication industry including mobile networks. Although AI / ML techniques in general are quite mature nowadays, some of the relevant aspects of the technology are still evolving while new complementary techniques are frequently emerging.

[0006] BRIEF DESCRIPTION OF THE DRAWINGS

[0007] Embodiments will be readily understood by the following detailed description in conjunction with the accompanying drawings. To facilitate this description, like reference numerals designate like structural elements. Embodiments are illustrated by way of example, and not by way of limitation, in the figures of the accompanying drawings in which: Figure 1 illustrates an example system AI / ML workflow; Figure 2 illustrates an overview and service framework for ML training (MLT); Figures 3 and 4 illustrate example network resource model (NRM) fragments for MLT; Figure 5 illustrates an inheritance hierarchy for MLT related NRMs; Figure 6 depicts an example network architecture; Figure 7 depicts an example wireless network; Figure 8 depicts example hardware resources; Figure 9 depicts an example of management services (MnS); Figure 10 depicts an example AI / ML functional framework; and Figure 11 depicts an example process for practicing the various embodiments discussed herein.

[0008] DETAILED DESCRIPTION

[0009] 1. I / L MANAGEMENT AND CONFIGURATION ASPECTS

[0010] The present disclosure is generally related to wireless communications technologies, cloud computing, edge computing, artificial intelligence (Al) and machine learning (ML), and in particular, to technologies and techniques for ML training (MLT) configuration.

[0011] AI / ML applications and techniques are widely used in fifth generation (5G) systems (5GS), including 5G core (5GC) (e.g., Network Data Analytics Function (NWDAF) 662 in Figure 6), next generation radio access networks (NG-RANs) 604 (e.g., RAN intelligence functions; see e.g., [TS38300] and [TS38401]), and management system (e.g., management data analytics service (MDAS); see e.g., third generation partnership project (3GPP) technical specification (TS) 28.104, [TS28533], [TS28535], [TS28536], and [TS28550]). Aspects of AI / ML training support consumer requested training and producer initiated training (see e.g., 3GPP TS 28.105 (“[TS28105]”), 3GPP TR 28.908 (“[TR28908]”), and / or the like). The present disclosure describes various aspects related to configuration of MLT.

[0012] 1.1. AI / ML OPERATIONAL WORKFLOW

[0013] Figure 1 shows an example AI / ML operational workflow 100 of the operational steps in the lifecycle of an ML entity. The workflow 100 involves four main phases, including: a training phase 101, an emulation phase 102, a deployment phase 103, and an inference phase 104. Example operational tasks for each phase are described infra.

[0014] 1.1.1. TRAINING PHASE

[0015] Prior to the training phase 101, it is assumed that various AI / ML model aspects are defined or configured for individual AI / ML models. Examples of such AI / ML model aspects include AI / ML model architecture aspects (e.g., the architecture and / or topology of the AI / ML model, such as the number and type of layers, activation functions, model complexity, and / or the like), data preparation tasks (e.g., data preprocessing, augmentation, synthesis, normalization, and / or the like), loss function and / or optimization function to optimize during training, hyperparameter settings / values, model parameter settings / values, and / or the like.

[0016] The training phase 101 includes ML training (MLT) 110 and ML testing 112 (or ML model testing 112). In some implementations, some or all of the MLT 110 and / or ML testing 112 operational tasks may be performed by an MLT MnS-P, although in other implementations at least some of the MLT 110 and / or ML testing 112 operational tasks are performed by an MLT MnS- C. In the training phase 101, the ML entity is generated based on the learning from training data, while performance and trustworthiness are evaluated based on testing data and validation data.

[0017] MLT 110 involves learning by a machine from the training data to generate a (new or updated) ML entity (see e.g., [TS28105]) that could be used for inference. The MLT 110 may also include the validation of the generated ML entity to evaluate the performance variance of the ML entity when performing on the training data and validation data. If the validation result does not meet the expectation (e.g., the variance is not acceptable), the ML entity needs to be re-trained. This is the initial step of the workflow. Additional aspects of the MLT MnS are specified in [TS28105].

[0018] MLT management allows an MnS-C to request ML entity training, consume and control the producer-initiated training, and manage the ML entity training / retraining process. As discussed in more detail infra, MLT management capabilities can also include training performance management and / or setting a policy (e.g., an MLT policy) for the producer- initiated ML entity training.

[0019] The ML testing 112 involves testing the validated ML entity (or the ML model) with testing data to evaluate the performance of the trained ML entity for selection for inference. Additionally or alternatively, the ML testing 112 (or model testing) involves performing one or more processes to validate ML model performance using testing data (or testing data set). When the performance of the trained ML entity meets the expectations on both training data and validation data, the ML entity is finally tested to evaluate the performance on testing data. If the testing result meets the expectation, the ML entity may be counted as a candidate for use towards the intended use case or task, otherwise the ML entity may need to be further (re)trained.

[0020] ML testing management allows an MnS-C to request ML entity testing, and to receive the testing results for a trained ML entity. It may also include capabilities for selecting the specific performance metrics to be used or reported by the ML testing function. The MnS-C may also be allowed to trigger ML entity (re-)training based on the ML entity testing performance requirements and / or conditions, constraints, parameters, and / or criteria defined in the MLT policy (or a separate ML testing policy) for the producer-initiated ML entity training.

[0021] In some cases, the ML entity may need to be verified, which is a special case of testing to check whether the ML entity (or ML model) works when deployed in or at the target node (e.g., an Al / ML inference function, inference engine, intelligent agent, and / or the like). In some implementations, the ML entity (or ML model) verification involves verifying ML entity (or ML model) performance when deployed and / or online in the intended or desired environment. In some examples, the verification includes or involves inference monitoring, wherein ML inferences / predictions are collected and anal6ed (e.g., by collecting ModelPerformance data as discussed in [TS28105]). In these implementations, the verification results may be part of the ML entity (or ML model) validation results, or may be reported in a same or similar manner as the validation results as discussed herein. In some implementations, the verification process may be skipped, for example, in case the input and output data, data types, and / or formats have been unchanged from the last ML entity.

[0022] For MLT (see e.g., [TS28105]), MLT can be initiated by an MnS-C and / or an MnS-P. In some implementations, the operational tasks of the training phase 101 (or MLT 110 and ML testing 112), emulation phase 102 (or ML emulation 120), and / or deployment phase 103 (or ML entity loading 130) may be performed by the MLTF 210 of Figure 2, the MTF 1010 of Figure 10, and / or an NWDAF-MTLF (see e.g., discussed of NWDAF 662 of Figure 6, infra).

[0023] The MLT function (e.g., MLT 110, MLTF 210, MTF 1010, NWDAF-MTLF, and / or the like) may be located in the Operations, Administration and Maintenance (0AM) system and / or in a 3GPP NF (e.g., RAN 604, RAN node 614, NWDAF 662 or NWDAF-MTLF, and / or the like). MLT can require a significant amount of resources. The producer- initiated MLT can be controlled to reduce the amount of resources consumed and / or to ensure appropriate functioning of the system, such as in cases where the MLT function is co-located with other functions (e.g., inference function and / or the like). 1.1.2. EMULATION PHASE

[0024] The ML emulation phase 102 includes ML emulation 120 where an ML entity is run, operated, or executed for inference in an emulation environment. The emulation phase 102 allows the inference performance of the ML entity in the emulation environment to be evaluated prior to applying (or deploying) the ML entity to the target network or system. In some examples, the emulation phase 102 is considered optional and can be skipped in the AI / ML operational workflow.

[0025] Before the ML entity is applied in the production network, the MnS inference consumer may want to receive results of inference in one or more environments that emulate (to different extents) the expected inference characteristics, in a process that may be termed as Inference emulation. The Inference emulation phase enables this.

[0026] After the ML entity is validated and tested during development, the MnS-C 205 may wish to receive information from an inference emulation process that indicates if the ML entity or the associated ML inference function is working correctly under certain runtime context. The management system should have the capabilities enabling an MnS-C 205: to receive the results from running inference through an AI / ML inference emulation environment available at the emulation MnS-P 210.

[0027] 1.1.3. DEPLOYMENT PHASE

[0028] The ML deployment phase 103 includes ML deployment operational tasks, which involve deploying the trained and tested ML entity to the target inference function. The target inference function uses the subject ML entity for inference. In some implementations, the deployment phase 103 may not be needed, for example, when the training function and inference function are in the same entity. In some implementations, the ML deployment operational tasks operational tasks are performed by the MLT MnS-P 210 in conjunction with the MLT MnS-C 205 (e.g., where the MLT MnS-P 210 deploys or otherwise provides the trained / tested ML entity to the MLT MnS-C 205 or other target node).

[0029] In some implementations, the deployment phase 103 (or the ML deployment operational tasks) includes ML entity loading 130. The ML entity loading 130 involves or includes one or more processes (e.g., sequence(s) of atomic actions) of making a trained ML entity available for use at the target AI / ML inference function. In some examples, the relationship between loading process(es) 130, ML model transfer process(es), and NWDAF process(es) may be for future study (FFS), and / or may be implementation- specific.

[0030] 1.1.4. INFERENCE PHASE

[0031] The inference phase 104 includes AI / ML inference 140. The AI / ML inference 140 involves or includes performing, determining, or generating inferences / predictions using the ML entity by the AI / ML inference function. In some implementations, the ML inference 140 operational tasks operational tasks are performed by the MLT MnS-C 205 or other target node to which the trained and tested ML entity is deployed. As examples, the MLT MnS-C 205 that performs AI / ML inference can be the the MIF 1015 of Figure 10 and / or the MIF 1145 of Figure 11.

[0032] 1.2. FUNCTIONALITY AND SERVICE FRAMEWORK FOR ML TRAINING

[0033] Figure 2 depicts an example of MLT capability provided via MLT MnS in the context of SBMA to the authorized consumer(s) (e.g., MLT MnS-C 205) by an MLT MnS-P. An MLT function (MLTF) 210 playing the role of MLT MnS-P 210 may consume various data for MLT purposes. As examples, the MLTF 210 may be the same or similar as the MTF 1010 of Figure 10, a model training logical function (MTLF) of an NWDAF 662, and / or the AI / ML model entities discussed in U.S. App. No. 18 / 358,288 filed on 25 Jul. 2023.

[0034] The internal business logic of MLT 215 leverages the current and historical data from various data sources 220, including those listed herein and / or in [TS28105] to monitor networks and / or services that are relevant to the ML model, prepare the data, trigger and conduct (e.g., execute, run, operate, and / or the like) the training, and / or performing other tasks, actions, and / or processes. Examples of such data can include performance measurements (PMs) as per [TS28552], [TS32425] and key performance indicators (KPIs) as per [TS28554]; trace data, minimization drive tests (MDT), radio link failure (RLF) reports / data, RRC connection establishment failure event (RCEF) reports / data as per 3GPP TS 32.422 and 3GPP TS 32.423; quality if experience (QoE) and service experience data as per 3GPP TS 28.405 and 3GPP TS 28.406; analytics data offered by NWDAF 662 as discussed herein and / or per [TS23288]; alarm information and notifications as per [TS28532]; connection management (CM) information and notifications; MDA reports from MDA MnS-Ps as discussed herein and / or as per [TS28104]; management data from non-3GPP systems; and / or other data that can be used for training purposes.

[0035] 1.3. CONFIGURATION MANAGEMENT FOR ML TRAINING PHASE

[0036] 1.3.1. DESCRIPTION

[0037] As defined in [TS28105], MLT can be initiated by an MnS-C or an MnS-P. An MLT function (e.g., MLT 110, MLTF 210, MTF 1010, NWDAF-MTLF, and / or the like) may be located in a management system (e.g., 0AM system, edge computing management system, cloud orchestrator, and / or the like), in an NF (e.g., RAN 604, RAN node 614, NWDAF 662 or NWDAF- MTLF, and / or the like), and / or an AF (e.g., AF 660, and / or other suitable functional element(s)), and / or a remote system (e.g., application server(s) 638, edge computing server(s), and / or the like). MLT can require a significant amount of resources. The producer- initiated MLT can be controlled to reduce the amount of resources consumed and / or to ensure appropriate functioning of the system, such as in cases where the MLT function is co-located with other functions (e.g., inference function and / or the like).

[0038] 1.3.2. USE CASES

[0039] 13.2.1. ML TRAINING INITIATED BY PRODUCER

[0040] The MLT or re-training may be initiated by the MLT MnS-P 210, for example, as a result of performance evaluation of the ML entity, based on feedback (see e.g., Figure 10), when new training data is received from the MnS-C 205 and / or from a data source 220, and / or when new training data (which may or may not be from the MnS-C 205) describing a new network status and / or that events have become available. Therefore, monitoring mechanisms for monitoring the performance and / or the KPIs of the ML entity can be employed, and thresholds that the MLT MnS-C 205 configured for the MLT MnS-P 210 to trigger the training or re-training can be employed.

[0041] When the MLT MnS-P 210 decides to start the MLT, the MLT MnS-P 210 performs one or more of the following actions / operations: selects the training data; trains the ML model using the selected training data; provides the training results (including the identifier of the ML entity generated from the initially trained ML model or the version number of the ML entity associated with the re-trained model, training performance, and / or the like) to the MLT MnS-C(s) 205 who have subscribed to receive the MLT results.

[0042] I.3.2.2. CONTROL OF PRODUCER-INITIATED ML TRAINING

[0043] For producer-initiated MLT, the MnS-P 210 has its own algorithm to trigger and perform MLT. In some examples, the MnS-P 210 may locally implement this MLT trigger algorithm. In other examples, the MLT trigger algorithm is implemented at a different (e.g., remote) entity / element (e.g., 0AM, AF, NF, and / or the like) that causes the MnS-P 210 to trigger MLT (e.g., via a suitable API, web service, and / or other connection / communication means).

[0044] However, the MnS-C 205 may expect the training to be performed under certain desired conditions or constraints. These desired conditions or constraints can include, for example, desired inference performance (e.g., as determined using one or more performance metrics); desired network conditions; desired platform configurations, arrangements, and / or parameters (e.g., number and / or arrangement of compute nodes for distributed training / learning and / or the like); desired platform conditions or contexts (e.g., in terms of resource usage and / or the like); desired application or service conditions or contexts; desired time periods and / or time intervals; desired environmental conditions; and / or the like. Therefore, the MnS-C 205 may need to be able to control the MLT (e.g., producer-initiated MLT) for a desired use case and / or to achieve a desired predictive accuracy. To better control the MLT, the MnS-C 205 can provide a policy (e.g., an MLT policy or the like) or configuration containing, specifying, or defining various conditions for the MnS-P 210 to trigger the MLT and / or to otherwise control the performance of the MLT. For purposes of the present disclosure, the “conditions” specified or defined in an MLT policy can include or refer to any number or combination of conditions, constraints, parameters (e.g., model parameters, hyperparameters, platform / hardware configuration parameters, network parameters, and / or any other parameters), properties, rulesets, configurations, data and / or dataset(s), and / or other aspects for triggering MLT execution / operation and / or for controlling various aspects of the MLT performance.

[0045] Examples of conditions for controlling the MLT include time conditions, inference performance conditions, network conditions, platform / system conditions, application and / or service conditions, and / or other suitable conditions. The time conditions are conditions related to the occurrence of one or more time periods and / or time intervals. The time conditions can be defined / specified in terms of one or more times and / or dates, time units (e.g., number of milliseconds, seconds, minutes, hours, days, weeks, months, and so forth), and / or intervals of time units.

[0046] The inference performance conditions are conditions related to inference performance of an existing ML entity running in / at an inference function, such as when an inference performance meets or does not meet one or more specified targets (e.g., target PMs, KPIs, and / or the like). The inference performance conditions can be defined / specified in terms of one or more PM and / or KPI thresholds, PM and / or KPI ranges, and / or the like. The inference performance conditions can be defined / specified using any PMs and / or KPIs, such as any of those mentioned herein.

[0047] The network conditions are conditions related to a network environment, such as when one or more network specified network / radio conditions have changed. The network conditions can be defined / specified in terms of one or more network / radio measurement thresholds, network / radio measurement ranges, and / or the like. Examples of the network conditions include a number of active user equipment (UEs), a number of UEs with one or more desired capabilities, a number of changes of neighbor cells, a number of handovers, a number of radio link failure (RLF) reports, a number of RRC connection establishment failure event (RCEF) reports, signal strength measurement thresholds, and / or signal quality measurement thresholds. The network conditions can be defined / specified using any network or radio measurement(s), such as PMs (see e.g., [TS28552]), KPIs (see e.g., [TS28554]), performance threshold monitoring events (see e.g., [TS28532]), fault supervision events (see e.g., [TS28532]), and / or any measurements / metrics mentioned herein.

[0048] The platform / system conditions are conditions related to a configuration, arrangement, and / or topology of a platform running / operating an existing ML entity, and / or one or more specified aspects of platform(s) operating / running the existing ML entity (e.g., resource usage / utilization, internal and / or external environmental conditions (e.g., internal temperature, external temperature surrounding the platform, and / or the like), and so forth) of the existing ML entity have changed have changed. The platform / system conditions can be defined / specified in terms of one or more platform-related measurement / metric thresholds, platform-related measurement / metric ranges, and / or the like; The platform / system conditions can be defined / specified using any platform-related measurement(s) / metric(s) and / or telemetry data can be used, such as performance measurements (see e.g., [TS28552]), KPIs (see e.g., [TS28554]), performance threshold monitoring events (see e.g., [TS28532]), fault supervision events (see e.g., [TS28532]), hardware platform measurements / metrics and / or telemetry data (see e.g., Intel® VTune™ Profiler User Guide, INTEL CORP., version 2024.0 (11 Nov. 2023)), and / or any measurements / metrics mentioned herein.

[0049] The application and / or service conditions are conditions related to one or more application requirements and / or service contexts or other aspects. The application and / or service conditions can be defined / specified in terms of one or more application / service measurement / metric thresholds, application / service measurement / metric ranges, and / or the like; any application / service metric(s) / measurement(s) can be used, such as performance measurements / metrics (e.g., response time / latency, throughput, concurrent users, resource utilization, and / or the like), reliability metrics (e.g., uptime / downtime, error rates, fault tolerance, and / or the like), scalability metrics (e.g., horizontal and / or vertical scalability), security metrics (e.g., incident rates, vulnerabilities, authentication and authorization metrics, and / or the like), user experience metrics, infrastructure metrics (e.g., server uptime, network latency, capacity, and / or the like), and / or any measurements / metrics / telemetry data mentioned herein.

[0050] In one example, the MnS-C 205 may want to avoid the MLT during busy traffic times (e.g., such as when the MLT function is located in an NF) and only allow the MLT to occur within a preconfigured time window. In another example, the MnS-C 205 may choose to deactivate the MLT if the training performance consistently cannot meet one or more desired performance requirements. The MnS-C 205 can define one or more conditions for controlling the MLT in the MLT policy such as, for example, inference performance metrics and / or threshold, network conditions, platform conditions, application / service / platform requirements and / or conditions, environmental conditions, and / or the like.

[0051] Additionally or alternatively, controlling various aspects of the MLT performance can include triggering, starting, stopping, pausing, saving, logging, and terminating MLT execution and / or operation. Additional or alternative examples of MLT control operations / commands can include callbacks (e.g., a set of functions to be applied at given stages of the training procedure, such as saving the model at one or more model checkpoints, early stopping based on patience value(s), logging, and / or the like), dynamic hyperparameter tuning (e.g., changing one or more hyperparameters such as learning rate, batch size, epochs, regularization parameters, and / or other ML model / algorithm- specific settings during MLT), dynamic adjustment of model parameter settings / values during MLT, dynamic adjustment of performance metrics to be monitored during MLT, distributed leaming / training aspects (e.g., changing the number, type(s), and / or specific compute nodes to perform distributed MLT), (e.g., adjusting layer weights and / or the like), model saving and loading, retraining using different model configurations and / or parameters (e.g., using new or different training dataset(s) and / or specified hyperparameter and / or model parameter configurations / settings), and / or the like. Any of the example MLT control operations / commands can be used for initial MLT and / or retraining a specified ML model.

[0052] The MLT policy can be embodied as any suitable data structure having any suitable data format. In some examples, individual conditions can be associated or related to one or more MLT control operations / commands such that, when a specified condition is detected to have occured, the MLT MnS-P 210 performs the corresponding MLT control operation(s) / command(s).

[0053] 1.3.3. POTENTIAL REQUIREMENTS

[0054] Potential requirements for controlling producer-initiated MLT can include the following: REQ-MLTRAIN_CFG-1: The MLT MnS-P 210 should have a capability to allow an authorized MnS-C 205 to configure activation, deactivation, and retraining related policies.

[0055] REQ-MLTRAIN_CFG-2: The MLT MnS-P 210 should have a capability to allow the authorized MnS-C 205 to activate and deactivate the MLTF (e.g., MLT 110, MLTF 210, MTF 1010, NWDAF-MTLF, and / or the like).

[0056] REQ-MLTRAIN_ACT-1: The MLT MnS-P 210 should have a capability to inform an authorized MnS-C 205 about the activation and deactivation of the MLTF (e.g., MLT 110, MLTF 210, MTF 1010, NWDAF-MTLF, and / or the like).

[0057] 1.3.4. EXAMPLE SOLUTIONS

[0058] 13.4.1. ML TRAINING POLICY CONFIGURATION

[0059] A data type and / or abstract class describing a policy for controlling MLT and / or the MLTF (e.g., MLT 110, MLTF 210, MTF 1010, NWDAF-MTLF, and / or the like), and this data type and / or abstract class can be used and / or inherited by an MOI representing the MLTF (e.g., MLTrainingFunct ion as discussed herein and / or as defined in [TS28105]). This policy can be referred to an “MnS policy”, “MLT policy”, “MLT configuration”, and / or the like.

[0060] The policy contains conditions, such as any of those discussed herein, for controlling and / or triggering the MLTF or the MLT process(es) at the MLTF. In particular, the policy includes one or more conditions for triggering the MLT. As examples, these conditions can include PM threshold(s) (e.g., PM levels or values indicating a desired inference performance, a range of acceptable or unacceptable PM values, and / or the like), MLT parameters (e.g., model convergence thresholds, number of training epochs and / or iterations, training time periods / limits, and / or the like), network conditions (e.g., number of active UEs, number of UEs with certain capabilities, number of changes of neighbor cells, number of handovers, number of RLFs and / or RCEFs, signal strength / quality measurement levels / thresholds, and / or the like), platform / system conditions (e.g., amount of resources consumed for training, time periods / limits for running training process(es) or tasks, number of nodes or clusters used for training, and / or the like), and / or any other conditions, constraints, parameters, and / or properties, including any of those discussed herein.

[0061] The MLT MnS-P 210 monitors the performance of the ML entity and / or for occurrence of the conditions and triggers the MLT (e.g., MLT activation or deactivation) according to the configured policy. In some examples, this can include MLT activation and deactivation, ML model tuning aspects, and / or other aspects of MLT.

[0062] I.3.4.2. ML TRAINING ACTIVATION AND DEACTIVATION

[0063] 13.4.2.1. FRAMEWORK FOR MLT ACTIVATION AND DEACTIVATION

[0064] This section describes the general framework for activation and deactivation of an MLTF (e.g., MLT 110, MLTF 210, MTF 1010, NWDAF-MTLF, and / or the like) and / or the MLT taking place at or by an MLTF.

[0065] A data type and / or abstract class can be defined to describe activation conditions, constraints, parameters, and / or properties and / or deactivation conditions, constraints, parameters, and / or properties. In some examples, this data type and / or abstract class can be specified or defined in the aforementioned policy. Additionally or alternatively, this data type and / or abstract class can be used or inherited by the MOI representing the MLTF (e.g., MLTrainingFunction defined in [TS28105]).

[0066] This general framework supports general properties for all types of activation / deactivation including, for exmaple, activation type. This data type and / or abstract class is extended with the attributes supporting these specific types of activation. In some examples, the activation type can be instant activation / deactivation and / or scheduled activation / deactivation, as described infra.

[0067] 13.4.2.2. INSTANT ACTIVATION / DEACTIVATION

[0068] Instant activation / deactivation involves the MLTF (e.g., MLT 110, MLTF 210, MTF 1010, NWDAF-MTLF, and / or the like) being instantly activated or deactivated. For example, a command or instruction issued by the MLT MnS-P 210 to the MLTF (e.g., via suitable API(s), web service(s), and / or other connection / communication means) can cause the instant (or almost instant) activation or deactivation of the MLT. The instant activation / deactivation mechanism can also be referred to as “asynchronous activation / deactivation”, “trigger-based activation / deactivation”, and / or the like.

[0069] In some implementations, the generic framework described in section 1.3.4.2.1 is extended with the following attributes to support instant activation / deactivation: switch for "activated" and "deactivated" status. One advantage of the instant activation / deactivation is that it allows for MLT to be adjusted “on-the-fly” so that the MLT can be dynamically adjusted based on current / existing conditions.

[0070] I.3.4.2.3. SCHEDULE-BASED ACTIVATION / DEACTIVATION

[0071] Schedule-based activation / deactivation involves the MLTF (e.g., MLT 110, MLTF 210, MTF 1010, NWDAF-MTLF, and / or the like) being activated or deactivated based on a given schedule. For example, a scheduler implemented by the MLT MnS-P 210, the MLTF, or some other NF / AF can schedule specific times and / or dates to activate or deactivate MLT. In some examples, this scheduler can be implemented by the model management function 1140 of Figure 11. In these ways, the MLT can take place at times of low network traffic to avoid or reduce the likelihood of network congestion, or avoid or reduce the likelihood of overload at the MLTF and / or in the network.

[0072] In some implementations, the generic framework described in section 1.3.4.2.1 is extended with the following attributes to support the schedule-based activation / deactivation: the schedule for activation / deactivation.

[0073] One advantage of the schedule-based activation / deactivation is that it is fully NRM-based approach and reuses the existing provisioning MnS operations for MLT configuration. This solution extends the existing IOC representing the MLT function (e.g., MLT rainingFunct i on defined in [TS28105]) with attributes defined by data type and / or abstract class for controlling the MLT, thus the changes are minimal on the existing NRMs. In these ways, the schedule-based activation / deactivation mechanism is a feasible solution.

[0074] 1.4. CLASS DEFINITIONS

[0075] 1.4.1. MLEntity

[0076] The IOC MLEnt ity represents the ML entity and / or an ML model. The MLEntity may contain 3 types of contexts: Trainingcontext which is the context under which the MLEntity has been trained, the ExpectedRunTimeContext which is the context where an MLEntity is expected to be applied or / and the RunTimeContext which is the context where the ML entity is being applied. It also contains a reference named retrainingEvent sMonitorRef which is a pointer to ThresholdMnonitor MOI. This indicates the list of performance measurements and the corresponding thresholds that are monitored and used to identify the need for re-training by the MnS-P. After the MLEntity MOI has been instantiated, the MnS-C can request MnS-P to instantiate a ThresholdMonitor MOI and update the reference in the MLEntity MOI that can be used by the MnS-P to decide on the re-training of the MLEnt ity. The MnS-P can be an MLT MnS-P 210 or an ML inference MnS-P.

[0077] Example attributes are shown by table 1.4.1-1 and example attribute constraints are shown by table 1.4.1-2. Additionally, the common notifications defined in table 1.4.3-1 are valid for this

[0078] IOC, without exceptions or additions.

[0079] Table 1.4.1 -2

[0080] 1.4.2. MLTrainingFunction The IOC MLTrainingFunct ion represents the entity that undertakes MLT and is also the container of the MLTrainingRequest IOC(s). The entity represented by MLTrainingFunction MOI supports training of one or more MLEnt ity ( s ) . Example attributes are shown by table 1.4.2-1. Additionally, the common notifications defined in table 1.4.3-1 are valid for this IOC, without exceptions or additions. Table 1.4.2-1

[0081] 1.4.3. NOTIFICATIONS

[0082] Table 1.4.3-1 shows a list of notifications, defined in [TS28532], that an MnS-C may receive. The notification header attribute ob j ectClas s / ob j ect l nstance captures the DN of an instance of a class defined in the present disclosure.

[0083] Table 1.4.3-1

[0084] 1.5. CLASS DIAGRAMS

[0085] Figures 3 and 4 depict example NRM fragments for MLT using UML diagrams, and Figure 5 illustrates an inheritance hierarchy for MLT related NRMs. Various aspects of UML semantics are described in 3GPP TS 32.156. Figures 3 and 4 show information object classes (IOCS) that can be used for managing MLT, and their relations. The depicted a set of classes (e.g., IOCS) that encapsulate the information relevant to ML model training. Various aspects of these classes (e.g., IOCs) are discussed herein and / or defined in [TS28105].

[0086] 2. NETWORK, SYSTEM, AND DEVICE CONFIGURATIONS AND ARRANGEMENTS

[0087] Figure 6 depicts an example network architecture 600. The network 600 may operate in a manner consistent with 3GPP technical specifications for LTE or 5G / NR systems. However, the example embodiments are not limited in this regard and the described examples may apply to other networks that benefit from the principles described herein, such as future 3GPP systems, WiMAX systems, GSMA systems, WiFi systems, and / or the like.

[0088] The network 600 includes a UE 602, which is any mobile or non-mobile computing device designed to communicate with a RAN 604 via an over-the-air connection. The UE 602 is communicatively coupled with the RAN 604 by a Uu interface, which may be applicable to both LTE and NR systems. Examples of the UE 602 include, but are not limited to, a smartphone, tablet computer, wearable device (e.g., smart watch, fitness tracker, smart glasses, smart clothing / fabrics, head-mounted displays, smart shows, and / or the like), desktop computer, workstation, laptop computer, servers, in-vehicle infotainment system, in-car entertainment system, instrument cluster, head-up display (HUD) device, extended reality (XR) device (e.g., including augmented reality, virtual reality (VR), and / or mixed reality), onboard diagnostic device, dashtop mobile equipment, mobile data terminal, electronic engine management system, engine management system, electronic / engine control unit / module, embedded system, sensor, microcontroller, control module, networked appliance, machine-type communication device, machine-to-machine (M2M), Internet of Things (loT) device, smart appliance, flying drone or unmanned aerial vehicle (UAV), terrestrial drone or autonomous vehicle, robot, electronic signage, single-board computer (SBC) (e.g., Raspberry Pi, Arduino, Intel Edison, and the like), plug computers, and / or any type of computing device such as any of those discussed herein. In some examples, the UE 602 can include desktop computers. The UE 602 may be the same or similar to any of the other UEs discussed herein such as, for example, UE 702, hardware resources 800, and / or the like.

[0089] In some examples, the network 600 includes a set of UEs 602 coupled directly with one another via a proximity services (ProSe), PC5, sidelink (SL) interface, which involves communication between two or more UEs 602 using 3GPP technology without traversing a network node. Here, the SL interface includes, for example, one or more SL logical channels (e.g., sidelink broadcast control channel (SBCCH), sidelink control channel (SCCH), and sidelink traffic channel (STCH)); one or more SL transport channels (e.g., sidelink shared channel (SL- SCH) and sidelink broadcast channel (SL-BCH)); and one or more SL physical channels (e.g., physical sidelink shared channel (PSSCH), physical sidelink control channel (PSCCH), physical sidelink Feedback channel (PSFCH), physical sidelink broadcast channel (PSBCH), and / or the like). The UE 602 may perform blind decoding attempts of SL channels / links according to the various examples herein.

[0090] In some examples, the UE 602 can communicate with an AP 606 via an over-the-air (OTA) connection. The AP 606 manages a WLAN connection between the UE 602 and the AP 606, which is consistent with any IEEE 802 protocol (e.g., IEEE 802.11 and / or the like). Additionally, the UE 602, RAN 604, and AP 606 may utilize cellular- WLAN aggregation / integration (e.g., LWA / LWIP), which may serve to offload some / all network traffic from the RAN 604.

[0091] The RAN 604 includes one or more network access nodes (NANs) 614 (also referred to as “access network nodes”, “RAN nodes”, and / or the like). The NANs 614 terminate air-interface(s) for the UE 602 by providing access stratum protocols including RRC, PDCP, RLC, MAC, and PHY / L1 protocols. In this manner, the NANs 614 enable data / voice connectivity between the CN 640 and the UE 602. The NANs 614 may be a macrocell base station or a low power base station for providing femtocells, picocells or other like cells having smaller coverage areas, smaller user capacity, or higher bandwidth compared to macrocells; or some combination thereof. In these implementations, an NAN 614 be referred to as a BS, gNB, RAN node, eNB, ng-eNB, NodeB, RSU, TRP, and the like. The RAN 604 may have an NG-RAN architecture as discussed in 3GPP TS 38.401.

[0092] The RAN 604 (or individual NANs 614) may provide the air interface over a licensed spectrum or an unlicensed spectrum. To operate in the unlicensed spectrum, the nodes may use LA A, eLAA, and / or feLAA mechanisms based on CA technology with PCells / S Cells. Prior to accessing the unlicensed spectrum, the nodes may perform medium / carrier-sensing operations based on, for example, a listen-before-talk (LBT) protocol.

[0093] The set of NANs 614 are coupled with one another via respective Xn interfaces. The Xn interfaces, which may be separated into control / user plane interfaces in some examples, allow the NANs 614 to communicate information related to handovers, data / context transfers, mobility, load management, interference coordination, and the like. The NANs 614 manage one or more cells, cell groups, component carriers (CCs), and the like to provide the UE 602 with an air interface for network access. The UE 602 may be simultaneously connected with a set of cells provided by the same or different NANs 614 of the RAN 604 or a different RAN 604. For example, the UE 602 and RAN 604 may use carrier aggregation (CA) to allow the UE 602 to connect with a set of CCs, each corresponding to a primary cell (PCell) or secondary cell (SCell). The NG-RAN 604 supports multi-radio DC (MR-DC) operation where a UE 602 is configured to utilize radio resources provided by two distinct schedulers, located in at least two different NG-RAN nodes 614 connected via a non-ideal backhaul, one NG-RAN node 614 providing NR access and the other NG-RAN node 614 providing either E-UTRA or NR access. Further details of MR-DC operation, including conditional PSCell addition (CPA) and conditional PSCell change (CPC), can be found in 3GPP TS 36.300 (“[TS36300]”), [TS38300], and 3GPP TS 37.340.

[0094] Individual UEs 602 can be configured to measure or collect radio information, and provide the radio information to one or more NANs 614. The radio information may be in the form of one or more measurement reports, and / or may include, for example, signal strength measurements, signal quality measurements, and / or the like. Each measurement report can be tagged with a timestamp and the location of the measurement (e.g., the UEs 602 current location). For example, the UE 602 can perform reference signal (RS) measurement and reporting procedures to provide the network with information about the quality of one or more wireless channels and / or the communication media in general, and this information can be used to optimize various aspects of the communication system. As examples, the measurement and reporting procedures performed by the UE 602 can include those discussed in 3GPP TS 38.211 (“[TS38211]”), 3GPP TS 38.212 (“[TS38212]”), 3GPP TS 38.213 (“[TS38213]”), 3GPP TS 38.214 (“[TS38214]”), 3GPP TS 38.215 (“[TS38215]”), 3GPP TS 38.101-1 (“[TS38101-1]”), 3GPP TS 38.104 (“[TS38104]”), 3GPP TS 38.113 (“[TS38113]”), 3GPP TS 38.133 (“[TS38133]”), 3GPP TS 38.331 (“[TS38331]”), and / or other the like. The physical signals and / or reference signals can include demodulation reference signals (DM-RS), phase-tracking reference signals (PT-RS), positioning reference signal (PRS), channel-state information reference signal (CSI-RS), synchronization signal block (SSB), primary synchronization signal (PSS), secondary synchronization signal (SSS), sounding reference signal (SRS), and / or the like. Examples of the measurements performed / collected by individual UEs 602 and / or included in measurement reports can include one or more of the following: bandwidth (BW), network or cell load, latency, jitter, round trip time (RTT), number of interrupts, out-of-order delivery of data packets, transmission power, bit error rate, bit error ratio (BER), Block Error Rate (BLER), packet error ratio (PER), packet loss rate, packet reception rate (PRR), data rate, peak data rate, end-to-end (e2e) delay, signal-to-noise ratio (SNR), signal-to-noise and interference ratio (SINR), signal-plus-noise-plus-distortion to noise-plus-distortion (SINAD) ratio, carrier-to-interference plus noise ratio (CINR), Additive White Gaussian Noise (AWGN), energy per bit to noise power density ratio (Eb / No), energy per chip to interference power density ratio (Ec / Io), energy per chip to noise power density ratio (Ec / No), peak-to-average power ratio (PAPR), reference signal received power (RSRP), reference signal received quality (RSRQ), received signal strength indicator (RSSI), received channel power indicator (RCPI), received signal to noise indicator (RSNI), Received Signal Code Power (RSCP), average noise plus interference (ANPI), GNSS timing of cell frames for UE positioning, GNSS code measurements, GNSS carrier phase measurements; Accumulated Delta Range (ADR), channel interference measurements, thermal noise power measurements, received interference power measurements, power histogram measurements, channel load measurements, STA statistics, and / or other like measurements. Other measurements may be additionally or alternatively used, such as those discussed in [TS36214], [TS38215], 3GPP TS 38.314 (“[TS38314]”), 3GPP TS 28.552 (“[TS28552]”), 3GPP TS 32.425 (“[TS32425]”), IEEE 802.11, and / or the like. Additionally or alternatively, any of the aforementioned measurements (or combination of measurements) may be collected by one or more NANs 614 and / or other network nodes.

[0095] As alluded to previously, the NG-RAN 614 provides a 5G-NR air interface (e.g., Uu interface), which may have the following characteristics: variable SCS; CP-OFDM for DL, CP- OFDM and DFT-s-OFDM for UL; polar, repetition, simplex, and Reed-Muller codes for control and LDPC for data. The 5G-NR air interface may rely on CSI-RS, PDSCH / PDCCH DMRS similar to the LTE air interface. The 5G-NR air interface may not use a CRS, but may use PBCH DMRS for PBCH demodulation; PTRS for phase tracking for PDSCH; and tracking reference signal for time tracking. The 5G-NR air interface may operating on FR1 bands that include sub-6 GHz bands or FR2 bands that include bands from 24.25 GHz to 52.6 GHz. The 5G-NR air interface may include an SSB that is an area of a downlink resource grid that includes PSS / SSS / PBCH.

[0096] The 5G-NR air interface may utilize BWPs for various purposes. For example, BWP can be used for dynamic adaptation of the SCS. For example, the UE 602 can be configured with multiple BWPs where each BWP configuration has a different SCS. When a BWP change is indicated to the UE 602, the SCS of the transmission is changed as well. Another use case example of BWP is related to power saving. In particular, multiple BWPs can be configured for the UE 602 with different amount of frequency resources (e.g., PRBs) to support data transmission under different traffic loading scenarios. A BWP containing a smaller number of PRBs can be used for data transmission with small traffic load while allowing power saving at the UE 602 and in some cases at the gNB 614a. A BWP containing a larger number of PRBs can be used for scenarios with higher traffic load.

[0097] The RAN 604 is communicatively coupled to CN 640 that includes network elements and / or network functions (NFs) to provide various functions to support data and telecommunications services to customers / subscribers (e.g., UE 602). The components of the CN 640 may be implemented in one physical node or separate physical nodes. In some examples, NFV may be utilized to virtualize any or all of the functions provided by the network elements of the CN 640 onto physical compute / storage resources in servers, switches, and the like. A logical instantiation of the CN 640 may be referred to as a network slice, and a logical instantiation of a portion of the CN 640 may be referred to as a network sub-slice.

[0098] In the example of Figure 6, the CN 640 is a 5GC 640 including an Authentication Server Function (AUSF) 642, Access and Mobility Management Function (AMF) 644, Session Management Function (SMF) 646, User Plane Function (UPF) 648, Network Slice Selection Function (NSSF) 650, Network Exposure Function (NEF) 652, Network Repository Function (NRF) 654, Policy Control Function (PCF) 656, Unified Data Management (UDM) 658, Unified Data Repository (UDR), Application Function (AF) 660, and Network Data Analytics Function (NWDAF) 662 coupled with one another over various interfaces as shown. The NFs in the 5GC 640 are briefly introduced as follows.

[0099] The NWDAF 662 is an NF capable of collecting data from UEs 602, other NF(s) in 5GC 640, an Operations, Administration and Maintenance (0AM) entities / functions, MnS (see e.g., Figure 9), MnF (see e.g., Figure 9), AFs 660, DNs 636, server(s) 638, cloud computing services, edge compute nodes and / or edge networks, and / or other entities / elements that can be used for analytics. The NWDAF 662 includes one or more of the following functionalities: support data collection from NFs and AFs 660; support data collection from 0AM; NWDAF service registration and metadata exposure to NFs and AFs 660; support analytics information provisioning to NFs and AFs 660; support ML model training and provisioning to NWDAF(s) 662 (e.g., those containing analytics logical function). Some or all of the NWDAF functionalities can be supported in a single instance of an NWDAF 662. The NWDAF 662 also includes an analytics reporting capability, which comprises means that allow discovery of the type of analytics that can be consumed by an external party and / or the request for consumption of analytics information generated by the NWDAF 662. The NWDAF 662 can collect data from NF(s) and / or other entities / elements / functions over an Nnf service-based interface associated with the NF(s) and / or other entities / elements / functions. The NWDAF 662 belongs to the same PLMN as the NF that provides the data. The Nnf interface is defined for the NWDAF 662 to request subscription to data delivery for a particular context, cancel subscription to data delivery, and request a specific report of data for a particular context. The 5GS architecture also allows the NWDAF 662 to retrieve management data from an 0AM entity by invoking 0AM services.

[0100] The NWDAF 662 interacts with different entities for different purposes, such as one or more of the following: data collection based on subscription to events provided by AMF 644, SMF 646, PCF 656, UDM 658, NSACF, AF 660 (directly or via NEF 652) and 0AM; analytics and data collection using the Data Collection Coordination Function (DCCF); retrieval of information from data repositories (e.g., UDR via UDM 658 for subscriber-related information); data collection of location information from LCS system; storage and retrieval of information from an Analytics Data Repository Function (ADRF); analytics and data collection from a Messaging Framework Adaptor Function (MFAF); retrieval of information about NFs (e.g., from NRF 654 for NF-related information); on-demand provision of analytics to consumers, as specified in [TS23288] § 6; and / or provision of bulked data related to analytics ID(s). NWDAF discovery and selection procedures are discussed in [TS23501] § 6.3.13 and [TS23288] § 5.2.

[0101] A single instance or multiple instances of NWDAF 662 may be deployed in a PLMN. If multiple NWDAF 662 instances are deployed, the architecture supports deploying the NWDAF 662 as a central NF, as a collection of distributed NFs, or as a combination of both. If multiple NWDAF 662 instances are deployed, an NWDAF 662 can act as an aggregate point (e.g., aggregator NWDAF 662) and collect analytics information from other NWDAFs 662, which may have different serving areas, to produce the aggregated analytics (e.g., per analytics ID), possibly with analytics generated by itself. When multiple NWDAFs 662 exist, not all of them need to be able to provide the same type of analytics results. For example, some of the NWDAFs 662 can be specialized in providing certain types of analytics. An analytics ID information element is used to identify the type of supported analytics that NWDAF 662 can generate. In some implementations, NWDAF 662 instance(s) can be collocated with a 5GS NF.

[0102] The NWDAF 662 may contain an analytics logical function (AnLF) and / or a model training logical function (MTLF). The NWDAF 662 can contain only an MTLF, only an AnLF, or both logical functions. The 5GS architecture allows an NWDAF containing an AnLF (referred to herein as “NWDAF-ANLF”) to use trained ML model provisioning services from the same or different NWDAF containing an MTLF (also referred to herein as “NWDAF-MTLF”). The Nnwdaf interface is used by the NWDAF-AnLF to request and subscribe to trained ML model provisioning services provided by the NWDAF-MTLF. The NWDAF 662 provides an Nnwdaf_MLModelProvision service enables an NF service consumer (NFc) to receive a notification when an ML model matching the subscription parameters becomes available in the NWDAF-MTLF (see e.g., [TS23288] § 7.5). The NWDAF 662 provides an Nnwdaf_MLModelInfo service that enables an NFc to request and get ML Model information from the NWDAF-MTLF (see e.g., [TS23288] § 7.6).

[0103] The AnLF is a logical function in the NWDAF 662 that performs inference, derives analytics information (e.g., derives statistics, inferences, and / or predictions based on analytics consumer requests) and exposes analytics services (e.g., Nnwdaf_AnalyticsSubscription or Nnwdaf_AnalyticsInfo). Analytics information are either statistical information of the past events, or predictive information. The MTLF is a logical function in the NWDAF 662 that trains AI / ML models and exposes new training services (e.g., providing trained ML model) as defined in clauses 7.5 and 7.6 of [TS23288].

[0104] Since multiple NWDAF 662 instances may be deployed in a network, an NFc can utilize the NRF 654 to discover NWDAF 662 instance(s) unless NWDAF information is available by other means (e.g., locally configured on NFcs). NFcs may make an additional query to the UDM 658, when supported. An NWDAF selection function in an NFc selects an NWDAF 662 (or NWDAF-MTLF and / or NWDAF-AnLF) instance based on the available NWDAF 662 instances, a list of supported analytics ID(s) (e.g., possibly per supported service) stored / from an NRF 654, NWDAF capabilities (e.g., analytics aggregation capability, analytics metadata provisioning capability, ML model training capabilities, ML model deployment capabilities, and / or the like), and / or other NRF 654 registration elements of the NF profile. Additional aspects of NWDAF 662 functionality are defined in 3GPP TS 23.288 (“[TS23288]”).

[0105] The AUSF 642 stores data for authentication of UE 602 and handle authentication-related functionality. The AUSF 642 may facilitate a common authentication framework for various access types.

[0106] The AMF 644 allows other functions of the 5GC 640 to communicate with the UE 602 and the RAN 604 and to subscribe to notifications about mobility events w.r.t the UE 602. The AMF 644 is also responsible for registration management (e.g., for registering UE 602), connection management, reachability management, mobility management, lawful interception of AMF-related events, and access authentication and authorization. The AMF 644 provides transport for SM messages between the UE 602 and the SMF 646, and acts as a transparent proxy for routing SM messages. AMF 644 also provides transport for SMS messages between UE 602 and an SMSF. AMF 544 interacts with the AUSF 642 and the UE 602 to perform various security anchor and context management functions. Furthermore, AMF 644 is a termination point of a RAN-CP interface, which includes the N2 reference point between the RAN 604 and the AMF 644. The AMF 644 is also a termination point of NAS (Nl) signaling, and performs NAS ciphering and integrity protection.

[0107] The AMF 644 also supports NAS signaling with the UE 602 over an N3IWF interface. The N3IWF provides access to untrusted entities. N3IWF may be a termination point for the N2 interface between the (R)AN 604 and the AMF 644 for the control plane, and may be a termination point for the N3 reference point between the (R)AN 604 and the 648 for the user plane. As such, the AMF 644 handles N2 signaling from the SMF 646 and the AMF 644 for PDU sessions and QoS, encapsulate / de-encapsulate packets for IPSec and N3 tunneling, marks N3 user-plane packets in the UL, and enforces QoS corresponding to N3 packet marking taking into account QoS requirements associated with such marking received over N2. N3IWF may also relay UL and DL control-plane NAS signaling between the UE 602 and AMF 644 via an N 1 reference point between the UE 602 and the AMF 644, and relay UL and DL user-plane packets between the UE 602 and UPF 648. The N3IWF also provides mechanisms for IPsec tunnel establishment with the UE 602. The AMF 644 may exhibit an Namf service-based interface, and may be a termination point for an N14 reference point between two AMFs 644 and an N17 reference point between the AMF 644 and a 5G-EIR (not shown by Figure 6). In addition to the functionality of the AMF 644 described herein, the AMF 644 may provide support for Network Slice restriction and Network Slice instance restriction based on NWDAF analytics.

[0108] The SMF 646 is responsible for SM (e.g., session establishment, tunnel management between UPF 648 and NAN 614); UE IP address allocation and management (including optional authorization); selection and control of UP function; configuring traffic steering at UPF 648 to route traffic to proper destination; termination of interfaces toward policy control functions; controlling part of policy enforcement, charging, and QoS; lawful intercept (for SM events and interface to LI system); termination of SM parts of NAS messages; DL data notification; initiating AN specific SM information, sent via AMF 644 over N2 to NAN 614; and determining SSC mode of a session. SM refers to management of a PDU session, and a PDU session or “session” refers to a PDU connectivity service that provides or enables the exchange of PDUs between the UE 602 and the DN 636. The SMF 646 may also include the following functionalities to support edge computing enhancements (see e.g., [TS23548]): selection of EASDF 661 and provision of its address to the UE as the DNS server for the PDU session; usage of EASDF 661 services as defined in [TS23548]; and for supporting the application layer architecture defined in [TS23558], provision and updates of ECS address configuration information to the UE. Discovery and selection procedures for EASDFs 661 is discussed in [TS23501] § 6.3.23.

[0109] The UPF 648 acts as an anchor point for intra-RAT and inter-RAT mobility, an external PDU session point of interconnect to data network 636, and a branching point to support multihomed PDU session. The UPF 648 also performs packet routing and forwarding, packet inspection, enforces user plane part of policy rules, lawfully intercept packets (UP collection), performs traffic usage reporting, perform QoS handling for a user plane (e.g., packet filtering, gating, UL / DL rate enforcement), performs UL traffic verification (e.g., SDF-to-QoS flow mapping), transport level packet marking in the UL and DL, and performs DL packet buffering and DL data notification triggering. UPF 648 may include an UL classifier to support routing traffic flows to a data network.

[0110] The NSSF 650 selects a set of network slice instances serving the UE 602. The NSSF 650 also determines allowed NSSAI and the mapping to the subscribed S-NSSAIs, if needed. The NSSF 650 also determines an AMF set to be used to serve the UE 602, or a list of candidate AMFs 644 based on a suitable configuration and possibly by querying the NRF 654. The selection of a set of network slice instances for the UE 602 may be triggered by the AMF 644 with which the UE 602 is registered by interacting with the NSSF 650; this may lead to a change of AMF 644. The NSSF 650 interacts with the AMF 644 via an N22 reference point; and may communicate with another NSSF in a visited network via an N31 reference point (not shown).

[0111] The NEF 652 securely exposes services and capabilities provided by 3GPP NFs for third party, internal exposure / re-exposure, AFs 660, edge computing networks / frame works, and the like. In such examples, the NEF 652 may authenticate, authorize, or throttle the AFs 660. The NEF 652 stores / retrieves information as structured data using the Nudr interface to a Unified Data Repository (UDR). The NEF 652 also translates information exchanged with the AF 660 and information exchanged with internal NFs. For example, the NEF 652 may translate between an AF-Service-Identifier and an internal 5GC information, such as DNN, S-NSSAI, as described in [TS23501] § 5.6.7. In particular, the NEF 652 handles masking of network and user sensitive information to external AF's 660 according to the network policy. The NEF 652 also receives information from other NFs based on exposed capabilities of other NFs. This information may be stored at the NEF 652 as structured data, or at a data storage NF using standardized interfaces. The stored information can then be re-exposed by the NEF 652 to other NFs and AFs, or used for other purposes such as analytics. For example, NWDAF analytics may be securely exposed by the NEF 652 for external party, as specified in [TS23288]. Furthermore, data provided by an external party may be collected by the NWDAF 662 via the NEF 652 for analytics generation purpose. The NEF 652 handles and forwards requests and notifications between the NWDAF 662 and AF(s) 660, as specified in [TS23288].

[0112] The NRF 654 supports service discovery functions, receives NF discovery requests from NF instances, and provides information of the discovered NF instances to the requesting NF instances. The NRF 654 also maintains NF profiles of available NF instances and their supported services. The NF profile of NF instance maintained in the NRF 654 includes the following information: NF instance ID; NF type; PEMN ID in the case of PEMN, PEMN ID + NID in the case of SNPN; Network Slice related Identifier(s) (e.g., S-NSSAI, NSI ID); an NF’s network address(es) (e.g., FQDN, IP address, and / or the like), NF capacity information, NF priority information (e.g., for AMF selection), NF set ID, NF service set ID of the NF service instance; NF specific service authorization information; names of supported services, if applicable; endpoint address(es) of instance(s) of each supported service; identification of stored data / information (e.g., for UDR profile and / or other NF profiles); other service parameter(s) (e.g., DNN or DNN list, EADN DNN or EADN DNN list, notification endpoint for each type of notification that the NF service is interested in receiving, and / or the like); location information for the NF instance (e.g., geographical location, data center, and / or the like); TAI(s); NF load information; Routing Indicator, Home Network Public Key identifier, for UDM 658 and AUSF 642; for UDM 658, AUSF 642, and NSSAAF in the case of access to an SNPN using credentials owned by a Credentials Holder with AAA Server, identification of Credentials Holder (e.g., the realm of the Network Specific Identifier based SUPI); for UDM 658 and AUSF 642, and if UDM 658 / AUSF 642 is used for access to an SNPN using credentials owned by a Credentials Holder, identification of Credentials Holder (e.g., the realm if network specific identifier based SUPI is used or the MCC and MNC if IMSI based SUPI is used); for AUSF 642 and NSSAAF in the case of SNPN Onboarding using a DCS with AAA server, identification of DCS (e.g., the realm of the Network Specific Identifier based SUPI); for UDM 658 and AUSF 642, and if UDM 658 / AUSF 642 is used as DCS in the case of SNPN Onboarding, identification of DCS ((e.g., the realm if Network Specific Identifier based SUPI, or the MCC and MNC if IMSI based SUPI); one or more GUAMI(s), in the case of AMF 644; for the UPF 648, see [TS23502] § 5.2.7.2.2; UDM Group ID, range(s) of SUPIs, range(s) of GPSIs, range(s) of internal group identifiers, range(s) of external group identifiers for UDM 658; UDR Group ID, range(s) of SUPIs, range(s) of GPSIs, range(s) of external group identifiers for UDR; AUSF Group ID, range(s) of SUPIs for AUSF 642; PCF Group ID, range(s) of SUPIs for PCF 656; HSS Group ID, set(s) of IMPIs, set(s) of IMPU, set(s) of IMSIs, set(s) of PSIs, set(s) of MSISDN for HSS; event ID(s) supported by AFs 660, in the case of NEF 652; event Exposure service supported event ID(s) by UPF 648; application identifier(s) supported by AFs 660, in the case of NEF 652; range(s) of external identifiers, or range(s) of external group identifiers, or the domain names served by the NEF, in the case of NEF 652 (e.g., used when the NEF 652 exposes AF information for analytics purpose as detailed in [TS23288]; additionally the NRF 654 may store a mapping between UDM Group ID and SUPI(s), UDR Group ID and SUPI(s), AUSF Group ID and SUPI(s) and PCF Group ID and SUPI(s), to enable discovery of UDM 658, UDR, AUSF 642 and PCF 656 using SUPI, SUPI ranges as specified in [TS23501] § 6.3, and / or interact with UDR to resolve the UDM Group ID / UDR Group ID / AUSF Group ID / PCF Group ID based on UE identity (e.g., SUPI)); IP domain list as described in 3GPP TS 29.510 § 6.1.6.2.21, Range(s) of (UE) IPv4 addresses or Range(s) of (UE) IPv6 prefixes, Range(s) of SUPIs or Range(s) of GPSIs or a BSF Group ID, in the case of BSF; SCP Domain the NF belongs to; DCCF Serving Area information, NF types of the data sources, NF Set IDs of the data sources, if available, in the case of DCCF; supported DNAI list, in the case of SMF 646; for SNPN, capability to support SNPN Onboarding in the case of AMF and capability to support User Plane Remote Provisioning in the case of SMF 646; IP address range, DNAI for UPF 648; additional V2X related NF profile parameters are defined in 3GPP TS 23.287; additional ProSe related NF profile parameters are defined in 3GPP TS 23.304; additional MBS related NF profile parameters are defined in 3GPP TS 23.247; additional UAS related NF profile parameters are defined in 3GPP TS 23.256; among many others discussed in [TS23501]. In some examples, service authorization information provided by an OAM system is also included in the NF profile in the case that, for example, an NF instance has an exceptional service authorization information.

[0113] For NWDAF 662, the NF profile includes: supported analytics ID(s), possibly per service, NWDAF serving area information (e.g., a list of TAIs for which the NWDAF can provide services and / or data), Supported Analytics Delay per Analytics ID (if available), NF types of the NF data sources, NF Set IDs of the NF data sources, if available, analytics aggregation capability (if available), analytics metadata provisioning capability (if available), ML model filter information parameters S-NSSAI(s) and area(s) of interest for the trained ML model(s) per analytics ID(s) (if available), federated learning (FL) capability type (e.g., FL server or FL client, if available), Time interval supporting FL (if available). The NWDAF's 662 Serving Area information is common to all its supported analytics IDs. The analytics IDs supported by the NWDAF 662 may be associated with a supported analytics delay, for example, the analytics report can be generated with a time (including data collection delay and inference delay) in less than or equal to the supported analytics delay. The determination of supported analytics delay, and how the NWDAF 662 avoid updating its Supported Analytics Delay in NRF frequently may be NWDAF-implementation specific.

[0114] The PCF 656 provides policy rules to control plane functions to enforce them, and may also support unified policy framework to govern network behavior. The PCF 656 may also implement a front end to access subscription information relevant for policy decisions in a UDR of the UDM 658. In addition to communicating with functions over reference points as shown, the PCF 656 exhibit an Npcf service-based interface.

[0115] The UDM 658 handles subscription-related information to support the network entities’ handling of communication sessions, and stores subscription data of UE 602. For example, subscription data may be communicated via an N8 reference point between the UDM 658 and the AMF 644. The UDM 658 may include two parts, an application front end and a UDR. The UDR may store subscription data and policy data for the UDM 658 and the PCF 656, and / or structured data for exposure and application data (including PFDs for application detection, application request information for multiple UEs 602) for the NEF 652. The Nudr service-based interface may be exhibited by the UDR to allow the UDM 658, PCF 656, and NEF 652 to access a particular set of the stored data, as well as to read, update (e.g., add, modify), delete, and subscribe to notification of relevant data changes in the UDR. The UDM 658 may include a UDM-FE, which is in charge of processing credentials, location management, subscription management and so on. Several different front ends may serve the same user in different transactions. The UDM-FE accesses subscription information stored in the UDR and performs authentication credential processing, user identification handling, access authorization, registration / mobility management, and subscription management. In addition to communicating with other NFs over reference points as shown, the UDM 658 may exhibit the Nudm service-based interface.

[0116] Edge Application Server Discovery Function (EASDF) 661 exhibits an Neasdf servicebased interface, and is connected to the SMF 646 via an N88 interface. One or multiple EASDF instances may be deployed within a PLMN, and interactions between 5GC NF(s) and the EASDF 661 take place within a PLMN. The EASDF 661 includes one or more of the following functionalities: registering to NRF 654 for EASDF 661 discovery and selection; handling the DNS messages according to the instruction from the SMF 646; and / or terminating DNS security, if used. Handling the DNS messages according to the instruction from the SMF 646 includes one or more of the following functionalities: receiving DNS message handling rules and / or BaselineDNSPattem from the SMF 646; exchanging DNS messages from / with the UE 602; forwarding DNS messages to C-DNS or L-DNS for DNS query; adding EDNS client subnet (ECS) option into DNS query for an FQDN; reporting to the SMF 646 the information related to the received DNS messages; and / or buffering / discarding DNS messages from the UE 602 or DNS Server. The EASDF has direct user plane connectivity (e.g., without any NAT) with the PSA UPF over N6 for the transmission of DNS signaling exchanged with the UE. The deployment of a NAT between EASDF 661 and PSA UPF 648 may or may not be supported. Additional aspects of the EASDF 661 are discussed in [TS23548].

[0117] The AF 660 is a functional element that provides service and / or application related information to NFcs (e.g., NWDAF 662, DCCF, MFAF, NEF 652, event consumer AF, LMF, and / or the like). The AF 660 allows NFcs to subscribe / unsubscribe from periodic notifications and / or notifications related to the detection of subscribed event(s) (e.g., asynchronous event(s)), for example, using various service operations such as those discussed in 3GPP TS 29.517. These notifications / events may be based on data collected from an AF 660, either directly from the AF 660 or via NEF 652. The data collected from an AF 660 is used as input for analytics by the NWDAF 662. The details for the data collected from an AF 660 as well as interactions between NEF 652, AF 660 and NWDAF 662 are described in [TS23288].

[0118] The AF 660 may also provide application influence on traffic routing, provide access to NEF 652, interact with the policy framework for policy control, and / or influence UPF 648 (re)selection and traffic routing. Based on operator deployment, when an AF 660 is considered to be a trusted entity, the network operator may permit the AF 660 to interact directly with relevant NFs. The 5GC 640 may enable edge computing by selecting operator / 3rd party services to be geographically close to a point that the UE 602 is attached to the network. This may reduce latency and load on the network. In edge computing implementations, the 5GC 640 may select a UPF 648 close to the UE 602 and execute traffic steering from the UPF 648 to DN 636 via the N6 interface. This may be based on the UE subscription data, UE location, and information provided by the AF 660, which allows the AF 660 to influence UPF (re)selection and traffic routing.

[0119] The data network (DN) 636 represents any network (or collection of networks) hosting data-centric services such as, for example, operator services, the internet, third-party services, or enterprise networks. The data-centric services may be offered as a service to the UE 602. Additionally or alternatively, the data-centric services may be provided by one or more application (app) servers 638. The DN 636 may be an operator external public or private packet data network (PDN), or an intra-operator PDN, for example, for provision of IMS services. In some implementations, the DN 636 may represent one or more local area DNs (EADNs), which are DNs 636 that is / are accessible by a UE 602 in one or more specific areas / locations, that provides connectivity to a specific DNN, and / or whose availability is provided to the UE 602. Outside of these specific areas, the UE 602 is not able to access the LADN 636.

[0120] In some implementations, the DN 636 may be an edge DN 636, which is an LADN that supports the architecture for enabling edge applications (see e.g., [5GEdge]). In these examples, the app server 638 act as edge compute node(s) and / or represents physical hardware systems / devices providing app server functionality and / or the app software resident in a cloud or edge network that performs app / edge / cloud server function(s). In some examples, the app server 638 provides an edge hosting environment that supports edge application server execution. Additionally or alternatively, the edge compute nodes may be included in / with, or co-located with one or more RANs 604 or RAN nodes 614. For example, the edge compute nodes can provide a connection between the RAN 604 and UPF 648 in the 5GC 640.

[0121] The edge compute nodes (also referred to as “edge hosts” or “edge servers”) provide a distributed computing environment for application and service hosting, and also provide storage and processing resources so that data and / or content can be processed in close proximity to subscribers (e.g., users of UEs 602) for faster response times. The edge compute nodes also support multitenancy run-time and hosting environment(s) for applications (e.g., edge apps) and / or network function virtualization (NFV) instances that may provide infrastructure services (e.g., processing wireless connections to and from the RAN 614 and UPF 648, protocol stack processing, and / or the like), content delivery services (e.g., content caching, mobile big data analytics, computational offloading, mapping and navigation services, among others), and / or other services. In some implementations, the edge compute nodes include an edge platform and / or virtualization infrastructure (VI) that provide compute, storage, and network resources for app / NFV instances. The VI provides virtualized environments and virtualized resources for edge hosts, and the edge computing applications may run as virtual machines (VMs) and / or virtualization containers on top of the VI. In some examples, the VI may be implemented using NFV technologies, such as those discussed in ETSI GR NFV 001, ETSI GS NFV 002, ETSI GR NFV 003, ETSI GR NFV 003, ETSI GS NFV 006, ETSI GS NFV-INF 001, ETSI GS NFV-INF 003, ETSI GS NFV-INF 004, ETSI GS NFV-MAN 001, and / or Israel et al., OSM Release FIVE Technical Overview, ETSI OPEN SOURCE MANO, OSM White Paper, 1st ed. (Jan. 2019). The VI may be implemented using other virtualization technologies and / or service orchestration and automation platforms, such as those discussed in E2E Network Slicing Architecture, GSMA, Official Doc. NG.127, vl.O (03 Jun. 2021), Open Network Automation Platform (ONAP) documentation, Release Istanbul, v9.0.1 (17 Feb. 2022), and / or 3GPP Service Based Management Architecture (SBMA) as discussed in [TS28533] and / or [5GEdge].

[0122] The edge compute nodes may include or be part of an edge system / network that employs one or more edge computing technologies (ECTs) (also referred to as an “edge computing framework” or the like). An edge system / network includes a collection of edge servers and edge management systems (not shown) necessary to run edge apps / NFV instances within an operator network or a subset of an operator network.

[0123] In one example implementation, the ECT is and / or operates according to the MEC framework, as discussed in ETSI GR MEC 001, ETSI GS MEC 003, ETSI GS MEC 009, ETSI GS MEC 010-1, ETSI GS MEC 010-2, ETSI GS MEC Oi l, ETSI GS MEC 012, ETSI GS MEC 013, ETSI GS MEC 014, ETSI GS MEC 015, ETSI GS MEC 016, ETSI GS MEC 021, ETSI GR MEC 024, ETSI GS MEC 028, ETSI GS MEC 029, ETSI MEC GS 030, and ETSI GR MEC 031 (collectively referred to herein as “[MEC]”). In another example implementation, the ECT is and / or operates according to the O-RAN framework, as described in O-RAN Working Group 1 (Use Cases and Overall Architecture): O-RAN Architecture Description, O-RAN Architecture Description vlO.OO, Release R003 (Oct. 2023); O-RAN Working Group 2 (Non-RT RIC and Al interface WG) Non-RT RIC Architecture, v04.00, Release R003 (Oct. 2023); O-RAN Working Group 3 (Near-Real-time RAN Intelligent Controller and E2 Interface Working Group): Near-RT RIC Architecture, v05.00, Release R003 (Oct. 2023), and / or any other specifications released by the O-RAN Alliance (collectively referred to as “[O-RAN]”).

[0124] In another example implementation, the ECT is and / or operates according to the 3rd Generation Partnership Project (3GPP) System Aspects Working Group 6 (SA6) Architecture for enabling Edge Applications as discussed in 3GPP TS 23.222, 3GPP TS 23.401, 3GPP TS 23.434, 3GPP TS 23.501 (“[TS23501]”), 3GPP TS 23.502 (“[TS23502]”), 3GPP TS 23.548 (“[TS23548]”), 3GPP TS 23.558 (“[TS23558]”), 3GPP TS 23.682, 3GPP TR 23.700- 98, 3GPP TS 28.104 (“[TS28104]”), 3GPP TS 28.105 (“[TS28105]”), 3GPP TS 28.312, 3GPP TS 28.532 (“[TS28532]”), 3GPP TS 28.533 (“[TS28533]”), 3GPP TS 28.535, 3GPP TS 28.536, 3GPP TS 28.538, 3GPP TS 28.541 (“[TS28541]”), 3GPP TS 28.545 (“[TS28545]”), 3GPP TS 28.550 (“[TS28550]”), 3GPP TS 28.554 (“[TS28554]”), 3GPP TS 28.622 (“[TS28622]”), 3GPP TS 29.122, 3GPP TS 29.222, 3GPP TS 29.522, 3GPP TR 28.908, 3GPP TS 33.122 (collectively referred to as “[5GEdge]”). In another example implementation, the ECT is and / or operates according to the Multi-Access Management Services (MAMS) framework as discussed in Kanugovi et al., Multi-Access Management Services (MAMS), INTERNET ENGINEERING TASK FORCE (IETF), Request for Comments (RFC) 8743 (Mar. 2020) and Zhu et al., Generic Multi-Access (GMA) Convergence Encapsulation Protocols, IETF RFC 9188 (Feb. 2022) (collectively referred to as “[MAMS]”).

[0125] The interfaces of the 5GC 640 include reference points and service-based interfaces. A reference point, at least in some examples, is a point at the conjunction of two non-overlapping functional groups, elements, or entities. The reference points in the 5GC 640 include: Nl, N2, N3, N4, N5, N6, N7, N8, N9, N10, Ni l, N12, N13, N14 (between two AMFs 644; not shown), N15, N16, and N22. Other reference points not shown in Figure 6 can also be used, such as any of those discussed in [TS23501]. The service-based representation of Figure 6 represents NFs within the control plane that enable other authorized NFs to access their services. A service-based interface (SBI), at least in some examples, is an interface over which an NF can access the services of one or more other NFs. In some implementations, the service-based interfaces are API-based interfaces (e.g., HTTP / 2, RESTful, SOAP, and / or any other API or web service) that can be used by an NF to call or invoke a particular service or service operation. The SBIs in the 5GC 640 include: Namf, Nsmf, Nnef, Npcf, Nudm, Naf, Nnrf, Nnssf, Nausf. Other service-based interfaces (e.g., Nudr, N5g-eir, and Nudsf) not shown in Figure 6 can also be used, such as any of those discussed in [TS23501].

[0126] Although not shown by Figure 6, the system 600 may also include NFs that are not shown such as, for example, UDR, Unstructured Data Storage Function (UDSF), Network Slice Admission Control Function (NSACF), Network Slice- specific and Stand-alone Non-Public Network (SNPN) Authentication and Authorization Function (NSSAAF), UE radio Capability Management Function (UCMF), 5G-Equipment Identity Register (5G-EIR), CHarging Function (CHF), Time Sensitive Networking (TSN) AF 660, Time Sensitive Communication and Time Synchronization Function (TSCTSF), DCCF, Analytics Data Repository Function (ADRF), MFAF, Non-Seamless WEAN Offload Function (NSWOF), Service Communication Proxy (SCP), Security Edge Protection Proxy (SEPP), Non-3GPP InterWorking Function (N3IWF), Trusted Non-3GPP Gateway Function (TNGF), Wireline Access Gateway Function (W-AGF), and / or Trusted WLAN Interworking Function (TWIF) as discussed in [TS23501].

[0127] Figure 7 illustrates a wireless network 700, which includes a UE 702 in wireless communication with a NAN 704. The UE 702 may be the same or similar to, and substantially interchangeable with any of the of the UEs discussed herein such as, for example, UE 602, hardware resources 800, and / or the like. The NAN 704 may be the same or similar to, and substantially interchangeable with any of the NANs discussed herein such as, for example, AP 606, NANs 614, RAN 604, hardware resources 800, and / or the like.

[0128] The UE 702 can communicatively couple with the NAN 704 via connection 706. The connection 506 is an air interface to enable communicative coupling, and can be consistent with cellular communications protocols (e.g., LTE, 5G / NR, mmWave or sub-6GHz frequencies, and / or any other access network protocol). The connection 506 may correspond to the Uu interface described w.r.t Figure 4.

[0129] The UE 702 includes a host platform 708 coupled with a modem platform 710. The host platform 708 includes application processing circuitry 712, which may be coupled with protocol processing circuitry 714 of the modem platform 710. The application processing circuitry 712 may run various applications for the UE 702 that source / sink application data. The application processing circuitry 712 may further implement one or more layer operations to transmit / receive application data to / from a data network. These layer operations includes transport (e.g., user datagram protocol (UDP), QUIC (Quick UDP Internet Connections), transmission control protocol (TCP), GPRS Tunneling (GTP), and / or some other transport layer protocol) operations and network / Intemet (e.g., internet protocol (IP), IPSec, routing information protocol (RIP), external gateway protocol (EGP), internet control message protocol (ICMP), internet group management protocol (IGMP), and / or some other network and / or Internet layer protocol) operations. The protocol processing circuitry 714 may perform one or more protocol layer operations to facilitate transmission or reception of data over the connection 706. The protocol layer operations implemented by the protocol processing circuitry 714 includes, for example, operations for some or all of the following layers: physical layer (PHY) (see e.g., 3GPP TS 38.201), medium access control (MAC) (see e.g., 3GPP TS 38.321), radio link control layer (RLC) (see e.g., 3GPP TS 38.322), packet data convergence protocol (PDCP) (see e.g., 3GPP TS 38.323), Service Data Adaptation Protocol (SDAP) (see e.g., 3GPP TS 37.324), radio resource control (RRC) (see e.g., 3GPP TS 38.331 (“[TS38331]”), and non-access stratum (NAS) (see e.g., 3GPP TS 24.301 and / or 3GPP TS 24.501).

[0130] The modem platform 710 may further include digital baseband circuitry 716 that may implement one or more layer operations that are “below” layer operations performed by the protocol processing circuitry 714 in a network protocol stack. These operations includes, for example, PHY operations including one or more of HARQ functions, scrambling / descrambling, encoding / decoding, layer mapping / de-mapping, modulation symbol mapping, received symbol / bit metric determination, multi-antenna port precoding / decoding, which includes one or more of space-time, space-frequency or spatial coding, reference signal generation / detection, preamble sequence generation and / or decoding, synchronization sequence generation / detection, control channel signal blind decoding, and / or other related functions, including any of those discussed herein and / or in 3GPP TS 36.201, 3GPP TS 38.201, [TS38211], [TS38212], [TS38213], [TS38214], and / or any other standards / specifications, including any of those mentioned herein. In some examples, the protocol processing circuitry 714 includes one or more instances of control circuitry (not shown) to provide control functions for the transmit / receive components.

[0131] The modem platform 710 may further include transmit circuitry 718, receive circuitry 720, RF circuitry 722, and RF front end (RFFE) 724, which includes or connect to one or more antenna panels 726. Briefly, the transmit circuitry 718 includes a digital-to- analog converter, mixer, intermediate frequency (IF) components, and / or the like; the receive circuitry 720 includes an analog-to-digital converter, mixer, IF components, and / or the like; the RF circuitry 722 includes a low-noise amplifier, a power amplifier, power tracking components, and / or the like; RFFE 724 includes filters (e.g., surface / bulk acoustic wave filters), switches, antenna tuners, beamforming components (e.g., phase-array antenna components), and / or the like The selection and arrangement of the components of the transmit circuitry 718, receive circuitry 720, RF circuitry 722, RFFE 724, and antenna panels 726 (referred generically as “transmit / receive components” or “Tx / Rx components”) may be specific to details of a specific implementation such as, for example, whether communication is TDM or FDM, in mmWave or sub-6 gHz frequencies, and / or the like. In some examples, the transmit / receive components may be arranged in multiple parallel Tx / Rx chains, may be disposed in the same or different chips / modules, and / or the like.

[0132] A UE reception may be established by and via the antenna panels 726, RFFE 724, RF circuitry 722, receive circuitry 720, digital baseband circuitry 716, and protocol processing circuitry 714. In some examples, the antenna panels 726 may receive a transmission from the NAN 704 by receive-beamforming signals received by a set of antennas / antenna elements of the one or more antenna panels 726. A UE transmission may be established by and via the protocol processing circuitry 714, digital baseband circuitry 716, transmit circuitry 718, RF circuitry 722, RFFE 724, and antenna panels 726. In some examples, the transmit components of the UE 704 may apply a spatial filter to the data to be transmitted to form a transmit beam emitted by the antenna elements of the antenna panels 726.

[0133] Similar to the UE 702, the NAN 704 includes a host platform 728 coupled with a modem platform 730. The host platform 728 includes application processing circuitry 732 coupled with protocol processing circuitry 734 of the modem platform 730. The modem platform may further include digital baseband circuitry 736, transmit circuitry 738, receive circuitry 740, RF circuitry 742, RFFE circuitry 744, and antenna panels 746. The components of the NAN 704 may be similar to and substantially interchangeable with like-named components of the UE 702. In addition to performing data transmission / reception as described above, the components of the NAN 704 may perform various logical functions that include, for example, RNC functions such as radio bearer management, UL and DL dynamic radio resource management, and data packet scheduling. Examples of the antenna elements of the antenna panels 726 and / or the antenna elements of the antenna panels 746 include planar inverted-F antennas (PIFAs), monopole antennas, dipole antennas, loop antennas, patch antennas, Yagi antennas, parabolic dish antennas, omni-directional antennas, and / or the like.

[0134] Figure 8 illustrates components capable of reading instructions from a machine-readable or computer-readable medium (e.g., a non-transitory machine-readable storage medium) and perform any one or more of the methodologies discussed herein. Specifically, Figure 8 shows hardware resources 800 including one or more processors (or processor cores) 810, one or more memory / storage devices 820, and one or more communication resources 830, each of which may be communicatively coupled via a bus 840 or other interface circuitry. For examples where node virtualization (e.g., NFV) is utilized, a hypervisor 802 may be executed to provide an execution environment for one or more network slices / sub- slices to utilize the hardware resources 800. In some examples, the hardware resources 800 may be implemented in or by an individual compute node, which may be housed in an enclosure of various form factors. In other examples, the hardware resources 800 may be implemented by multiple compute nodes that may be deployed in one or more data centers and / or distributed across one or more geographic regions.

[0135] The processors 810 may include processors (or cores) 810-1 to 810- / ? (where p is a number). Individual processors 810-1 to 810- / ? may be, for example, a central processing unit (CPU), a reduced instruction set computing (RISC) processor, a complex instruction set computing (CISC) processor, a graphics processing unit (GPU), a DSP such as a baseband processor, an ASIC, an FPGA, a radio-frequency integrated circuit (RFIC), a microprocessor or controller, a multi-core processor, a multithreaded processor, an ultra-low voltage processor, an embedded processor, an xPU, a data processing unit (DPU), an Infrastructure Processing Unit (IPU), a network processing unit (NPU), another processor (including any of those discussed herein), and / or any suitable combination thereof. Each processor (or core) 810-1 to 810- / ? may be the same as, or different from, each other processor (or core) 810-1 to 810- / ?.

[0136] The memory / storage devices 820 may include main memory, disk storage, or any suitable combination thereof. The memory / storage devices 820 may include, but are not limited to, any type of volatile, non-volatile, semi-volatile memory, and / or any combination thereof. As examples, the memory / storage devices 620 can be or include random access memory (RAM), static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), magnetoresistive RAM (MRAM), conductive bridge Random Access Memory (CB-RAM), spin transfer torque (STT)-MRAM, phase change RAM (PRAM), core memory, dual inline memory modules (DIMMs), microDIMMs, MiniDIMMs, block addressable memory device(s) (e.g., those based on NAND or NOR technologies), read-only memory (ROM), programmable ROM (PROM), erasable PROM (EPROM), electrically EPROM (EEPROM), flash memory, non-volatile RAM (NVRAM), solid-state storage, magnetic disk storage mediums, optical storage mediums, memory devices that use chalcogenide glass, multi-threshold level NAND flash memory, NOR flash memory, single or multi-level Phase Change Memory (PCM) and / or phase change memory with a switch (PCMS), NVM devices that use chalcogenide phase change material (e.g., chalcogenide glass), a resistive memory, nano wire memory, ferroelectric transistor random access memory (FeTRAM), anti-ferroelectric memory, magnetoresistive random access memory (MRAM) memory that incorporates memristor technology, phase change RAM (PRAM), resistive memory including the metal oxide base, the oxygen vacancy base and the conductive bridge Random Access Memory (CB-RAM), or spin transfer torque (STT)-MRAM, a spintronic magnetic junction memory based device, a magnetic tunneling junction (MTJ) based device, a Domain Wall (DW) and Spin Orbit Transfer (SOT) based device, a thyristor based memory device, and / or a combination of any of the aforementioned memory devices, and / or other memory.

[0137] The communication resources 830 may include interconnection or network interface controllers, components, or other suitable devices to communicate with one or more peripheral devices 804 or one or more databases 806 or other network elements via a network 808. For example, the communication resources 830 may include wired communication components (e.g., for coupling via USB, Ethernet, and / or the like), cellular communication components, NFC components, Bluetooth® components, WiFi® components, and other communication components.

[0138] Instructions 850 may comprise software, a program, an application, an applet, an app, or other executable code for causing at least any of the processors 810 to perform any one or more of the methodologies discussed herein. The instructions 850 may reside, completely or partially, within at least one of the processors 810 (e.g., within the processor’s 810 cache memory), the memory / storage devices 820, or any suitable combination thereof. Furthermore, any portion of the instructions 850 may be transferred to the hardware resources 800 from any combination of the peripheral devices 804 or the databases 806. Accordingly, the memory of processors 810, the memory / storage devices 820, the peripheral devices 804, and the databases 806 are examples of computer-readable and machine-readable media.

[0139] In some examples, the peripheral devices 804 may represent one or more sensors such as, for example, exteroceptive sensors, proprioceptive sensors, and / or exproprioceptive sensors (e.g., sensors that capture, measure, or correlate internal states and external states). Examples of such sensors include, inter alia, inertia measurement units (IMU) comprising accelerometers, gyroscopes, and / or magnetometers; microelectromechanical systems (MEMS) or nanoelectromechanical systems (NEMS) comprising 3-axis accelerometers, 3-axis gyroscopes, and / or magnetometers; level sensors; flow sensors; temperature sensors / thermistors; pressure sensors; barometric pressure sensors; gravimeters; altimeters; image sensors / cameras; light detection and ranging (LiDAR) sensors; proximity sensors; depth sensors, ambient light sensors; optical light sensors; ultrasonic transceivers; microphones; power, energy, environmental (PEE) sensor(s); gas sensors; and the like.

[0140] Additionally or alternatively, the peripheral devices 604 may represent one or more actuators such as, for example, soft actuators (e.g., actuators that changes its shape in response to a stimuli such as, for example, mechanical, thermal, magnetic, and / or electrical stimuli), hydraulic actuators, pneumatic actuators, mechanical actuators, electromechanical actuators (EMAs), microelectromechanical actuators, electrohydraulic actuators, linear actuators, linear motors, rotary motors, DC motors, stepper motors, servomechanisms, electromechanical switches, electromechanical relays (EMRs), power switches, valve actuators, piezoelectric actuators and / or biomorphs, thermal biomorphs, solid state actuators, solid state relays (SSRs), shape-memory alloy-based actuators, electroactive polymer-based actuators, relay driver integrated circuits (ICs), solenoids, impactive actuators / mechanisms (e.g., jaws, claws, tweezers, clamps, hooks, mechanical fingers, humaniform dexterous robotic hands, and / or other gripper mechanisms that physically grasp by direct impact upon an object), propulsion actuators / mechanisms (e.g., wheels, axles, thrusters, propellers, engines, motors (e.g., those discussed previously), clutches, and the like), projectile actuators / mechanisms (e.g., mechanisms that shoot or propel objects or elements), and / or audible sound generators, visual warning devices, and / or other like electromechanical components.

[0141] Figure 9 depicts an example of management services (MnS) deployment 900. MnS is a Service Based Management Architecture (SBMA). An MnS is a set of offered management capabilities (e.g., capabilities for management and orchestration (MANO) of network and services). The entity producing an MnS is referred to as an MnS producer (MnS-P) and the entity consuming an MnS is referred to as an MnS consumer (MnS-C). An MnS provided by an MnS-P can be consumed by any entity with appropriate authorization and authentication. As shown by Figure 9, the MnS-P offers its services via a standardized service interface composed of individually specified MnS components (e.g., MnS-C).

[0142] A MnS is specified using different independent components. A concrete MnS includes at least two of these components. Three different component types are defined, including MnS component type A, MnS component type B, and MnS component type C. The MnS component type A is a group of management operations and / or notifications that is agnostic with regard to the entities managed. The operations and notifications as such are hence not involving any information related to the managed network. These operations and notifications are called generic or network agnostic. For example, operations for creating, reading, updating and deleting managed object instances, where the managed object instance to be manipulated is specified only in the signature of the operation, are generic.

[0143] MnS component type B refers to management information represented by information models representing the managed entities. A MnS component type B is also called Network Resource Model (NRM). Examples of MnS component type B include network resource models (see e.g., [TS28622]) and network resource models (see e.g., [TS28541]). MnS component type C is performance information of the managed entity and fault information of the managed entity. Examples of management service component type C include alarm information (see e.g., [TS28532] and [TS28545]) and performance data (see e.g., [TS28552], [TS28554], and [TS32425]).

[0144] An MnS-P is described by a set of metadata called MnS-P profile. The profile holds information about the supported MnS components and their version numbers. This may include also information about support of optional features. For example, a read operation on a complete subtree of managed object instances may support applying filters on the scoped set of objects as optional feature. In this case, the MnS profile should include the information if filtering is supported.

[0145] Figure 9 also depicts an example management function (MnF) deployment 910. The MnF is a logical entity playing the roles of MnS-C and / or MnS-P. An MnF with the role of management service exposure governance is referred to as an “Exposure governance management function” or “exposure governance MnF”. An MnS produced by an MnF 910 may have multiple consumers. The MnF 910 may consume multiple MnS from one or multiple MnS-Ps. In the MnF deployment 910, the MnF plays both roles (e.g., MnS-P and MnS-C). An MnF can be deployed as a separate entity or embedded in an NF to provide MnS(s). For example, MnF deployment scenario 920 shows an example where the MnF is deployed as a separate entity to provide MnS(s) and MnF deployment scenario 930 in which an MnF is embedded in an NF to provide MnS(s). In these examples, the MnFs may interact by consuming MnS produced by other MnFs. Figure 9 also depicts an example MDA service (MDAS or MDA MnS) deployment 950. Management data analytics (MDA), as a key enabler of automation and intelligence, is considered a foundational capability for mobile networks and services management and orchestration. The MDA provides a capability of processing and analysing data related to network and service events and status including, for example, performance measurements, KPIs, trace data, minimization drive tests (MDT) reports, radio link failure (RLF) reports, RRC connection establishment failure event (RCEF) reports, QoE reports, alarms, configuration data, network analytics data, and service experience data from AFs 660, and / or the like, to provide analytics output, (e.g., statistics, predictions, inferences, root cause analysis issues, and / or the like), and may also include recommendations to enable necessary actions for network and service operations. The MDA output is provided by the MDAS-P to the corresponding consumer(s) (e.g., MDAS-C / MDA MnS- C) that requested the analytics.

[0146] The MDA can identify ongoing issues impacting the performance of the network and services, and help to identify in advance potential issues that may cause potential failure and / or performance degradation. The MDA can also assist to predict the network and service demand to enable the timely resource provisioning and deployments which would allow fast time-to-market network and service deployments. The MDAS includes the services exposed by the MDA, which can be consumed by various consumers including, for example, MnFs (e.g., MnS-Ps and / or MnS- Cs for network and service management), NFs (e.g., NWDAF 662 and / or any other NFs / NEs discussed herein), SON functions, network and service optimization tools / functions, SLS assurance functions, human operators, AFs 660, and / or the like. For purposes of the present disclosure, the terms MDAS and MDA MnS may be used interchangeably.

[0147] The MDAS in the context of SBMA enables any authorized consumer to request and receive analytics. A management function (MDAF) may play the roles of MDA MnS-P, MDA MnS-C, other MnS-C, NWDAF consumer, and Location Management Function (LMF) service consumer, and may also interact with other non-3GPP management systems.

[0148] The internal business logic related to MDA leverages current and historical data related to: PM as per [TS28552] KPIs as per [TS28554]; trace data, including MDT / RLF / RCEF, as per 3GPP TS 32.422 and 3GPP TS 32.423; QoE and service experience data as per 3GPP TS 28.405 and 3GPP TS 28.406; analytics data offered by an NWDAF 662 as per [TS23288] including 5GC data and external web / app-based information (e.g., web crawler that provides online news) from an AF 660; alarm information and notifications as per [TS28532]; CM information and notifications; UE location information provided by LMF as per 3GPP TS 23.273; MDA reports from other MDA MnS-Ps; and management data from non-3GPP systems. Additionally or alternatively, the MDAF and / or the MDA internal business logic includes the MDA capability for the energy saving analysis discussed herein.

[0149] Analytics output from the MDA internal business logic are made available by the management functions (MDAFs) playing the role of MDA MnS-Ps to authorized consumers including, but not limited to, other MnFs, NFs / NEs, NWDAF 662, SON functions, optimization tools, and human operators. Historical analytics reports may be saved and retrieved for use at later times by a MDA MnS-C, and historical analytics input (enabling) data (along with current analytics input data) may be used for analytics by MDA MnS-P. Such a historical data usage may be applicable to both or one of the MDA MnS-P and MDA MnS-C side. In some examples, “historical data” refers to (a) historical analytics reports that have been produced in the past, and (b) historical analytics input (enabling) data that had been collected in the past.

[0150] In some examples, the MDA process may utilize AI / ML technologies, such as any of those discussed herein. An MDAF may optionally be deployed as one or more AI / ML inference function(s) in which the relevant ML entities are used for inference per the corresponding MDA capability. Specifications for MDA ML entity training to enable ML entity deployments are given in [TS28105].

[0151] 3. ARTIFICI L INTELLIGENCE AND MACHINE LEARNING ASPECTS

[0152] Figure 10 depicts an example functional framework 1000 for RAN and / or NF intelligence. The functional framework 1000 includes a data collection function (DCF) 1005, an model training function (MTF) 1010 (also referred to as “MLTF 1010”), model inference function (MLIF) 1015 (also referred to as “MLIF 1015”), and an actor 1020. The various AI / ML-related components, functions, elements, or entities shown by Figure 10 may be implemented as hardware, software, firmware, and / or some combination thereof. In some examples, one or more of the AI / ML-related elements are implemented as part of the same hardware (e.g., IC, chip, SoC, SiP, multi-processor chip, multi-tile package, and / or the like), software (e.g., program, process, engine, service, framework, middleware, and / or the like), or firmware as at least one other component, function, element, or entity.

[0153] In some implementations, the DCF 1005, MTF 1010, and / or MIF 1015 can be formed or embodied as an ML function (MLF). In other implementations, a first MLF includes the DCF 1005 and MTF 1010 and a second MLF includes the MIF 1015. In these implementations, the first MLF and the second MLF can communicate with one another using any other access technologies and interfaces discussed herein. Other arrangements / configurations are possible in other implementations. In one example, the first MLF corresponds to an MnF and / or MnS-P and the second MLF corresponds to another MnF and / or an MnS-C, or vice versa (see e.g., Figure 10). Additionally or alternatively, the first MLF corresponds to a set of functions / entities shown by Figures 1-4 and the second MLF corresponds to a different set of functions / entities shown by Figures 1-4. In these examples, the sets of MLFs may be mutually exclusive, or some or all of the MLFs in each of the sets of MLFs may overlap or be shared. In another example, the first MLF and / or the second MLF is / are implemented by respective UEs (e.g., UE 602, UE 702). Additionally or alternatively, the first MLF and / or the second MLF is / are implemented by a same UE or different UEs. In another example, the first MLF and / or the second MLF is / are implemented by respective RANs (e.g., RAN 604) or respective NANs (e.g., AP 606, NAN 614, NAN 704). Additionally or alternatively, the first MLF is implemented as or by a UE and the second MLF is implemented by as or by a RAN node, or vice versa.

[0154] The DCF 1005 is a function that provides input data to the MTF 1010 and MIF 1015. As examples, the DCF 1005 can collect and store RAN configuration parameters, NF configuration parameters, measurement data, RLM data, key performance indicators (KPIs), SLAs, model performance metrics, knowledge base data, ground truth data, ML model parameters, hyperparameters, and / or other data for model training, update, and inference. AI / ML algorithmspecific data preparation (e.g., data pre-processing and cleaning, formatting, and transformation) may or may not be carried out in the DCF 1005. Examples of input data may include measurements from UEs 602, RAN nodes 614, and / or additional or alternative network entities; feedback from actor 1020; and / or output(s) from AI / ML model(s). The input data fed to the MTF 1010 is training data, and the input data fed to the MIF 1015 is inference data. In some implementations, the DCF 1005 is, includes, or otherwise has access to a data repository that is responsible for data collection and storage. The collected data is stored into the repository, and the stored data can be discovered and extracted or otherwise obtained by other functions / elements from the data repository. In some implementations, the DCF 1005 is responsible for data preparation tasks (e.g., data selection, filtering, cleaning, formatting, transformation, translation, augmentation, synthesizing (e.g., artificially generated), and / or pre-processing (e.g., normalization)), which may be performed prior to packaging or otherwise forming the datasets to be provided to the MTF 1010 or the MIF 1015.

[0155] The MTF 1010 is a function that performs AI / ML model training, validation, testing, and / or other aspects of the training phase 101. An AI / ML model (or set of models, ML pipeline, and / or the like) may be trained, validated, and tested using respective training datasets, validation datasets, testing datasets obtained from the DCF 1005. The MTF 1010 produces trained, tested, and / or validated AI / ML model(s) that are ready for deployment. The produced trained and tested models can be stored in a model repository (not shown).

[0156] The MTF 1010 may also perform some or all aspects of the emulation phase 102 and / or deployment phase 103. The MTF 1010 performs model deployment by initially deploying a trained, validated, and tested AI / ML model to the MIF 1015. The MTF 1010 performs model updates by delivering updated AI / ML model(s) to the MIF 1015. The training and updating may include model tuning, re-training, re-testing, and / or re-validating the AI / ML model(s). Examples of the AI / ML model(s) can include any other mentioned herein.

[0157] The MTF 1010 may generate model performance metrics as part of the model training, testing, and / or validation procedure. Examples of the model performance metrics are discussed infra. The MTF 1010 may also be responsible for data preparation tasks (e.g., data selection, filtering, cleaning, formatting, transformation, translation, augmentation, synthesizing (e.g., artificially generated), and / or pre-processing (e.g., normalization)), which may be performed on the data delivered by the DCF 1005, if required.

[0158] In some implementations, the MTF 1010 includes or has access to a model repository that is responsible for AI / ML models’ (both trained and un-trained) storage and exposure. Various model data can be stored in the model repository such as, for example, trained / updated model(s), model parameters, hyperparameters, and / or model metadata. Examples of model metadata include model performance metrics, hardware platform / configuration data, model execution conditions, constraints, parameters, properties, and / or the like. In some examples, the model data can also include inferences made when operating an emulated or deployed ML model. Examples of AI / ML models and other ML model aspects are discussed herein.

[0159] The MIF 1015 is a function that operates a trained AI / ML model and generates AI / ML model inference output(s). For purposes of the present disclosure, the term “inference” refers to the process of using trained AI / ML model(s) to generate statistical inferences, predictions, decisions, probabilities and / or probability distributions, actions, configurations, policies, data analytics, outcomes, optimizations, and / or the like based on new, unseen data (e.g., “input inference data”). In some examples, the inference process can include feeding input inference data (e.g., from the DCF 1005) into a trained ML model, forward passing the input inference data through the ML model’s architecture / topology wherein the ML model performs computations on the data using its learned parameters (e.g., weights, biases, and / or the like) to generate interfence(s), and outputing those inference(s) to an actor 1020. In some examples, the inference process can include data preparation tasks (e.g., such as any of those mentioned herein) before the forward pass, wherein the input inference data is pre-processed or transformed to match the format required by the ML model. In some examples, the MIF 1015 performs some or all aspects of the emulation phase 102 and / or deployment phase 103.

[0160] The MIF 1015 produces an inference output, which is the inferences generated or otherwise produced when the MIF 1015 operates the AI / ML model using the inference data. The MIF 1015 provides the inference output to the actor 1020. Details of inference output are use case specific and may be based on the specific type of AI / ML model being used. The MIF 1015 may also be responsible for data preparation (e.g., data pre-processing and cleaning, formatting, and transformation) based on inference data delivered by the DCF 1005, if required. The MIF 1015 may provide model performance feedback to the MTF 1010 when applicable. The model performance feedback may include various performance metrics (e.g., any of those discussed herein) related to producing inferences. The model performance feedback may be used for monitoring the performance of the AI / ML model, when available.

[0161] The actor 1020 is a function that receives the inference output from the MIF 1015, and triggers or otherwise performs corresponding actions based on the inference output. The actor 1020 may trigger actions directed to other entities and / or to itself. In some examples, the actor 1020 is a network energy saving (NES) function, a mobility robustness optimization (MRO) function, a load balancing optimization (LBO) function, handover (HO) optimization and / or conditional HO (CHO) optimization, physical cell identifier (PCI) configuration, automatic neighbor relation (ANR) management, random access (RACH) optimization, radio resource management (RRM) optimization, and / or some other SON function. In these examples, the inference output is related to NES, MRO, LBO HO / CHO optimization, PCI configuration, ANR management, RACH optimization, RRM optimization, and / or related to some other SON function, and the actor 1020 is one or more RAN nodes 614, UEs 602, VNE 101, PEE sensors, one or more NFs, and / or some other entities / elements discussed herein that perform various operations based on the output inferences.

[0162] The actor 1020 may also provide feedback to the DCF 1005 for storage. The feedback includes information related to the actions performed by the actor 1020. The feedback may include any information that may be needed to derive training data (and / or testing data and / or validation data), inference data, and / or data to monitor the performance of the AI / ML model and its impact to the network through updating of KPIs, performance counters, and the like.

[0163] In some implementations, the MLF (e.g., the first or second MLFs mentioned previously) includes or has access to a model management function (or “MLMF”), which is responsible for management of the AI / ML model(s) produced by the MTF 1010 and / or operated by the MIF 1015. The MLMF may perform various management tasks, such as include deployment of trained models, monitoring ML entity performance, reporting ML entity validation and / or performance data, and / or any of the management functions / aspects discussed herein. In model deployment, the MLMF may allocate and schedule hardware and / or software resources for inference, based on received trained and tested models. In performance monitoring, the MLMF may decide, based on model performance KPIs and / or metrics, to stop, pause, or terminate the running model, start model re-training and / or tuning, select another model, and / or perform other functions / actions / tasks. In examples, the MLMF may be able to configure model management policies for performing the various model management tasks. As alluded to previously, the model performance feedback produced by the MIF 1015 include performance metrics (e.g., accuracy, momentum, precision, quantile, recall / sensitivity, model bias, run-time latency, resource consumption, and / or other suitable metrics / measures, such as any of those discussed herein) of deployed and executing models based on the inference(s) for monitoring purposes. Additionally, the model performance metrics produced by the MTF 1010 may include the same or similar performance metrics based on training, testing, and / or validation of the ML model(s). These performance measurements may be stored in the data repository and / or reported according to the validation reporting mechanisms discussed herein. In some implementations, a performance measurement function may be included in the framework 1000 (or in an MLF) to measure, calculate, and / or predict the performance metrics for the MTF 10105 and / or the MIF 1015.

[0164] 3.1. PERFORMANCE METRICS / MEASUREMENTS

[0165] The performance metrics that are measured, calculated, and / or predicted by the performance measurement function may be based on the particular AI / ML task and the other inputs / parameters of the ML entity. The performance metrics may include model-based metrics and platform-based metrics. The model-based metrics are metrics related to the performance of the model itself and / or without considering the underlying hardware platform. The platform-based metrics are metrics related to the performance of the underlying hardware platform when operating the ML model.

[0166] The model-based metrics may be based on the particular type of AI / ML model and / or the Al / ML domain. For example, regression-related metrics may be predicted for regression-based ML models. Examples of regression-related metrics include error value, mean error, mean absolute error (MAE), mean reciprocal rank (MRR), mean squared error (MSE), root MSE (RMSE), correlation coefficient (R), coefficient of determination (R2), Golbraikh and Tropsha criterion, and / or other like regression-related metrics such as those discussed in Naser et al., Insights into Performance Fitness and. Error Metrics for Machine Learning, arXiv:2006.00887vl (17 May 2020) (“[Naser]”).

[0167] In another example, correlation-related metrics may be predicted for correlation-related metrics Examples of correlation-related metrics include accuracy, precision (also referred to as positive predictive value (PPV)), mean average precision (mAP), negative predictive value (NPV), recall (also referred to as true positive rate (TPR) or sensitivity), specificity (also referred to as true negative rate (TNR) or selectivity), false positive rate, false negative rate, F score (e.g., Fi score, F2 score, Fp score, and / or the like), Matthews Correlation Coefficient (MCC), markedness, receiver operating characteristic (ROC), area under the ROC curve (AUC), distance score, and / or other like correlation-related metrics, such as those discussed in [Naser]. Additional or alternative model-based metrics may also be predicted such as, for example, cumulative gain (CG), discounted CG (DCG), normalized DCG (NDCG), signal-to-noise ratio (SNR), peak SNR (PSNR), structural similarity (SSIM), Intersection over Union (loU), perplexity, bilingual evaluation understudy (BLEU) score, inception score, Wasserstein metric, Frechet inception distance (FID), string metric, edit distance, Eevenshtein distance, Damerau-Eevenshtein distance, number of evaluation instances (e.g., iterations, epochs, or episodes), learning rate (e.g., the speed at which the algorithm reaches (converges to) optimal weights), learning rate decay (or weight decay), number and / or type of computations, number and / or type of multiply and accumulates (MACs), number and / or type of multiply adds (MAdds) operations and / or other like performance metrics related to the performance of the ME model.

[0168] Examples of the platform-based metrics include latency, response time, throughput (e.g., rate of processing work of a processor or platform / system), availability and / or reliability, power consumption (e.g., performance per Watt, and / or the like), transistor count, execution time (e.g., amount of time to obtain an inference, and / or the like), memory footprint, memory utilization, processor utilization, processor time, number of computations, instructions per second (IPS), floating point operations per second (FLOPS), and / or other like performance metrics related to the performance of the ML model and / or the underlying hardware platform to be used to operate the ML model.

[0169] Additionally or alternatively, proxy metrics (e.g., a metric or attribute used as a stand-in or substitute for another metric or attribute) can be used for predicting the ML model performance. For any of the aforementioned performance metrics, the total, mean, and / or some other distribution of such metrics may be predicted and / or measured using any suitable data collection and / or measurement mechanism(s). Aditionally or alternatively, any of the PMs and / or performance metrics discussed in [TS28552], [TS28554], [TS32425], and / or the like, can be used for purposes of the present disclosure.

[0170] 4. EXAMPLE IMPLEMENTATIONS

[0171] Figure 11 shows an example process 1100 to be performed by a service producer (e.g., an MnS-P). The process 1100 includes receiving a request to configure a policy for controlling an MLT process for an AI / ML model (1101); collecting data based on the configured policy (1102); determining whether any of the condition(s) defined / specified in the policy have occurred; if no condition(s) in the policy have occurred, continuing to monitor for occurrence of one or more conditions (1103); if any of the condition(s) in the policy have occurred, controlling the MLT process based on the specified condition(s) (1104). The example operations of process 1100 can be arranged in different orders, one or more of the depicted operations may be combined and / or divided / split into multiple operations, depicted operations may be omitted, and / or additional or alternative operations may be included in any of the depicted processes. Additional examples of the presently described methods, devices, systems, and networks discussed herein include the following, non-limiting example implementations. Each of the following non-limiting examples may stand on its own or may be combined in any permutation or combination with any one or more of the other examples provided below or throughout the present disclosure.

[0172] Example 1 includes a method of operating a service producer for machine learning training (MLT), the method comprising: receiving, from a service consumer, a request to configure a policy for controlling MLT of a machine learning (ML) model; collecting data based on the configured policy; and controlling the MLT of the ML model according to the configured policy and based on the collected data.

[0173] Example 2 includes the method of example 1 and / or some other example(s) herein, wherein the method includes: sending, to the service consumer, an indication of whether configuration of the policy was successful.

[0174] Example 3 includes the method examples 1-2 and / or some other example(s) herein, wherein the request to configure the policy is a request to create the policy; a request to modify one or more conditions, parameters, or attributes of the policy; or a request to delete the policy.

[0175] Example 4 includes the method examples 1-3 and / or some other example(s) herein, wherein the controlling includes: triggering the MLT based on occurrence of one or more conditions specified by the configured policy.

[0176] Example 5 includes the method examples 1-4 and / or some other example(s) herein, wherein the controlling includes: stopping or pausing the MLT based on occurrence of one or more conditions specified by the configured policy.

[0177] Example 6 includes the method example 5 and / or some other example(s) herein, wherein the one or more conditions include one or more performance metric (PM) thresholds related to inference performance.

[0178] Example 7 includes the method example 6 and / or some other example(s) herein, wherein the one or more performance metric thresholds include one or more of: one or more PM values corresponding to a desired inference performance or a range of acceptable or unacceptable PM values for one or more PMs.

[0179] Example 8 includes the method examples 4-7 and / or some other example(s) herein, wherein the one or more conditions include one or more network conditions under which the MLT is to be performed.

[0180] Example 9 includes the method example 8 and / or some other example(s) herein, wherein the one or more network conditions include one or more of: a number of active user equipment (UEs), a number of UEs with one or more desired capabilities, a number of changes of neighbor cells, a number of handovers, a number of radio link failure (RLF) reports, a number of RRC connection establishment failure event (RCEF) reports, signal strength measurement thresholds, or signal quality measurement thresholds.

[0181] Example 10 includes the method examples 4-9 and / or some other example(s) herein, wherein the one or more conditions include one or more platform conditions.

[0182] Example 11 includes the method example 10 and / or some other example(s) herein, wherein the one or more platform conditions include one or more of: a number of nodes or clusters used for the MLT, an amount of compute resources consumed for the MLT, or environmental conditions related to one or more platforms performing the MLT.

[0183] Example 12 includes the method examples 4-11 and / or some other example(s) herein, wherein the one or more conditions include one or more time conditions.

[0184] Example 13 includes the method example 12 and / or some other example(s) herein, wherein the one or more time conditions include occurrence of one or more time periods or occurrence of one or more time intervals.

[0185] Example 14 includes the method examples 1-13 and / or some other example(s) herein, wherein the policy is defined by a data type for an attribute of a Managed Object Instance (MOI) representing an MLT function.

[0186] Example 15 includes the method examples 1-14 and / or some other example(s) herein, wherein the policy is defined by an abstract class inherited by the MOI representing an MLT function.

[0187] Example 16 includes the method examples 1-15 and / or some other example(s) herein, wherein the method includes: receiving an activation / deactivation request from the service consumer to activate or deactivate the MLT function; sending, to the consumer, an indication indicating whether the activation / deactivation request is accepted; and activating or deactivating the MLT function according to the activation / deactivation request.

[0188] Example 17 includes the method example 16 and / or some other example(s) herein, wherein the activation / deactivation request is a request to activate or deactivate the MLT function in response to receipt of the activation / deactivation request.

[0189] Example 18 includes the method example 16 and / or some other example(s) herein, wherein the activation / deactivation request is a request to activate or deactivate the MLT function according to a schedule.

[0190] Example 19 includes the method examples 14-18 and / or some other example(s) herein, wherein the MLT function is a machine learning training function (MLTF) contained by a Network Data Analytics Function (NWDAF).

[0191] Example 20 includes the method examples 14-19 and / or some other example(s) herein, wherein the MLT function is the service producer.

[0192] Example 21 includes the method examples 1-20 and / or some other example(s) herein, wherein the service producer is an MLT management services (MnS) producer and the service consumer is an MLT MnS consumer.

[0193] Example 22 includes an apparatus comprising means to perform one or more elements of a method described in or related to any of examples 1 -21 , or any other method or process described herein. Example 23 includes one or more non-transitory computer-readable media comprising instructions to cause an electronic device, upon execution of the instructions by one or more processors of the electronic device, to perform one or more elements of a method described in or related to any of examples 1-21, or any other method or process described herein. Example 24 includes an apparatus comprising logic, modules, or circuitry to perform one or more elements of a method described in or related to any of examples 1 -21 , or any other method or process described herein. Example 25 includes a method, technique, or process as described in or related to any of examples 1-21, or portions or parts thereof. Example 26 includes an apparatus comprising: one or more processors and one or more computer-readable media comprising instructions that, when executed by the one or more processors, cause the one or more processors to perform the method, techniques, or process as described in or related to any of examples 1-21, or portions thereof. Example 27 includes a signal as described in or related to any of examples 1-21, or portions or parts thereof. Example 28 includes a datagram, packet, frame, segment, protocol data unit (PDU), or message as described in or related to any of examples 1-21, or portions or parts thereof, or otherwise described in the present disclosure. Example 29 includes a signal encoded with data as described in or related to any of examples 1-21, or portions or parts thereof, or otherwise described in the present disclosure. Example 30 includes a signal encoded with a datagram, packet, frame, segment, protocol data unit (PDU), or message as described in or related to any of examples 1-21, or portions or parts thereof, or otherwise described in the present disclosure. Example 31 includes an electromagnetic signal carrying computer-readable instructions, wherein execution of the computer-readable instructions by one or more processors is to cause the one or more processors to perform the method, techniques, or process as described in or related to any of examples 1-21, or portions thereof. Example 32 includes a computer program comprising instructions, wherein execution of the program by a processing element is to cause the processing element to carry out the method, techniques, or process as described in or related to any of examples 1-21, or portions thereof. Example 33 includes a signal in a wireless network as shown and described herein. Example 34 includes a method of communicating in a wireless network as shown and described herein. Example 35 includes a system for providing wireless communication as shown and described herein. Example 36 includes a device for providing wireless communication as shown and described herein.

[0194] Any of the above-described examples may be combined with any other example (or combination of examples), unless explicitly stated otherwise. The foregoing description of one or more implementations provides illustration and description, but is not intended to be exhaustive or to limit the scope of embodiments to the precise form disclosed. Modifications and variations are possible in light of the above teachings or may be acquired from practice of various embodiments.

[0195] 5. TERMINOLOGY

[0196] For the purposes of the present document, the following terms and definitions are applicable to the examples and embodiments discussed herein. Additionally, the terminology discussed in 3GPP TS 28.500 (“[TS28500]”), [TS28105], [TR28908], and 3GPP TS 21.905 may also be applicable to the examples and embodiments discussed herein.

[0197] As used herein, the singular forms “a,” “an” and “the” are intended to include plural forms as well, unless the context clearly indicates otherwise. It will be further understood that the terms “comprises” and / or “comprising,” when used in this specification, specific the presence of stated features, integers, steps, operations, elements, epochs, iterations, stages, and / or components, but do not preclude the presence or addition of one or more other features, integers, steps, operation, elements, components, and / or groups thereof. The phrase “A and / or B” means (A), (B), or (A and B). For the purposes of the present disclosure, the phrase “A, B, and / or C” means (A), (B), (C), (A and B), (A and C), (B and C), or (A, B and C). The phrase “X(s)” means one or more X or a set of X. The description may use the phrases “in an embodiment,” “In some embodiments,” “in one implementation,” “In some implementations,” “in some examples”, and the like, each of which may refer to one or more of the same or different embodiments, implementations, and / or examples. Furthermore, the terms “comprising,” “including,” “having,” and the like, as used with respect to the present disclosure, are synonymous.

[0198] The terms “coupled,” “communicatively coupled,” along with derivatives thereof are used herein. The term “coupled” may mean two or more elements are in direct physical or electrical contact with one another, may mean that two or more elements indirectly contact each other but still cooperate or interact with each other, and / or may mean that one or more other elements are coupled or connected between the elements that are said to be coupled with each other. The term “directly coupled” may mean that two or more elements are in direct contact with one another. The term “communicatively coupled” may mean that two or more elements may be in contact with one another by a means of communication including through a wire or other interconnect connection, through a wireless communication channel or ink, and / or the like.

[0199] The term “measurement” at least in some examples refers to the observation and / or quantification of attributes of an object, event, or phenomenon. Additionally or alternatively, the term “measurement” at least in some examples refers to a set of operations having the object of determining a measured value or measurement result, and / or the actual instance or execution of operations leading to a measured value. Additionally or alternatively, the term “measurement” at least in some examples refers to data recorded during testing. The term “metric” at least in some examples refers to a quantity produced in an assessment of a measured value. Additionally or alternatively, the term “metric” at least in some examples refers to data derived from a set of measurements. Additionally or alternatively, the term “metric” at least in some examples refers to set of events combined or otherwise grouped into one or more values. Additionally or alternatively, the term “metric” at least in some examples refers to a combination of measures or set of collected data points. Additionally or alternatively, the term “metric” at least in some examples refers to a standard definition of a quantity, produced in an assessment of performance and / or reliability of the network, which has an intended utility and is carefully specified to convey the exact meaning of a measured value.

[0200] The term “scheduler” at least in some examples refers to an entity or element that creates or generates a list of times, sequence(s), and / or an order in which tasks, events, actions, jobs, and / or the like are intended to take place. Additionally or alternatively, the term “scheduler” at least in some examples refers to an entity or element that schedules, coordinates, and / or manages tasks, events, actions, jobs, and / or the like to run or execute at specific times, under certain conditions or parameters, and / or based on (pre)defined or configured priorities. Additionally or alternatively, the term “scheduler” at least in some examples refers to an entity or element that assigns or otherwise manages resources and / or timings to perform tasks, events, actions, jobs, and / or the like. The term “network scheduler” at least in some examples refers to a node, element, or entity that manages network packets in transmit and / or receive queues of one or more protocol stacks of network access circuitry (e.g., a network interface controller (NIC), baseband processor, and the like). The term “network scheduler” at least in some examples can be used interchangeably with the terms “packet scheduler”, “queueing discipline” or “qdisc”, and / or “queueing algorithm”.

[0201] The term “network function” or “NF” at least in some examples refers to a functional block within a network infrastructure that has one or more external interfaces and a defined functional behavior. The term “NF instance” at least in some examples refers to an identifiable instance of an NF. The term “NF service” at least in some examples refers to functionality exposed by an NF through a service-based interface and consumed by other authorized NFs. The term “NF service instance” at least in some examples refers to an identifiable instance of the NF service. The term “NF service operation” at least in some examples refers to an elementary unit that an NF service is composed of. The term “NF service set” at least in some examples refers to a group of interchangeable NF service instances of the same service type within an NF instance; in some examples, the NF service instances in the same NF service set have access to the same context data. The term “NF set” at least in some examples refers to a group of interchangeable NF instances of the same type, supporting the same services and the same network slice(s); in some examples, the NF instances in the same NF Set may be geographically distributed but have access to the same context data. The term “network instance” at least in some examples refers to information identifying a domain; in some examples, a network instance is used by a UPF for traffic detection and routing. The term “network service” or “NS” at least in some examples refers to a composition or collection of NF(s) and / or network service(s), defined by its functional and behavioral specification(s).

[0202] The term “Application Function” or “AF” at least in some examples refers to an element or entity that interacts with a 3GPP core network in order to provide services. Additionally or alternatively, the term “Application Function” or “AF” at least in some examples refers to an edge compute node or ECT framework from the perspective of a 5G core network.

[0203] The term “access technology” at least in some examples refers to the technology used for the underlying physical connection to a communication network. The term “radio access technology” or “RAT” at least in some examples refers to the technology used for the underlying physical connection to a radio based communication network. The term “radio technology” at least in some examples refers to technology for wireless transmission and / or reception of electromagnetic radiation for information transfer. The term “RAT type” at least in some examples may identify a transmission technology and / or communication protocol used in an access network.

[0204] The term “service consumer” or “consumer” at least in some examples refers to an entity that consumes one or more services. The term “service producer” or “producer” at least in some examples refers to an entity that offers, serves, or otherwise provides one or more services. The term “service provider” or “provider” at least in some examples refers to an organization or entity that provides one or more services to at least one service consumer. For purposes of the present disclosure, the terms “service provider” and “service producer” may be used interchangeably even though these terms may refer to difference concepts.

[0205] The term “information element” or “IE” at least in some examples refers to a structural element containing one or more fields. Additionally or alternatively, the term “information element” or “IE” at least in some examples refers to a field or set of fields defined in a standard or specification that is used to convey data and / or protocol information.

[0206] The term “reference” at least in some examples refers to data useable to locate other data and may be implemented a variety of ways (e.g., a pointer, an index, a handle, a key, an identifier, a hyperlink, and / or the like). The terms “configuration”, “policy”, “ruleset”, and / or “operational parameters”, at least in some examples refer to a machine -readable information object that contains instructions, conditions, parameters, criteria, data, metadata, and / or other information that is / are relevant to a component, device, system, network, service producer, service consumer, and / or other element / entity.

[0207] The term “data set” or “dataset” at least in some examples refers to a collection of data; a “data set” or “dataset” may be formed or arranged in any type of data structure. In some examples, one or more characteristics can define or influence the structure and / or properties of a dataset such as the number and types of attributes and / or variables, and various statistical measures (e.g., standard deviation, kurtosis, and / or the like). The term “data structure” at least in some examples refers to a data organization, management, and / or storage format. Additionally or alternatively, the term “data structure” at least in some examples refers to a collection of data values, the relationships among those data values, and / or the functions, operations, tasks, and the like, that can be applied to the data. Examples of data structures include primitives (e.g., Boolean, character, floating-point numbers, fixed-point numbers, integers, reference or pointers, enumerated type, and / or the like), composites (e.g., arrays, records, strings, union, tagged union, and / or the like), abstract data types (e.g., data container, list, tuple, associative array, map, dictionary, set (or dataset), multiset or bag, stack, queue, graph (e.g., tree, heap, and the like), and / or the like), routing table, symbol table, quad-edge, blockchain, purely-functional data structures (e.g., stack, queue, (multi) set, random access list, hash consing, zipper data structure, and / or the like).

[0208] The term “association” at least in some examples refers to a model of relationships between Managed Objects. Associations can be implemented in several ways, such as: name bindings, reference attributes, and association objects.

[0209] The term “Information Object Class” or “IOC” at least in some examples refers to a representation of the management aspect of a network resource. Additionally or alternatively, the term “Information Object Class” or “IOC” at least in some examples refers to a description of the information that can be passed / used in management interfaces. In some examples, their representations are technology agnostic software objects. Additionally or alternatively, an IOC has attributes that represents the various properties of the class of objects. Additionally or alternatively, IOC can support operations providing network management services invocable on demand for that class of objects. Additionally or alternatively, an IOC may support notifications that report event occurrences relevant for that class of objects. In some examples, an IOC is modelled using the stereotype "Class" in the UML meta-model.

[0210] The term “Managed Object” or “MO” at least in some examples refers to an instance of a Managed Object Class (MOC) representing the management aspects of a network resource. Its representation is a technology specific software object. In some examples, an MO is called an “MO instance” or “MOI”. Additionally or alternatively, the term “Managed Object” or “MO” at least in some examples refers to a class of technology specific software objects. In some examples, an MOC is the same as an IOC except that the former is defined in technology specific terms and the latter is defined in technology agnostic terms. MOCs are used / defined in SS level specifications. In some examples, IOCS and / or MOCs are used / defined in IS level specifications.

[0211] The term “Management Information Base” or “MIB” at least in some examples refers to an instance of an NRM and has some values on the defined attributes and associations specific for that instance. In some examples, an MIB includes a name space (describing the MO containment hierarchy in the MIB through Distinguished Names), a number of MOs with their attributes, and a number of associations between the MOs.

[0212] The term “name space” at least in some examples refers to a collection of names. In some examples, a name space is restricted to a hierarchical containment structure, including its simplest form - the one-level, flat name space. In some examples, all MOs in an MIB are included in the corresponding name space and the MIB / name space shall only support a strict hierarchical containment structure (with one root object). An MO that contains another is said to be the superior (parent); the contained MO is referred to as the subordinate (child). The parent of all MOs in a single name space is called a Local Root. The ultimate parent of all MOs of all managed systems is called the global root.

[0213] The term “network resource” at least in some examples refers to a discrete entity represented by an IOC for the purpose of network and service management. In some examples, a network resource may represent intelligence, information, hardware and / or software of a telecommunication network. The term “Network Resource Model” or “NRM” at least in some examples refers to a collection of IOCs, inclusive of their associations, attributes and operations, representing a set of network resources under management.

[0214] The term “self-organizing network” or “SON” at least in some examples refers to a type of network architecture or system that is designed to automate the planning, configuration, optimization, and / or healing processes of a wireless network with little or no direct human intervention (see e.g., 3GPP TS 32.500, 3GPP TS 32.522, 3GPP TS 32.541, 3GPP TS 32.551, 3GPP TS 28.310, 3GPP TS 28.313, 3GPP TS 28.627, and 3GPP TS 28.628).

[0215] The term “performance indicator” at least in some examples refers to performance data aggregated over a group of NFs that is derived from performance measurements collected at the NFs that belong to the group. In some examples, performance indicators are derived, collected or aggregated according to an aggregation method identified in a performance indicator definition.

[0216] The term “artificial intelligence” or “Al” at least in some examples refers to any intelligence demonstrated by machines, in contrast to the natural intelligence displayed by humans and other animals. Additionally or alternatively, the term “artificial intelligence” or “Al” at least in some examples refers to the study of “intelligent agents” and / or any device that perceives its environment and takes actions that maximize its chance of successfully achieving a goal.

[0217] The terms “artificial neural network”, “neural network”, or “NN” refer to an ML technique comprising a collection of connected artificial neurons or nodes that (loosely) model neurons in a biological brain that can transmit signals to other arterial neurons or nodes, where connections (or edges) between the artificial neurons or nodes are (loosely) modeled on synapses of a biological brain. The artificial neurons and edges typically have a weight that adjusts as learning proceeds. The weight increases or decreases the strength of the signal at a connection. Neurons may have a threshold such that a signal is sent only if the aggregate signal crosses that threshold. The artificial neurons can be aggregated or grouped into one or more layers where different layers may perform different transformations on their inputs. Signals travel from the first layer (the input layer), to the last layer (the output layer), possibly after traversing the layers multiple times. NNs are usually used for supervised learning, but can be used for unsupervised learning as well. Examples of NNs include deep NN (DNN), feed forward NN (FFN), deep FNN (DFF), convolutional NN (CNN), deep CNN (DCN), deconvolutional NN (DNN), a deep belief NN, a perception NN, recurrent NN (RNN) (e.g., including Long Short Term Memory (LSTM) algorithm, gated recurrent unit (GRU), echo state network (ESN), and the like), spiking NN (SNN), deep stacking network (DSN), Markov chain, perception NN, diffusion models (e.g., generative adversarial network (GAN), latent consistency models, state-of-the-art diffusion models (SDXL), stable diffusion models, adversarial diffusion distillation models (ADD), and / or the like), large language models (LLMs), transformers (e.g., Generative Pre-trained Transformers (GPT), Bidirectional Encoder Representations from Transformers (BERT), Robustly Optimized BERT Approach (RoBERTa), BigScience Large Open-science Open-access Multilingual Language Model (BLOOM), Text-to- Text Transfer Transformer (T5), XLNet, Conditional Transformer Language Model (CTRL), Perceiver, and the like), stochastic NNs (e.g., Bayesian Network (BN), Bayesian belief network (BBN), a Bayesian NN (BNN), Deep BNN (DBNN), Dynamic BN (DBN), probabilistic graphical model (PGM), Boltzmann machine, restricted Boltzmann machine (RBM), Hopfield network or Hopfield NN, convolutional deep belief network (CDBN), and the like), Linear Dynamical System (LDS), Switching LDS (SLDS), Optical NNs (ONNs), an NN for reinforcement learning (RL) and / or deep RL (DRL), attention and / or self-attention mechanisms, and / or the like.

[0218] The term “machine learning model” or “ML model” at least in some examples refers to an application, program, process, algorithm, and / or function that is capable of making predictions, inferences, or decisions based on an input data set and / or is capable of detecting patterns based on an input data set. Additionally or alternatively, the term “machine learning model” or “ML model” at least in some examples refers to a mathematical algorithm that can be "trained" by data (or otherwise learn from data) and / or human expert input as examples to replicate a decision an expert would make when provided that same information. In some examples, a “machine learning model” or “ML model” is trained on a training data to detect patterns and / or make predictions, inferences, and / or decisions. In some examples, a “machine learning model” or “ML model” is based on a mathematical and / or statistical model. For purposes of the present disclosure, the terms “ML model”, “Al model”, “AI / ML model”, and the like may be used interchangeably.

[0219] The term “machine learning entity” or “ML entity” at least in some examples refers to an entity that is either an ML model and / or contains an ML model and ML model-related metadata that can be managed as a single composite entity. In some examples, ML model-related metadata may include, for example, the applicable runtime context for the ML model. Additionally or alternatively, the term “machine learning entity” or “ML entity” at least in some examples refers to a manageable artifact of an ML mode. The term “Al decision entity”, “machine learning decision entity”, or “ML decision entity” at least in some examples refers to an entity that applies a non-AI and / or non-ML based logic for making decisions that can be managed as a single composite entity.

[0220] The term “machine learning training”, “ML training”, or “MLT” at least in some examples refers to capabilities and associated end-to-end (e2e) processes to enable an ML training function to perform ML model training (e.g., as defined herein). In some examples, ML training (or ML training capabilities) include interaction with other parties / entities to collect and / or format the data required for ML model training. Additionally or alternatively, the term “machine learning training”, “ML training”, or “MLT” at least in some examples refers to one or more e2e processes to enable an MLT function to perform ML model initial training or re-training.

[0221] The term “machine learning model training”, “ML model training”, or “MLT” at least in some examples refers to capabilities of an ML training function to take data, run the data through an ML model, derive associated loss, optimization, and / or objective / goal, and adjust the parameterization of the ML model based on the computed loss, optimization, and / or objective / goal. Additionally or alternatively, the term “machine learning model training” or “ML model training” at least in some examples refers to one or more processes performed by an ML training function to take training data, run it through an ML model, derive the associated loss and adjust the parameterization of that ML model based on the computed loss.

[0222] The term “machine learning initial training” or “ML initial training” at least in some examples refers to the ML model training that generates the initial version of an ML entity. The term “machine learning re-training” or “ML re-training” at least in some examples refers to the ML model training that generates a new version of a trained ML entity using the same type, but different values or distributions of training data used to train the model associated to the previous version of the ML entity. In some examples, a new version of a trained ML entity supports the same type of inference as the previous version of the ML entity, e.g., the data type of inference input and data type of inference output remain unchanged between the two versions of the ML entity.

[0223] The term “machine learning training function”, “ML training function”, or “MLT function” at least in some examples refers to a (logical) function (or set of (logical) functions) with MLT capabilities. The term “AI / ML inference function”, “ML inference function”, “MLIF”, or “MIF” at least in some examples refers to a (logical) function (or set of (logical) functions) that employs an ML model and / or Al decision entity to conduct inference. Additionally or alternatively, the term “AI / ML inference function” or “ML inference function” at least in some examples refers to an inference framework used to run a compiled model in the inference host. In some examples, an “AI / ML inference function” or “ML inference function” may also be referred to an “model inference engine”, “ML inference engine”, or “inference engine”.

[0224] The term “model parameter” and / or “parameter” in the context of ML, at least in some examples refer to values, characteristics, and / or properties that are learnt during training. Additionally or alternatively, the term “model parameter” and / or “parameter” at least in some examples refers to a configuration variable that is internal to an AI / ML model and whose value can be estimated from the given data. The term “hyperparameter” at least in some examples refers to characteristics, properties, and / or parameters for an ML process that are not generally learnt during ML model training. The term “tuning” or “tune” at least in some examples refers to a process of adjusting model parameters or hyperparameters of an ML model in order to improve its performance. Additionally or alternatively, the term “tuning” or “tune” at least in some examples refers to a optimizing an ML model’s model parameters and / or hyperparameters.

[0225] The term “objective function” at least in some examples refers to a function to be maximized or minimized for a specific optimization problem. In some examples, an objective function is defined by its decision variables and an objective, wherein the objective is the value, target, or goal to be optimized, and each decision variable is a variable that represents a decision to be made.

[0226] The term “optimization” at least in some examples refers to an act, process, or methodology of making something (e.g., a design, system, or decision) as fully perfect, functional, or effective as possible. In some examples, optimization includes mathematical procedures such as finding the maximum or minimum of a function.

[0227] The terms “regression algorithm” and / or “regression analysis” in the context of ML at least in some examples refers to a set of statistical processes for estimating the relationships between a dependent variable (often referred to as the “outcome variable”) and one or more independent variables (often referred to as “predictors”, “covariates”, or “features”). Examples of regression algorithms / models include logistic regression, linear regression, gradient descent (GD), stochastic GD (SGD), and the like.

[0228] The term “reinforcement learning” or “RL” at least in some examples refers to a goal- oriented learning technique based on interaction with an environment. In RL, an agent aims to optimize a long-term objective by interacting with the environment based on a trial and error process. Examples of RL algorithms include Markov decision process, Markov chain, multiarmed bandit learning, temporal difference learning, deep RL, and Q-leaming. Examples Q- leaming algorithms include Deep Q-Network (DQN), Double DQN, Dueling DQN, Q* (“Q star”), Production-Ready Reinforcement Learning Agent (PEARL), and / or the like.

[0229] The term “supervised learning” at least in some examples refers to an ML technique that aims to learn a function or generate an ML model that produces an output given a labeled data set. The term “unsupervised learning” at least in some examples refers to an ML technique that aims to learn a function to describe a hidden structure from unlabeled data and / or builds / generates models from a set of data that contains only inputs and no desired output labels. Examples of unsupervised learning approaches / methods include K-means clustering, hierarchical clustering, mixture models, density-based spatial clustering of applications with noise (DBSCAN), ordering points to identify the clustering structure (OPTICS), anomaly detection methods (e.g., local outlier factor, isolation forest, and / or the like), expectation-maximization algorithm (EM), method of moments, topic modeling, and blind signal separation techniques (e.g., principal component analysis (PCA), independent component analysis, non-negative matrix factorization, singular value decomposition). In some examples, unsupervised training methods include backpropagation, Hopfield learning rule, Boltzmann learning rule, contrastive divergence, wake sleep, variational inference, maximum likelihood, maximum a posteriori, Gibbs sampling, backpropagating reconstruction errors, and hidden state reparameterizations. The term ’’semisupervised learning at least in some examples refers to ML algorithms that develop ML models from incomplete training data, where a portion of the sample input does not include labels.

[0230] Although many of the examples discussed herein are provided with use of specific cellular / mobile network terminology, including with the use of 4G / 5G 3GPP network components (or expected terahertz -based 6G / 6G+ technologies), these examples may be applied to many other deployments of wide area and local wireless networks, as well as the integration of wired networks (including optical networks and associated fibers, transceivers, and / or the like). Purthermore, various standards (e.g, 3GPP, ETSI, IEEE, and / or the like) may define various message formats, PDUs, MAC CEs, containers, frames, and / or other data structures, as comprising a sequence of optional or mandatory containers, frames, data elements (DEs), data frames (DFs), information elements (IES), information object classes (IOCS), managed object classes (MOCs), paramters, attributes, and / or other elements. However, the requirements of any particular standard should not limit the examples discussed herein, and as such, any combination of containers, frames, DFs, DEs, IEs, IOCs, MOCs, parameters, attributes, values, actions, features, and / or other elements are possible in various examples, including any combination of containers, frames, DFs, DEs, IEs, IOCs, MOCs, parameters, attributes, values, actions, features, and / or other elements that are strictly required to be followed in order to conform to such standards or any combination of containers, frames, DFs, DEs, IEs, IOCs, MOCs, parameters, attributes, values, actions, features, and / or other elements strongly recommended and / or used with or in the presence / absence of optional elements.

[0231] Moreover, the present disclosure provides various examples of names / labels for various systems, sub-systems, devices, planes, layers, protocols, components, operations, containers, frames, DFs, DEs, IEs, IOCs, MOCs, parameters, attributes, values, actions, features, and other elements / data structures. However, the specific names or labels used regarding the various systems, sub-systems, devices, planes, layers, components, operations, parameters, attributes, IEs, IOCs, MOCs, and other elements / data structures, are provided for the purpose of discussion and illustration, rather than limitation. The various systems, sub-systems, devices, planes, layers, components, operations, parameters, attributes, IEs, IOCs, MOCs, and other elements / data structures can have alternative names or labels to those provided herein. Furthermore, additional or alternative embodiments, implementations, and / or iterations of 3GPP specifications and / or other relevant standards / specifications may name certain elements / entities different to those discussed herein, but still fall within the context of the present disclosure.

[0232] Aspects of the inventive subject matter may be referred to herein, individually and / or collectively, merely for convenience and without intending to voluntarily limit the scope of this application to any single aspect or inventive concept. Although specific aspects have been shown and described herein, the present disclosure covers any and all adaptations or variations and any arrangement capable of achieving the same purpose may be substituted for the specific aspects shown and described herein. Combinations of the described aspects and other aspects not specifically described herein will be apparent to those of skill in the art upon reviewing the present disclosure.

Claims

CLAIMS1. A method of operating a service producer for machine learning training (MLT), the method comprising: receiving, from a service consumer, a request to configure a policy for controlling MLT of a machine learning (ML) model; collecting data based on the configured policy; and controlling the MLT of the ML model according to the configured policy and based on the collected data.

2. The method of claim 1, wherein the triggering includes: sending, to the service consumer, an indication of whether configuration of the policy was successful.

3. The method of claims 1-2, wherein the request to configure the policy is a request to create the policy; a request to modify one or more conditions, parameters, or attributes of the policy; or a request to delete the policy.

4. The method of claims 1-3, wherein the controlling includes: triggering the MLT based on occurrence of one or more conditions specified by the configured policy.

5. The method of claims 1-4, wherein the controlling includes: stopping or pausing the MLT based on occurrence of one or more conditions specified by the configured policy.

6. The method of claim 5, wherein the one or more conditions include one or more performance metric (PM) thresholds related to inference performance.

7. The method of claim 6, wherein the one or more performance metric thresholds include one or more of: one or more PM values corresponding to a desired inference performance or a range of acceptable or unacceptable PM values for one or more PMs.

8. The method of claims 4-7, wherein the one or more conditions include one or more network conditions under which the MLT is to be performed.

9. The method of claim 8, wherein the one or more network conditions include one or more of: a number of active user equipment (UEs), a number of UEs with one or more desired capabilities, a number of changes of neighbor cells, a number of handovers, a number of radio link failure (RLF) reports, a number of RRC connection establishment failure event (RCEF) reports, signal strength measurement thresholds, or signal quality measurement thresholds.

10. The method of claims 4-9, wherein the one or more conditions include one or more platform conditions.

11. The method of claim 10, wherein the one or more platform conditions include one or more of: a number of nodes or clusters used for the MLT, an amount of compute resources consumed for the MLT, or environmental conditions related to one or more platforms performing the MLT.

12. The method of claims 4-11, wherein the one or more conditions include one or more time conditions.

13. The method of claim 12, wherein the one or more time conditions include occurrence of one or more time periods or occurrence of one or more time intervals.

14. The method of claims 1-13, wherein the policy is defined by a data type for an attribute of a Managed Object Instance (MOI) representing an MLT function.

15. The method of claims 1-13, wherein the policy is defined by an abstract class inherited by the MOI representing an MLT function.

16. The method of claims 1-15, wherein the method includes: receiving an activation / deactivation request from the service consumer to activate or deactivate the MLT function; sending, to the consumer, an indication indicating whether the activation / deactivation request is accepted; and activating or deactivating the MLT function according to the activation / deactivation request.

17. The method of claim 16, wherein the activation / deactivation request is a request to activate or deactivate the MLT function in response to receipt of the activation / deactivation request.

18. The method of claim 16, wherein the activation / deactivation request is a request to activate or deactivate the MLT function according to a schedule.

19. The method of claims 14-18, wherein the MLT function is a machine learning training function (MLTF) contained by a Network Data Analytics Function (NWDAF).

20. The method of claims 14-19, wherein the MLT function is the service producer.

21. The method of claims 1-20, wherein the service producer is an MLT management services (MnS) producer and the service consumer is an MLT MnS consumer.

22. One or more computer readable media comprising instructions, wherein execution of the instructions by processor circuitry is to cause the processor circuitry to perform the method of claims 1-21.

23. A computer program comprising the instructions of claim 22.

24. An electromagnetic signal carrying the instructions of claim 22.

25. An apparatus comprising means for performing the method of claims 1-21.

Images