Controlling inference on a communications network

Abstract

A method of controlling a neural network model performing an inference action in a communications network wherein the method includes: receiving a set of values from a hidden layer of the neural netwo

Description

FIELD

[01] The subject-matter of the present disclosure relates to methods of controlling a neural network model performing an inference action in a communications network. BACKGROUND

[02] As we move towards the development of 6G networks, there is a growing need for efficient and fast machine-learning models that can be deployed on resource-constrained devices and in environments with limited network bandwidth or dynamic connectivity conditions. Deep neural networks have shown remarkable performance in various machine learning tasks, but their deployment on 6G networks presents a significant challenge given their requirement for significant computational resources and that are often deployed on resource-constrained devices or in environments with limited or varying network bandwidth. In such scenarios, the latency and energy consumption of these models can be prohibitive. One approach to address this challenge is to use early exit techniques.

[03] Early exit is a technique that allows a neural network to make a prediction before it has fully processed the input. This technique can significantly reduce the inference time and energy consumption of deep neural networks. When a network has multiple layers, early exit can be implemented by adding decision points at various layers, allowing the network to output a prediction when it reaches a certain confidence threshold. For example, consider a neural network that classifies images into different categories. Instead of running the entire network for every image, we can split the network into several smaller sub-networks, each of which makes a prediction about the image. These sub-networks can have different levels of complexity, with simpler sub-networks making less accurate predictions than more complex ones.

[04] After each sub-network makes its prediction, we can evaluate the accuracy of the prediction and decide whether to continue with the next sub-network or stop the inference process altogether. If the accuracy of the intermediate prediction is sufficient for the problem needs, we can stop the inference process early, saving time and computational resources.

[05] The advantage of early exit is that it allows us to optimize the inference process for a given accuracy requirement. This allows us to trade off computation time and resource usage for accuracy. This can be particularly useful in scenarios where resources are limited, such as on embedded devices (loTs) or in real-time applications (XR, VR, V2X).

[06] Early exit has gained significant attention in recent years due to the need to deploy deep neural networks on resource-constrained devices and in environments with limited network bandwidth. This technique can significantly reduce the inference time and energy consumption of deep neural networks, making them more practical for real-world applications.

[07] The 3rd Generation Partnership Project (3GPP) is currently investigating the integration of machine learning models into telecommunications networks. As we look towards the development of 6G technology, significant improvements are expected in data rates, latency, capacity, and reliability. This integration of machine learning is anticipated to enable new and innovative applications in areas such as Al, loT, and edge computing. Although still in the research and development phase, 6G holds great promise for enhancing the capabilities of telecommunications networks and driving advancements in the field.

[08] It is an aim of the present invention to address such problems and improve on the prior art. SUMMARY

[09] The disclosure relates to accelerating machine learning, ML, (e.g. neural network) model inference over wireless networks using early exit points.

[10] According to an aspect of the present disclosure, there is provided a method of controlling a neural network model performing an inference action in a communications network, the method iteratively comprising: receiving a set of values from a hidden layer of the neural network model; calculating a confidence metric associated with the set of received values; comparing the confidence metric to an confidence threshold; and controlling the neural network model to: exit inference when confidence metric is greater than or equal to the confidence threshold, and continue to a subsequent layer of the neural network model when the confidence metric is less than the confidence threshold; receiving a condition associated with the inference action; and dynamically adjusting the confidence threshold based on the received condition.

[11] Advantageously, dynamically adjusting the confidence thresholds improves inference performance where conditions permit, and improves efficiency of inference when conditions do not permit. In addition, when conditions do not permit full inference, e.g. latency requirements are too stringent, a risk of inference failure is reduced.

[12] In an embodiment, the method may be a computer-implemented method.

[13] In an embodiment, controlling the neural network model to exit inference comprises: bypassing any remaining hidden layers of the neural network model; redirecting the received set of values from the hidden layer to an output, fully-connected, layer of the neural network model; and outputting and output from the output, fully-connected, layer of the neural network model.

[14] In an embodiment, the condition is a condition selected from a list of conditions including: a condition of the communications network, a configuration of the communications network, a constraint of user equipment operating on the communications network, a capability of user equipment operating on the communications network, and a key performance indicator of the neural network model.

[15] In an embodiment, the method further comprises: deactivating the neural network model based on the received condition and / or the comparison.

[16] In an embodiment, the method further comprises: activating the neural network model based on the received condition and / or the comparison.

[17] In an embodiment, the method further comprises: controlling the neural network model to stop inference based on the received condition and / or the comparison.

[18] In an embodiment, controlling the neural network model to stop inference based on the received condition and / or the comparison is based on a specific criterion.

[19] In an embodiment, the specific criterion may be selected from a list of criterion including persistent unfavourable network condition, predefined thresholds not being met for a set duration, or repeated failure to meet confidence thresholds.

[20] In an embodiment, the method of any preceding claim, performed by an adaptive inference framework model.

[21] In an embodiment, wherein the adaptive inference framework model is configured to adjust its behaviour based on real-time analysis of network conditions, device performance metrics, and model inference key performance indicators (KPIs), such as accuracy, inference latency, etc.

[22] In an embodiment, the adaptive inference framework model is one of a rules-based model (e.g. a set of hard coded rules, or a set of heuristics), a machine learning model, and a reinforcement learning model.

[23] In an embodiment, calculating the confidence metric comprises calculating a cross-entropy loss associated with the set of values from the hidden layer, wherein the cross-entropy loss is the confidence metric.

[24] In an embodiment, calculating the confidence metric comprises calculating an L2 loss associated with the set of values, wherein the L2 loss is the confidence metric.

[25] According to an aspect of the present disclosure, there is provided a transitory, or non-transitory, computer readable medium having instructions stored thereon that when executed by at least one processor, cause the at least one processor to perform the method of any preceding aspect or embodiment.

[26] According to an aspect of the present disclosure, there is provided an adaptive inference framework model for controlling a neural network model performing an inference action in a communications network, comprising: an input configured to receive a set of values from a hidden layer of the neural network model; a processor configured to: calculate a confidence metric associated with the set of received values, compare the confidence metric to a confidence threshold, and generate a command to control the neural network model to: exit inference when confidence metric is greater than or equal to the confidence threshold, and continue to a subsequent layer of the neural network model when the confidence metric is less than the confidence threshold, receive a condition associated with the inference action; and dynamically adjust the confidence threshold based on the received condition; and an output configured to transmit the command to the neural network model.

[27] Applying Early-Exit mechanism to 6G can contribute to the goals of this nextgeneration technology in several ways: 1. Extreme data rates and low latency: One of the key objectives of 6G is to achieve significantly higher data rates and lower latencies compared to 5G. Early-Exit Networks can help in this regard by providing faster and more efficient Al model inference, reducing the time required for processing Al tasks and minimizing the latency in Al-driven applications. 2. Al-driven networks: 6G envisions a shift towards Al-driven network management and orchestration, which can benefit from the adaptive nature of Early-Exit Networks. By adjusting the computational complexity depending on the input data, Early-Exit Networks can optimize resource allocation in Al-driven network management, ensuring efficient utilization of available resources and improved network performance. 3. Massive loT and edge computing: 6G aims to support a massive number of connected devices, with diverse requirements in terms of latency, power consumption, and data processing. Early-Exit Networks can be particularly useful in this context by providing energy-efficient and adaptive Al models that can run on resource-constrained devices, facilitating loT and edge computing deployments. 4. Enhanced security and privacy: As 6G networks will be increasingly reliant on Al-driven applications and services, ensuring security and privacy is paramount. Early-Exit Networks can potentially contribute to this area by enabling faster and more efficient processing of security-related Al tasks, such as anomaly detection or intrusion prevention, helping to maintain a secure and reliable network. 5. Holographic and immersive experiences: 6G aims to enable advanced applications like holographic communication and virtual reality, which demand high data rates, low latency, and reliable performance. By providing faster and more efficient Al model inference, Early-Exit Networks can help facilitate these applications by reducing the time required for Al-driven image and video processing tasks, ensuring a seamless and immersive user experience.

[28] In summary, while 6G is still in the research and development stage, Early-Exit Networks can potentially contribute to its goals by providing faster, more efficient, and more adaptive Al models. These characteristics align well with the objectives of 6G, such as extreme data rates, low latency, Al-driven networks, massive loT, and edge computing, and can help enhance the overall performance of future 6G networks and applications. BRIEF DESCRIPTION OF DRAWINGS

[29] The subject-matter of the present disclosure is best described with reference to the accompanying figures, in which:

[30] Figure 1 shows a block diagram of a communications network on which methods of one or more embodiments are implemented;

[31] Figure 2 shows a block diagram of a computer used to perform the methods associated to the one or more embodiments; and

[32] Figure 3 shows a workload flowchart of Creation / Execution of an early exit, machine learning, framework, according to at least one embodiment. DESCRIPTION OF EMBODIMENTS

[33] As part of the 3GPP RAN1 study on AI / ML for NR air-interface in (RP-221348), RAN1 will be looking into aspects such as performance, complexity, and potential specification impact of using AI / ML for the selected use cases of CSI feedback enhancement, Beam management, and Positioning accuracy.

[34] More specifically, RAN1 has been investigating the following: Characterize the defining stages of AI / ML related algorithms and associated complexity: o Model generation, e.g., model training (including input / output, pre- / post-process, online / offline as applicable), model validation, model testing, as applicable o Inference operation, e.g., input / output, pre- / post-process, as applicable Characterize lifecycle management of AI / ML model: e.g., model training, model deployment, model inference, model monitoring, model updating Dataset(s) for training, validation, testing, and inference For the use cases under consideration: 1) Evaluate performance benefits of AI / ML based algorithms for the agreed use cases in the final representative set: o KPIs: Determine the common KPIs and corresponding requirements for the AI / ML operations. Determine the use-case specific KPIs and benchmarks of the selected use-cases. ■ Performance, inference latency and computational complexity of AI / ML based algorithms should be compared to that of a state-of-the-art baseline ■ Overhead, power consumption (including computational), memory storage, and hardware requirements (including for given processing delays) associated with enabling respective AI / ML scheme, as well as generalization capability should be considered.

[35] Regarding the model inference, RAN1 has discussed several aspects related to the benefit(s) and requirement(s) / specification impact(s) of AI / ML model inference. For example, for the use case of positioning accuracy enhancement, considering UE-side and / or Network-side model inference. RANl#110-e meeting (August 22nd - 26th, 2022): Agreement Study aspects in terms of potential benefit(s) and requirementfs) / specification impact(s) of AI / ML model training and inference in AI / ML for positioning accuracy enhancement considering at least • UE-side or Network-side inference o Note: model inference at both UE and network side is not precluded where proponent(s) are encouraged to clarify their AI / ML approaches

[36] Moreover, RAN1 has discussed the impact of AI / ML inference complexity and inference latency on the performance benefits of using AI / ML: RANl#110b-e meeting (October 10th - 19th, 2022) Agreement: The following are additionally considered for the initial list of common KPIs (if applicable) for evaluating performance benefits ofAI / ML • Clarification on inference complexity o Note: Inference complexity includes complexity for pre- and postprocessing. • FFS: Power consumption, latency (e.g., Inference latency) Moreover, RANI agreed that inference latency, as a model capability, will have specification impact, for example, for the use case of AI / ML based positioning accuracy. Agreement Regarding AI / ML model indication [ / configuration], to study and provide inputs on potential specification impact at least for the following aspects on conditions / criteria of AI / ML model for A! ML based positioning accuracy enhancement Model capability, e.g., positioning accuracy quality and model inference latency.

[37] Early exit (EE) is an effective technique that can improve the efficiency, responsiveness, and adaptability of machine learning models deployed in wireless networks. By allowing machine learning models to terminate their computations sooner when the required accuracy level is achieved. Early exit can reduce the computational complexity and energy consumption of the models, while improving their latency and reliability in real-time applications.

[38] There are several reasons why early exit solutions should be adopted in telecommunications networks. First, early exit can enable the deployment of more complex and sophisticated machine learning models that would otherwise be computationally infeasible on mobile devices or network edge servers. By reducing the computational complexity and energy consumption of these models, early exit can expand the range of applications and use cases that can be supported in wireless networks, including those requiring real-time decision-making such as autonomous driving, telemedicine, and gaming.

[39] Second, early exit can improve the scalability and resource allocation of machine learning models in radio access networks (RAN). By allowing models to adapt their computations to the available resources and network conditions, early exit can ensure that the models remain responsive and adaptable to changing user needs and network environments. This can also reduce the latency and bandwidth requirements for machine learning applications, making them more accessible and cost-effective for end-users.

[40] Third, early exit can improve the energy efficiency and sustainability of machine learning models in telecommunications networks. By reducing the computational complexity and energy consumption of the models, early exit can help to mitigate the environmental impact of machine learning applications, while also reducing the operating costs for service providers and end-users.

[41] The primary goal of this invention is to develop a network function that integrates adaptive inference with early-exit machine learning frameworks, enabling efficient and optimized utilization of computational and communication resources across various networking environments. This network function should provide faster inference times, reduced latency, improved resource allocation, and network-level optimization while maintaining the accuracy and performance of the machine-learning models. Moreover, it should be scalable and adaptable to handle a wide range of tasks, data distributions, and network conditions.

[42] The invention can be summarized as follows, where this is meant to be a summary but not a restriction or limitation of the proposed solutions - as such the summary itself is to be considered as a set of solutions:

[43] A new (and / or existing) network function (NF), and / or logical functions, with a system for adaptive inference with early-exit ML frameworks, comprising, for example, one or more of: a. a network communication module configured to receive input data from network nodes and / or devices and transmit output predictions and / or intermediate results; b. an early-exit ML framework, including multiple exit points and corresponding exit decision metrics, enabling adaptive inference based on the confidence of predictions at each exit point; c. a confidence threshold determination module configured to calculate and dynamically adjust confidence thresholds for each exit point, considering factors such as network conditions, device constraints and / or conditions, and application requirements. In other words, the condition associated with the inference action may be any of those factors, which may alternatively, and / or additionally, described as a condition of a communications network, a configuration of the communications network, a constraint of user equipment operating on the communications network (and involved in the inference task), capability of user equipment operating on the communications network (and involved in the inference task), and a key performance indicator (KPI) such as a KPI of the neural network model.

[44] Condition of the Communications Network: If network latency is high, it may be beneficial to exit early to avoid delays, so we can set lower confidence thresholds. On the contrary, if we know that latency is low, the confidence threshold can be increased to exit with higher confidence values.

[45] Configuration of the Communications Network: In a network with limited bandwidth, early exit can reduce the amount of data transmitted, thus conserving bandwidth and speeding up the overall process. Again, if we have limited bandwidth the confidence thresholds can be lowered.

[46] User Equipment Constraint: If the user device (e.g., a smartphone) has limited battery life, early exit helps conserve power by reducing the computational load.

[47] User Equipment Capability: A device with limited processing power may benefit from early exits to prevent overload and ensure smooth operation.

[48] Key Performance Indicator (KPI) of the Neural Network Model: If a KPI such as accuracy, latency, or throughput is being optimized, early exit decisions might be based on balancing these metrics. For instance, if the application is used for self-driving cars, which require high level of confidence on the prediction, the confidence levels might remain high, even though the other conditions might suggest otherwise to meet this KPI.

[49] There can be conditions that are limiting and can stop the overall inference, for example, at the runtime, one can realize that the network bandwidth is low and that the UE capability is limited, meaning that it won’t be able to obtain the inference without violating the latency constraint. In such scenarios, one can stop and kill the overall inference process.

[50] The method may comprise controlling the neural network model to stop inference based on the received condition and / or the comparison. Controlling the neural network model to stop inference based on the received condition and / or the comparison is based on a specific criterion. In an embodiment, the specific criterion may be selected from a list of criterion including persistent unfavourable network condition, predefined thresholds not being met for a set duration, or repeated failure to meet confidence thresholds.

[51] Other actions that may be implemented may be activating and deactivating the neural network mode. Activating and deactivating the neural network model may involve making the neural network model enabled so that it can perform inference or the neural network model may be disabled to prevent inference from being performed.

[52] The network communication module includes data pre-processing and encoding functionalities to convert raw input data into a format suitable for the early-exit machine learning framework.

[53] We define a set of confidence metrics to be used for the early-exit machine learning framework, such as, softmax probability or entropy-based measures. These metrics will be used to evaluate prediction confidence at each exit point to decide whether to exit a current inference execution. Those metrics may also be termed L2 loss associated with a set of values output from a hidden layer of the neural network model and cross-entropy loss associated with the set of values output from the hidden layer.

[54] The confidence threshold determination module employs ML and / or reinforcement learning (RL) and / or data-driven techniques and / or heuristics to dynamically adjust confidence thresholds based on current network and device conditions, device constraints, model application, and device performance data, among other possible metrics.

[55] The method for adaptive inference leveraging early-exit machine learning proposed in this invention comprises the steps of (but not limited to those steps, or modification of those steps, or additional steps, or a modified sequence of those steps or performing steps in parallel, etc.): a. receiving input data from network nodes and / or devices; for example, monitoring network conditions, device conditions, edge server conditions, etc. b. processing the input data through an early-exit machine learning framework with multiple exit points and corresponding classifiers and / or exit decision metrics; c. evaluating the confidence of predictions at each exit point and comparing it to dynamically adjusted confidence thresholds; d. exiting the inference process and providing output predictions and / or intermediate results when the confidence at an exit point exceeds the corresponding threshold; e. optimizing the allocation of computational and communication resources for the early-exit machine learning framework.

[56] In other words, the steps may include receiving a set of values from a hidden layer of the neural network model; calculating a confidence metric associated with the set of received values; comparing the confidence metric to an confidence threshold; and controlling the neural network model to: exit inference when confidence metric is greater than or equal to the confidence threshold, and continue to a subsequent layer of the neural network model when the confidence metric is less than the confidence threshold; receiving a condition associated with the inference action; and dynamically adjusting the confidence threshold based on the received condition.

[57] The set of values may be output values from the nodes within the hidden layer of the neural network model. Since the neural network model is a deep learning neural network, it may be considered to have a plurality of such hidden layers. Those layers may be connected in series, in other words, outputs, or values, or a first hidden layer may be used as inputs to the second hidden layer, outputs, or values, of the second hidden layer may be used as inputs to the third hidden layer, and so forth until the final, fully-connected, layer. In other embodiments, there may be skip connections so values from some nodes of some hidden layers may skip one or more subsequent layers. In any case, the set of values described above may be current values in the sense that they correspond to the latest set of values that have been calculated.

[58] In other words, controlling the neural network model to exit inference comprises: bypassing any remaining hidden layers of the neural network model; redirecting the received set of values from the hidden layer to an output, fully-connected, layer of the neural network model; and outputting and output from the output, fully-connected, layer of the neural network model.

[59] It will be appreciated that the methods described herein are thus performed by an adaptive inference framework model. The adaptive inference framework model is configured to adjust its behaviour based on real-time analysis of network conditions, device performance metrics, and model inference key performance indicators (KPIs), such as accuracy, inference latency, etc.

[60] As indicated above, the adaptive inference framework model is one of a rules-based model, a machine learning model, and a reinforcement learning model.

[61] In cases where the confidence at an exit point is below or equal to the dynamically adjusted threshold, the inference process will continue to execute through subsequent layers until it reaches the final layer of the machine learning model for a more comprehensive prediction.

[62] The method for adaptive inference leveraging early-exit machine learning proposed in this invention maybe terminated (or stopped or switched (back) to a normal inference learning process), based on an indication, other than the conditions (metrics, thresholds, parameters, KPIs, etc., disclosed herein) from at least another newly defined (and / or exiting) network entity (and / or network function), for example, in the case that early-exit learning process not exceeding (or achieving) the dynamically defined confidence threshold, in another example, not exceeding (or achieving) the threshold for a given time period (or a defined early-exit timer), and / or not exceeding (or achieving) the threshold consistently (e.g. for a given number of instances or occurrences, etc.).

[63] In this section we provide a detailed description of the invention, we will explore the components, operation, and implementation of the new (or existing) network function for adaptive inference with early-exit ML frameworks. The primary aim of this invention is to optimize computational and communication resources in various networking environments.

[64] With reference to Figure 1, present embodiments relate to a method of controlling a neural network model performing an inference action in a communications network 10. The communications network includes a plurality of nodes 12. The nodes may include user equipment, UE, cellular base stations, and the like. The method of controlling a neural network model performing an inference action in the communications network 10 may be a method of controlling a neural network model performing an inference action on data transmitted between a first node and a second node. The inference action may take place using the neural network model stored in storage at either the first node or the second node, or may be stored on storage between the first and second node at an intermediate node.

[65] With reference to Figure 2, the method may be a computer-implemented method. In other words, the method may be implemented on a computer 20. The computer 20 includes at least one processor 22, and storage 24. The storage includes instructions stored as electronic data that when executed by the at least one processor 22, causes the at least one processor 22 to perform the methods described herein. The storage may be considered non-transitory computer-readable media. When the instructions are provided as a signal for loading onto the storage, the signal may be considered transitory computer-readable media.

[66] To this end we define the following modules:

[67] Network Communication module:

[68] The network communication module serves as the primary interface between the early-exit ML framework and the rest of the network, ensuring seamless communication and data exchange. This module is responsible for receiving input data from network nodes or devices, transmitting the output predictions or intermediate results, and handling data-related operations. Here, we will elaborate on the key aspects of the network communication module: 1. Data reception and transmission: The network communication module receives input data from various sources, such as network nodes, devices, or sensors, and transmits the output predictions or intermediate results back to the appropriate recipients. 2. Data pre-processing: Before the input data can be processed by the early-exit machine learning framework, it may require pre-processing to ensure it is in a suitable format. The network communication module can include data pre-processing functionalities, such as data normalization, feature extraction, or data augmentation, to convert raw input data into a format compatible with the ML framework. 3. Data buffering and prioritization: The network communication module can implement data buffering and prioritization mechanisms to handle varying data loads, network congestion, or latency requirements. For instance, the module can buffer incoming data temporarily if the early-exit ML framework is not ready to process it or prioritize processing data with higher importance or stricter latency requirements. 4. Security and privacy: The network communication module should ensure the security and privacy of the data being transmitted, especially in sensitive applications or regulated environments. This can involve implementing encryption and decryption methods, secure authentication, and access control mechanisms to protect the data during transmission and storage.

[69] By handling these aspects, the network communication module ensures that input data is appropriately processed and transmitted, allowing the ML framework to focus on adaptive inference and resource optimization.

[70] Early-exit ML framework module:

[71] The early-exit ML framework is the main component of the network(s) function(s) for adaptive inference. It employs any AI / ML model with multiple predefined exit points strategically placed after specific layers or blocks within the network. This design enables the framework to adaptively process input data and provide predictions based on the complexity of the data, resulting in more efficient and faster inference. Here, we will elaborate on the key aspects of the early-exit ML framework: 1. Multiple exit points: The early-exit ML framework incorporates multiple exit points throughout its architecture, with each exit point associated with one or more distinct confidence prediction strategies. These exit points are strategically placed, for example, after specific layers or blocks in a neural network, allowing the framework to obtain predictions at different levels of processing complexity. This design enables the model to adapt its computational complexity based on input data, balancing accuracy and efficiency. 2. Prediction confidence obtainment: At each exit point, the early-exit ML framework evaluates the confidence of the predictions made. This evaluation can be performed using one or multiple confidence metrics. For example, for classifications solutions, softmax probability or entropy-based measures, could quantify the certainty of the predicted class. These confidence metrics are listed below. The framework uses this confidence information to determine whether the prediction at the current exit point is sufficiently reliable or if additional processing is needed. 3. Adaptive inference process: The adaptive inference process in the early-exit ML framework leverages the confidence thresholds determined by the confidence threshold determination module. If the prediction confidence at a specific exit point exceeds the threshold, the framework stops processing and returns the prediction. Otherwise, the input data continues to be processed through the remaining layers of the neural network. This adaptive mechanism enables the framework to efficiently process input data, providing faster inference for simpler data instances while maintaining accuracy for more complex instances. 4. Training and fine-tuning: Training the early-exit ML framework involves optimizing the AI / ML model and each exit point confidence prediction, to provide accurate predictions. The training process can be performed using techniques such as multi-objective optimization, where the objectives include minimizing the overall loss across all exit points and balancing the trade-off between accuracy and computational efficiency. Additionally, the framework can be fine-tuned to adapt to specific application requirements, network environments, or device constraints.

[72] In summary, the early-exit ML framework is a crucial component of the network function for adaptive inference, enabling efficient and optimized utilization of computational resources by adapting the inference process based on input data complexity and confidence thresholds. This design allows the framework to provide faster inference times, reduced latency, and improved resource allocation, making it well-suited for various networking environments and applications with heterogeneous requirements.

[73] Confidence metrics

[74] Confidence metrics are quantitative measures that indicate the level of certainty or reliability associated with a prediction made by a machine learning model. They provide insights into how trustworthy a prediction is, helping to make informed decisions about whether to accept the prediction as is or continue processing to potentially improve the prediction quality. Following we list some of the most used confidence metrics: 1. Probability estimates: In classification tasks, probability estimates represent the model's estimated likelihood that a given input belongs to a particular class. The maximum probability value across all classes can serve as the confidence metric for the current prediction. The probability can be computed using, for example a softmax function. 2. Margin: The margin is calculated as the difference between the probability estimates of the top two most likely classes. A larger margin indicates higher confidence in the prediction, as the model clearly distinguishes between the top two classes. 3. Entropy: Entropy measures the uncertainty in the probability distribution of the model's predictions. Lower entropy values correspond to higher confidence, indicating that the model assigns high probability to a specific class and low probability to other classes. 4. Bayesian uncertainty: Bayesian uncertainty measures the model's uncertainty in its predictions by considering the distribution of its weights or parameters. Models that provide a measure of Bayesian uncertainty, such as Bayesian neural networks, can use this metric as a confidence indicator. Lower Bayesian uncertainty values indicate higher confidence in the predictions. 5. Ensemble agreement: Ensemble agreement can be used as a confidence metric when multiple models or sub-models are employed in an ensemble. The agreement between the predictions of these models or sub-models serves as a confidence indicator, with a higher level of agreement among the ensemble members implying higher confidence in the predictions. 6. Conformal prediction intervals: Conformal prediction is a technique that provides confidence intervals for the predictions made by a model. These intervals can be used as a confidence metric, where narrower intervals indicate higher confidence in the predictions.

[75] These confidence metrics can be employed in various machine learning applications and frameworks, including early-exit frameworks, to assess the reliability of predictions and make informed decisions about whether to accept a prediction or continue processing for potential improvement.

[76] Confidence Threshold Determination module:

[77] The confidence threshold determination module is responsible for dynamically calculating and updating the confidence thresholds for each exit point in the early-exit machine learning framework. These thresholds are crucial in determining whether the prediction at a given exit point is sufficiently reliable or if additional processing is needed. The module takes into account various factors, such as network and device conditions, device constraints, and application requirements, to determine the optimal threshold values. Here, we detail three categories of KPIs (network, device, and model) and provide examples of KPIs for each category:

[78] Network KPIs: 1. Network latency: Network latency represents the time it takes for data to travel from one point to another within the network. High latency can affect the real-time performance of the early-exit ML framework, especially in applications with strict latency requirements. The confidence threshold module can use this KPI to adapt the thresholds, prioritizing faster inference by allowing early exits at lower thresholds when latency is high. 2. Bandwidth utilization: Bandwidth utilization indicates the proportion of available network bandwidth currently being used. High bandwidth utilization can lead to network congestion, potentially affecting the early-exit ML framework's performance. The module can take this KPI into account to adapt the thresholds, potentially lowering them during periods of high bandwidth utilization to reduce the data transmission load. 3. Packet loss rate: Packet loss rate measures the percentage of data packets that are lost during transmission in the network. High packet loss rates can impact the quality and reliability of the data being processed by the early-exit ML framework. The confidence threshold module can use this KPI to adjust the thresholds, potentially increasing them to ensure that only high-confidence predictions are transmitted when packet loss is high. 4. Network jitter: Network jitter represents the variation in latency experienced by data packets during transmission. High jitter can lead to unpredictable performance of the early-exit ML framework. The module can consider this KPI when determining confidence thresholds, potentially adopting more conservative thresholds to account for the uncertainty introduced by jitter. 5. Network throughput: Network throughput is the rate at which data is successfully transmitted through the network. High throughput is desirable, as it indicates that the network can handle large amounts of data efficiently. The confidence threshold module can use this KPI to inform its decisions, potentially allowing for higher thresholds (and thus, more processing) when the network throughput is high.

[79] Device KPIs: 1. Processing power utilization: Processing power utilization measures the proportion of the device's available processing capabilities (CPU or GPU) currently being used. High processing power utilization can indicate that the device is under heavy workload, potentially affecting the early-exit ML framework performance. The module can adapt the thresholds based on this KPI, potentially lowering them to prioritize faster inference. This will reduce processing load during high utilization periods. 2. Memory usage: Memory usage refers to the proportion of available memory (RAM) currently being used. High memory usage can lead to performance issues, as the device may struggle to allocate memory for additional tasks or processes. The confidence threshold module can take this KPI into account when determining thresholds. This can help reduce the memory footprint of the early-exit ML framework during periods of high memory usage. 3. Energy consumption: Energy consumption measures the amount of power consumed by the device during operation. In devices with limited power sources, such as battery-powered devices, energy efficiency is crucial. The confidence threshold module can use this KPI to adjust the thresholds, potentially allowing early exits at lower thresholds. This will reduce energy consumption and extend battery life. 4. Device temperature: Device temperature is an indicator of the device's thermal state, which can affect its performance and longevity. High device temperatures can result from heavy workloads or prolonged operation, potentially leading to thermal throttling or device damage. The module can consider this KPI when determining confidence thresholds. It can adopt more conservative thresholds to reduce the processing load and allow the device to cool down.

[80] Model KPIs: 1. Prediction accuracy: Prediction accuracy is a measure of how well the model's predictions align with the actual outcomes or the ground truth. High prediction accuracy is desirable, as it indicates that the model is performing well. The confidence threshold module can use this KPI to adjust the thresholds, potentially allowing for higher thresholds (and thus, more processing) when prediction accuracy is high. 2. Inference latency: Inference latency represents the time it takes for the early-exit ML framework to generate a prediction for a given input. Lower latency is preferable, especially in applications with strict latency requirements. The module can take this KPI into account when determining confidence thresholds, potentially lowering them to prioritize faster inference during periods of high latency. 3. Model complexity: Model complexity refers to the number of layers, nodes, or parameters in the machine learning model. Models with higher complexity may provide more accurate predictions but also require more computational resources and processing time. The confidence threshold module can consider this KPI when determining thresholds. It can also adjust them to balance the trade-off between accuracy and computational efficiency based on the model's complexity. 4. Confidence score distribution: The distribution of confidence scores generated by the early-exit ML framework can provide insights into the model's overall performance and reliability. The module can analyze this distribution to identify trends or patterns that may inform the confidence threshold determination process. For example, a high concentration of low-confidence predictions suggesting a need for more aggressive early exits. 5. Exit point utilization: Exit point utilization is a measure of how frequently each exit point is used during the inference process. This KPI can help the confidence threshold module identify potential bottlenecks or inefficiencies in the early-exit ML framework, allowing it to adjust the thresholds to balance the workload across all exit points more effectively.

[81] By considering these KPIs, the confidence threshold determination module can dynamically adjust the confidence thresholds for each exit point in the early-exit machine learning framework, ensuring a balance between accuracy, computational efficiency, and resource utilization.

[82] Once the relevant KPIs are obtained, we will describe the process through which the confidence threshold determination module obtains a threshold for each early exit point: 1. Analyze the information: The module analyzes the collected information to identify trends, patterns, or correlations that can inform the confidence threshold determination process. For example, it may identify that higher network latency correlates with a need for more aggressive early exits, or that certain device constraints require lower confidence thresholds to maintain energy efficiency. 2. Calculate the initial thresholds: Using the gathered and analyzed information, the module calculates an initial confidence threshold for each exit point. This calculation can be based on heuristics, predefined rules, or data-driven models that take into account the some or all of the aforementioned KPIs. For example, the module may determine that higher exit points should have higher confidence thresholds, as they represent more complex processing and are expected to provide more accurate predictions. 3. Optimize the thresholds: The module can further refine the initial confidence thresholds by employing optimization techniques, such as gradient-based optimization, genetic algorithms, or particle swarm optimization. These techniques can help the module find the optimal thresholds that balance the trade-off between prediction accuracy, computational efficiency, and resource utilization for each exit point under the current network and device conditions and application requirements. 4. Update the thresholds: The module continuously monitors the performance of the early-exit ML framework and the network function, as well as any changes in the network conditions, device constraints, or application requirements. Based on this monitoring, the module can update the confidence thresholds for each exit point dynamically, ensuring that the thresholds remain relevant and effective in adapting the inference process to the current conditions and requirements. 5. Provide the thresholds to the early-exit ML framework: Once the confidence thresholds for each exit point are determined, the module provides these values to the early-exit ML framework. The framework then uses these thresholds during the adaptive inference process to decide whether the prediction at a given exit point is sufficiently reliable or if additional processing is needed.

[83] By following this process, the confidence threshold determination module can dynamically obtain and update the confidence thresholds for each exit point in the early-exit machine learning framework. This adaptability enables the framework to balance prediction accuracy, computational efficiency, and resource utilization according to the current network and device conditions and application requirements.

[84] Creation / Execution of a Early-Exit ML Frameworks Workload

[85] Figure 3 shows an example flow chart of a possible setup message exchange in relation to the creation / execution of a EE ML framework workload proposed in this invention. The Figure shows message exchange between the network entity in charge of EE ML framework and / or the UE willing to perform inference exploiting EE ML framework. In one example, the subscription request and response procedure is as follows: > Step 1: The UE that aims to exploit EE solutions in a model sends, for an example, a EEMLRegistration message (or EEMLRegistration Request message, or any other message naming) to the NF(s) and / or logical function(s) responsible for the early-exit ML framework, specifying one or more of the information such as: ■ UE ID: The specific unique identifier of the UE. ■ Priority level of the inference task (e.g., high priority for urgent use cases or low priority for less critical use cases). ■ EEML session ID: The session early exit unique identifier. ■ UE's capabilities: o A list of UE specs that can be monitored and , e.g., CPU specs, GPU specs, RAM, remaining battery, current battery consumption, etc. o Indication of UE support of the new capability of AI / ML Early-exit inference. ■ Network monitoring conditions. A list of network conditions the UE can monitor, e.g., RSRP, RSRQ, SINR, Throughput etc. ■ Functionality ID: is a unique identifier that categorizes the specific type of inference tasks for the early-exit ML framework. It aids in task routing, resource allocation, and configuration management within the NF. This ID also facilitates monitoring, logging, and ensures interoperability across different parts of the system. It serves as a crucial element for optimized and efficient task handling. ■ Network configuration for the UE to support the early-exit procedure. ■ Model ID: If known the model unique identifier. ■ Model type: The types of AI / ML models, such as supervised, unsupervised, or reinforcement learning, etc. ■ Model architecture: The architecture of the AI / ML model, such as deep neural networks or decision trees. ■ Model application: The purpose of the model. > Step 2: The NF processes the received information and calculates the appropriate early-exit ML framework configuration parameters, including the number of exit points, the confidence metric(s) for each exit point, the processing layers, and other model-specific parameters. The NF sends to the UE, for example, a EEMLRegistrationReply message (or EEMLRegistration Response / Acknowledge message, and / or any other suitable naming) to the UE containing the configuration details for the early-exit ML framework. In another example, the NF may responds to the UE with a failure message EEMLRegistrationFailure message, and / or any other suitable naming). The NF may also send a cause value for the failure to perform the registration request from the UE. > Step 3: The EEML NF collects, from the UE and / or any existing possible monitoring NF (and / or logical function) using a EEMLMontiroing message(or EEMLMontiroing Request message, or any other message naming), network KPIs (one or more) such as, but not limited to: ■ Latency. ■ Throughput. ■ Packet loss. ■ Jitter. ■ Others. It also retrieves device KPIs (one or more) such as, but not limited to: ■ Battery life. ■ CPU usage. ■ Memory usage. ■ Others. and finally, model KPIs (one or more) such as, but not limited to: ■ Prediction accuracy. ■ Inference latency. ■ Model complexity. ■ Confidence score distribution. ■ Others. > Step 4: Based on the collected KPIs, the NF uses a predetermined algorithm and / or model to determine the optimal confidence thresholds for each exit point in the early-exit ML framework. > Step 5: The NF sends a message to the UE containing the confidence thresholds for each exit point, for example, EEMLConfidenceThr message(or EMLConfidenceThr Response message, or any other message naming). This message may also contain instructions for execution, resource allocation, and any other modelspecific configuration details. > Step 6: Upon the reception of the EEMLConfidenceThr. The UE processes the input data using the early-exit ML framework, generating predictions at each exit point and calculating the associated confidence metrics. The UE compares the confidence metrics at each exit point with the received thresholds, stopping the processing and obtaining the prediction result if the confidence metric meets or exceeds the threshold. > Step 7: If further inference is needed, the process goes back to step 3. > Step 8: The UE transmits an EEMLstop message(or EEMLstopIndication message, and / or any other message naming)message and the EE ML to signal the conclusion of the EE ML process, indicating that either a confidence threshold has been met or the inference process has successfully traversed through the entire model. The EE ML NF may also send a feedback message, referred to as EEMLfeedback (or EEMLfeedbacklndication message, or EEMLfeedbackReport, and / or any other message naming), to the UE regarding the prediction's accuracy, latency, or other performance metrics. This feedback can be used to adjust the confidence thresholds or resource allocation for future inferences, enabling the early-exit ML framework to adapt to changing conditions and requirements. > Step 9: Alternative Termination Criteria. EE ML process continuously monitors the performance of the ML model based on various KPIs and / or confidence metrics, and / or other specific requirement. The EE ML process evaluates whether the inference process is meeting the dynamically defined confidence thresholds. Several scenarios can trigger alternative termination criteria: ■ Time-Based Evaluation: The EE ML NF starts a timer when the early-exit ML process begins. If the confidence threshold is not met or exceeded within this predefined time period, the EE ML NF implements the actions described above. ■ Consistency Check evaluation: The EE ML NF keeps track of the number of instances where the confidence at an exit point falls below the dynamically defined threshold. If this occurs consistently for a given number of instances or occurrences, the NF evaluates the need for alternative action, such as, the actions described above. ■ External Indicators: The EE ML NF may also consider signals or indicators from other newly defined (and / or existing) network entities or functions. These could be flags or messages indicating network congestion, device limitations, or other exceptional conditions that may affect the early-exit process. ■ Decision Making: Based on the above evaluations, the EE ML NF makes a decision. If any of the alternative termination criteria are met, the NF sends an EEMLSwitch message (or EEMLSwitchRequest message, or EEMLSwitchCommand message, and / or any other suitable message naming) to the UE. This message instructs the UE to terminate the early-exit process and / or revert to a normal inference learning process, among other possible actions. ■ Configuration Update: Along with the EEMLSwitch message (or in another separate message) the NF may also send, for example, updated configuration parameters for the normal inference process, including new confidence thresholds, layer configurations, or other model-specific details. ■ Logging and Reporting: The NF logs the event and the reason for the switch for future analysis and possible optimization of the early-exit ML framework

[86] The following should be taken into account when considering the proposals, examples and methods described in this disclosure herein: 1. The execution of steps in all examples, algorithms, procedures, and figures can be in any order, and modification and / or addition of other steps is also possible. 2. All proposals, examples, and / or Figures, text and description, may apply to the term model, model functionality and / or functionality. Additionally, the terms Model or Functionality can be used interchangeably, and / or jointly (together), for example, Model ID and / or Functionality ID. 3. Figures naming including messages, signaling, and / or procedures can be expressed using the term “Model” and / or “Functionality” throughout this invention. 4. The proposed methods, solutions, examples, figures, etc., may also apply for the eNB, gNB, and / or NG-RAN and all related dedicated (newly defined and / or existing) RRC signaling, procedures, messages, and / or lEs, and X2, Xn, S1, NG, F1, E1, and / or other interfaces signaling and messages, and / or other related network entities (e.g. MME, UPF, SMF, AMF, and / or newly defined entities and / or functions, etc.), in addition to possible interaction with NWDAF (for analytics and predictions purposes). 5. Additionally, all proposals, examples, description, solutions, and / or Figures may use, dedicated (newly defined and / or existing) NAS existing signaling, procedures, messages, and / or lEs. Moreover, system information (periodic and / or on-demand) using existing and / or newly defined SIBs, and / or other type of signalling between any network entity / function and / or the UE(s). 6. The proposed approaches also apply to non-3GPP entities. 7. The concepts, proposals, solutions, methods, and / or examples, presented in this disclosure, apply to various type of communication systems, such as 4G, 4G-Advanced, 5G, 5G-Advanced, and 6G. Moreover, the above may also apply (in full or part or modified) to systems of Non-Terrestrial Networks (NR-NTN and / or loT-NTN), in addition to Terrestrial Networks (TN).

[87] Advantages &Novel features of the invention

[88] The present invention relates to the field of machine learning, and more specifically, to a network function that enables adaptive inference using early-exit machine learning frameworks for optimizing computation and communication resources in various networking environments.

[89] This novel network function comprises (but not limited to these advantages): • A network communication module that receives input data from network nodes or devices and transmits the output. • An early-exit machine learning framework, including multiple exit points and corresponding confidence metrics, enabling adaptive inference based on the 5 confidence of predictions at each exit point. • A confidence threshold determination module that calculates and adjusts the confidence thresholds for each exit point dynamically, based on factors such as network conditions, device constraints, and application requirements.

[90] The network function for adaptive inference with early-exit machine learning 10 frameworks offers several advantages, including: • Faster inference times and reduced latency, improving the performance of Al-driven applications and services in networking environments. • Energy-efficient Al model deployment, particularly suitable for resource-constrained devices and edge computing. 15 • Improved resource allocation and network-level optimization, enhancing the overall performance and efficiency of the network.

[91] Acronyms and Definitions 3GPP 3rd Generation Partnership Project 5G 5th Generation 5GC 5G Core 5QI 5G QoS Identifier 5GS 5G System 5GSM 5G System Session Management 5GMM 5G System Mobility Management AF Application Function Al Artificial Intelligence AM Acknowledged Mode AMF Access and Mobility Management Function AS Application Server ASP Application Service Provider AUSF Authentication Server Function CDN Content Delivery Network DCAF Data Collection Application Function DNAI Data Network Access Identifier DNN Data Network Name DNS Domain Name Server DRB Data Radio Bearer EE Early Exit eNB Evolved Node B EPC Evolved Packet Core FEC Forward Error Correction FQDN Fully Qualified Domain Name GBR Guaranteed Bit Rate gNB GPSI Next generation Node B Generic Public Subscription Identifier HSS Home Subscriber Service IAB Integrated Access and Backhaul ID Identity / ldentifier lloT Industrial Internet of Things IMEI International Mobile Equipment Identities IP l-SMF LADN LL SSM MBMS MBS MBSF MBSTF MB-SMF MB-UPF ML MME MN MNF MNO MT NAS NEF NRF NG-RAN NG-eNB NSA NSSF NTN NW NWDAF OS OSAPP PCF PCO PDR PDU PTM PTP QFI Internet Protocol Intermediate SMF Local Area Data Network Lower Layer SSM Multimedia Broadcast / Multicast Service Multicast / Broadcast Service Multicast / Broadcast Service Function Multicast / Broadcast Service Transport Function Multicast / Broadcast Session Management Function Multicast / Broadcast User Plane Function Machine Learning Mobility Management Entity Master Node Monitoring Network Function Mobile Network Operator Mobile Termination Non-Access Stratum Network Exposure Function Network Repository Function Next Generation Radio Access Network Next Generation eNB Non-Standalone Network Slice Selection Function Non-Terrestrial Networks Network Network Data Analytics Function Operating System OS Application Policy Control Function Protocol Configuration Options Packet Detection Rule Protocol Data Unit Point To Multipoint Point to Point QoS Flow Identifier (ID) QoS Quality of Service RACH Random Access Channel RAN Radio Access Network RRC Radio Resource Control RSD Route Selection Descriptor RSRP Reference Signal Received Power RSRQ Reference Signal Received Quality RSS Reference Signal Strength RSSI Received Signal Strength Indicator. SA Standalone SDAP Service Data Adaptation Protocol SDU Service Data Unit SGW Serving Gateway SIM Subscriber Identity Module SLA Service Level Agreement SM Session Management SMF Session Management Function SN Secondary Node S-NSSAI Single Network Slice Selection Assistance Information SSB Synchronization Signal Block SSM Source Specific IP Multicast address SSC Session and Service Continuity SRB Signaling Radio Bearer SUPI Subscription Permanent Identifier TA Tracking Area TAI Tracking Area Identity TE Terminal Equipment TM Transparent Mode TMGI Temporary Mobile Group Identity TS Technical Specification UDM Unified Data Manager UDR Unified Data Repository UE User Equipment UL Uplink UM Unacknowledged Mode UP User Plane UPF User Plane Function UR LLC Ultra-Reliable and Low-Latency Communication URSP UE Route Selection Policy 5

Claims

1. A method of controlling a neural network model performing an inference action in a communications network, the method iteratively comprising:receiving a set of values from a hidden layer of the neural network model; calculating a confidence metric associated with the set of received values;comparing the confidence metric to a confidence threshold; andcontrolling the neural network model to:exit inference when confidence metric is greater than or equal to the confidence threshold, andcontinue to a subsequent layer of the neural network model when the confidence metric is less than the confidence threshold;receiving a condition associated with the inference action; anddynamically adjusting the confidence threshold based on the received condition.

2. The method of Claim 1, wherein controlling the neural network model to exit inference comprises:bypassing any remaining hidden layers of the neural network model;redirecting the received set of values from the hidden layer to an output, fully-connected, layer of the neural network model; andoutputting an output from the output, fully-connected, layer of the neural network model.

3. The method of Claim 1 or Claim 2, wherein the condition is a condition selected from a list of conditions including: a condition of the communications network, a configuration of the communications network, a constraint of user equipment operating on the communications network, a capability of user equipment operating on the communications network, and a key performance indicator of the neural network model.

4. The method of any preceding claim, further comprising:deactivating the neural network model based on the received condition and / or the comparison.

5. The method of any preceding claim, further comprising: activating the neural network model based on the received condition and / or the comparison.

6. The method of any preceding claim, further comprising: controlling the neural network model to stop inference based on the received condition and / or the comparison.

7. The method of any preceding claim, performed by an adaptive inference framework model.

8. The method of Claim 7, wherein the adaptive inference framework model is one of a rules-based model, a machine learning model, and a reinforcement learning model.

9. The method of any preceding claim, wherein calculating the confidence metric comprises calculating a cross-entropy loss associated with the set of values form the hidden layer, wherein the cross-entropy loss is the confidence metric.

10. The method of any of Claims 1 to 8, wherein calculating the confidence metric comprises calculating an L2 loss associated with the set of values, wherein the L2 loss is the confidence metric.

11. A transitory, or non-transitory, computer readable medium having instructions stored thereon that when executed by at least one processor, cause the at least one processor to perform the method of any preceding claim.

12. An adaptive inference framework model for controlling a neural network model performing an inference action in a communications network, comprising:an input configured to receive a set of values from a hidden layer of the neural network model;a processor configured to:calculate a confidence metric associated with the set of received values, compare the confidence metric to a confidence threshold, andgenerate a command to control the neural network model to:exit inference when confidence metric is greater than or equal to5 the confidence threshold, andcontinue to a subsequent layer of the neural network model when the confidence metric is less than the confidence threshold, receive a condition associated with the inference action; and dynamically adjust the confidence threshold based on the received10 condition; andan output configured to transmit the command to the neural network model.15

Images