Data parameters and prediction horizon optimization for artificial intelligence based fault prediction

Abstract

A method, system and apparatus are disclosed. According to one aspect, a method in a network node configured to determine one or more parameters usable for training an artificial intelligence (AI) prediction model is described. The method includes automatically determining automatically determining a data sampling rate, a training data size, and a prediction horizon for training the AI prediction model. The data sampling rate, the training data size, and the prediction horizon meet one or more objectives. The method also includes performing one or more actions based on the data sampling rate, the training data size, and the prediction horizon.

Description

[0001] DATA PARAMETERS AND PREDICTION HORIZON OPTIMIZATION FOR ARTIFICIAL INTELLIGENCE BASED FAULT PREDICTION

[0002] TECHNICAL FIELD

[0003] The present disclosure relates to artificial intelligence (Al) and adjustment of parameters associated with Al models, and in particular, to determination and / or adjustment of data parameters and prediction horizon for Al-based fault prediction.

[0004] BACKGROUND

[0005] Artificial Intelligence (Al) and Machine Learning (ML), which is an application of Al, may both be referred to as (AI / ML) and be used for predicting performance associated with infrastructure (e.g., central processing unit (CPU), memory, network, etc.) and application faults / problems in cloud environments (i.e., cloud networks). Such a prediction may be data-centric and require collecting a large amount of data from the cloud environments. The data represents the operational statistics of the cloud infrastructure and / or applications, often in a time sequence manner.

[0006] The performance of an AI / ML model may be directly affected by the characteristics of the collected data. More specifically, the total number of data samples available for training a prediction model, i.e. the sampling rate, and the prediction horizon can affect the performance of the prediction model. Depending on the type of the AI / ML model, the model performance is, in many cases, dependent on the size of collected data, i.e., model accuracy becomes higher when training with a larger number of data samples. However, data collection may require extensive storage and network resources.

[0007] A cloud service provider may request a model that is trained with minimum resource consumption and that can predict as accurately as possible so that the provider may avoid revenue loss associated with missing true faults and avoid unnecessary efforts in cases of false fault alarms. Frequently, the two needs are in conflict, i.e., the resource consumption for data collection and storage is higher for prediction models with higher accuracy. Further, the cloud service provider may desire the prediction model to provide enough lead time to take appropriate action to prevent the fault from occurring. That is, a longer prediction horizon may be more desirable.

[0008] In addition, the conventional prediction horizon requirement usually conflicts with the prediction accuracy. Prediction models typically show better accuracy with a shorter prediction horizon. Generally, determining the best data sampling rate, the training data size, and the prediction horizon for a prediction model that meets the requirements of the cloud service provider relies on experiments and human experts. In other words, the conventional determination is a manual process. In large scale, distributed, heterogenous networks (such as edge clouds), different AI / ML models can be deployed in different edge sites for predicting the local faults. With a large number and type of edge sites, it becomes impracticable for human experts to test and select the best sampling rate, training data size, and prediction horizon for each model.

[0009] Some conventional technology provides adjustment of the data characteristic parameters such as data size, prediction horizon, and sampling rate. There are works that try limited combinations of these parameters and evaluate the model using the selected combinations. The combination that results in the highest performance may be selected. Further, these conventional technologies may be applicable to specific use cases, datasets, and / or ML models.

[0010] For example, the training dataset size, prediction horizon, and the granularity of time (sampling rate) may have an impact on the accuracy of an ML-based quality of service (QoS) forecasting model. A limited number of prediction horizons and sampling rates may be manually tried to train the forecasting model and evaluate the performance of model by measuring Mean Absolute Error (MAE). Similarly, different sampling rates, ML methods, features and hyper-parameters may be evaluated to determine influence accuracy of a power forecaster for a wind turbine, which may include experimenting with two ML models and four sampling rates and prediction horizons to tune the parameters.

[0011] Further, some conventional processes attempt to maximize the performance of an ML model while minimizing its resource consumption by tuning the configuration of the prediction model to optimize their objectives.

[0012] For example, a framework for Hyper-Parameter Optimization (HPO) and Neural Architecture Search (NAS) which may be referred to as Resource-Aware Automated ML (RA-AutoML) may be used to attempt to optimize multiple objectives (e.g., accuracy and inference time of an ML model) while considering resource constraints. In addition, a multi-objective AutoML process may be used to find a trade-off between accuracy and computational complexity of the ML model. The processes of two examples may include using Non-Dominated Sorting Genetic Algorithm (NSGA)-II-based algorithms to solve the multi-objective optimization problem. Other conventional technologies may employ a method that includes detecting a lowered performance of the ML-based model when a performance metric (such as precision, recall, or prediction accuracy) of the ML- based model is below a threshold, determining whether the lowered performance is correlated with a sample rate of the received data, and if yes, increasing the sample rate of the received data.

[0013] Further, a method for selecting data samples that is representative to a large population dataset may be performed. The method may include identifying the numerical data columns, categorical data columns, determining the predictor variable (input), and predictand variable (output) via applying linear regression on the numerical data columns, generating sequential model based on predictor and predictand variables, generating stratified samples on categorical data columns via stratified sampling, generating sample key based on sequential model and stratified samples, and generating sample dataset representative of population data set, based on sample key.

[0014] Moreover, machine learning model performance may be predicted from the model parameter values using the parameter values (e.g., weight values) as inputs. A machine- learned performance prediction model that has been trained to predict performance values of machine-learned models based on their parameter values (e.g., weight values and / or hyperparameter values) may be used.

[0015] In other words, conventional technology utilizes a subset of the problem, based on human experiments, i.e., focusing on determining only the prediction horizon or sampling rate among limited alternatives. More specifically, some of the limitations of conventional technologies include:

[0016] • Manually trying only a limited number of ML models, training data size, prediction horizons, and sampling rates to decide on the values of these parameters.

[0017] • Tuning hyper-parameters and architecture of the prediction model in an attempt to optimize objectives.

[0018] • Providing parameter values that generally include neural network weight values and / or hyperparameter values.

[0019] • Increasing the sampling rate which may result in unnecessary resource consumption and affect ML model performance if the performance degradation is caused by some other factors (such as data drift).

[0020] • Evaluating sample selections from a population dataset and assuming that there is a population dataset. SUMMARY

[0021] Some embodiments advantageously provide methods, systems, and apparatuses for determination and / or adjustment and / or optimization of data parameters and prediction horizon for artificial intelligence (Al) based fault prediction.

[0022] One or more embodiments provide a method that optimizes and automates the selection of sampling rate, training data size, prediction horizon, such that the model performance is maximized while the resource consumption for data collection and storage is minimized, and other the requirements or conditions such as requirements of a cloud service provider are fulfilled.

[0023] In some embodiments, the data size, prediction horizon, and sampling rate, is determined for automatically training a prediction model while minimizing resource consumption and maximizing the model performance.

[0024] In some other embodiments, a number of ML models, training data size, prediction horizons, and sampling rates are automatically tuned. Further, multiple objectives may be optimized / adjusted, e.g., accuracy and resource consumption, while determining data parameters. In addition, the data parameters such as training data size, prediction horizon, and sampling rate may be tuned for optimizing their objectives.

[0025] In some embodiments, a relationship between ML model performance and data sampling rate during ML model runtime is determined.

[0026] One or more embodiments provide a system and method for determining data sampling rate, training data size, and prediction horizon for training an ML-based fault prediction model while optimizing multiple objectives, i.e., the model performance, the prediction horizon, and the data collection and storage resource consumption for training the model. The method may be beneficial to a cloud service provider as the method may automatically find the parameters that can fulfill the requirements established by the cloud service provider.

[0027] According to one aspect, a prediction optimizer is described. The prediction optimizer may be referred to as a network node or a prediction optimizer unit. The network node (e.g., configured as prediction optimizer, comprising the prediction optimizer unit, etc.) receives parameter optimization requests from an AI / ML Model Management System, obtains parameter boundaries, resource constraints and model performance requirements from a Cloud Management System, and outputs combinations (e.g., tuples) of the data sampling rate, training data size, and prediction horizon parameters for AI / ML Model Management System to train a prediction model. The network node (and / or the prediction optimizer unit) may include and / or be configured with an Optimization Request Handler that handles the optimization requests, collects inputs. The Optimization Request Handler sends the collected parameter boundaries to a Search Space Identifier to initialize the search space and forwards all other inputs to the Prediction Optimization Function that explores the search space and finds a set of parameter tuples that optimize both accuracy and resource consumption. The result is then sent to a Decision Maker that selects one or multiple tuples and outputs the selected results.

[0028] In some embodiments, a Prediction Optimizer System that determines the parameter combinations of data sampling rate, training data size and prediction horizon that optimize the AI / ML based prediction model performance, and the data collection and storage resource usage is described. The network node (e.g., configured as prediction optimizer, comprising the prediction optimizer unit, etc.) of the system may be configured to one or more of:

[0029] Receive an optimization request with a selected AI / ML prediction model from an AI / ML management system.

[0030] Collect resource constraints, and the data size and sampling rate boundaries from a Cloud Management system.

[0031] Request for minimal accuracy requirement and prediction horizon boundaries from the Cloud Management system.

[0032] Request the AI / ML Management system for maximum data samples allowed with minimum data sampling rate based on the boundaries.

[0033] Request the AI / ML Management system to select features for the data samples collected.

[0034] Search for the optimal prediction horizon, data sampling rate and data size so that the resource constraints and accuracy minimal requirements are met, and resource usage and accuracy are optimized.

[0035] Return / output the values for the parameter tuples (training data size, prediction horizon, data sampling rate) to the AI / ML Management System that may be used to prepare the prediction model for deployment.

[0036] One or more embodiments of the present disclosure are beneficial at least because the following is provided: • Automated Solution: The Prediction Optimizer system and / or the network node (and / or prediction optimizer unit) automatically determines the values for data and time related parameters for training a fault prediction model, which may help a cloud service provider train the model without relying on a human expert or extensive experiments. This can also facilitate the deployment of prediction models in large-scale heterogenous networks.

[0037] • Model Performance and Data storage Resource Consumption Balance: The Prediction Optimizer System and / or the network node (and / or prediction optimizer unit) determines the data-related parameters for training a fault prediction model while considering not only the performance of the prediction model, but also the resource consumption for collecting and storing the data that will be used for training the model.

[0038] • Applicable to various prediction model types: The Prediction Optimizer System and / or the network node (and / or prediction optimizer unit) may be configured to determine the data-related parameters for training various types of ML models that should be deployed to predict faults in a large-scale cloud environment.

[0039] According to one aspect, a method in a network node configured to determine one or more parameters usable for training an artificial intelligence (Al) prediction model is described. The method includes automatically determining automatically determining a data sampling rate, a training data size, and a prediction horizon for training the Al prediction model. The data sampling rate, the training data size, and the prediction horizon meet one or more objectives. The method also includes performing one or more actions based on the data sampling rate, the training data size, and the prediction horizon.

[0040] In some embodiments, the one or more objectives include one or more of a model performance and a data collection network and storage resource utilization with one or both of a minimal prediction accuracy and a maximum data collection resource utilization constraint for training the Al prediction model.

[0041] In some other embodiments, automatically determining includes searching for the data sampling rate, the training data size, and the prediction horizon that meet one or more resource constraints and accuracy minimal requirements and optimize resource usage and Al prediction model accuracy.

[0042] In some embodiments, the method further includes, to perform the automatic determination, obtaining one or more of: (A) the Al prediction model; (B) one or more resource constraints, one or more data size boundaries, and one or more data sampling rate boundaries; (C) a minimal accuracy requirement and one or more prediction horizon boundaries; (D) data collection for one or more maximum data samples that are allowed with a minimum data sampling rate based on the one or more data size boundaries and the one or more sampling rate boundaries; and (E) a feature selection set to select one or more features for one or more data samples associated with the one or more maximum data samples.

[0043] In some other embodiments, the one or more actions includes determining a set of combinations of a plurality of combinations of the data sampling rate, a training data size, and the prediction horizon used for training the Al prediction model and optimizing model performance and resource consumption.

[0044] In some embodiments, the one or more actions include causing another network node to one or more of: (A) use the data sampling rate and the training data size to collect data; (B) use the prediction horizon to build the Al prediction model, where the Al prediction model is a fault prediction model; (C) deploy the fault prediction model to a cloud system; and (E) report fault prediction results.

[0045] In some other embodiments, the one or more objectives include a first optimized objective and a second optimized objective that is contradicting to the first optimized objective.

[0046] In some embodiments, the method further includes one or more of: (A) initializing a set of solutions including the data sampling rate, the training data size, and the prediction horizon; (B) estimating one or more resources used for collecting and storing training data based on the set of solutions; (C) determining an accuracy of the Al prediction model based on the set of solutions; (D) selecting at least one solution that dominates other solutions of the set of solutions; (E) determining whether a search for an optimal solution is to continue based on search termination criteria and the selected at least one solution; (F) if the search termination criteria is not met, updating the data sampling rate, the training data size, and the prediction horizon of the selected at least one solution; and (G) if the search termination criteria is met, terminating the search and returning the latest solutions that dominate the other solutions or an empty set.

[0047] In some other embodiments, the automatic determination is based on one or both of a Non-dominated Sorting Algorithm (NSGA) and a Particle Swarm Optimization (PSO).

[0048] In some embodiments, the Al prediction model is trained to predict call drop rates associated with one or more user equipment (UEs) using a core network. According to another aspect, a network node configured to determine one or more parameters usable for training an artificial intelligence (Al) prediction model. The network node is configured to automatically determine a data sampling rate, a training data size, and a prediction horizon for training the Al prediction model. The data sampling rate, the training data size, and the prediction horizon meet one or more objectives. The network node is further configured to perform one or more actions based on the data sampling rate, the training data size, and the prediction horizon.

[0049] In some embodiments, the one or more objectives include one or more of a model performance and a data collection network and storage resource utilization with one or both of a minimal prediction accuracy and a maximum data collection resource utilization constraint for training the Al prediction model.

[0050] In some other embodiments, automatically determining includes searching for the data sampling rate, the training data size, and the prediction horizon that meet one or more resource constraints and accuracy minimal requirements and optimize resource usage and Al prediction model accuracy.

[0051] In some embodiments, for performing the automatic determination, the network node is configured to obtain one or more of: (A) the Al prediction model; (B) one or more resource constraints, one or more data size boundaries, and one or more data sampling rate boundaries; (C) a minimal accuracy requirement and one or more prediction horizon boundaries; (D) data collection for one or more maximum data samples that are allowed with a minimum data sampling rate based on the one or more data size boundaries and the one or more sampling rate boundaries; and (E) a feature selection set to select one or more features for one or more data samples associated with the one or more maximum data samples.

[0052] In some other embodiments, the one or more actions includes determining a set of combinations of a plurality of combinations of the data sampling rate, a training data size, and the prediction horizon used for training the Al prediction model and optimizing model performance and resource consumption.

[0053] In some embodiments, the one or more actions include causing another network node to one or more of: (A) use the data sampling rate and the training data size to collect data; (B) use the prediction horizon to build the Al prediction model, where the Al prediction model is a fault prediction model; (C) deploy the fault prediction model to a cloud system; and (D) report fault prediction results. In some other embodiments, the one or more objectives includes a first optimized objective and a second optimized objective that is contradicting to the first optimized objective.

[0054] In some embodiments, the network node is further configured to one or more of: (A) initialize a set of solutions comprising the data sampling rate, the training data size, and the prediction horizon; (B) estimate one or more resources used for collecting and storing training data based on the set of solutions; (C) determine an accuracy of the Al prediction model based on the set of solutions; (D) select at least one solution that dominates other solutions of the set of solutions; (E) determine whether a search for an optimal solution is to continue based on search termination criteria and the selected at least one solution; (F) if the search termination criteria is not met, update the data sampling rate, the training data size, and the prediction horizon of the selected at least one solution; and (G) if the search termination criteria is met, terminate the search and return the latest solutions that dominate the other solutions or an empty set.

[0055] In some other embodiments, the automatic determination is based on one or both of a Non-dominated Sorting Algorithm (NSGA) and a Particle Swarm Optimization (PSO).

[0056] In some embodiments, the Al prediction model is trained to predict call drop rates associated with one or more user equipment (UEs) using a core network.

[0057] According to one aspect, a system is described. The system includes a first network node configured to determine one or more parameters usable for training an artificial intelligence (Al) prediction model, and communicate with a second network node, a third network node, and a fourth network node. The first network node is configured to receive, from the third network node, the Al prediction model, receive, from the second network node, one or more resource constraints, one or more data size boundaries, and one or more sampling rate boundaries, and receive, from the second network node, a minimal accuracy requirement and one or more prediction horizon boundaries. The first network node is further configured to receive, from third network node, data collection for one or more maximum data samples that are allowed with a minimum data sampling rate based on the one or more data size boundaries and the one or more sampling rate boundaries and receive, from the third network node, a feature selection set to select one or more features for one or more data samples associated with the one or more maximum data samples.

[0058] In addition, the first network node is configured to automatically determine a data sampling rate, a training data size, and a prediction horizon for training the Al prediction model. The data sampling rate, the training data size, and the prediction horizon meet one or more objectives. The automatic determination is based on the one or more resource constraints, the one or more data size boundaries, the one or more sampling rate boundaries, the minimal accuracy requirement, the one or more prediction horizon boundaries, the data collection, and the feature selection set. One or more actions are performed by the first network node based on the data sampling rate, the training data size, and the prediction horizon. The third network node is configured to collect data using the data sampling rate and the training data size, build the Al prediction model using the prediction horizon, where the Al prediction model is a fault prediction model, and deploy the fault prediction model. The fourth network node is configured to report fault prediction results to the second network node.

[0059] In some embodiments, one or more of: (A) the second network node is cloud management system; (B) the second network node is configured to perform resource management and fault management; (C) the third network node is an Al management system; (D) the third network node is configured to perform data collection, feature selection, and model management; and (E) the fourth network node is comprised in a cloud system configurable for management by one or more entities.

[0060] BRIEF DESCRIPTION OF THE DRAWINGS

[0061] A more complete understanding of the present embodiments, and the attendant advantages and features thereof, will be more readily understood by reference to the following detailed description when considered in conjunction with the accompanying drawings wherein:

[0062] FIG. 1 is a schematic diagram of an example network architecture illustrating a communication system according to principles disclosed herein;

[0063] FIG. 2 is a block diagram of a network node in communication with other network nodes according to some embodiments of the present disclosure;

[0064] FIG. 3 is a schematic diagram of another example network architecture illustrating a communication system according to principles disclosed herein;

[0065] FIG. 4 is a block diagram of another network node in communication with a user equipment over a wireless connection according to some embodiments of the present disclosure;

[0066] FIG. 5 is a flowchart of an example process in a network node configured as a prediction optimizer according to some embodiments of the present disclosure; FIG. 6 shows an example system and process for optimizing data and time parameters of fault prediction models in a cloud system according to some embodiments of the present disclosure;

[0067] FIG. 7 shows an overview of an example prediction optimizer unit according to some embodiments of the present disclosure;

[0068] FIG. 8 shows example search spaces according to some embodiments of the present disclosure;

[0069] FIG. 9 is a flowchart of an example prediction optimization process according to some embodiments of the present disclosure; and

[0070] FIG. 10 is a flowchart of another example prediction optimization process according to some embodiments of the present disclosure.

[0071] DETAILED DESCRIPTION

[0072] Before describing in detail exemplary embodiments, it is noted that the embodiments reside primarily in combinations of apparatus components and processing steps related to determination and / or adjustment and / or optimization of data parameters and prediction horizon for artificial intelligence (Al) based fault prediction. Accordingly, components have been represented where appropriate by conventional symbols in the drawings, showing only those specific details that are pertinent to understanding the embodiments so as not to obscure the disclosure with details that will be readily apparent to those of ordinary skill in the art having the benefit of the description herein.

[0073] As used herein, relational terms, such as “first” and “second,” “top” and “bottom,” and the like, may be used solely to distinguish one entity or element from another entity or element without necessarily requiring or implying any physical or logical relationship or order between such entities or elements. The terminology used herein is for the purpose of describing particular embodiments only and is not intended to be limiting of the concepts described herein. As used herein, the singular forms “a”, “an” and “the” are intended to include the plural forms as well, unless the context clearly indicates otherwise. It will be further understood that the terms “comprises,” “comprising,” “includes” and / or “including” when used herein, specify the presence of stated features, integers, steps, operations, elements, and / or components, but do not preclude the presence or addition of one or more other features, integers, steps, operations, elements, components, and / or groups thereof. In embodiments described herein, the joining term, “in communication with” and the like, may be used to indicate electrical or data communication, which may be accomplished by physical contact, induction, electromagnetic radiation, radio signaling, infrared signaling or optical signaling, for example. One having ordinary skill in the art will appreciate that multiple components may interoperate, and modifications and variations are possible of achieving the electrical and data communication.

[0074] In some embodiments described herein, the term “coupled,” “connected,” and the like, may be used herein to indicate a connection, although not necessarily directly, and may include wired and / or wireless connections.

[0075] The terminology used herein is for the purpose of describing particular embodiments only and is not intended to be limiting of the concepts described herein. As used herein, the singular forms “a”, “an” and “the” are intended to include the plural forms as well, unless the context clearly indicates otherwise. It will be further understood that the terms “comprises,” “comprising,” “includes” and / or “including” when used herein, specify the presence of stated features, integers, steps, operations, elements, and / or components, but do not preclude the presence or addition of one or more other features, integers, steps, operations, elements, components, and / or groups thereof.

[0076] The term “network node” used herein can be any kind of network node comprised in a network. For example, a network node may include a node in an access network, core network, cloud network, etc., may be any computing device such as a server, and may be virtualized.

[0077] The functions described herein as being performed by a network node may be distributed over a plurality of network nodes. In other words, it is contemplated that the functions of the network node described herein are not limited to performance by a single physical device and, in fact, can be distributed among several physical devices.

[0078] In some embodiments, the term “total number or data samples” is used and may refer to training data size. In some other embodiments, the term “sampling rate” is used and may refer to the time difference between two consecutive collected data samples. In some embodiments, the term “prediction horizon” may be used and may refer to a number of steps that may be predicted as part of an Al process. For example, a prediction horizon may be the number of future steps a model should or may predict.

[0079] In some other embodiments, the term Al is used and may refer to Al and / or ML. For example an Al-based fault prediction model may refer to a fault prediction model that is based on Al and / or ML. Referring to the drawing figures, in which like elements are referred to by like reference numerals, there is shown in FIG. 1 a schematic diagram of a communication system 10, according to an embodiment, which comprises a network 12, and a network 14. Further, communication system 10 comprises a plurality of network nodes 16a, 16b, 16c, 16d (referred to collectively as network nodes 16). Network nodes 16a, 16b, 16c are comprised in the network 12, and network node 16d is comprised in the second network 14. Each network node 16 is connectable to another network node 16. Further, each network node 16a, 16b, 16c is connectable to the second network 14 (and / or network node 16d comprised in the second network 14) over a wired or wireless connection 18.

[0080] Also, it is contemplated that a network node 16 can be in simultaneous communication and / or configured to separately communicate with more than one network node 16 and more than one type of network node 16. For example, a network node 16 can have dual connectivity with another network node 16 that more than once communication protocol or communication link.

[0081] A network node 16 (e.g., network node 16a) is configured to include a prediction optimizer unit 20 which is configured to perform any step and / or task and / or process and / or method and / or feature described in the present disclosure, e.g., network node functions and / or prediction optimization functions. Although the prediction optimizer unit 20 is shown as comprised in network node 16a, the present disclosure is not limited as such, and the prediction optimizer unit may be comprised in any network node or any component of system 10. In one nonlimiting example, prediction optimizer unit 20 may be comprised in network node 16d in network 14. In another nonlimiting example, network node 16 comprising the prediction optimizer unit 20 is radio base station and / or may be configured for performing edge computing functions.

[0082] Example implementations, in accordance with an embodiment, of the network node 16 and other network nodes 16 discussed in the preceding paragraphs will now be described with reference to FIG. 2.

[0083] The communication system 10 includes a network node 16 and includes hardware 22 enabling the network node 16 to communicate with the other network nodes 16. The hardware 22 may include a communication interface 24 for setting up and maintaining at least a wireless / wired connection with the other network nodes 16, which may be located in the network 12, network 14, or any other network. The communication interface 24 may be formed as or may include transmitters and / or receivers. In the embodiment shown, the hardware 22 of the network node 16 further includes processing circuitry 26. The processing circuitry 26 may include a processor 28 and a memory 30. In particular, in addition to or instead of a processor, such as a central processing unit, and memory, the processing circuitry 26 may comprise integrated circuitry for processing and / or control, e.g., one or more processors and / or processor cores and / or FPGAs (Field Programmable Gate Array) and / or ASICs (Application Specific Integrated Circuitry) adapted to execute instructions. The processor 28 may be configured to access (e.g., write to and / or read from) the memory 30, which may comprise any kind of volatile and / or nonvolatile memory, e.g., cache and / or buffer memory and / or RAM (Random Access Memory) and / or ROM (Read-Only Memory) and / or optical memory and / or EPROM (Erasable Programmable Read-Only Memory).

[0084] Thus, the network node 16 further has software 32 stored internally in, for example, memory 30, or stored in external memory (e.g., database, storage array, network storage device, etc.) accessible by the network node 16 via an external connection. The software 32 may be executable by the processing circuitry 26. The processing circuitry 26 may be configured to control any of the methods and / or processes described herein and / or to cause such methods, and / or processes to be performed, e.g., by network node 16. Processor 28 corresponds to one or more processors 28 for performing network node 16 functions described herein. The memory 30 is configured to store data, programmatic software code and / or other information described herein. In some embodiments, the software 32 may include instructions that, when executed by the processor 28 and / or processing circuitry 26, causes the processor 28 and / or processing circuitry 26 to perform the processes described herein with respect to network node 16. For example, processing circuitry 26 of the network node 16 may include a prediction optimizer unit 20 which is configured to perform any step and / or task and / or process and / or method and / or feature described in the present disclosure, e.g., network node functions and / or prediction optimization functions.

[0085] In some embodiments, the inner workings of the network node 16 may be as shown in FIG. 2 and independently, the surrounding network topology may be that of FIG. 1.

[0086] The connection between the network node 16 and the other network nodes 16 is in accordance with the teachings of the embodiments described throughout this disclosure. More precisely, the teachings of some of these embodiments may improve the Al model parameters, data rate, latency, and / or power consumption and thereby provide benefits such as reduced user waiting time, relaxed restriction on file size, better responsiveness, extended battery lifetime, etc.

[0087] Although FIGS. 1 and 2 show various “units” such as prediction optimizer unit 20 as being within a respective processor, it is contemplated that these units may be implemented such that a portion of the unit is stored in a corresponding memory within the processing circuitry. In other words, the units may be implemented in hardware or in a combination of hardware and software within the processing circuitry.

[0088] In some embodiments, the network node is comprised in a radio network which may be associated with a Third Generation Partnership Project (3GPP) system. The 3GPP has developed and is developing standards for Fourth Generation (4G) (also referred to as Long Term Evolution (LTE)) and Fifth Generation (5G) (also referred to as New Radio (NR)) wireless communication systems. Such systems provide, among other features, broadband communication between network nodes, such as base stations, and user equipment (UE), as well as communication between network nodes and between UEs. The 3GPP is also developing standards for Sixth Generation (6G) wireless communication networks.

[0089] The network node may comprise any of base station (BS), radio base station, base transceiver station (BTS), base station controller (BSC), radio network controller (RNC), g Node B (gNB), evolved Node B (eNB or eNodeB), Node B, multi-standard radio (MSR) radio node such as MSR BS, multi-cell / multicast coordination entity (MCE), relay node, donor node controlling relay, radio access point (AP), transmission points, transmission nodes, Remote Radio Unit (RRU) Remote Radio Head (RRH), a core network node (e.g., mobile management entity (MME), self-organizing network (SON) node, a coordinating node, positioning node, MDT node, etc.), an external node (e.g., 3rd party node, a node external to the current network), nodes in distributed antenna system (DAS), a spectrum access system (SAS) node, an element management system (EMS), etc. The network node may also comprise test equipment. The term “radio node” used herein may be used to also denote a user equipment (UE) such as a wireless device (WD) or a radio network node.

[0090] In some embodiments, the non-limiting terms wireless device (WD) or a user equipment (UE) are used interchangeably. The UE herein can be any type of wireless device capable of communicating with a network node or another UE over radio signals, such as a wireless device (WD). The UE may also be a radio communication device, target device, device to device (D2D) UE, machine type UE or UE capable of machine to machine communication (M2M), low-cost and / or low-complexity UE, a sensor equipped with UE, Tablet, mobile terminals, smart phone, laptop embedded equipped (LEE), laptop mounted equipment (LME), USB dongles, Customer Premises Equipment (CPE), an Internet of Things (loT) device, or a Narrowband loT (NB-IOT) device etc.

[0091] Also, in some embodiments the generic term “radio network node” is used. It can be any kind of a radio network node which may comprise any of base station, radio base station, base transceiver station, base station controller, network controller, RNC, evolved Node B (eNB), Node B, gNB, Multi-ccll / multicast Coordination Entity (MCE), relay node, access point, radio access point, Remote Radio Unit (RRU) Remote Radio Head (RRH).

[0092] Note that although terminology from one particular wireless system, such as, for example, 3GPP LTE and / or New Radio (NR), may be used in this disclosure, this should not be seen as limiting the scope of the disclosure to only the aforementioned system. Other wireless systems, including without limitation Wide Band Code Division Multiple Access (WCDMA), Worldwide Interoperability for Microwave Access (WiMax), Ultra Mobile Broadband (UMB) and Global System for Mobile Communications (GSM), may also benefit from exploiting the ideas covered within this disclosure.

[0093] Note further, that functions described herein as being performed by a user equipment or a network node may be distributed over a plurality of user equipments and / or network nodes. In other words, it is contemplated that the functions of the network node and user equipment described herein are not limited to performance by a single physical device and, in fact, can be distributed among several physical devices.

[0094] Unless otherwise defined, all terms (including technical and scientific terms) used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this disclosure belongs. It will be further understood that terms used herein should be interpreted as having a meaning that is consistent with their meaning in the context of this specification and the relevant art and will not be interpreted in an idealized or overly formal sense unless expressly so defined herein.

[0095] FIG. 3 shows a schematic diagram of another embodiment of a communication system 10 such as a 3GPP-type cellular network that may support standards such as LTE and / or NR (5G). The embodiment of FIG. 3 comprises an access network 12, such as a radio access network, and a core network 14. The access network 12 comprises a plurality of network nodes 16a, 16b, 16c, such as NBs, eNBs, gNBs or other types of wireless access points, each defining a corresponding coverage area 40a, 40b, 40c (referred to collectively as coverage areas 40). Each network node 16a, 16b, 16c is connectable to the core network 14 (and / or network node 16d) over a wired or wireless connection 18. A first user equipment (UE) 42a located in coverage area 40a is configured to wirelessly connect to, or be paged by, the corresponding network node 16a. A second UE 42b in coverage area 40b is wirelessly connectable to the corresponding network node 16b. While a plurality of UEs 42a, 42b (collectively referred to as UEs 42) are illustrated in this example, the disclosed embodiments are equally applicable to a situation where a sole UE 42 is in the coverage area 40 or where a sole UE 42 is connecting to the corresponding network node 16. Note that although only two UEs 42 and three network nodes 16 are shown for convenience, the communication system may include many more UEs 42 and network nodes 16.

[0096] Also, it is contemplated that a UE 42 can be in simultaneous communication and / or configured to separately communicate with more than one network node 16 and more than one type of network node 16. For example, a UE 42 can have dual connectivity with a network node 16 that supports LTE and the same or a different network node 16 that supports NR. As an example, UE 42 can be in communication with an eNB for LTE / E-UTRAN and a gNB for NR / NG-RAN.

[0097] A network node 16 (e.g., eNB or gNB) is configured to include a prediction optimizer unit 20 which is configured to perform any step and / or task and / or process and / or method and / or feature described in the present disclosure, e.g., network node functions and / or prediction optimization functions. A user equipment 42 is configured to include a UE management unit 44 which is configured to perform any step and / or task and / or process and / or method and / or feature described in the present disclosure, e.g., UE functions.

[0098] Example implementations, in accordance with an embodiment, of the UE 42 and network node 16 discussed in the preceding paragraphs will now be described with reference to FIG. 4.

[0099] The communication system 10 includes a network node 16 provided in a communication system 10 and includes hardware 22 enabling it to communicate with the other network nodes 16 and the UE 42. The hardware 22 may include a radio interface 46 for setting up and maintaining at least a wireless connection 50 with a UE 42 located in a coverage area 40 served by the network node 16. The radio interface 46 may be formed as or may include, for example, one or more RF transmitters, one or more RF receivers, and / or one or more RF transceivers. The radio interface 46 includes an array of antennas 48 to radiate and receive signal(s) carrying electromagnetic waves.

[0100] In the embodiment shown, the hardware 22 of the network node 16 may further include communication interface 24, processing circuitry 26, processor 28, memory 30, prediction optimizer unit 20, and software 32, each one being configurable as described with respect to FIG. 2.

[0101] The communication system 10 further includes the UE 42 already referred to. The UE 42 may have hardware 52 that may include a radio interface 54 configured to set up and maintain a wireless connection 50 with a network node 16 serving a coverage area 40 in which the UE 42 is currently located. The radio interface 54 may be formed as or may include, for example, one or more RF transmitters, one or more RF receivers, and / or one or more RF transceivers. The radio interface 54 includes an array of antennas 56 to radiate and receive signal(s) carrying electromagnetic waves.

[0102] The hardware 52 of the UE 42 further includes processing circuitry 58. The processing circuitry 58 may include a processor 60 and memory 62. In particular, in addition to or instead of a processor, such as a central processing unit, and memory, the processing circuitry 58 may comprise integrated circuitry for processing and / or control, e.g., one or more processors and / or processor cores and / or FPGAs (Field Programmable Gate Array) and / or ASICs (Application Specific Integrated Circuitry) adapted to execute instructions. The processor 60 may be configured to access (e.g., write to and / or read from) memory 62, which may comprise any kind of volatile and / or nonvolatile memory, e.g., cache and / or buffer memory and / or RAM (Random Access Memory) and / or ROM (Read-Only Memory) and / or optical memory and / or EPROM (Erasable Programmable Read-Only Memory).

[0103] Thus, the UE 42 may further comprise software 70, which is stored in, for example, memory 62 at the UE 42, or stored in external memory (e.g., database, storage array, network storage device, etc.) accessible by the UE 42. The software 70 may be executable by the processing circuitry 58. The software 70 may include a client application 72. The client application 72 may be operable to provide a service to a human or non-human user via the UE 42.

[0104] The processing circuitry 58 may be configured to control any of the methods and / or processes described herein and / or to cause such methods, and / or processes to be performed, e.g., by UE 42. The processor 60 corresponds to one or more processors 60 for performing UE 42 functions described herein. The UE 42 includes memory 62 that is configured to store data, programmatic software code and / or other information described herein. In some embodiments, the software 70 and / or the client application 72 may include instructions that, when executed by the processor 60 and / or processing circuitry 58, causes the processor 60 and / or processing circuitry 58 to perform the processes described herein with respect to UE 42. For example, the processing circuitry 58 of the user equipment 42 may include UE management unit 44 which is configured to perform any step and / or task and / or process and / or method and / or feature described in the present disclosure, e.g., UE functions.

[0105] In some embodiments, the inner workings of the network node 16 and UE 42 may be as shown in FIG. 4 and independently, the surrounding network topology may be that of FIG. 2.

[0106] The wireless connection 50 between the UE 42 and the network node 16 is in accordance with the teachings of the embodiments described throughout this disclosure. More precisely, the teachings of some of these embodiments may improve the data rate, latency, and / or power consumption and thereby provide benefits such as reduced user waiting time, relaxed restriction on file size, better responsiveness, extended battery lifetime, etc. In some embodiments, a measurement procedure may be provided for the purpose of monitoring data rate, latency and other factors on which the one or more embodiments improve.

[0107] Although FIGS. 3 and 4 show various “units” such as prediction optimizer unit 20 and UE management unit 44 as being within a respective processor, it is contemplated that these units may be implemented such that a portion of the unit is stored in a corresponding memory within the processing circuitry. In other words, the units may be implemented in hardware or in a combination of hardware and software within the processing circuitry.

[0108] FIG. 5 is a flowchart of an example process in a network node 16. One or more blocks described herein may be performed by one or more elements of network node 16 such as by one or more of processing circuitry 36 (including the prediction optimizer unit 20), processor 28, and / or communication interface 24 and / or radio interface 46. Network node 16 such as via processing circuitry 26 and / or processor 28 and / or communication interface 24 and / or radio interface 46 is configured to determine one or more parameters usable for training an artificial intelligence (Al) prediction model. Network node 16 is further configured to automatically determine (SI 00) a data sampling rate, a training data size, and a prediction horizon for training the Al prediction model. The data sampling rate, the training data size, and the prediction horizon meet one or more objectives. Further, the network node 16 is configured to perform (SI 02) one or more actions based on the data sampling rate, the training data size, and the prediction horizon.

[0109] In some embodiments, the one or more objectives include one or more of a model performance and a data collection network and storage resource utilization with one or both of a minimal prediction accuracy and a maximum data collection resource utilization constraint for training the Al prediction model.

[0110] In some other embodiments, automatically determining includes searching for the data sampling rate, the training data size, and the prediction horizon that meet one or more resource constraints and accuracy minimal requirements and optimize resource usage and Al prediction model accuracy.

[0111] In some embodiments, the method further includes, to perform the automatic determination, obtaining one or more of: (A) the Al prediction model; (B) one or more resource constraints, one or more data size boundaries, and one or more data sampling rate boundaries; (C) a minimal accuracy requirement and one or more prediction horizon boundaries; (D) data collection for one or more maximum data samples that are allowed with a minimum data sampling rate based on the one or more data size boundaries and the one or more sampling rate boundaries; and (E) a feature selection set to select one or more features for one or more data samples associated with the one or more maximum data samples.

[0112] In some other embodiments, the one or more actions includes determining a set of combinations of a plurality of combinations of the data sampling rate, a training data size, and the prediction horizon used for training the Al prediction model and optimizing model performance and resource consumption.

[0113] In some embodiments, the one or more actions include causing another network node to one or more of: (A) use the data sampling rate and the training data size to collect data; (B) use the prediction horizon to build the Al prediction model, where the Al prediction model is a fault prediction model; (C) deploy the fault prediction model to a cloud system; and (E) report fault prediction results.

[0114] In some other embodiments, the one or more objectives include a first optimized objective and a second optimized objective that is contradicting to the first optimized objective.

[0115] In some embodiments, the method further includes one or more of: (A) initializing a set of solutions including the data sampling rate, the training data size, and the prediction horizon; (B) estimating one or more resources used for collecting and storing training data based on the set of solutions; (C) determining an accuracy of the Al prediction model based on the set of solutions; (D) selecting at least one solution that dominates other solutions of the set of solutions; (E) determining whether a search for an optimal solution is to continue based on search termination criteria and the selected at least one solution;

[0116] (F) if the search termination criteria is not met, updating the data sampling rate, the training data size, and the prediction horizon of the selected at least one solution; and (G) if the search termination criteria is met, terminating the search and returning the latest solutions that dominate the other solutions or an empty set.

[0117] In some other embodiments, the automatic determination is based on one or both of a Non-dominated Sorting Algorithm (NSGA) and a Particle Swarm Optimization (PSO).

[0118] In some embodiments, the Al prediction model is trained to predict call drop rates associated with one or more user equipment (UEs) 42 using a core network 14.

[0119] In some embodiments, model performance and data resource utilization are examples of optimization objectives, and prediction accuracy and a data collection resource utilization are examples of constraints for performing optimization.

[0120] Having described the general process flow of arrangements of the disclosure and having provided examples of hardware and software arrangements for implementing the processes and functions of the disclosure, the sections below provide details and examples of arrangements for determination and / or adjustment and / or optimization of data parameters and prediction horizon for artificial intelligence (Al) based fault prediction.

[0121] In some embodiments, the term “prediction optimizer” is used and may refer to a network node 16 and / or prediction optimizer unit 20. That is, network node 16 and / or prediction optimizer unit 20 may be configured to perform the functions of the prediction optimizer described herein.

[0122] FIG. 6 shows an example system and process for optimizing data and time parameters of fault prediction models in a system 10 (e.g., cloud system). System 10 includes network node 16a (and / or prediction optimizer unit 20) configured to perform prediction optimizer functions, network node 16b (Cloud Management System), network node 16c (AI / ML Management System), network node 16d (Monitoring System), and network 12, 14 which may include other network nodes 16 such as network node 16e and be a Cloud System such as under management by one or more entities.

[0123] Network node 16a (and / or prediction optimizer unit 20) may be the Prediction Optimizer and be configured to return the training data size, prediction horizon, and sampling rate for training a fault prediction model, so that the performance of the model and the amount of resource used for collecting and storing the training data is balanced or optimized. Network node 16a (and / or prediction optimizer unit 20) can be realized as a component that interacts with network node 16b (Cloud Management System) and network node 16c (AI / ML Model Management System). Network node 16c (AI / ML Model Management System) may be configured for training and managing the AI / ML models and performing tasks such as data collection, feature selection, model training and management, etc. Network node 16b (Cloud Management System) may be configured to perform management tasks of cloud systems including resource management and fault management. The example process to optimize the data parameters may include one or more of the following steps:

[0124] S200. Fault management in network node 16b (Cloud Management System) requests for a fault prediction model from Network node 16c (AI / ML Model Management System).

[0125] S202. Network node 16c (AI / ML Model Management System) provides a selected model to network node 16a (and / or prediction optimizer unit 20).

[0126] S204. Network node 16a (and / or prediction optimizer unit 20) requests for resource constraints, and data size and sampling rate boundaries from the Resource Management component of network node 16b (Cloud Management System).

[0127] S206. Network node 16a (and / or prediction optimizer unit 20) requests for minimal accuracy requirement and prediction horizon boundaries from the Fault Management component of network node 16b (Cloud Management System).

[0128] S208. Network node 16a (and / or prediction optimizer unit 20) requests Data Collection of network node 16c (AI / ML Model Management System) for maximum data samples allowed with minimum data sampling rate based on the boundaries from S204.

[0129] S210. Network node 16a (and / or prediction optimizer unit 20) requests Feature Selection of network node 16c (AI / ML Model Management System) to select features for the data samples collected in S208.

[0130] S212. Network node 16a (and / or prediction optimizer unit 20) searches for the optimal prediction horizon, data sampling rate and data size so that the resource constraints and accuracy minimal requirements are met, and resource usage and accuracy are optimized.

[0131] S214. Model Management of network node 16c (AI / ML Model Management System) uses the data sampling rate and data size to collect data and uses the prediction horizon to build a Fault Prediction Model, and optimize model parameters using e.g., AutoML.

[0132] S216. Model Management of network node 16c (AI / ML Model Management System) deploys Fault Prediction Model to the network 12, 14, and / or network node 16e (Cloud System).

[0133] S218. The Fault Prediction Model reports the fault prediction results to Fault Management of network node 16b (Cloud Management System).

[0134] FIG. 7 shows an overview of an example prediction optimizer unit 20 according to some embodiments of the present disclosure. Prediction optimizer unit 20 (Prediction Optimizer) of network node 16 may be configured for automatically optimizing training data size, prediction horizon, and sampling rate for training a fault prediction model. Further, prediction optimizer unit 20 (Prediction Optimizer) of network node 16 may comprise and perform functions associated with one or more module or units or components, i.e., Optimization Request Handler 100, Search Space Identifier 102, Prediction Optimization Function 104, And Decision Maker 106. The inputs of the Prediction Optimizer system are the optimization request for a given prediction model, constraints, boundaries, and training data. The outputs of the system is one or more optimal combinations of the prediction horizon, data sampling rate and data size used for training the prediction model. These parameters optimize the model performance and resource consumption.

[0135] Optimization Request Handler 100 is configured for receiving the inputs of the prediction optimizer system. The inputs are the prediction optimization request of a given prediction model, resource consumption constraint showing the maximum amount of resource available for storing training data, accuracy constraint that shows the minimum performance of the prediction model that is acceptable, prediction horizon and data sampling rate and data size boundaries, and a maximum amount of training data with the minimum sampling rate within the boundaries.

[0136] Search Space Identifier 102 is configured to use the data sampling rate, data size, and prediction horizon boundaries to identify and initialize the search space of the optimizer (i.e., prediction optimizer unit 20). In some embodiments, the search space refers to a range, boundary, or a plurality of numbers / values and may be provided by a user such as a network operator. In some other embodiments, a search space corresponds to one of the data sampling rate, data size, and prediction horizon. For example, a first search space may correspond to the data sampling rate, a second search space may correspond to the data size, and a third search space may correspond to the prediction horizon. FIG. 8 shows example search spaces 200, where each search space 200 corresponds to a corresponding one of the data sampling rate 202, data size 204, and prediction horizon 206. More specifically, data sampling rate 202 (or data sampling rate boundary) corresponds to search space 200a, data size 204 (or data size boundary) corresponds to search space 200b, and prediction horizon 206 (or prediction horizon boundary) corresponds to search space 200c. That is, the boundary of the data sampling rate 202, data size 204, and prediction horizon 206 may be used to identify and initialize the corresponding search space 200 of the optimizer (i.e., prediction optimizer unit 20). The sampling rate 202 may be given in samples per time unit, the data size 204 may be given in number of samples, and the prediction horizon 206 may be given in time units. The optimizer (i.e., prediction optimizer unit 20) should search in each space to find the corresponding sampling rates, data size, and prediction horizons that minimize resource consumption while maximizing prediction model’s performance (e.g., accuracy).

[0137] Although the search spaces 200 are shown as having the same size, the embodiments are not limited as such and the search spaces 200 may have different sizes. Similarly, although the data sampling rate 202, data size 204, and prediction horizon 206 are shown as having the same size / boundary, the embodiments are not limited as such and the data sampling rate 202, data size 204, and prediction horizon 206 may have different sizes / boundaries.

[0138] Prediction Optimization Function 104, which may be a core component of the prediction optimizer system, is configured for finding the data sampling rate, data size, and prediction horizon values that optimize the resource consumption and model’s performance. The input of this component is the search space initialized by the Search Space Identifier 102, resource consumption constraint, accuracy constraints, and the training data received from the Optimization Request Handler 100. The output of this component is the values of the data sampling rate, data size, and prediction horizon that optimize resource consumption and accuracy. This component may be configured to determine and / or adjust and / or optimize at least two contradicting objectives at the same time (multi-objective), i.e., that improving one objective results in worsening the other objective. Therefore, instead of single values, the output is a set of data sampling rate, training data size and prediction horizons, i.e., a pareto front.

[0139] Decision Maker 106 is configured to receive the pareto front from the Prediction Optimization Function 104 as input and selects top k data sampling rate, data size, and prediction horizon value combinations (tuples) as the output, which could further be used for training the prediction model. This component can consider objective priorities or existing trade-off policies to make decisions.

[0140] Prediction Optimization Function 104

[0141] FIG. 9 is a flowchart of an example prediction optimization process (e.g., performed by Prediction Optimization Function 104) according to some embodiments of the present disclosure. At step S300, upon receiving the inputs, first, at least a solution of a set of solutions (data sampling rate, data size, and prediction horizon) within the identified search space is identified. At step S302, considering the input training data and the initialized solution (sampling rate, data size, and prediction horizon), how much resources would be consumed to collect and store the training data is estimated.

[0142] Resource consumption of storing training data can be proportional to the size of training data, sampling rate, and feature size. Therefore, the resource consumption can be estimated using an approximation function or a simple machine learning model that is trained offline to map the size of training data, sampling rate, and feature size to resource consumption.

[0143] Upon resource estimation, the Prediction Optimization Function 104 may use the prediction model, data, and the solution to calculate the accuracy of the prediction model, at step S304. This can be performed by training a prediction model or using a surrogate model to estimate the accuracy of the model considering its parameters.

[0144] At step S306, Prediction Optimization Function 104 may use the resource and accuracy values calculated in the previous steps to select the solution(s) that dominate(s) other solutions. One solution dominating other solutions refers to the first solution being at least as good as other solutions with respect to one objective (e.g., accuracy) and strictly better than other solutions with respect to the other objectives (e.g., resource consumption).

[0145] At step S308, the search termination criteria are checked to determine whether the search for optimal solution(s) should continue or not. The search termination criteria can be a maximum number of iterations or checking convergence through rate of changes in objective space. If the search termination condition returns false, the optimization function updates, at step S310, the solution by changing the values of data sampling rate, data size, and prediction horizon, and it repeats resource estimation, accuracy calculation, and dominated solution selection steps to continue searching for a better solution.

[0146] Otherwise, if the search termination condition returns true at step S312, the search for the solution would be terminated at step S314, and the latest solutions that dominated all other solution would be returned as the output of the Prediction Optimization Function 104. In some embodiments, the prediction optimization function may not find the pareto optimal solution and return non-pareto optimal, sub-optimal or local optimum solutions instead. In some other embodiments, the pareto optimal solution is not found if the parameters of the optimization algorithm that is performing the search is not properly tuned considering the complexity of the problem (size of search space, complexity of the prediction model, etc.), or the algorithm cannot explore the search space completely.

[0147] Further, if there are no feasible solutions considering the constraints on the accuracy and resource consumption set by the cloud service provider, at step S316, the Prediction Optimization Function 104 returns an empty set, where the cloud service provider would need to adjust the constraints. The pseudocodes of example algorithms for the Prediction Optimization Function 104 are described in the following sections.

[0148] Non-dominated Sorting Algorithm (NSGA)-II-based Optimization Algorithm

[0149] The following is an example algorithm for the Prediction Optimization Function based on NSGA-II for optimizing the resource consumption and accuracy of the prediction model by configuring data sampling rate, data size, and prediction horizon. NSGA-II is an algorithm that is usable to solve multi-objective optimization problems.

[0150] The example algorithm is as follows:

[0151] Example Algorithm NSGA-II-based Optimization Algorithm _

[0152] Input: Data ( ), Prediction Model (Af), Search Space (P), Resource Constraint (Rc), Accuracy Constraint (Ac)

[0153] Output: Pareto Front of data sampling rate (S), data size (A), and prediction horizon (H)

[0154] 1: Initialize the population Po of size L randomly within search space P, at time t02: while t < maximum iteration

[0155] 3: for each solution in the population do

[0156] 4: A <— calculate prediction accuracy considering M, D, S, N, H

[0157] 5: R <— estimate resource consumption considering D, S, N, H

[0158] 6: if A < Acor R>Rcthen 7: Discard the solution

[0159] 8: end if

[0160] 9: end for

[0161] 10: Find non-dominated fronts considering A and R for all solutions

[0162] 11: Select top L solutions (S, N, and H) as next population Pt+1

[0163] 12: Select L / 2 parents from population Pt+1and replace them with their off-

[0164] 13: t = t + l

[0165] 14: end while

[0166] 15: return solutions S, N and H in population Pt

[0167] In some embodiments, the NSGA-II-based optimization algorithm starts by randomly initializing a population of size L, which consists of the solutions (data sampling rate, data size, prediction horizon) within search space (P). To find the optimal solution, the fitness of each solution in the current population is assessed. This is done through measuring the accuracy (A) of a prediction model trained using the data-related parameters in the solution and estimating the resource consumption (R) for these parameters. If the accuracy and resource usage do not meet the specified constraints, the solution is discarded. Once the fitness of all solutions are evaluated, top solutions are identified using non-dominated sorting. To obtain the next generation, L / 2 of the top L solutions are selected to generate off-springs through cross-over and mutation. The new generation goes through fitness evaluations and repeats the process of evolution until a maximum number of iterations is reached. The non-dominated solutions in the final population are the optimal solutions found by the algorithm.

[0168] Particle Swarm Optimization (PSO)-based Optimization Algorithm The following is another example algorithm (i.e., example algorithm 3) for the

[0169] Prediction Optimization Function 104 based on PSO. This algorithm optimizes the resource consumption and accuracy of the prediction model through data sampling rate and prediction horizon configuration.

[0170] Algorithm: PSO-based Optimization Algorithm

[0171] Input: Data ( ), Prediction Model (M), Search Space (P), Resource Constraint Output: Pareto Front of data sampling rate (S), data size (A), and prediction horizon

[0172] 1: Initialize the velocity and set of particles with S, N and H values with space P 2: while t < maximum iteration

[0173] 3: for each particle do 4: A <— calculate prediction accuracy considering M, D, S, N, H

[0174] 5: R <— estimate resource consumption considering D, S, N, H

[0175] 6: if A < Acor R>Rcthen

[0176] 7: Discard the solution

[0177] 8: end if

[0178] 9: Find non-dominated solution considering A and R and update personal

[0179] 10: end for

[0180] 11: Update global best by choosing the particle whose solution is non-

[0181] 12: Update particle position (S, N and H) considering velocity and global best

[0182] 13: t = t + l

[0183] 14: End

[0184] 15: return the latest non-dominated solutions (S, N and H front)

[0185] In some embodiments, the PSO-based optimization algorithm starts by randomly sampling a set of solutions or particles (data sampling rate, prediction horizon, training data size) from the search space (P) and assigning a random velocity to each solution. The fitness of each solution is assessed through measuring the accuracy (A) of a prediction model trained using the data-related parameters in the particle, and estimating the resource consumption (R) for these parameters. If the accuracy and resource usage do not meet the specified constraints, the particle is discarded. The non-dominated particles are identified using the value of A and R. To find the optimal answer, the non-dominated particles are identified as global best solutions. The position of each particle (data sampling rate, prediction horizon, training data size) is updated to move toward the non-dominated solutions or global bests considering the velocity of that particle. The process of assessing the fitness of new particles and updating their position continues until a maximum number of iterations is reached. The algorithm returns the non-dominated particles after the last update of the particle positions.

[0186] Example of Prediction Optimizer system in 5G core

[0187] Network node 16 (e.g., comprising prediction optimizer unit 20) may be deployed a network such as core network 14 (in a 5G core network) or any other network such as a cloud network.

[0188] FIG. 10 is a flowchart of another example prediction optimization process, where 5G core is deployed in cloud and a prediction model is to be trained to predict call drop rates of users (e.g., UEs 42) using 5G core (e.g., core network 14). Network node 16a (and / or Prediction Optimizer unit 20), network node 16b (Cloud Management System), and network node 16c (AI / ML Model Management System) are shown. Network node 16a (and / or Prediction Optimizer unit 20) may include Optimization Request Handler 100, Search Space Identifier 102, Prediction Optimization Function 104, and Decision maker 106. The process to optimize data sampling rate, data size, and prediction horizon may include one or more of the following steps:

[0189] S400. Network node 16b (Cloud Management System) sends a request to network node 16c (AI / ML Model Management System) for a model that can predict call drop rate.

[0190] S402. The network node 16c (AI / ML Model Management System) decides the type of prediction model (LSTM) and sends an optimization request to the Optimization Request Handler 100 of network node 16a (and / or Prediction Optimizer unit 20).

[0191] S404. To get information for the optimization process, the Optimization Request Handler 100 sends a request to the network node 16b (Cloud Management System) to get the constraints and boundaries associated with this request. The constraints include the maximum amount of storage allowed and minimum acceptable accuracy. The boundaries include the ranges for prediction horizon, data sampling rate, and data size.

[0192] S406. The network node 16b (Cloud Management System) responds to this request, clarifying that the maximum amount of resources consumption should be less than 10MB, the minimum acceptable accuracy is 85%, the model should predict at least 2 minutes and at most 1 hour into the future (prediction horizon), the sampling rate should be between 1 sample every 5 seconds and 1 sample every 1 hour, and the data size can range between 1000 and 5000 data samples.

[0193] S408. The Optimization Request Handler 100 further requests the network node 16c (AI / ML Model Management System) for 5000 samples of training data with minimum sampling rate, 5 seconds.

[0194] S410. The network node 16c (AI / ML Model Management System) answers this request by returning 5000 data samples that includes 10 features and the call drop rate key performance indicator (KPI).

[0195] S412. The optimization process starts by initializing (e.g., with Search Space Identifier 102) a search space for prediction horizon between 2 to 60 minutes, data sampling rate between one sample every 5 to 3600 seconds, and data size between 1000 and 5000.

[0196] S414. The Optimization Request Handler 100 sends an optimization request to the Prediction Optimization Function 104 for an LSTM model with 10MB and 85% constraints on resource and accuracy respectively and maximum 5000 training data samples.

[0197] S416. The Prediction Optimization Function 104 sends a request to the Search Space Identifier 102 to ask for the optimization search space.

[0198] S418. The Search Space Identifier 102 answers this request by returning the search space identified in step S412.

[0199] S420. The Prediction Optimization Function 104 uses an NSGA-II-based Algorithm or other algorithm to find prediction horizon, data sampling rate, and data size pareto front that maximizes the accuracy of the call drop rate prediction (LSTM) model while minimizing the resource consumption for storing the data samples.

[0200] S422. The pareto front is sent to the Decision Maker 106 to finalize the value of prediction horizon, data sampling rate, and training data size for training the model.

[0201] S424. The Decision Maker 106 returns the solution (a 5-minute-long prediction horizon, 10 seconds sampling rate, and 4000 data samples) to the Optimization Request Handler 100.

[0202] S426. The Optimization Request Handler 100 sends the optimized solution to the AI / ML Management System.

[0203] S428. The network node 16c (AI / ML Model Management System) collects 4000 data samples with a 10 second sampling rate and trains an LSTM-based call drop rate prediction model that can predict 5 minutes ahead.

[0204] S430. The network node 16c (AI / ML Model Management System) returns the call drop rate prediction model to the Cloud Management System for deployment.

[0205] In some embodiments, system 10 (Prediction Optimizer System) and / or network node 16 comprises software functions that can be implemented in cloud infrastructure. The system components can be implemented e.g., as containers in Kubernetes or virtual machines (VMs) in OpenStack. As will be appreciated by one of skill in the art, the concepts described herein may be embodied as a method, data processing system, computer program product and / or computer storage media storing an executable computer program. Accordingly, the concepts described herein may take the form of an entirely hardware embodiment, an entirely software embodiment or an embodiment combining software and hardware aspects all generally referred to herein as a “circuit” or “module.” Any process, step, action and / or functionality described herein may be performed by, and / or associated to, a corresponding module, which may be implemented in software and / or firmware and / or hardware. Furthermore, the disclosure may take the form of a computer program product on a tangible computer usable storage medium having computer program code embodied in the medium that can be executed by a computer. Any suitable tangible computer readable medium may be utilized including hard disks, CD-ROMs, electronic storage devices, optical storage devices, or magnetic storage devices.

[0206] Some embodiments are described herein with reference to flowchart illustrations and / or block diagrams of methods, systems and computer program products. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions may be provided to a processor of a general purpose computer (to thereby create a special purpose computer), special purpose computer, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, create means for implementing the functions / acts specified in the flowchart and / or block diagram block or blocks.

[0207] These computer program instructions may also be stored in a computer readable memory or storage medium that can direct a computer or other programmable data processing apparatus to function in a particular manner, such that the instructions stored in the computer readable memory produce an article of manufacture including instruction means which implement the function / act specified in the flowchart and / or block diagram block or blocks.

[0208] The computer program instructions may also be loaded onto a computer or other programmable data processing apparatus to cause a series of operational steps to be performed on the computer or other programmable apparatus to produce a computer implemented process such that the instructions which execute on the computer or other programmable apparatus provide steps for implementing the functions / acts specified in the flowchart and / or block diagram block or blocks.

[0209] It is to be understood that the functions / acts noted in the blocks may occur out of the order noted in the operational illustrations. For example, two blocks shown in succession may in fact be executed substantially concurrently or the blocks may sometimes be executed in the reverse order, depending upon the functionality / acts involved. Although some of the diagrams include arrows on communication paths to show a primary direction of communication, it is to be understood that communication may occur in the opposite direction to the depicted arrows.

[0210] Computer program code for carrying out operations of the concepts described herein may be written in an object oriented programming language such as Python, Java® or C++. However, the computer program code for carrying out operations of the disclosure may also be written in conventional procedural programming languages, such as the "C" programming language. The program code may execute entirely on the user's computer, partly on the user's computer, as a stand-alone software package, partly on the user's computer and partly on a remote computer or entirely on the remote computer. In the latter scenario, the remote computer may be connected to the user's computer through a local area network (LAN) or a wide area network (WAN), or the connection may be made to an external computer (for example, through the Internet using an Internet Service Provider).

[0211] Many different embodiments have been disclosed herein, in connection with the above description and the drawings. It will be understood that it would be unduly repetitious and obfuscating to literally describe and illustrate every combination and subcombination of these embodiments. Accordingly, all embodiments can be combined in any way and / or combination, and the present specification, including the drawings, shall be construed to constitute a complete written description of all combinations and subcombinations of the embodiments described herein, and of the manner and process of making and using them, and shall support claims to any such combination or subcombination.

[0212] Abbreviations that may be used in the preceding description include:

[0213] AI / ML Artificial Intelligence / Machine Learning

[0214] AutoML Automated Machine Learning

[0215] HPO Hyper-Parameter Optimization KPI Key Performance Indicator

[0216] LSTM Long Short-Term Memory

[0217] MAE Mean Absolute Error

[0218] NAS Neural Architecture Search NSGA Non-Dominated Sorting Algorithm

[0219] PSO Particle Swarm Optimization

[0220] QoS Quality-of- Service

[0221] It will be appreciated by persons skilled in the art that the embodiments described herein are not limited to what has been particularly shown and described herein above. In addition, unless mention was made above to the contrary, it should be noted that all of the accompanying drawings are not to scale. A variety of modifications and variations are possible in light of the above teachings and following claims.

Claims

What is claimed is:

1. A method in a network node (16) configured to determine one or more parameters usable for training an artificial intelligence, Al, prediction model, the method comprising: automatically determining (SI 00) a data sampling rate, a training data size, and a prediction horizon for training the Al prediction model, the data sampling rate, the training data size, and the prediction horizon meeting one or more objectives; and performing (SI 02) one or more actions based on the data sampling rate, the training data size, and the prediction horizon.

2. The method of Claim 1, wherein the one or more objectives include one or more of a model performance and a data collection network and storage resource utilization with one or both of a minimal prediction accuracy and a maximum data collection resource utilization constraint for training the Al prediction model.

3. The method of any one of Claims 1 and 2, wherein automatically determining includes: searching for the data sampling rate, the training data size, and the prediction horizon that meet one or more resource constraints and accuracy minimal requirements and optimize resource usage and Al prediction model accuracy.

4. The method of any one of Claims 1-3, wherein the method further includes, to perform the automatic determination, obtaining one or more of: the Al prediction model; one or more resource constraints, one or more data size boundaries, and one or more data sampling rate boundaries; a minimal accuracy requirement and one or more prediction horizon boundaries; data collection for one or more maximum data samples that are allowed with a minimum data sampling rate based on the one or more data size boundaries and the one or more sampling rate boundaries; and a feature selection set to select one or more features for one or more data samples associated with the one or more maximum data samples.

5. The method of any one of Claims 1-4, wherein the one or more actions includes: determining a set of combinations of a plurality of combinations of the data sampling rate, a training data size, and the prediction horizon used for training the Al prediction model and optimizing model performance and resource consumption.

6. The method of any one of Claims 1-5, wherein the one or more actions include causing another network node (16) to one or more of: use the data sampling rate and the training data size to collect data; use the prediction horizon to build the Al prediction model, the Al prediction model being a fault prediction model; deploy the fault prediction model to a cloud system; and report fault prediction results.

7. The method of any one of Claims 1-6, wherein the one or more objectives includes a first optimized objective and a second optimized objective that is contradicting to the first optimized objective.

8. The method of any one of Claims 1-7, wherein the method further includes one or more of: initializing a set of solutions comprising the data sampling rate, the training data size, and the prediction horizon; estimating one or more resources used for collecting and storing training data based on the set of solutions; determining an accuracy of the Al prediction model based on the set of solutions; selecting at least one solution that dominates other solutions of the set of solutions; determining whether a search for an optimal solution is to continue based on search termination criteria and the selected at least one solution; if the search termination criteria is not met, updating the data sampling rate, the training data size, and the prediction horizon of the selected at least one solution; and if the search termination criteria is met, terminating the search and returning the latest solutions that dominate the other solutions or an empty set.

9. The method of any one of Claims 1-8, wherein the automatic determination is based on one or both of a Non-dominated Sorting Algorithm, NSGA, and a Particle Swarm Optimization, PSO.

10. The method of any one of Claims 1-9, wherein the Al prediction model is trained to predict call drop rates associated with one or more user equipment, UEs, (42) using a core network (14).

11. A network node (16) configured to determine one or more parameters usable for training an artificial intelligence, Al, prediction model, the network node (16) being configured to: automatically determine a data sampling rate, a training data size, and a prediction horizon for training the Al prediction model, the data sampling rate, the training data size, and the prediction horizon meeting one or more objectives; and perform one or more actions based on the data sampling rate, the training data size, and the prediction horizon.

12. The network node (16) of Claim 11, wherein the one or more objectives include one or more of a model performance and a data collection network and storage resource utilization with one or both of a minimal prediction accuracy and a maximum data collection resource utilization constraint for training the Al prediction model.

13. The network node (16) of any one of Claims 11 and 12, wherein automatically determining includes: searching for the data sampling rate, the training data size, and the prediction horizon that meet one or more resource constraints and accuracy minimal requirements and optimize resource usage and Al prediction model accuracy.

14. The network node (16) of any one of Claims 11-13, wherein for performing the automatic determination, the network node (16) is configured to obtain one or more of: the Al prediction model; one or more resource constraints, one or more data size boundaries, and one or more data sampling rate boundaries; a minimal accuracy requirement and one or more prediction horizon boundaries;data collection for one or more maximum data samples that are allowed with a minimum data sampling rate based on the one or more data size boundaries and the one or more sampling rate boundaries; and a feature selection set to select one or more features for one or more data samples associated with the one or more maximum data samples.

15. The network node (16) of any one of Claims 11-14, wherein the one or more actions includes: determining a set of combinations of a plurality of combinations of the data sampling rate, a training data size, and the prediction horizon used for training the Al prediction model and optimizing model performance and resource consumption.

16. The network node (16) of any one of Claims 11-15, wherein the one or more actions include causing another network node (16) to one or more of: use the data sampling rate and the training data size to collect data; use the prediction horizon to build the Al prediction model, the Al prediction model being a fault prediction model; deploy the fault prediction model to a cloud system; and report fault prediction results.

17. The network node (16) of any one of Claims 11-16, wherein the one or more objectives include a first optimized objective and a second optimized objective that is contradicting to the first optimized objective.

18. The network node (16) of any one of Claims 11-17, wherein the network node (16) is further configured to one or more of: initialize a set of solutions comprising the data sampling rate, the training data size, and the prediction horizon; estimate one or more resources used for collecting and storing training data based on the set of solutions; determine an accuracy of the Al prediction model based on the set of solutions; select at least one solution that dominates other solutions of the set of solutions; determine whether a search for an optimal solution is to continue based on search termination criteria and the selected at least one solution;if the search termination criteria is not met, update the data sampling rate, the training data size, and the prediction horizon of the selected at least one solution; and if the search termination criteria is met, terminate the search and return the latest solutions that dominate the other solutions or an empty set.

19. The network node (16) of any one of Claims 11-18, wherein the automatic determination is based on one or both of a Non-dominated Sorting Algorithm, NSGA, and a Particle Swarm Optimization, PSO.

20. The network node (16) of any one of Claims 11-19, wherein the Al prediction model is trained to predict call drop rates associated with one or more user equipment, UEs, (42) using a core network (14).

21. A system (10) comprising a first network node (16) configured to determine one or more parameters usable for training an artificial intelligence, Al, prediction model, and communicate with a second network node (16), a third network node (16), and a fourth network node (16): the first network node (16) being configured to: receive, from the third network node (16), the Al prediction model; receive, from the second network node (16), one or more resource constraints, one or more data size boundaries, and one or more sampling rate boundaries; receive, from the second network node (16), a minimal accuracy requirement and one or more prediction horizon boundaries; receive, from third network node (16), data collection for one or more maximum data samples that are allowed with a minimum data sampling rate based on the one or more data size boundaries and the one or more sampling rate boundaries; receive, from the third network node (16), a feature selection set to select one or more features for one or more data samples associated with the one or more maximum data samples; automatically determine a data sampling rate, a training data size, and a prediction horizon for training the Al prediction model, the data sampling rate, the training data size, and the prediction horizon meeting one or more objectives, the automatic determination being based on the one or more resource constraints, the one or more data size boundaries, the one or more sampling rate boundaries, the minimal accuracyrequirement, the one or more prediction horizon boundaries, the data collection, and the feature selection set; and perform one or more actions based on the data sampling rate, the training data size, and the prediction horizon; the third network node (16) being configured to: collect data using the data sampling rate and the training data size; build the Al prediction model using the prediction horizon, the Al prediction model being a fault prediction model; and deploy the fault prediction model. the fourth network node (16) being configured to: report fault prediction results to the second network node (16);22. The system (10) of Claim 21, wherein one or more of: the second network node (16) is cloud management system; the second network node (16) is configured to perform resource management and fault management; the third network node (16) is an Al management system; the third network node (16) is configured to perform data collection, feature selection, and model management; and the fourth network node (16) is comprised in a cloud system configurable for management by one or more entities.

Images