Machine-learning model(s) for estimating ran functionality machine learning model impact on performance measurement counters

Abstract

An apparatus 100 comprising: means for receiving a network configuration 106 derived from a plurality of machine-learning, ML models, each ML model directed towards a respective one or more radio access network, RAN functionalities; means for receiving a plurality of predicted performance, PM measurement counters output 108 from a plurality of ML performance measurement models, each ML prediction measurement model corresponding to one of the plurality of ML models; and means for processing, using a common ML performance measurement counter model 102, the network configuration and the plurality of predicted performance measurement counters to determine a model output comprising, for one or more performance measurement counters, a respective plurality of impact scores 112, wherein each impact score is indicative of a predicted impact of a corresponding ML model in the plurality of ML models on the respective performance measurement counter of said impact score for the network configuration. The apparatus may further comprise means for executing the plurality of ML models on respective measurement data to generate a plurality of respective RAN functionality predictions; and means for generating, from the plurality of respective RAN functionality predictions, the network configuration.

Description

Technical Field This specification relates to the execution of machine-learning models on user devices. Background Performance Measurement (PM) Counters are defined in 3GPP TS 28.552. PM counters are typically triggered in a gNB, stored in PM file with a configured measurement period sample, and then uploaded via a Northbound interface out of the gNB toward a mobile operator Operations, Administration and Management (OAM) system. PM counters are used by operators for network correctness monitoring and taking some actions to fix the possible identified issues / problems. Typically, network monitoring based on PM counters is / may be followed with some changes in a network configuration. Artificial Intelligence (AI) / Machine Learning (ML) models are intended to replace human impacts in configuring a network. For example, in the New Generation - Radio Access Network (NG-RAN), different AI / ML models may be deployed within a 5G Node B gNB (e.g., at the gNB-Central Unit - gNB-CU) and are intended to be (concurrently) used for different functionalities. Examples include Mobility Optimization, Load Balancing and Network Energy Saving (see, for example, ref. 3GPP TR 37.817). 3GPP RAN3 Rei. 19 AI / ML for NG-RAN work further includes two new use cases for AI / ML models, namely, Network Slicing and Coverage and Capacity Optimization (CCO). Summary According to a first aspect of this specification, there is described apparatus comprising: means for receiving a network configuration derived from a plurality of machine-learning models, each machine-learning model directed towards a respective one or more radio access network functionalities; means for receiving a plurality of predicted performance measurement counters output from a plurality of machinelearning performance measurement models, each machine-learning prediction measurement model corresponding to one of the plurality of machine-learning models; and means for processing, using a common machine-learning performance measurement counter model, the network configuration and the plurality of predicted performance measurement counters to determine a model output comprising, for one or more performance measurement counters, a respective plurality of impact scores. Each impact score is indicative of a predicted impact of a corresponding machinelearning model in the plurality of machine-learning models on the respective performance measurement counter of said impact score for the network configuration. The apparatus may further comprise: means for executing the plurality of machinelearning models on respective measurement data to generate a plurality of respective radio access network functionality predictions; and means for generating, from the plurality of respective radio access network functionality predictions, the network configuration. The means for executing the plurality of machine-learning models on the respective measurement data to generate the plurality of respective radio access network functionality predictions may comprise: means for executing two or more of the plurality of machine-learning models in parallel. The apparatus may further comprise: means for executing the plurality of machinelearning performance measurement models on respective radio access network functionality predictions output by a respective machine-learning model in the plurality of machine-learning models. The one or more radio access network functionalities may comprise one or more of: energy saving; load balancing; and / or user equipment mobility optimisation. The apparatus may further comprise: means for disabling one or more machinelearning models in the plurality of machine-learning models based at least in part on the respective impact scores of said one or more machine-learning models. The apparatus may further comprise: means for selecting between two or more machine-learning models in the plurality of machine-learning models based at least in part on the respective impact scores of said one or more machine-learning models, wherein said two or more machine-learning models are directed towards the same radio access network functionality. The apparatus may further comprise: means for incrementing a performance measurement counter bin based at least in part on one or more of the impact scores in the plurality of impact scores. The common machine-learning performance measurement counter model may be hosted in a gNB. The common machine-learning performance measurement counter model may be hosted in a Management Data Analytics Function, MDAF, in an Operations Administration Management, OAM, system. The apparatus may further comprise: means for receiving input data for the common machine-learning performance measurement from a gNB. According to a further aspect of this specification, there is described apparatus comprising: means for jointly training a plurality of pairs of machine-leaning models, wherein each pair of machine-learning models comprises a respective first machinelearning model for providing one or more radio access functionalities and a respective second machine-learning model for predicting one or more performance measurement counters based on the output of the respective first machine-learning model of the pair; means for generating a set of training data for a combined performance measurement counter model using the plurality of pairs of machine-learning models, the training data comprising a set of training examples, each training example comprising: a network configuration parameter; one or more corresponding performance measurement counters output by a respective one or more second machine-learning models; and one or more combined performance measurement counter values; and means fortraining the combined performance measurement counter model using the training data to predict a plurality of impact scores for each of one or more performance measurement counters, each impact score indicative of a predicted impact of a corresponding machine learning model in the plurality of machine-learned models on a respective performance measurement counter. The means for generating a set of training data fora combined performance measurement counter model using the plurality of pairs of machine-learning models may comprise: means for generating the set of training data using a linearised in parts method. The linearised in parts method may comprise: fora plurality of first machinelearning models: incrementing a network configuration parameter associated with a first machine-learning model by one granularity step; and determining a one or more respective combined performance measurement counter values for said first machinelearning models. The linearised in parts method may further comprise: for each of the plurality of first machine-learning models, determining a corresponding performance measurement counter for the network configuration parameter using the second machine-learning model corresponding to the first machine-learning model. Determining one or more respective combined performance measurement counter values for said first machine-learning models may comprise simulating a network environment with a set of network configuration parameters comprising the incremented network configuration parameter. The means fortraining the combined performance measurement counter model may comprise: means for applying a supervised learning method to the combined performance measurement counter model using the training data. According to a further aspect of this specification, there is described apparatus comprising: means for generating, from a set of network measurements and using a plurality of machine-learning models, a set of network actions, each network action associated with a respective time, wherein each machine learning model directed towards a respective one or more radio access network functionalities; and means for processing, using a combined performance measurement counter model, the set of network actions to generate one or more average performance measurement counters over a time interval spanned by the respective time of the earliest network action in the set of network actions and the respective time of the latest network action in the set of network actions. The means for generating, from a set of network measurements and using a plurality of machine-learning models, the set of network actions may comprise: means for executing the one or more machine-learning models on respective sets of input data to generate respective sets of output data; and means for generating, based on the respective sets of output data, one or more respective network actions. The apparatus may further comprise: means for performing the one or more respective network actions. The apparatus may further comprise: means for generating a time-ordered set of input data for the combined performance measurement counter model, the generating comprising time-ordering the set of network actions and the respective output data. The combined performance measurement counter model may processe the time-ordered set of input data to generate the one or more average performance measurement counters. The time-ordered set of input data for the combined performance measurement counter model may further comprise one or more of: a set of action times; a set of time differences between consecutive actions; a set of delay times between respective output data times and the corresponding action being taken; and / or a set of delay times between input data to a machine-learning model being received and the respective set of output data being generated. Generating one or more average performance measurement counters may be further based on: one or more network parameters; and / or one or more individual performance measurement counter predictions. According to a further aspect of this specification, there is described a computer implemented method comprising: receiving a network configuration derived from a plurality of machine-learning models, each machine-learning model directed towards a respective one or more radio access network functionalities; receiving a plurality of predicted performance measurement counters output from a plurality of machinelearning performance measurement models, each machine-learning prediction measurement model corresponding to one of the plurality of machine-learning models; and processing, using a common machine-learning performance measurement counter model, the network configuration and the plurality of predicted performance measurement counters to determine a model output comprising, for one or more performance measurement counters, a respective plurality of impact scores. Each impact score is indicative of a predicted impact of a corresponding machine-learning model in the plurality of machine-learning models on the respective performance measurement counter of said impact score for the network configuration. According to a further aspect of this specification, there is described a computer implemented method comprising: jointly training a plurality of pairs of machineleaning models, wherein each pair of machine-learning models comprises a respective first machine-learning model for providing one or more radio access functionalities and a respective second machine-learning model for predicting one or more performance measurement counters based on the output of the respective first machine-learning model of the pair; generating a set of training data fora combined performance measurement counter model using the plurality of pairs of machine-learning models, the training data comprising a set of training examples, each training example comprising: a network configuration parameter; one or more corresponding performance measurement counters output by a respective one or more second machine-learning models; and one or more combined performance measurement counter values; and training the combined performance measurement counter model using the training data to predict a plurality of impact scores for each of one or more performance measurement counters, each impact score indicative of a predicted impact of a corresponding machine learning model in the plurality of machine-learning models on a respective performance measurement counter. According to a further aspect of this specification, there is described a computer implemented method comprising: generating, from a set of network measurements and using a plurality of machine-learning models, a set of network actions, each network action associated with a respective time, wherein each machine learning model directed towards a respective one or more radio access network functionalities; and processing, using a combined performance measurement counter model, the set of network actions to generate one or more average performance measurement counters over a time interval spanned by the respective time of the earliest network action in the set of network actions and the respective time of the latest network action in the set of network actions. According to a further aspect of this specification, there is described a computer program product (e.g., a non-transitory computer readable medium) storing computer readable instructions that, when executed by a computer, cause the computer to perform any one or more of the methods described herein. Brief Description of the Drawings Example embodiments will now be described by way of non-limiting example, with reference to the accompanying drawings, in which: FIG. 1 shows an overview of an example system / method 100 for predicting the impact of machine learning models on PM counters; FIG. 2 shows an overview of an example method 200 fortraining a common AI / ML model for predicting impact of AI / ML models dedicated to RAN functionalities to PM Counters; FIG. 3 shows an example of a method for generating training data for a common AI / ML model for predicting impact of AI / ML models dedicated to RAN functionalities to PM Counters using a "linearised in parts" process; FIG. 4 shows an overview of a further example system / method 100 for predicting the impact of AI / ML models dedicated to RAN functionalities on PM counters; FIG. 5 shows a flow diagram of an example method for predicting the impact of AI / ML models dedicated to RAN functionalities on PM counters; FIG. 6 shows a flow diagram of an example method fortraining a common AI / ML model for predicting impact of AI / ML models dedicated to RAN functionalities to PM Counters; FIG. 7 shows a flow diagram of a further example method for predicting the impact of AI / ML models dedicated to RAN functionalities on PM counters; FIG. 8 shows an apparatus / system according to some example embodiments, which may form at least a part of a user device, network node, base station, or gNB; and FIG. 9 shows a non-transitory media according to some embodiments. Detailed Description Currently, (i.e., without AI / ML capability in the NG-RAN), rule-based systems for network configuration in a Radio Access Network (RAN) may be run in parallel, as their configuration is performed by the mobile network operator. The operator has full control and, thus, may evaluate how the impact of an executed network configuration change has on the related PM counter(s) evolves overtime. AI / ML-based systems are expected to provide a number of advantages in network configuration (see, for example, 3GPP TR 37.817). As an example, AI / ML-based load balancing can predict cell load in a gNB at a future point in time and consequently take offloading actions by handing over some UEs to neighbouring gNBs. This can provide higher quality user experience and improve the system capacity than, e.g., a reactive rules-based approach. As a further example, AI / ML-based network energy saving can predict the energy efficiency and load state at a future point in time, which can be used to make better decisions on cell activation / deactivation for energy saving. Based on the predicted load, the system may dynamically configure the energy-saving strategy (e.g., the switch-off timing and granularity, offloading actions) to keep a balance between system performance and energy efficiency and to reduce the energy consumption. However, since different RAN functionality (or task) specific AI / ML models can be concurrently operating during the inference phase (i.e., during NG-RAN operation), more than one AI / ML model may simultaneously impact the same PM counter(s). This may be the case even in the case where one AI / ML model is intended to be used for a very specific functionality by issuing inference output, e.g., in the form of predictions or recommendations to an Actor entity (i.e., an NG-RAN node) that can take an action based on (or influenced by) this inference output. That is, an AI / ML model in the NG-RAN, when used in inference phase may impact a PM counter either intentionally (i.e., as part of the functionality to be optimized) or unintentionally. Typical example may be an AI / ML model dedicated to UE mobility optimization in the network via proper tuning of the parameters to avoid too late / too early handover, handover to wrong cell and / or ping pong effects. In parallel, there may be other AI / ML model(s) dedicated to load balancing and / or energy saving and / or other RAN functionalities which may also change some mobility conditions. In other words, with AI / ML, in contrast to rule-based systems, the possible network configuration changes may be performed automatically, and may even be done at the same time for some of functionalities. Thus, it can be challenging for the operator to hold an AI / ML model dedicated to a RAN functionality accountable for PM counter improvement (or drop). Such induced ambiguities may further contribute to lack of trust mobile network operators may have on AI / ML technology, as it is a newly introduced feature. Especially some conservative operators may still prefer to have at least full network monitoring possibility and, thus, see how the executed AI / ML-driven (or, AI / ML-based) network configuration impacts the PM counter(s) over operation time. An exemplary advantage of such (AI / ML-based) performance monitoring capability, as reflected by (evolving) PM Counter impact information, is that it can trigger fallback to non-AI / ML-based network operation for at least some RAN functionalities. This specification describes systems, apparatus and methods for allowing an operator (e.g., via the OAM) to: • understand the impact of each AI / ML-based RAN functionality in a gNB based on the same measured PM counter(s); • detect and mitigate conflicting AI / ML-based actions in a gNB based on the same measured PM counter(s), i.e., to detect whether one AI / ML-based RAN functionality has positive effect while other AI / ML-based RAN functionality has negative effect on the same measured PM counter; • compare non-AI / ML-based RAN functionality A versus AI / ML-based RAN functionality A based on the same measured PM counter(s), i.e., to detect whether AI / ML-based RAN functionality A in gNodeB has a positive or negative effect on the measured PM counter, and to activate / deactivate AI / ML-based RAN functionality on the basis of said comparison; • compare one AI / ML-based RAN functionality A (ML model version 1) versus another AI / ML-based RAN functionality A (ML model version 2) based on the same measured PM counter, i.e., to detect whether one AI / ML-based RAN functionality A in gNodeB has better effect on the measured PM counter compared to another AI / ML- based RAN functionality A (of a different AI / ML model version), and to select a ML model based on said comparison; • determine if a particular AI / ML-based RAN functionality in the gNB needs to be paused / deactivated (e.g., to avoid conflicts, to avoid negative impacts) by configuring the gNodeB to fall back to the non-AI / ML-based RAN functionality; and • determine if a particular AI / ML model needs to be updated / re-trained due to negative impact of the AI / ML-based RAN functionality on the PM counter; this may be in relation to the performance of the AI / ML model itself. FIG. 1 shows an overview of an example system / method 100 for predicting the impact of machine learning models on PM counters. A common AI / ML model for PM counter(s) 102 (also referred to herein as a "AI / ML model for predicting impact of AI / ML models dedicated to RAN functionalities to PM Counters") receives input data 104 comprising a set of network configuration parameters 106 and a set of PM counter predictions 108. In some examples, the input data 104 further comprises an indication 110 of machine-learning models from which the set of network configuration parameters 106 were derived. The common AI / ML model for PM counter(s) 102 processes the input data 104 to generate a set of output data 112. The set of output data 112 comprises a plurality of impact scores 114A-N for each of one or more PM counters, where each impact score indicates an impact of a corresponding AI / ML model dedicated to a RAN functionality on its respective PM counter. The network configuration parameters 106 may comprise a set of network parameters derived from the output of a plurality of AI / ML models dedicated to a RAN functionalities. Each of the latter models is dedicated to a respective RAN functionality, e.g., UE mobility optimization, load balancing, energy saving and / or other RAN functionalities. Although each machine-learning model is dedicated to a respective network functionality, the machine-learning model can impact other network functionalities as well. Each AI / ML model dedicated to a RAN functionality receives respective input data comprising a set of one or more network measurements and processes the input data to generate a set of output data comprising one or more predicted and / or target network configuration parameters. A plurality of the AI / ML models dedicated to a RAN functionalities may be run in parallel. Examples of such AI / ML models dedicated to a RAN functionalities are described in 3GPP TR 37.817. For an AI / ML-based energy saving use case, the outputs of an AI / ML models dedicated to this functionality may comprise one or more of: (i) a beam on-off prediction; (ii) a cell on-off prediction; (iii) a candidate beam to handover the traffic before switching off the serving beam; (iv) a candidate cell to handover the traffic before switching off the serving cell; (v) a cell energy cost prediction; (vi) a gNodeB energy cost prediction; (vii) a cell load prediction; and / or (viii) an energy efficiency prediction. For an AI / ML-based load balancing use case, the outputs of an AI / ML models dedicated to this functionality may comprise one or more of: (i) a selection of a target cell for load balancing; (ii) predicted UE(s) selected to be handed over to target NG-RAN node; and / or (iii) a predicted resource status information of neighbouring NG-RAN node(s). Foran AI / ML-based mobility use case, the outputs of an AI / ML models dedicated to this functionality may comprise one or more of: (i) a UE trajectory prediction (Latitude, longitude, altitude, cell ID of UE over a future period of time); (ii) a predicted handover target node, candidate cells in Condition handover; (iii) a UE traffic prediction; and / or (iv) a predicted resource reservation time window for conditional handover. Many other examples are possible. The PM counter predictions are each output from a respective machine-learning model for PM counter prediction that corresponds to one of the plurality of AI / ML models dedicated to RAN functionalities. Each machine-learning model for PM counter prediction processes respective input data comprising the output of the corresponding AI / ML model dedicated to a RAN functionality to generate the PM counter prediction for said AI / ML model dedicated to a RAN functionality. The respective input data for a machine-learning model for PM counter prediction may further comprise the input to the corresponding AI / ML model dedicated to a RAN functionality, e.g., the set of one or more network measurements. The AI / ML models dedicated to a RAN functionality and the machine-learning models for PM counter prediction may have been jointly trained, as described in relation to FIG. 2. The AI / ML models dedicated to RAN functionalities and the machine-learning models for PM counter prediction may comprise one or more of: a neural network (e.g., a fully connected network, a convolutional neural network, a recurrent neural network, a transformer network, a residual network, and / or the like); a decision tree model; a random forest model; a support vector machine; and / or the like. The network configuration parameters 106 and the PM counter predictions 108 are input into a common AI / ML model for PM counter(s) 102. In some examples, an indication of the model identities 110 of the AI / ML models dedicated to RAN functionalities may also be input into the AI / ML model for PM counter(s) 102, e.g., a vector whose components each correspond to a respective AI / ML model dedicated to a RAN functionality, with a value of one indicating that the AI / ML models dedicated to a RAN functionality corresponding to that vector component is enabled, and a value of zero indicating that the AI / ML model dedicated to a RAN functionality corresponding to that vector component is disabled. The common AI model for PM counter(s) 102 processes the input data 104 to generate one or more impact scores 114A-N for each of one or more performance measurement counters. Each impact score indicates a contribution of a respective AI / ML model dedicated to a RAN functionality on a respective performance measurement counter. In some examples, an impact score may be a numerical score, e.g., a number between zero and one indicating a fraction of an overall PM counter value attributable to the respective AI / ML model dedicated to a RAN functionality. Alternatively, an impact score may be a categorical value, e.g., "low", "medium", "high" indicating a level of contribution to an overall PM counter value attributable to the respective AI / ML model dedicated to a RAN functionality. Based on the impact scores, a network may take one or more network actions. For example, for each ML model deployed in network (e.g., a gNB) its impact on a specific network performance measurement may be determined for a specific time interval. The network may maintain a record of impact scores in the form of a plurality of bins. When an impact score is determined, the bin with the range corresponding to the ML model impact is selected and its value incremented by one. For example, for a ML model that generated inferences between time t and t-5 minutes having an impact of 0.8 on a specific gNB performance measurement, the counter corresponding to the bin "0.7 - 1" is incremented by one. As another example, for a ML model that generated inferences between time t and t-5 minutes having an impact of 0.2 on a specific gNB performance measurement, the counter corresponding to the bin "LOW" is incremented by one. The measurement can be split into sub-counters per performance measurement. In some examples, the one or more impact scores may be compared to one or more threshold values, and a network action taken based on the result of the comparison. One difference between "legacy" inference and the inference operations described herein is that, in the frameworks described herein there is no need for the network node (e.g., an NG RAN node) to provide explicit inference requests during each period of interest. However, the network node may decide to deactivate the "Common AI / ML model for PM counter(s)" at times, e.g., when the energy consumption does not allow concurrent operation of the AI / ML models dedicated to RAN functionalities and the "Common AI / ML model for PM counter(s)". FIG. 2 shows an overview of an example method 200 fortraining a common AI / ML model for predicting impact of AI / ML models dedicated to RAN functionalities to PM Counters, such as the common AI / ML model for PM counter(s) 102 described in relation to FIG. 1. The method is divided into two stages 202A, 202B. In the first stage 202A, a set of AI / ML models dedicated to RAN functionalities 204A-N are trained jointly with a corresponding set of AI / ML models for PM counter(s) 206A-N. In the second stage 202B, a common AI / ML model for predicting impact of AI / ML models dedicated to RAN functionalities to PM Counters 208 is trained. The method may be performed by one or more data processing apparatus operating in one or more locations. In a first stage 202A, a plurality of AI / ML models 204A-N dedicated to a corresponding plurality of RAN functionalities is trained, i.e., one AI / ML model per RAN functionality. These AI / ML models dedicated to RAN functionalities 204A-N are used to train a corresponding plurality of individual PM counter(s) AI / ML models 206A-N, where each individual PM counter(s) AI / ML model 206A-N predicts an impact on one or more PM counters of executing its respective AI / ML model dedicated to a RAN functionality individually (i.e., not executed in parallel with one or more other AI / ML models dedicated to RAN functionalities). In stage 1 202A, supervised training methods are used to train the AI / ML models dedicated to RAN functionalities 204A-N and AI / ML models for PM counter(s) 206A-N. This training may be performed in an offline environment. For example, for training an AI / ML model 204A-N dedicated to a given RAN functionality, a set of training data comprising a plurality of training examples may be used. Each training example comprises a set of input training data, e.g., a set of network measurements, and a corresponding ground truth output, e.g., a known network state (such as current or future network state) corresponding to the set of network measurements or a known action taken in response to the set of network measurements. During training, input training data is processed by an AI / ML model 204A-N for a given RAN functionality to generate a candidate output. The candidate output is compared to the ground truth output for that input training data, e.g., using a loss / objective function (such as an LI, L2 or cross-entropy loss), and updates to parameters of the AI / ML model 204A-N determined based on the comparison, e.g., by applying an optimisation routine to the objective function (such as stochastic gradient descent). For training an AI / ML model for PM counter(s) 206A-N, a supervised learning method may be used that considers all the input training data of the AI / ML model dedicated to a RAN functionality 204A-N and corresponding output of the AI / ML model 204A-N dedicated to a RAN functionality as input training data for the AI / ML model for PM counter(s) 206A-N. In some examples, the input training data may further comprise an action based on the output of the corresponding AI / ML model 204A-N dedicated to the RAN functionality (which may change each monitored PM counter from slightly to significantly). Each set of input training corresponds to one or more ground truth PM counter impact scores, indicating the impact executing of the AI / ML model 204A-N and / or performing the action based on its output. Since some extreme configuration parameters of the AI / ML model dedicated to the RAN functionality may be difficult to obtain from live network, such model training may be executed in a simulation environment. The ground truth impact score may be based on the values of the PM counters obtained from simulation, reflecting the result of the taken AI / ML-based action. In other words, the training data fortraining an AI / ML model for PM counter(s) 206A-N may comprise a plurality of training examples, each training example comprising a set of input training data and a corresponding set of ground truth PM counter impact scores. The input training data may comprise: (i) the input data for the corresponding AI / ML model 204A-N dedicated to a given RAN functionality (e.g.., the set of network measurements); (ii) the output of the AI / ML model 204A-N for a given RAN functionality (e.g., a predicted network state or action); and / or (iii) a network action taken based on the output of the AI / ML model 204A-N fora given RAN functionality. During training, input training data is processed by an AI / ML model 204A-N for PM counter(s) 206A-N to generate a candidate set of PM counter impact scores. The candidate set of PM counter impact scores is compared to the ground truth set of PM counter impact scores for that input training data, e.g., using a loss / objective function (such as an LI, L2 or cross-entropy loss), and updates to parameters of the AI / ML model 204A-N determined based on the comparison, e.g., by applying an optimisation routine to the objective function (such as stochastic gradient descent). Each AI / ML model for PM counters 206A-N is dedicated to a given RAN functionality, exactly as the corresponding AI / ML model 204A-N dedicated to that functionality (e.g., Mobility Optimization), and does cover the impact of other concurrently running AI / ML models 204A-N dedicated to other RAN functionalities. This can be beneficial to operators, as the operator may use a given AI / ML model for PM counters to estimate the expected network performance impact of the given AI / ML model 204A-N dedicated to a RAN functionality. The initially trained AI / ML models for PM counters 206A-N from the first step 202A are also used to train the Common AI / ML model for PM counters 208, as described in relation to the second stage 204B, to reflect the case when not only one AI / ML model 204A-N dedicated to a RAN functionality is running in time. In a second stage 202B, a common AI / ML model for predicting impact of AI / ML models dedicated to RAN functionalities to PM Counters208 is trained based on the outputs of the plurality of AI / ML models dedicated to RAN functionalities 204A-N and / or the plurality of individual PM counter(s) AI / ML models 206A-N. A "linearised in parts" method may be used to generate training data for the common AI / ML model for predicting impact of AI / ML models dedicated to RAN functionalities to PM Counters 208 from the plurality of AI / ML models dedicated to RAN functionalities 204A-N and / or the plurality of individual PM counter(s) AI / ML models 206A-N, as described in relation to FIG. 3. Following the first stage 202A training procedure, a second stage 202B of training the Common AI / ML model for PM counters (i.e., the model described in relation to Fig. 1) is executed. When AI / ML models 204A-N dedicated to different RAN functionalities are running in parallel, a change of a network configuration parameter as a result of the outcome of one AI / ML model dedicated to a RAN functionality 204A-N, may impact other RAN functionalities. The AI / ML models 204A-N dedicated to those RAN functionalities may then react to modify the same network configuration parameter. The manner of this reaction is difficult to predict and highly non-linear. To address such non-linearities, a so called "linearized in parts" approach to obtain the reliable input and output data needed to train the Common AI / ML model for PM counters. The second stage training may be performed in an offline environment. Alternatively, the second stage training may be performed in an online environment, e.g., using a continual learning method (e.g., training and inference phases occur concurrently). To train the Common AI / ML model for PM counters 208, a supervised learning approach may be used, with input and output (label) data features being the following: input training data feature(s) is / are the network configuration parameter(s), obtained as output of the AI / ML model dedicated to the given RAN functionality. In some examples, the input data may further comprise PM counter values output of each AI / ML model for PM counters that is tailored to a given AI / ML model dedicated to a RAN functionality output data (labels) are the corresponding values of the PM counter # 1 used in supervised learning process, for example as described in relation to FIG 3. The initial / first version of the Common AI / ML model for PM counters 208 may be obtained from simulation environment, i.e., trained offline. This allows all possible configuration / input parameters combinations of all possible AI / ML models 204A-N for all possible RAN functionalities to be obtained with a given granularity change level. Following versions of the Common AI / ML models for PM counters 208 may be obtained online based on live network data either in gNB or OAM, e.g., by fine-tuning the initial / first version of the Common AI / ML model for PM counters 208 based on real-world data. Once the Common AI / ML model for PM counter(s) 208 is trained offline, this model 208 is then deployed for inference, e.g., at one or more gNBs or OAMs. FIG. 3 shows an example of a method 300 for generating training data for a Common AI / ML model for PM counters using a "linearised in parts" process. The method 300 may be performed by one or more data processing apparatus operating in one or more locations. For simplicity, FIG. 3 shows two AI / ML models dedicated to RAN functionalities 1 and 2, each one with only one network configuration parameter, and a single PM counter (PM counter #1). However, it will be appreciated that the method can be generalised to a plurality of AI / ML models dedicated to RAN functionalities, each with one or more network configuration parameters and one or more corresponding PM counters. The x-axis represents a change in the PM counter #1 value as a result of changing a first network configuration parameter, where the first network configuration parameter change is an output of the related AI / ML model dedicated to RAN functionality 1. Each grid point on x-axis represents the value of the PM counter #1 related to a first network configuration parameter change by one granularity step. Granularity step here means the resolution in changing the first network configuration parameter. Considering a single AI / ML model dedicated to RAN functionality 1 is active, the x-axis represents the AI / ML model for PM counters corresponding to AI / ML model dedicated to RAN functionality 1. The y-axis represents a change in the PM counter #1 value as a result of changing a second network configuration parameter, where the second network configuration parameter change is an output of the related AI / ML model dedicated to RAN functionality 2. The second network configuration parameter may be a different network configuration parameter the first network configuration parameter. Alternatively, the second network configuration parameter may be the same network configuration parameter the first network configuration parameter. Each grid point on y-axis represents the value of the PM counter #1 related to a second network configuration parameter change by one granularity step. Granularity step here means a granularity in changing the second network configuration parameter. Considering a single AI / ML model dedicated to RAN functionality 2 is active, the y-axis represents the AI / ML model for PM counters corresponding to AI / ML model dedicated to RAN functionality 2. To obtain reliable input and output data to train the proposed Common AI / ML model for PM counters, the network configuration parameters may be repeatedly altered by one step and respective values of the PM counter(s) determined via simulation. For example, in a first step, the first network configuration parameter (as output of the AI / ML model dedicated to RAN functionality 1) is changed by one granularity step. In a simulation environment, which reflects the gNB functionality in a real environment with a real traffic model, this network configuration parameter change will result in PM counter #1 value equal to Mil in this step for the AI / ML model dedicated to RAN functionality 1. In a second step, the second network configuration parameter (as output of the AI / ML model dedicated to RAN functionality 2) is changed by one granularity step. This will result in PM counter #1 value equal to M22 in this second step for the AI / ML model dedicated to RAN functionality 2. The PM counter #1 value for the AI / ML model dedicated to RAN functionality 1 is considered as unchanged, i.e. equal to Mil in this step. In addition to PM counter #1 in each step, also an "Impact score" to reflect a relative impact of each AI / ML model dedicated to a RAN functionality may be provided. In a third step, the first network configuration parameter of the AI / ML model dedicated to RAN functionality 1 is further changed by one granularity step. This will correspond to PM counter #1 value equal to M13 in this third step for the AI / ML model dedicated to RAN functionality 1. The PM counter #1 value for the AI / ML model dedicated to RAN functionality 2 is considered as unchanged, i.e. equal to M22 in this step. 5 In a fourth step, the second network configuration parameter of the AI / ML model dedicated to RAN functionality 2 is further changed by one granularity step. This will correspond to PM counter #1 value equal to M24 in this fourth step for the AI / ML model dedicated to RAN functionality 2. The PM counter #1 value for the AI / ML model 10 dedicated to RAN functionality 1 is considered as unchanged, i.e. equal to M13 in this step. One or more of these steps may be repeated a plurality of times to generate values of PM counter #1 for a plurality of combinations of network configuration parameters. 15 The resulting training data from the method of FIG. 3. Is summarised in the following table: Step # Input data Output data 1 Network configuration parameter = one granularity step (AI / ML model dedicated to RAN Functionality 1), and corresponding PM Counter #1 value (Output from AI / ML model for PM counters related to functionality 1 from 1st step of training) PM Counter #1 = Mil (AI / ML model dedicated to RAN Functionality 1) 2 Network configuration parameter = one granularity step (AI / ML model dedicated to RAN Functionality 1), and corresponding PM Counter #1 value (Output from AI / ML model for PM counters related to functionality 1 from 1st step of training) Network configuration parameter = one granularity step (AI / ML model dedicated to RAN Functionality 2), and corresponding PM Counter #1 value (Output from AI / ML model for PM counters related to functionality 2 from 1st step of training) PM Counter #1 = Mil (AI / ML model dedicated to RAN Functionality 1) PM Counter #1 = M22 (AI / ML model dedicated to RAN Functionality 2) 3 Network configuration parameter = two granularity steps (AI / ML model for Functionality 1), and corresponding PM Counter #1 value (Output from AI / ML model for PM counters related to functionality 1 from 1st step of training) Network configuration parameter = one granularity step (AI / ML model dedicated to RAN Functionality 2), and corresponding PM Counter #1 value (Output from AIML model for PM counters related to functionality 2 from 1st step of training) PM Counter #1 = M13 (AI / ML model dedicated to RAN Functionality 1) PM Counter #1 = M22 (AI / ML model dedicated to RAN Functionality 2) 4 Network configuration parameter = two granularity steps (AI / ML model dedicated to RAN Functionality 1), and corresponding PM Counter #1 value (Output from AI / ML model for PM counters related to functionality 1 from 1st step of training) Network configuration parameter = two granularity steps (AI / ML model dedicated to RAN Functionality 2), and corresponding PM Counter #1 value (Output from AI / ML model for PM counters related to functionality 2 from 1st step of training) PM Counter #1 = M13 (AI / ML model dedicated to RAN Functionality 1) PM Counter #1 = M24 (AI / ML model dedicated to RAN Functionality 2) The principle manifested in Fig. 3 may be generalized for any number of AI / ML models dedicated to RAN functionalities and any number of network configuration parameters and any number of impacted PM counters. The key principle is to apply the "Linearized 5 in parts" approach, which basically means to change in the given step only one network configuration parameter as output of the AI / ML model dedicated to the given RAN functionality by one granularity step and obtain the change of the corresponding PM counter(s). 10 FIG. 4 shows an overview of a further example system / method 400 for predicting the impact of AI / ML models dedicated to RAN functionalities on PM counters. A common AI / ML model for PM counter(s) 502 is stimulated to provide estimates of one or more average PM counters 506A-M (or one or more PM counter changes from a previous RAN monitoring period) when at least one action 508A-N taken by the network, where this action was based on the output 512A-N of one of the AI / ML models 504A-N dedicated to a RAN functionality optimization during a network runtime period. The AI / ML models 504A-N receive as input respective input data 510A-N, h, at a respective time, h, and process their respective input data 510A-N to provide respective output data 512A-N, 0,, at respective times t, + <5 / . The input data 504A-N for a given AI / ML model 504A-N comprises a set of measurement data relevant to the functionality of that AI / ML model 504A-N. The input data 504A-N may be in the form of a vector. The output data 512A-N for a given AI / ML model 504A-N may comprise a prediction of a network state, a prediction of a future measurement and / or a proposed network action. The output data 512A-N may be in the form of a vector. Based on the output 512A-N of the AI / ML models 504A-N, one or more actions 508A-N may be taken by the network. An action 508A-N taken based on the output 512A-N of a given AI / ML model 504A-N may be associated with a respective action time ti + 5; + a. Examples of actions include, but are not limited to: mobility-related actions, such as altering parameters for UE handovers or the like; load balancing-related actions; and / or energy use optimization-related actions. Data 514A-N relating to the set of actions 508A-N may be input into an ordering operation 516, which time-orders the set of actions 508A-N and the associated information to generated time-ordered input data 518 for the common AI / ML model for PM counter(s) 502. The data 514A-N may comprise, for each action 508A-N, the action 508A-N, the corresponding output data 512A-N from an AIL / ML model 504A-N, the time difference between the action and a previous (consecutive) action, a delay with which the action was taken after receiving the AI / ML model output, and a delay with which an AI / ML model output was issued after the model received input data. In some examples, the input data described in relation to FIG.l may additionally form part of the time-ordered input data 518 for the common AI / ML model for PM counter(s) 502. The common AI / ML model for PM counter(s) 502 processes the time-ordered input data to generate an average value of one or more PM counters over the timespan of the set of actions 508A-N. FIG. 5 shows a flow diagram 500 of an example method for predicting the impact of AI / ML models dedicated to RAN functionalities on PM counters. The method 500 may be performed by one or more computers operating in one or more locations. For example, the method 500 may be performed by one or more base stations, one or more gNBs, and / or the like. Each operation in the method may be performed by suitable means for performing that operation, for example as described in relation to FIG. 8. The method 500 may correspond to the method described in relation to FIG. 1. At operation 502, a network configuration derived from a plurality of AI / ML models dedicated to RAN functionalities is received. Each AI / ML model dedicated to a RAN functionality is directed towards a respective one or more radio access network functionalities, for example energy saving, load balancing, and / or user equipment mobility optimisation. The method may further comprise executing the plurality of AI / ML models dedicated to RAN functionalities on respective measurement data to generate a plurality of respective radio access network functionality predictions, e.g., each AI / ML model dedicated to a RAN functionality receives as input respective measurement data and processes the input measurement data to generate one or more radio access network functionality predictions. The network configuration may be generated from the plurality of respective radio access network functionality predictions. For example, the radio access network functionality predictions may themselves form the network configuration. Alternatively, one or more rules may be applied to the radio access network functionality predictions to generate the network configuration. Executing the plurality of AI / ML models dedicated to RAN functionalities may comprise executing two or more of these AI / ML models in parallel. At operation 504, a plurality of predicted performance measurement counters is received. The plurality of predicted performance measurement counters is output from a plurality of AI / ML models for PM counters, e.g., each AI / ML models for PM counters outputs one or more performance measurement counters. There is a 1-1 correspondence between the AI / ML models for PM counters and the AI / ML models dedicated to RAN functionalities. The method may further comprise generating the plurality of predicted performance measurement counters by executing the plurality of ML performance measurement models on respective radio access network functionality predictions output by a respective ML model in the plurality of ML models. Each ML performance measurement model corresponds to a respective one of the plurality of ML models used to generate the network functionality predictions, and estimates an individual PM counter impact of its corresponding ML model. At operation 506, the network configuration and the plurality of predicted performance measurement counters are processed, using a common ai / ml model for PM Counters , to determine a model output. The model output comprises, for one or more performance measurement counters, a respective plurality of impact scores. Each impact score is indicative of a predicted impact of a corresponding ML model in the plurality of ML models on the respective performance measurement counter of said impact score for the network configuration. For example, the common AI / ML model for PM Counters may receive input data comprising the plurality of predicted performance measurement counters and the network configuration, and process the input data to determine the impact scores. In some examples, the input data further comprises an indication of the identities of the ML models being used to generate the network configuration. The impact scores may be numerical scores indicating, for each impact score, the impact / contribution of a respective ML model to a PM counter. For example, an impact score may be a numerical score in the range [0, 1]. The contribution to a PM counter score attributable to a given ML model may be obtained by multiplying such an impact score by a measured / predicted overall value of said PM counter. Alternatively, the impact scores may be categorical values, e.g., "low impact", "medium impact", "high impact", or the like. One or more actions may be taken based on the impact scores output by the common ML model. The actions may be taken by a network. In some examples, one or more of the AI / ML models for a network functionality may be disabled based on one or more of its respective impact scores. For example, if an impact score of a ML model indicates that the ML model affects a PM counter negatively beyond a threshold value, then that ML model may be disabled. In some examples, a network may select between two or more ML models for the same network functionality based on one or more of their respective impact scores. For example, the ML model with the smallest negative impact, or largest positive impact, on one or more PM counters may be chosen, and the other ML models for the same functionality disabled. In some examples, one or more of the impact scores for a ML model may trigger the network to retrain the ML model. For example, if an impact score of a ML model indicates that the ML model affects a PM counter negatively beyond a threshold value, then that ML model may be selected for retraining. In some examples, one or more of the impact scores may trigger a network to disable some or all ML based functionality and revert to a rules-based approach. For example, if an impact score of a ML model indicates that the ML model affects a PM counter negatively beyond a threshold value, then the network reverts to using a rules-based approach forthat functionality. In some examples, one or more of the impact scores may trigger a PM counter bin in a plurality of PM counter bins to be incremented. For example, for a ML model that generated inferences between a time t and t-5 minutes having an impact of 0.8 on a specific gNB performance measurement, a counter corresponding to the bin "0.7 - 1" is incremented by one. As another example, for a ML model that generated inferences between time t and t-5 minutes having an impact of 0.2 on a specific gNB performance measurement, a counter corresponding to the bin "LOW" is incremented by one. FIG. 6 shows a flow diagram 600 of an example method for training a common ML prediction measurement model. The method 600 may be performed by one or more computers operating in one or more locations. For example, the method 600 may be performed by one or more base stations, one or more gNBs, and / or the like. Each operation in the method may be performed by suitable means for performing that operation, for example as described in relation to FIG. 8. The method 600 may correspond to the method described in relation to FIGs. 2 and / or 3. At operation 602, a plurality of pairs of ML models are jointly trained. Each pair of ML models comprises a respective first ML model for providing one or more radio access functionalities and a respective second ML model for predicting one or more performance measurement counters based on the output of the respective first ML model of the pair. Operation 602 may correspond to the first stage of FIG. 2. A supervised learning method may be used to jointly train the plurality of pairs of ML models on training data comprising a set of training examples. Each training example may comprise: a set of input data for the first ML model of a pair (e.g., network measurements); a ground truth output for the first ML model of the pair (e.g., a set of ground truth / target network parameters); and a corresponding ground truth performance measurement counter. During training, the first ML model of a pair generates a candidate output from a set of input data based on current values of parameters of the first ML model. The candidate output is compared to the corresponding ground truth output (e.g., using an objective / loss function) and updates to parameters of the first ML model determined based on the comparison (e.g., by applying an optimization routine to the objective / loss function). Similarly, the second ML model of a pair generates a candidate performance measurement counter from the output of the first ML model of the pair (and, in some examples, the input to the first ML pair). The candidate performance measurement counter is compared to the corresponding ground truth performance measurement counter (e.g., using an objective / loss function) and updates to parameters of the first ML model determined based on the comparison (e.g., by applying an optimization routine to the objective / loss function). At operation 604, a set of training data for a combined performance measurement counter model is generated using the plurality of pairs of ML models. The training data comprises a plurality of training examples. Each training example comprising: one or more network configuration parameters; one or more corresponding performance measurement counters output by a respective one or more second machine-learning models; and one or more (ground truth) combined performance measurement counter values. Operation 604 may, in some examples, correspond to the "linearized in parts" method described in relation to FIG. 3. The linearised in parts method may comprise, for a plurality of first ML models: incrementing a network configuration parameter associated with a first machinelearning model by one granularity step; and determining a one or more respective combined performance measurement counter values for said first ML models. This process may be repeated in a cycle to generate multiple combined performance measurement counter values for each of the first machine-learning models. In some examples, for each of the plurality of first ML models, a corresponding (individual) performance measurement counter for the network configuration parameter is determined using the second ML model corresponding to the first ML model. In some examples, determining the one or more respective combined performance measurement counter values for said first ML models comprises simulating a network environment with a set of network configuration parameters comprising the incremented network configuration parameter. In other examples, live network data may be used to derive the one or more respective combined performance measurement counter values. At operation 606, the combined performance measurement counter model is trained using the training data to predict a plurality of impact scores for each of one or more performance measurement counters. Each impact score is indicative of a predicted impact of a corresponding machine learning model in the plurality of machine-learned models on a respective performance measurement counter. Operation 606 may correspond to the second stage of FIG. 2. A supervised learning method may be used to train the combined performance measurement counter model. During training, the combined performance measurement counter model generates one or more candidate impact scores based on the one or more network configuration parameters and one or more corresponding performance measurement counters of a training example. The candidate impact score is compared to the corresponding ground truth combined performance measurement counter values (e.g., using an objective / loss function) and updates to parameters of the first ML model determined based on the comparison (e.g., by applying an optimization routine to the objective / loss function). FIG. 7 shows a flow diagram 700 of a further example method for predicting the impact of ML models on PM counters. The method 700 may be performed by one or more computers operating in one or more locations. For example, the method 700 may be performed by one or more base stations, one or more gNBs, and / or the like. Each operation in the method may be performed by suitable means for performing that operation, for example as described in relation to FIG. 8. The method 700 may correspond to the method described in relation to FIG. 4. At operation 702, a set of network actions is generated from a set of network measurements and using a plurality of ML models. Each network action is associated with a respective time. Each ML model is directed towards a respective one or more radio access network functionalities. The network actions may be generated by executing the one or more ML models on respective sets of input data (e.g., one or more sets of network measurements) to generate respective sets of output data (e.g., target / predicted network parameters). Based on the respective output data, one or more respective network actions may be generated, e.g., using one or more pre-defined rules. In some examples, the output data comprises a network action itself. The network may implement the one or more actions. In some examples, a time-ordered set of input data for the combined performance measurement counter model is generated. The generating comprises time-ordering the set of network actions and the respective output data. In some examples, the combined performance measurement counter model further comprises one or more of: a set of action times; a set of time differences between consecutive actions; a set of delay times between respective output data times and the corresponding action being taken; and / or a set of delay times between input data to a machine-learning model being received and the respective set of output data being generated. At operation 704, the set of network actions is processed using a combined performance measurement counter model to generate one or more average performance measurement counters over a time interval spanned by the respective time of the earliest network action in the set of network actions and the respective time of the latest network action in the set of network actions. The combined performance measurement counter model may process input data comprising the time-ordered set of input data. In some examples the input data to the combined performance measurement counter model further comprises one or more network parameters; and / or one or more individual performance measurement counter predictions. FIG. 8 shows an apparatus / system according to some example embodiments, which may form at least a part of a user device, network node, base station, or gNB. The apparatus may be configured to perform the operations described herein, for example operations described with reference to any disclosed process. The apparatus comprises at least one processor 800 and at least one memory 801 directly or closely connected to the processor. The memory 801 includes at least one random access memory (RAM) 801A and at least one read-only memory (ROM) 801B. Computer program code (software) 805 is stored in the ROM 801B. The apparatus may be connected to a transmitter (TX) and a receiver (RX). The apparatus may, optionally, be connected with a user interface (UI) for instructing the apparatus and / or for outputting data. The at least one processor 800, with the at least one memory 801 and the computer program code 805 are arranged to cause the apparatus to at least perform at least the method, or a part of the method, according to any preceding process. FIG. 9 shows a non-transitory media 900 according to some embodiments. The non-transitory media 900 is a computer readable storage medium. It may be e.g., a CD, a DVD, a USB stick, a blue ray disk, etc. The non-transitory media 900 stores computer program code, causing an apparatus to perform the method of any preceding process for example as disclosed in relation to the flow diagrams and related features thereof. Names of network elements, protocols, and methods are based on current standards. In other versions or other technologies, the names of these network elements and / or protocols and / or methods may be different, as long as they provide a corresponding functionality. For example, embodiments may be deployed in 2G / 3G / 4G / 5G networks and further generations of 3GPP but also in non-3GPP radio networks such as Wi-Fi. A memory may be volatile or non-volatile. It may be e.g., a RAM, a SRAM, a flash memory, a FPGA block ram, a DCD, a CD, a USB stick, and a blue ray disk. If not otherwise stated or otherwise made clear from the context, the statement that two entities are different means that they perform different functions. It does not necessarily mean that they are based on different hardware. That is, each of the entities described in the present description may be based on a different hardware, or some or all of the entities may be based on the same hardware. It does not necessarily mean that they are based on different software. That is, each of the entities described in the present description may be based on different software, or some or all of the entities may be based on the same software. Each of the entities described in the present description may be embodied in the cloud. Implementations of any of the above-described blocks, apparatuses, systems, techniques or methods include, as non-limiting examples, implementations as hardware, software, firmware, special purpose circuits or logic, general purpose hardware or controller or other computing devices, or some combination thereof. Some embodiments may be implemented in the cloud. It is to be understood that what is described above is what is presently considered the preferred embodiments. However, it should be noted that the description of the preferred embodiments is given by way of example only and that various modifications may be made without departing from the scope as defined by the 5 appended claims.

Claims

1. Apparatus comprising:means for receiving a network configuration derived from a plurality of machine-learning models, each machine-learning model directed towards a respective one or more radio access network functionalities;means for receiving a plurality of predicted performance measurement counters output from a plurality of machine-learning performance measurement models, each machine-learning prediction measurement model corresponding to one of the plurality of machine-learning models; andmeans for processing, using a common machine-learning performance measurement counter model, the network configuration and the plurality of predicted performance measurement counters to determine a model output comprising, for one or more performance measurement counters, a respective plurality of impact scores, wherein each impact score is indicative of a predicted impact of a corresponding machine-learning model in the plurality of machine-learning models on the respective performance measurement counter of said impact score for the network configuration.

2. The apparatus of claim 1, wherein the apparatus further comprises:means for executing the plurality of machine-learning models on respective measurement data to generate a plurality of respective radio access network functionality predictions; andmeans for generating, from the plurality of respective radio access network functionality predictions, the network configuration.

3. The apparatus of claim 2, wherein means for executing the plurality of machine-learning models on the respective measurement data to generate the plurality of respective radio access network functionality predictions comprises:means for executing two or more of the plurality of machine-learning models in parallel.

4. The apparatus of any preceding claim, wherein the apparatus further comprises:means for executing the plurality of machine-learning performance measurement models on respective radio access network functionality predictions output by a respective machine-learning model in the plurality of machine-learning models.

5. The apparatus of any preceding claim, wherein the one or more radio access network functionalities comprise one or more of: energy saving; load balancing; and / or user equipment mobility optimisation.

6. The apparatus of any preceding claim, further comprising:means for disabling one or more machine-learning models in the plurality of machine-learning models based at least in part on the respective impact scores of said one or more machine-learning models.

7. The apparatus of any preceding claim, further comprising:means for selecting between two or more machine-learning models in the plurality of machine-learning models based at least in part on the respective impact scores of said one or more machine-learning models, wherein said two or more machine-learning models are directed towards the same radio access network functionality.

8. The apparatus of any preceding claim, further comprising:means for incrementing a performance measurement counter bin based at least in part on one or more of the impact scores in the plurality of impact scores.

9. The apparatus of any preceding claim, wherein the common machine-learning performance measurement counter model is hosted in a gNB.

10. The apparatus of any of claims 1 to 8, wherein the common machine-learning performance measurement counter model is hosted in a Management Data Analytics Function, MDAF, in an Operations Administration Management, OAM, system, and wherein the apparatus further comprises:means for receiving input data for the common machine-learning performance measurement from a gNB.

11. Apparatus comprising:means for jointly training a plurality of pairs of machine-leaning models, wherein each pair of machine-learning models comprises a respective first machinelearning model for providing one or more radio access functionalities and a respective second machine-learning model for predicting one or more performance measurement counters based on the output of the respective first machine-learning model of the pair;means for generating a set of training data fora combined performance measurement counter model using the plurality of pairs of machine-learning models, the training data comprising a set of training examples, each training example comprising: a network configuration parameter; one or more corresponding performance measurement counters output by a respective one or more second machine-learning models; and one or more combined performance measurement counter values; andmeans fortraining the combined performance measurement counter model using the training data to predict a plurality of impact scores for each of one or more performance measurement counters, each impact score indicative of a predicted impact of a corresponding machine learning model in the plurality of machine-learned models on a respective performance measurement counter.

12. The apparatus of claim 11, wherein the means for generating a set of training data for a combined performance measurement counter model using the plurality of pairs of machine-learning models comprises:means for generating the set of training data using a linearised in parts method.

13. The apparatus of claim 12, wherein the linearised in parts method comprises: for a plurality of first machine-learning models:incrementing a network configuration parameter associated with a first machine-learning model by one granularity step; anddetermining a one or more respective combined performance measurement counter values for said first machine-learning models.

14. The apparatus of claim 13, wherein the linearised in parts method further comprises:for each of the plurality of first machine-learning models, determining a corresponding performance measurement counter for the network configuration parameter using the second machine-learning model corresponding to the first machine-learning model.

15. The apparatus of any of claims 13 or 14, wherein determining a one or more respective combined performance measurement counter values for said first machinelearning models comprises simulating a network environment with a set of network configuration parameters comprising the incremented network configuration parameter.

16. The apparatus of any of claims 11 to 15, wherein the means for training the combined performance measurement counter model comprises:means for applying a supervised learning method to the combined performance measurement counter model using the training data.

17. Apparatus comprising:means for generating, from a set of network measurements and using a plurality of machine-learning models, a set of network actions, each network action associated with a respective time, wherein each machine learning model directed towards a respective one or more radio access network functionalities; andmeans for processing, using a combined performance measurement counter model, the set of network actions to generate one or more average performance measurement counters over a time interval spanned by the respective time of the earliest network action in the set of network actions and the respective time of the latest network action in the set of network actions.

18. The apparatus of claim 17, wherein the means for generating, from a set of network measurements and using a plurality of machine-learning models, a set of network actions comprises:means for executing the one or more machine-learning models on respective sets of input data to generate respective sets of output data; andmeans for generating, based on the respective sets of output data, one or more respective network actions.

19. The apparatus of claim 18, further comprising:means for performing the one or more respective network actions.

20. The apparatus of any of claims 18 to 19, wherein the apparatus further comprises:means for generating a time-ordered set of input data for the combined performance measurement counter model, the generating comprising time-ordering the set of network actions and the respective output data,wherein the combined performance measurement counter model processes the time-ordered set of input data to generate the one or more average performance measurement counters.

21. The apparatus of claim 20, wherein the time-ordered set of input data for the combined performance measurement counter model further comprises one or more of: a set of action times; a set of time differences between consecutive actions; a set of delay times between respective output data times and the corresponding action being taken; and / or a set of delay times between input data to a machine-learning model being received and the respective set of output data being generated.

22. The apparatus of any of claims 17 to 21, wherein generating one or more average performance measurement counters is further based on: one or more network parameters; and / or one or more individual performance measurement counter predictions.

23. A computer implemented method comprising:receiving a network configuration derived from a plurality of machine-learning models, each machine-learning model directed towards a respective one or more radio access network functionalities;receiving a plurality of predicted performance measurement counters output from a plurality of machine-learning performance measurement models, each machine-learning prediction measurement model corresponding to one of the plurality of machine-learning models; andprocessing, using a common machine-learning performance measurement counter model, the network configuration and the plurality of predicted performance measurement counters to determine a model output comprising, for one or more performance measurement counters, a respective plurality of impact scores, wherein each impact score is indicative of a predicted impact of a corresponding machine-learning model in the plurality of machine-learning models on the respective performance measurement counter of said impact score for the network configuration.

24. A computer implemented method comprising:jointly training a plurality of pairs of machine-leaning models, wherein each pair of machine-learning models comprises a respective first machine-learning model for providing one or more radio access functionalities and a respective second machine-learning model for predicting one or more performance measurement counters based on the output of the respective first machine-learning model of the pair;generating a set of training data fora combined performance measurement counter model using the plurality of pairs of machine-learning models, the trainingdata comprising a set of training examples, each training example comprising: a network configuration parameter; one or more corresponding performance measurement counters output by a respective one or more second machine-learning models; and one or more combined performance measurement counter values; and training the combined performance measurement counter model using thetraining data to predict a plurality of impact scores for each of one or more performance measurement counters, each impact score indicative of a predicted impact of a corresponding machine learning model in the plurality of machine-learning models on a respective performance measurement counter.

25. A computer implemented method comprising:generating, from a set of network measurements and using a plurality of machine-learning models, a set of network actions, each network action associated with a respective time, wherein each machine learning model directed towards a respective one or more radio access network functionalities; andprocessing, using a combined performance measurement counter model, the set of network actions to generate one or more average performance measurement counters over a time interval spanned by the respective time of the earliest network action in the set of network actions and the respective time of the latest network action in the set of network actions.34

Images