Efficient collection of distributed data in RAN

Abstract

In one aspect, a computer-implemented method implemented by a first network node in a radio access network (RAN) is provided. The method includes obtaining an output from a machine learning (ML) model. The method includes obtaining an output feedback identifier for the output, wherein the output feedback identifier uniquely identifies the output. The method includes generating a first message, wherein the first message includes the output feedback identifier. The method comprises sending the first message comprising the output feedback identifier to a third network node.

Description

Technical Field

[0001] Embodiments are disclosed that relate to multi-agent reinforcement learning ("RL"), distributed optimization, self-organizing networks ("SON"), zero-touch automation, and mobility simulation for optimizing radio access networks. Background Art

[0002] 3GPP TR 37.817 (v17.0.0) “Study on enhancements to data collection for NR and EN-DC” (Release 17) (“TR 37.817”) provides a description of the principles of RAN intelligence enabled by AI. Figure 1 A functional framework for RAN intelligence is shown. Figure 1 An overview of AI functionality and optimized inputs and outputs for AI-enabled RAN is provided, as well as use cases and solutions for AI-enabled RAN. The study is based on current architectures and interfaces, and the use cases analyzed include network energy saving, load balancing, and mobility optimization.

[0003] For each use case, AI / ML model training is located in the OAM or in the gNB, specifically in the gNB-CU. Figures 2 to 4 The model training in OAM for these three use cases including network energy saving, load balancing and mobility optimization is described respectively.

[0004] TR 37.817 mentions the use of "feedback". Section 4.2 of TR37.817 states: "An actor is a function that receives output from a model's reasoning function and triggers or implements corresponding actions. An actor can trigger actions on other entities or itself. Feedback: Information that may be needed to derive training data, reasoning data, or monitor the performance of AI / ML models and their impact on the network by updating KPIs and performance counters."

[0005] As described below, both NG-RAN node 1 hosting model inference, and NG-RAN node 2 (which is any neighboring NG-RAN node of NG-RAN node 1) provide feedback to OAM related to network energy saving, load balancing or mobility optimization actions taken by NG-RAN. Sections 5.1.2.6, 5.2.2.6 and 5.3.2.6 in TR 37.817 list the types of feedback given for AI / ML based network energy saving, load balancing and mobility optimization use cases. In one example, the action taken by NG-RAN node 1 (the serving NG-RAN node) requires handing over at least one UE to NG-RAN node 2 (the target NG-RAN node). In this example, the feedback (provided to OAM in this case) includes the UE performance affected by the action (e.g., the performance of the UE being handed over), including QoS parameters such as throughput / bit rate, packet delay / latency, packet loss, etc. More details can be found in TR 37.817.

[0006] Figure 2 The signaling between the UE (208), NG-RAN node 1 (202), NG-RAN node 2 (204) and OAM (206) is shown. For the network energy saving use case, Section 5.1.2.2 of TR 37.817 mentions that the NG-RAN node 1 (202) uses the AI / ML model trained from the OAM (206) to make energy decisions.

[0007] Step 0: NG-RAN Node 2 is assumed to optionally have an AI / ML model that can provide input information to NG-RAN Node 1.

[0008] Step 1: NG-RAN node 1 configures measurement information on the UE side and sends a configuration message to the UE to implement the measurement process and reporting.

[0009] Step 2: The UE collects the indicated measurements, e.g., UE measurements related to RSRP, RSRQ, SINR of the serving cell and neighboring cells.

[0010] Step 3: The UE sends a measurement report message to NG-RAN node 1.

[0011] Step 4: NG-RAN Node 1 also sends UE measurement reports to OAM along with other input data for model training.

[0012] Step 5: NG-RAN node 2 (assumed to optionally have an AI / ML model) also sends input data for model training to OAM.

[0013] Step 6: Model training at OAM. Required measurements and input data from other NG-RAN nodes are leveraged to train AI / ML models for network energy saving.

[0014] Step 7: OAM deploys / updates the AI / ML model into the NG-RAN node. The NG-RAN node can also continue model training based on the AI / ML model received from OAM. Note: This step is not within the scope of RAN3 Rel-17.

[0015] Step 8: NG-RAN Node 2 sends the required input data to NG-RAN Node 1 for model inference based on AI / ML network energy saving.

[0016] Step 9: The UE sends a UE measurement report to NG-RAN node 1.

[0017] Step 10: Based on the local input of NG-RAN node 1 and the input received from NG-RAN node 2, NG-RAN node 1 generates model reasoning output (e.g., energy saving strategy, handover strategy, etc.).

[0018] Step 11: NG-RAN Node 1 sends model performance feedback to OAM (if applicable). Note: This step is outside the scope of RAN3.

[0019] Step 12: NG-RAN node 1 performs network energy saving actions based on the model inference output. If the output is a handover strategy, NG-RAN node 1 can select the most suitable target cell for each UE before performing the handover.

[0020] Step 13: NG-RAN Node 2 provides feedback to OAM.

[0021] Step 14: NG-RAN Node 1 provides feedback to OAM.

[0022] For the network energy saving use case, Section 5.1.2.6 of TR 37.817 states:

[0023] Feedback on network energy saving based on AI / ML

[0024] To optimize the performance of AI / ML-based network energy-saving models, the following feedback can be considered to be collected from NG-RAN nodes: (i) resource status of neighboring NG-RAN nodes; (ii) energy efficiency; (iii) UE performance affected by energy-saving actions (e.g., UE being handed over), including bit rate, packet loss and latency; and / or (iv) system KPIs (e.g., throughput, latency, RLF of current and neighboring NG-RAN nodes).

[0025] Figure 3The signaling between the UE (308), NG-RAN node 1 (302), NG-RAN node 2 (304), and OAM (306) is shown. Section 5.2.2.2 of TR 37.817 describes the high-level signaling flow for AI / ML use cases related to load balancing with model training in OAM (306) and model inference in NG-RAN node 1 (302).

[0026] Step 0: NG-RAN Node 2 is assumed to optionally have an AI / ML model, which can provide useful input information to NG-RAN Node 1, such as predicted resource status, etc.

[0027] Step 1: NG-RAN node 1 configures the UE to provide measurement and / or location information (e.g., RRM measurements, MDT measurements, speed, location).

[0028] Step 2: The UE collects the indicated measurements, e.g., UE measurements related to RSRP, RSRQ, SINR of the serving cell and neighboring cells.

[0029] Step 3: The UE reports the requested measurement and / or location information (e.g., UE measurements related to RSRP, RSRQ, SINR of the serving cell and neighboring cells, speed, location) to the NG-RAN node 1.

[0030] Step 4: NG-RAN node 1 also sends UE measurement reports and other input data for model training to OAM. NG-RAN node 2 also sends input data for model training to OAM.

[0031] Step 5: AI / ML model training is located in OAM. The required measurements and input data from other NG-RAN nodes are utilized to train the AI / ML model.

[0032] Step 6: OAM deploys / updates the AI / ML model into the NG-RAN node. Allowing the NG-RAN node to continue model training based on the AI / ML model received from OAM. Note: This step is outside the scope of RAN3 Rel-17.

[0033] Step 7: The UE collects and reports the requested measurement or location information to NG-RAN node 1.

[0034] Step 8: NG-RAN node 1 receives input information for load balancing model inference from neighboring NG-RAN node 2.

[0035] Step 9: NG-RAN Node 1 implements the model inference and generates a load balancing prediction or decision.

[0036] Step 10: NG-RAN 1 sends model performance feedback to OAM (if applicable). Note: This step is outside the scope of RAN3.

[0037] Step 11: NG-RAN Node 1 may take load balancing action and the UE moves from NG-RAN Node 1 to NG-RAN Node 2.

[0038] Step 12: NG-RAN Node 1 and NG-RAN Node 2 send feedback information to OAM.

[0039] For the load balancing use case, Section 5.2.2.6 of TR 37.817 further describes feedback for AI / ML-based load balancing and states as follows: In order to optimize the performance of the AI / ML-based load balancing model, the following feedback may be considered for collection from the NG-RAN nodes: (i) UE performance information from the target NG-RAN node (for those UEs switched from the source NG-RAN node); (ii) resource status information updates from the target NG-RAN node; and / or (iii) system KPIs (e.g., throughput, latency, RLF of current and neighboring NG-RAN nodes).

[0040] Figure 4 The signaling between the UE (408), NG-RAN node 1 (402), NG-RAN node 2 (404) and OAM (406) is shown. For the mobility optimization use case, Section 5.3.2.2 of TR 37.817 provides:

[0041] Step 0: The NG-RAN node 2 is assumed to optionally have an AI / ML model that can generate the required inputs such as resource status and utilization predictions / estimations etc.

[0042] Step 1: The NG-RAN node configures measurement information on the UE side and sends a configuration message including the configuration information to the UE.

[0043] Step 2: The UE collects the indicated measurements, e.g., UE measurements related to RSRP, RSRQ, SINR of the serving cell and neighboring cells.

[0044] Step 3: The UE sends a measurement report message including the required measurements to NG-RAN node 1.

[0045] Step 4: The NG-RAN node 1 sends input data for training to the OAM, where the input data for training includes required input information from the NG-RAN node 1 and measurements from the UE.

[0046] Step 5: The NG-RAN node 2 sends input data for training to the OAM, wherein the input data for training includes required input information from the NG-RAN node 2. If the NG-RAN node 2 executes the AI / ML model, the input data for training may include corresponding inference results from the NG-RAN node 2.

[0047] Step 6: Model training. The required measurements are utilized to train the AI / ML model for UE mobility optimization.

[0048] Step 7: OAM sends an AI / ML model deployment message to deploy the trained / updated AI / ML model into the NG-RAN node. The NG-RAN node can also continue model training based on the AI / ML model received from OAM. Note: This step is not within the scope of RAN3 Rel-17.

[0049] Step 8: NG-RAN node 1 obtains the measurement report as inference data for UE mobility optimization.

[0050] Step 9: The NG-RAN node 1 obtains input data for reasoning from the NG-RAN node 2 for UE mobility optimization, wherein the input data for reasoning includes required input information from the NG-RAN node 2. If the NG-RAN node 2 executes the AI / ML model, the input data for reasoning may include the corresponding reasoning result from the NG-RAN node 2.

[0051] Step 10: Model inference. The required measurements are used for model inference to output predictions, such as UE trajectory prediction, target cell prediction, target NG-RAN node prediction, etc.

[0052] Step 11: NG-RAN 1 sends model performance feedback to OAM (if applicable). Note: This step is outside the scope of RAN3.

[0053] Step 12: Based on the prediction, recommended action or configuration, NG-RAN node 1, the target NG-RAN node (represented by NG-RAN node 2 in this step in the flowchart) and the UE implement a mobility optimization / handover procedure to handover the UE from NG-RAN node 1 to the target NG-RAN node.

[0054] Step 13: NG-RAN node 1 sends feedback information to OAM.

[0055] Step 14: NG-RAN node 2 sends feedback information to OAM.

[0056] For the mobility optimization use case, Section 5.3.2.6 in TR 37.817 further describes the feedback as follows: In order to optimize the performance of the AI / ML-based mobility optimization model, the following data is required as feedback data: (i) QoS parameters of the switched UE, such as throughput, packet delay, etc.; (ii) resource status information updates from the target NG-RAN node; and / or (iii) performance information from the target NG-RAN node (the details of the performance information will be discussed in the specification work phase).

[0057] TR 37.817 also studies the case where both AI / ML model training and AI / ML model inference are located in NG-RAN (i.e. gNB). In this case, feedback is signaled from NG-RAN node 2 to NG-RAN node 1 hosting model training and model inference functions, but is the same as described above.

[0058] TR 37.817 also mentions the use of so-called “model performance feedback”. Section 4.2 of TR 37.817 states:

[0059] Model inference is a function that provides AI / ML model inference output (e.g., predictions or decisions). Where applicable, the model inference function can provide model performance feedback to the model training function. If necessary, the model inference function is also responsible for data preparation (e.g., data preprocessing and cleaning, formatting, and transformation) based on the inference data delivered by the data collection function.

[0060] Output: The inference output of the AI / ML model produced by the model inference function. (Note: the details of the inference output are use case specific.)

[0061] Model performance feedback: This can be used to monitor the performance of AI / ML models (when available). (Note: The details of the model performance feedback process are outside the scope of RAN3.) Summary of the invention

[0062] As can be seen in TR 37.817, feedback is intended to provide an indication about the performance of the AI / ML model, but has not yet been defined. One problem with the current technology is that a node (e.g., OAM) receiving feedback (information) receives feedback related to the same AI / ML model reasoning output (e.g., action) from two or more nodes (e.g., NG-RAN nodes). This means that the node receiving the feedback, or more precisely, for example, the model training function hosted therein, which is responsible for preparing training data, must identify and connect / combine all feedback received from multiple nodes related to the same output (e.g., AI / ML-based action). In practice, associating information including feedback received from several nodes with a specific output (e.g., an action triggered or taken by a specific node at a specific time) is a difficult task, because each node may run multiple use cases in parallel, and each node may report information related to the output (e.g., actions taken by itself and actions taken by other (e.g., neighboring) nodes for different use cases) to the node hosting the model training function and / or the model reasoning function.

[0063] The problem can be further explained with two examples. In one example, two NG-RAN nodes (herein referred to as NG-RAN nodes 1A and 1B) host model reasoning functions and use AI / ML models to implement mobility optimization for their respective (served) set of UEs. Both NG-RAN nodes decide to switch one of their respective UEs to a common third NG-RAN node, here referred to as NG-RAN node 2, at the same time or at a close time instance. NG-RAN node 2 will then observe the performance of the two switched UEs over a period of time and report it to OAM, but OAM does not know which UE was switched according to which of the mobility optimization actions taken by NG-RAN nodes 1A and 1B respectively, so it does not know which UE performance feedback received from NG-RAN node 2 corresponds to which of the two actions.

[0064] In another example, one NG-RAN node (here referred to as NG-RAN Node 1) implements AI / ML model reasoning for network energy conservation and mobility optimization in parallel. At a certain point, using its mobility optimization model, NG-RAN Node 1 decides to handover a UE to another NG-RAN node, again referred to as NG-RAN Node 2. Thereafter, using its network energy conservation model, NG-RAN Node 1 decides to also handover the remaining UEs to NG-RAN Node 2 (e.g., because the traffic / load in the cell of NG-RAN Node 2 is now below a certain threshold); NG-RAN Node 2 will then observe the UE performance of both groups of switched UEs and report it to OAM. However, again, OAM does not know which group of UEs was switched due to which action, so it does not know which UE performance feedback received from NG-RAN Node 2 corresponds to which action.

[0065] Another problem with the current specification is that model performance feedback metrics and other feedback metrics have not been specified. It is agreed that AI / ML models are implementation-specific and therefore will not be specified. For this reason, metrics or other indicators that represent model performance feedback cannot rely on a specific model implementation. For example, expressing model performance feedback as the processing power required for the AI / ML model to run would not be a correct metric, because the required processing power strictly depends on the implementation properties of the model, such as its complexity, output / action range, etc. Therefore, it cannot be regarded as an absolute indicator of how well the model is implemented. Therefore, the problem lies in defining metrics that represent AL / ML model performance in a way that is independent of the AI / ML model implementation.

[0066] Aspects of the present disclosure include a method that enables a first network node or a second network node (e.g., a RAN node, such as an NG-RAN node, a gNB-CU and / or a gNB-DU, or a UE) to provide information associated with the execution of an AI / ML model (implemented by the first network node or the second network node) to a third network node (e.g., another RAN node or an external system, such as an OAM or SMO system). The method enables the third node to effectively collect data associated with the execution of the AI / ML model from multiple network nodes affected by the model execution in order to monitor / evaluate model performance or generate new training data. The collected information associated with the execution of the AI / ML model may include information related to the output of the AI / ML model, performance feedback, etc.

[0067] In some embodiments, the output of the AI / ML model may represent an action associated with RAN functionality, such as UE switching, predicted by the AI / ML model and applied / taken by the first node or the second node. In other examples, the output of the AI / ML model may represent information associated with one or more actions that may be taken by the first or second node, such as an action value, so that an action can be selected based on the output of the AI / ML model. In some other cases, the output of the AI / ML model may represent one or more predictions of a measurable quantity or metric, for example, which may be used by the first node and / or signaled to the second node. In either case, data (such as feedback information) from one or more RAN nodes is required (e.g., by a third node) for the above purposes.

[0068] According to some embodiments, feedback (information) associated with an AI / ML model or its output is reported by a first node (e.g., an NG-RAN node) and / or one or more second nodes (e.g., an NG-RAN node or a UE) to a third network node. The method enables the third node to receive information from many (e.g., first and second) nodes, including feedback related to many outputs generated at different (first) nodes at different times (possibly time-delayed and on request), and still distinguish which specific output each feedback is related to. The latter enables the third node to evaluate the performance impact of each AI / ML output generated by various nodes in the system (e.g., the performance impact of AI / ML-based or AI / ML-triggered actions selected and applied by different nodes), and to determine the performance of the deployed AI / ML models.

[0069] According to one aspect, a computer-implemented method implemented by a first network node in a radio access network (RAN) is provided. The method includes obtaining an output from a machine learning (ML) model. The method includes obtaining an output feedback identifier for the output, wherein the output feedback identifier uniquely identifies the output. The method includes generating a first message, wherein the first message includes the output feedback identifier. The method includes sending the first message including the output feedback identifier to a third network node.

[0070] According to another aspect, a computer-implemented method implemented by a second network node is provided. The method includes receiving a first message generated by a first network node, the first message including an output from a machine learning (ML) model. The method includes obtaining an output feedback identifier associated with the output. The method includes collecting data associated with the output or an execution of the output. The method includes generating a second message, the second message including the collected data and the output feedback identifier. The method includes sending the second message to the first network node or a third network node.

[0071] According to yet another aspect, a computer-implemented method performed by a third network node is provided. The method includes receiving an output feedback identifier associated with an output from a machine learning (ML) model from a first network node or a second network node. The method includes receiving data associated with the output from the first network node or the second network node. The method includes associating the data with the output from the ML model using the output feedback identifier.

[0072] In another aspect, there is provided a network node having a processing circuit adapted to implement the above method. In another aspect, there is provided a computer program comprising instructions which, when executed by the processing circuit of the network node, cause the network node to implement the above method. In another aspect, there is provided a carrier embodying the computer program, wherein the carrier is one of an electronic signal, an optical signal, a radio signal and a computer-readable storage medium. BRIEF DESCRIPTION OF THE DRAWINGS

[0073] The accompanying drawings, which are incorporated herein and constitute a part of this specification, illustrate various embodiments.

[0074] Figure 1 A functional framework for RAN intelligence is shown.

[0075] Figure 2 AI / ML model training at OAM and model inference at NG-RAN for the first use case are shown.

[0076] Figure 3 AI / ML model training at OAM and model inference at NG-RAN for the second use case are shown.

[0077] Figure 4 AI / ML model training at OAM and model inference at NG-RAN for the third use case are shown.

[0078] Figure 5 A simplified flow chart according to some embodiments is shown.

[0079] Figure 6 A flow chart according to some embodiments is shown.

[0080] Figure 7 A flow chart according to some embodiments is shown.

[0081] Figure 8 A flow chart according to some embodiments is shown.

[0082] Fig. 9 Methods according to some embodiments are shown.

[0083] Fig.10 Methods according to some embodiments are shown.

[0084] Fig.11 Methods according to some embodiments are shown.

[0085] Fig.12 is a block diagram of a network node according to some embodiments.

[0086] Fig.13 A flow chart according to some embodiments is shown. DETAILED DESCRIPTION

[0087] One advantage of the techniques disclosed herein is that they enable a third node, e.g., an external system such as an OAM or SMO system, or an AI / ML model training function and / or AI / ML inference function hosted therein, to obtain information from several nodes in the network (e.g., UEs and gNBs), including feedback related to one or more (possibly multiple) outputs generated at different first nodes at different times (possibly time-delayed, upon request, etc.). The techniques also provide advantages over a solution in which information including feedback related to outputs generated at a first node is first reported (back) to the first node and collected at the first node, and then forwarded to a third node.

[0088] Among other benefits, other advantages may include:

[0089] The signaling overhead between the (first and second) nodes is reduced, since the information is directly signaled to the third node.

[0090] The signalling overhead between the (first and second) nodes is reduced, since no (fully specified) request for information from the first node to one or more second nodes is required, nor is a fully specified request including all details regarding the content, time and manner of reporting the information.

[0091] Signaling overhead between any nodes in the network is reduced because signaling the same information (same type of information from the same node, at the same time, etc.) multiple times to a third node can be avoided if the information is necessary to evaluate and / or learn from multiple outputs (e.g., actions taken) generated at the same time or at close time instances.

[0092] Resource utilization at the first node is reduced because it does not need to collect and process information sent by the second node.

[0093] A (potentially) more flexible and efficient data collection strategy, as the third node can request more relevant information when needed (e.g., for AI / ML model training).

[0094] The probability that the third node erroneously associates feedback information with an output (eg, action) to which it is not relevant or does not correspond is (significantly) reduced.

[0095] Faster data collection process, as the feedback information can reach the third node directly without having to be relayed / forwarded by the first node.

[0096] It is possible to perform more fine-grained causal analysis of actions and their consequences.

[0097] More direct observability of the AI / ML model outputs or actions taken that result in system / network performance. This enables better management / optimization of decisions about which actions should be taken based on which AI / ML outputs. For example, action A based on a specific AI / ML output may provide a specific benefit in terms of system performance. However, when another action B based on the same or different AI / ML output is taken, the overall performance benefit generated by actions A and B is reduced relative to taking only action A. Therefore, knowing each output / action feedback information allows the network to make decisions, for example, it is best to take only action A based on a specific AI / ML output, and leave action B to, for example, the legacy decision process.

[0098] Another advantage of the methods described herein is that an AI / ML model performance metric can be derived that determines how good / accurate a particular prediction performed by a given AI / ML model reasoning function is. One advantage of these methods is that such a model prediction error metric / indicator is independent of the type and / or implementation of the AI / ML model. Similarly, as with the above advantages, the model prediction error enables an understanding of how well the AI / ML model performs in predicting a particular parameter (e.g., a measurable quantity or metric). This understanding enables triggering specific actions, such as retraining, updating, dismissing, and / or replacing the AI / ML model. For example, retraining and updating can be performed in a selective manner, involving only a specific range of input data and / or training data for which predictions with high model prediction errors are recorded. These methods allow visibility of predictions with poor / bad performance / accuracy, thereby enabling decisions to be made in different parts of the system / network as to whether to consider such predictions for further processes and / or actions.

[0099] In some embodiments, a first node as described herein refers to a node that generates an output using an AI / ML model and uses / acts on the output or sends a signal to another node (e.g., a second node). In one example, the output is an action, such as a UE handover, and is applied by the first node.

[0100] In some embodiments, the second node as described herein refers to any other node affected by the output of the AI / ML model, for example, any other node involved in or affected in any other way by the action selected and implemented by the first node based on the AI / ML model. It should be clear to those skilled in the art that such a second node can also act as the first node with respect to its AI / ML-based actions (if it also uses the AI / ML model and the methods described herein are also applicable to those actions).

[0101] In some embodiments, the third node as described herein is a node hosting model training functionality and / or model inference functionality. Such a node receives feedback information related to the output of a specific AI / ML model, for example, an action as an output of (or triggered by) a specific AI / ML model, with the purpose of monitoring / evaluating model performance and taking appropriate actions when necessary, such as retraining, updating the model, or deactivating the model, or activating the model.

[0102] Unless otherwise specifically stated, the terms "feedback", "information", "feedback information", "information including feedback" and "data" are used interchangeably herein. Data may refer to any information required for model training or retraining / updating, such as training data, and any information required for model performance evaluation, such as performance feedback / metrics.

[0103] Unless otherwise specifically stated, the terms "AI / ML model" and "model" can be used interchangeably. The term "AI / ML model" can also be used as an abbreviation for "AI / ML model and algorithm". It can also refer to the AI / ML model itself and the supporting software packages or applications required to run it correctly, which can include, for example, software required for data preparation / preprocessing.

[0104] “Network node” or simply “node” may be a RAN node, OAM, core network (CN) node, SMO, network management system (NMS), non-real-time RAN intelligent controller (Non-RT RIC), real-time RAN intelligent controller (RT-RIC), gNB, eNB, en-gNB, ng-eNB, gNB-CU, gNB-CU-CP, gNB-CU-UP, eNB-CU, eNB-CU-CP, eNB-CU-UP, IAB node, IAB donor DU, IAB donor CU, IAB-DU, IAB-MT, O-CU, O-CU-CP, O-CU-UP, O-DU, O-RU, O-eNB, cloud-based network function, cloud-based centralized training node, cloud-based centralized inference node, or UE.

[0105] Figure 5 A simplified flow chart according to some embodiments is shown. Figure 5 Signaling between a first node (502), one or more second nodes (504), and a third node (506) is shown. At 501, the first node generates an AI / ML model reasoning output, such as an AI / ML-based optimization action, using an internally deployed AI / ML model, and then uses or acts on the output, or sends the output to at least another node, such as a second node. The one or more second nodes are all other nodes or entities that receive the output (e.g., an AI / ML-based action or an AI / ML-triggered action derived from the output), or are involved in the output, or are affected by the output in any way. At 503, the first node determines / publishes an output feedback identifier for the AI / ML model reasoning output generated at 501, and provides the output feedback identifier to the second node (optionally as part of 505). At 505, the first node uses the output and / or sends the output to the second node (e.g., applying an action such as UE switching). At 507 and 509, the first node and the second node collect data related to the output, respectively. At 511 and 513 , the first node and the second node provide output-related data (eg, feedback) and an output feedback identifier to the third node, respectively.

[0106] Method in a first network node

[0107] In some embodiments, a method is performed by a first node 502 in a communication network to implement distributed data collection related to an AI / ML model available in the first network node or the second network node 504. The first network node uses the AI / ML model deployed / executed at the first node to determine / publish an output feedback identifier for the AI / ML model inference output generated at the first node, the identifier uniquely identifying a specific output generated at the first node.

[0108] The output feedback identifier is sent to one or more second nodes 504 that receive the output, or are involved in the output, or are affected by the output in any way, for example, to one or more second nodes that will / should / can report information including feedback related to the output. The first node can signal the identifier before, during, or after signaling the output or using / acting on the output.

[0109] The output feedback identifier is sent to the third node 506. The first node may signal the identifier to the third node before, during, or after reporting information including feedback related to the output, e.g., the first node may signal the identifier to the third node together with other information or separately.

[0110] In one embodiment, the output feedback identifier may parameterize one or more outputs generated by execution of the AI / ML model by the first node (or the second network node). Each time a new output is generated, e.g., when the first node (or the second network node) selects an AI / ML output or action, a new (identifier) ​​value is assigned, e.g., from a predefined or configured range of available or expected values.

[0111] The output feedback identifier may also include one or more of the following information:

[0112] (i) instructions related to the AI / ML model, which are used to derive the output, optionally including the AI / ML model ID and version number,

[0113] (ii) an indication of the use case (or use cases) to which the AI / ML-based output (e.g., AI / ML-based action) applies / corresponds,

[0114] (iii) an indication relating to a particular output generated at the first node, such as an index uniquely identifying the output, provided that the output is a particular discrete output from a predefined set of outputs,

[0115] (iv) an indication related to another output (and / or action taken) previously generated at the first node or at a different node, e.g., where the other output or action led to that particular output,

[0116] (v) an indication of the time (e.g., a timestamp) at which the output was generated, e.g., the hour, day, month, and / or year, or, alternatively or additionally, the hour of the day, day of the week, etc.,

[0117] (vi) an indication relating to the area (e.g., area scope) in which / for which the output was generated, e.g., in which cell, RAN node, tracking area, RAN-based notification area, PLMN, etc.,

[0118] (vii) an indication associated with a first node generating an output,

[0119] (viii) an indication related to a group (one or more) of second nodes that receive or are affected by the output, for example, an indication of a group of UEs, such as a UE group tag, which may identify one or more UEs associated with one or a combination of: mobility triggered by load balancing (action), mobility triggered by coverage (i.e., signal strength / quality), or mobility triggered by energy saving (action), and / or

[0120] (ix) Slices applied / corresponding to AI / ML-based outputs (e.g., AI / ML-based actions).

[0121] In one embodiment, the first node may additionally receive a request from a third node to collect and provide necessary data associated with execution of the AI / ML model of the first node. As part of the request for data collection associated with execution of the AI / ML model, the third node may indicate one or more of: (i) an output feedback identifier for reporting data collected for a particular AI / ML model, and / or (ii) a configuration for determining an output feedback identifier for reporting data collected for a particular AI / ML model. In some embodiments, the first network node determines at least the output feedback identifier based on information received by the third network node.

[0122] In some embodiments, an output feedback identifier is signaled between nodes (or entities) in a mobile network, the identifier serving as a key to identify information including feedback related to a particular output or action taken by a first node using an AI / ML model deployed and executed at the first node. The identifier can parameterize the output or action generated using the AI / ML model at the first node. For each new output, such as an action, the first node takes / publishes a new identifier. Note that in some cases, the AI / ML model may be deployed and executed on another node different from the first node, and the output of the AI / ML model may instead be signaled from the other node to the first node in order to use / act on the output, for example, to select and / or apply an action.

[0123] In some embodiments, before, during, or after the output is signaled or used / acted upon (e.g., an application action), an output feedback identifier is signaled from a first node to one or more second nodes involved in or in any way affected by the output (e.g., an action). The identifier is then further signaled from the first node and the one or more second nodes to a third node when reporting necessary (usually requested) information associated with the execution of the AI / ML model. In an alternative embodiment, the output feedback identifier may also be signaled to the third node before information related to the execution of the AI / ML model is signaled.

[0124] In some embodiments, the action taken by the first node (e.g., gNB) is or involves / requires handing over a plurality of UEs to a second node (e.g., gNB), and the first node is interested in global / aggregate statistics related to the performance of all UEs being handed over. In one embodiment, the outbound feedback identifier is signaled in each handover request and is the same for all UEs. Optionally, at least a portion of the outbound feedback identifier (e.g., a portion including an indication of a group of UEs (e.g., a UE group tag)) is the same.

[0125] The output feedback identifier may include various information, such as the specific AI / ML model used to generate the output, for example defined as an AI / ML model ID and version number.

[0126] In some embodiments, the output feedback identifier is broadcast / sent to all relevant nodes, not only the node that performs the action due to the AI / ML model output, but also the nodes that may be affected by the action. When reporting information including feedback related to the output, the second node will include the most recently received identifier.

[0127] Method in a second network node

[0128] Aspects of the present disclosure also include a method performed by a second node in a communication network to enable distributed data collection related to an AI / ML model available in a first network node or a second network node.

[0129] The second network node may receive an output feedback identifier from the first network node, the output feedback identifier being used to report information associated with execution of the AI / ML model of the first node. The second network node may send information associated with execution of the AI / ML model of the first node and the output feedback identifier to the first node or the third node.

[0130] In one embodiment, the second node may receive a request from a third network node to collect and provide necessary data associated with the execution of the AI / ML model of the first node. As part of the request for data collection associated with the execution of the AI / ML model, the third node may indicate one or more of: (i) an output feedback identifier for reporting data collected for a specific AI / ML model, and / or (ii) a request or configuration for determining an output feedback identifier for reporting data collected for a specific AI / ML model. The second network node may determine at least the output feedback identifier based on information received by the third network node.

[0131] Method in a third network node

[0132] Aspects of the present disclosure also include a method performed by a third node in a communication network to enable distributed data collection related to an AI / ML model available in a first network node or a second network node.

[0133] The third node receives the output feedback identifier from the first node or from the second node. The third node may receive the identifier from the first node before, during, or after the first node reports information including feedback related to the output. Thus, it may receive the output feedback identifier together with other information or separately.

[0134] The third node or the AI / ML model training function hosted therein uses an output feedback identifier received from the first node and / or one or more second nodes, which provides information including feedback related to the output obtained by the first node (e.g., using the AI / ML model deployed / executed at the first node) to identify information received from different nodes and associate the information with the output.

[0135] In some embodiments, the third node may request the first node and / or the one or more second nodes to collect and provide necessary data associated with the execution of the AI / ML model. As part of a request for data collection associated with the execution of the AI / ML model, the third node may indicate one or more of the following: (i) what data should be collected, such as information including feedback related to a specific AI / ML model, (ii) an output feedback identifier for reporting data collected for a specific AI / ML model, (iii) a configuration for determining an output feedback identifier for reporting data collected for a specific AI / ML model, such as a range of output feedback identifiers, or a set of parameters for determining an output feedback identifier that can be interpreted by a third network node. In other embodiments, the third node may send a request only to the first node, which in turn may forward the request to one or more second nodes as needed. The one or more second nodes then collect the required data and provide it to the third node.

[0136] Output Feedback (Identifier) ​​Example

[0137] In some embodiments, the first node is a node that implements AI / ML model reasoning and a node that takes action based on AI / ML model output. In some embodiments, the first node may not be a node that implements AI / ML model reasoning, but may simply use, for example, AI / ML model reasoning output received from another / external node to derive an action. The embodiments described herein disclose how to signal information associated with the execution of an AI / ML model executed by / at a first node across multiple nodes, wherein such information may include information related to the output of the AI / ML model, performance feedback information, etc.

[0138] The output of the AI / ML model may represent an action associated with RAN functionality, such as UE handover, predicted by the AI / ML model and applied / taken by the first node or the second node. In other examples, the output of the AI / ML model may represent information associated with one or more actions that may be taken by the first or second node, such as an action value, so that an action may be selected based on the output of the AI / ML model. In some other cases, the output of the AI / ML model may represent one or more predictions of measurable quantities or metrics, which may be used by the first node and / or signaled to the second node, for example. In either case, data such as feedback information from one or more RAN nodes is required by, for example, a third node for the above purposes.

[0139] In one example, the action of the (source) RAN node is or causes the UE to be handed over to another (target) RAN node. Such action is based on the AI / ML model output generated by the RAN node running the AI / ML model inference. In this case, the RAN node running the AI / ML model inference is the first node, and at least the target RAN node and the UE are the second node.

[0140] In one example, the action may include determining parameter values ​​for radio transmission / reception, such as allocation of radio resources, allocation of link adaptation parameters (such as modulation order, rank, MCS index, etc.). In another example, the action may include mobility-related decisions, such as handover of a user equipment from one radio cell to another, load balancing decisions, energy saving configurations (for network nodes or user equipment), etc.

[0141] In the case where the output of the AI / ML model is a prediction of a measurable quantity or metric, the third node may request one or more model performance metrics, such as a mean absolute error or a model prediction error percentage, from the (first and second) nodes for a specific time or time window. In one embodiment, the first node sends the output to at least one second node with a signal, which can measure the predicted quantity or metric at a later point. In one example, the first node switches at least one UE to the second node and further signals a prediction of the amount of traffic that the at least one UE will generate in the next X seconds (e.g., in a handover preparation signaling). The second node may then measure the amount of traffic generated by the at least one UE and calculate at least one model performance metric based on the received prediction and the measured values ​​for one or more UEs. The second node may further signal the at least one model performance metric to the third node, for example, based on a corresponding request from the third node, optionally forwarded by the first node.

[0142] Figure 6 A flow chart according to some embodiments is shown. In some embodiments, Figure 6 Additional steps are shown, demonstrating how the techniques disclosed herein enable efficient collection of distributed data in a RAN. Figure 5 601, 603, 605, 607, 609 and steps 631 and 633 may be optional.

[0143] Figure 6Signaling between a first node (602), one or more second nodes (605A-B), and a third node (606) is shown.

[0144] Step 601 shows AI / ML model deployment. It can be assumed that the first node uses the ML model (e.g., RL agent) to select and implement one or more actions that involve or affect one or more second nodes in any way. It is also assumed that the ML model is trained and retrained / updated externally, that is, trained and retrained / updated by a third node (e.g., an external system such as an OAM or SMO system) or at a third node. This means that the AI / ML model training function responsible for training and updating the ML model is located / hosted at the third node (606). Another option is that the AI / ML model is retrained at the node hosting the AI / ML model reasoning function, in which case steps 627 and 629 will be signaled to such a node. Therefore, before the ML model can be used, it must be deployed in the first node (step 601). In one embodiment, the third node can dispatch and signal a portion of the output feedback identifier to the first node, which is later (in step 611) issued by the first node for each output generated at the first node using the deployed ML model (e.g., action taken by the first node). For example, the third node may assign and signal to the first node a prefix or suffix to be added to the output feedback identifier. In another embodiment, alternatively or additionally, the third node may signal this portion of the output feedback identifier to the first node (and optionally, one or more second nodes) in a feedback / data request (in steps 603, 605). In another embodiment, the third node may (pre)configure the first node (and optionally, the one or more second nodes) with a structure for the output feedback identifier. In another embodiment, the third node may (pre)configure the first node (and optionally, one or more second nodes) with multiple output feedback identifiers and conditions on when to use them.

[0145] Steps 607 and 609 illustrate performance monitoring. After receiving an optional feedback / data request from a third node (603, 605), the first node and one or more second nodes are requested or configured to provide information, including feedback related to one or more outputs (e.g., actions taken by the first node) generated at the first node (e.g., using a specific AI / ML model in a specific version), start monitoring the corresponding performance metrics or indicators, and collect information according to the request and / or 3GPP standards (if applicable). For the second node, in the first step, the monitoring may be limited to observing / checking whether it receives a specific output feedback identifier, and therefore may need to collect and report information. If the second node is a UE, it is possible that the UE is configured by the first node or at least one second node (e.g., a gNB) (to monitor performance and collect feedback), for example, if the output / action is selected using a specific AI / ML model in a specific version, it is requested to report information including feedback related to the output / action involving or affecting the UE.

[0146] Step 611 illustrates generating an AI / ML output. According to an example use case, assume that the first node uses an AI / ML model (deployed in step 601) trained or updated at a third node to generate an output, such as selecting an action, with the purpose of optimizing or improving RAN operations and processes to achieve a specific goal, such as RAN energy saving. The first node may be requested (603) to provide information including feedback related to the output, for example, because the output was generated using a specific AI / ML model in a specific version. Alternatively, the first node may be configured to provide feedback information in another manner, in which case no feedback / data request is required. In either case, the first node issues an output feedback identifier or the like at step 611, which uniquely identifies the exact output generated at the first node for which the first node and / or the second node should provide feedback information to the third node. In one example, the first network node may request the second network node (e.g., UE) to provide feedback information to the first network node based on the identified output. The first network node may then forward the received feedback information (possibly together with the output feedback identifier) ​​to the third network node.

[0147] The first node may publish an output feedback identifier for an output generated at the first node (e.g., an action taken or to be taken by the first node) based on an AI / ML model deployed / executed at the first node, the identifier uniquely identifying a specific output generated at the first node. The identifier may also include one or more of the following information: (i) an indication related to the AI / ML model that is used to derive the output, optionally including the AI / ML model ID and version number; (ii) an indication related to the use case (or multiple use cases) to which the AI / ML-based output (e.g., an AI / ML-based action) is applied / corresponds; (iii) an indication related to a specific output generated at the first node, for example, an index that uniquely identifies the output, provided that the output is a specific discrete output from a predefined set of outputs; (iv) an indication related to another output (and / or action taken) previously generated at the first node or at a different node, for example, where other outputs or actions lead to the specific output (v) an indication related to the time (e.g., a timestamp) at which the output was generated, for example, in which hour, day, month and / or year, or, alternatively or additionally, in which hour of the day, day of the week, etc.; (vi) an indication related to the area (e.g., area scope) in which / for which the output was generated, for example, in which cell, RAN node, tracking area, RAN-based notification area, PLMN, etc.; (vii) an indication related to the first node that generated the output; (viii) an indication related to a set (one or more) of second nodes that received or were affected by the output; and / or (ix) the slice to which the AI / ML-based output (e.g., AI / ML-based action) applies / corresponds.

[0148] The output feedback identifier may parameterize the output of the AI / ML model, for example, the output obtained using the AI / ML model at the first node. For each new output generated, such as a selected action, a new identifier is determined / published by the first node.

[0149] In one embodiment, the first node may also receive a feedback / data request message from a third node (step 603), which includes a request or configuration to provide an output feedback identifier constructed in a specific manner and / or an output feedback identifier including specific information. Therefore, the structure of the output feedback identifier and / or the information included may depend on the request or configuration provided by the third network node.

[0150] Step 613 shows the signaling of an output feedback identifier. The first node signals the output feedback identifier to one or more second nodes involved in the output generated in step 611 or affected in any way by the output, i.e., to one or more second nodes, such as neighboring gNBs and UEs, that will / should / can collect and report information including feedback related to the output. In one example, the action taken by the (first) gNB based on the output is a handover of the UE to a (second) neighboring gNB. Here, the set of second nodes includes at least the neighboring (target) gNB and the UE to be handed over, but it may also include other UEs served by the first gNB or the (second) neighboring gNB. The exact definition of the set of second nodes depends on the AI / ML-enabled use case and the AI / ML model design and must be known or signaled to the first node. The first node may signal the output feedback identifier before, during or after using and / or signaling the output, such as applying an action (such as UE handover). In one embodiment, the first node may signal the output feedback identifier in a handover request message over the Xn interface during the handover preparation process. This means that, in one example, step 613 may be a part of step 615 .

[0151] Steps 603 and 605 illustrate signaling of feedback / data request from the third node. In some embodiments, the third node may use the feedback / data request mechanism to request information associated with the execution of the AI / ML model (e.g., executed by / at the first node) from multiple nodes in the network, where such information may include information related to the output of the AI / ML model, performance feedback, network / system KPIs, etc.

[0152] In order to train, retrain / update or monitor the performance of the ML model, the third node or the AI / ML model training function located / hosted therein must first obtain new (training) data to generate new training samples / examples, which is generally referred to as experience in reinforcement learning (RL). This experience generally includes a tuple (s, a, r, s'), where s is the state of the environment before the action is applied, a is the action applied, r is the reward received, and s' is the state of the environment after the action is applied. In order to generate new experience, the AI / ML model training function must know all information, including feedback defining states s and s', and the reward r received for taking action a for a given state s, which then results in a state transition from s to s'. This means that the third node must obtain all information, including feedback related to the action taken by the first node, that is, the third node must receive all information defining the tuple (s, a, r, s'). To achieve this, the third node may, for example, request information from the first node and one or more second nodes (steps 603, 605). Such a request may include all the basic details about what, when and how to report information to the third node, or, alternatively or additionally, some of the details may be defined in the 3GPP standard. Optionally, the first node and the second node may have been configured to report feedback to the third node without receiving an explicit request. This means that the feedback / data request message (603, 605) may not always be required to trigger further steps that result in signaling feedback data / information to the third node.

[0153] In one embodiment, if the third node is only intended to monitor / evaluate the performance of the ML model, the third node may only need to receive the reward r or receive all information defining the reward r, and therefore only request this information accordingly. Alternatively or additionally, the third node may request specific information in addition to / independent of the reward r, or request information defining the reward r, such as specific network / system KPIs that enable monitoring and / or evaluation of network / system performance affected by the use of a specific AI / ML model.

[0154] It is understood that the first node is able to correctly adopt the ML model, that is, when it chooses action a, it enters state s. If in order to obtain state s, the first node requires specific information from the second node, it is assumed that the first node will request and receive the information from the second node by appropriate means, even if this is Figure 6 Not shown.

[0155] For supervised learning (SL) or another AI / ML approach (in addition to including RL), the training sample may be in the form of a tuple (i, m), where i is the input and m is the measurement, which is the ground truth for the output of the ML model. If the third node is intended to train or retrain / update the ML model, it must receive all the information defining the tuple (i, m). Conversely, if the third node is intended to monitor / evaluate the performance of the ML model, it must receive all the information defining the tuple (i, m, p), or at least all the information defining the tuple (m, p), where p is the prediction generated at the first node, i.e., a specific AI / ML model inference output. Alternatively or additionally, the third node may directly receive one or more AI / ML model performance metrics, such as a model prediction error (in percentage). As previously described, it is assumed that the third node can request this information.

[0156] In one embodiment, the third node may request or configure (via the first network node and / or the second network node) the first node and / or the second node and / or the UE to provide an output feedback identifier constructed in a specific manner.

[0157] In one variant, the output feedback identifier is set to a fixed value set by the third network node. In addition, it may be one or more or any combination of the following options: (i) an identifier of a job initiated by the OAM, (ii) an identifier associated with a (re)training or update process, (iii) an identifier of the AI / ML model (e.g., model ID and version number), (iv) an identifier of the AI / ML model vendor, (v) an identifier of the use case, and / or (vi) an identifier of an output or action (in the case where the third node is only interested in one or more specific outputs or actions).

[0158] In another variant, the output feedback identifier includes: (i) a fixed common part (i.e., a part of the feedback that is common to all parties involved in the output or action or all parties affected in any way by the output or action), and (ii) a dynamic part (determined at runtime by the network nodes and / or UEs involved in or affected in any way by the output or action). The common part may include one or more or any combination of the options listed for the case where the feedback identifier is set to a fixed value. The dynamic portion may be one or more or any combination of the following options: (i) an identifier of the network node / UE affected by the output or action, (ii) an identifier of the role played by the node with respect to the output or action (e.g., source / destination), (iii) a random identifier determined by the network node or UE, (iv) a UE context ID, (v) a timestamp, (vi) an indication (e.g., a label or Boolean value) indicating whether the output (e.g., the action or actions selected based on the output) succeeded or failed, (vii) one or more indications of performance associated with the output or action, such as throughput, packet loss, capacity efficiency / consumption, latency, etc., (viii) one or more indications of RAN-visible QoE metrics and RAN-visible QoE values ​​associated with the output or action (e.g., during UE mobility events), and / or (ix) an indication of whether the feedback is periodic, etc.

[0159] In another example, the identifier may have a long form and a truncated form, and depending on network conditions and resource status of the UE, for example, the appropriate form is used.

[0160] In an example related to mobility, the OAM requests the first and second network nodes to provide an output feedback identifier for a mobility event occurring between these network nodes. The OAM sends a request (as part of a request for receiving feedback information) to one or more of the network nodes and the UE involved in the mobility process to provide an output feedback identifier, the output feedback identifier comprising at least a timestamp associated with the action (e.g., when handover preparation started, when handover execution was completed), a tag indicating whether the handover preparation and execution were successful, an indication of whether the handover was incoming or outgoing, or equivalently, whether the network node was a source RAN node or a target RAN node for the handover.

[0161] The OAM may provide an initial version of the output feedback identifier that contains a fixed (public) portion that includes one or more of the following: a use case identifier (e.g., pointing to "mobility optimization"), an AI / ML model ID, an AI / ML process ID (e.g., used to later reconstruct that the requested feedback is being requested to train / retrain a specific AI / ML model).

[0162] The source RAN node and / or target RAN node and / or UE affected by the output (e.g., involved in an action selected based on the output) sends its output feedback identifier to the OAM (UE, e.g., via the target RAN node) during, upon, or after completing the action. For example, the source RAN node may include in its feedback a timestamp when handover preparation is initiated, the target RAN node may include in its feedback a timestamp corresponding to the handover completion time and a Boolean value indicating whether the handover is successful, and the UE may include in its feedback the layer 1 or application layer throughput experienced before, during, and / or after the UE mobility event.

[0163] In another example, the MR-DC procedure (eg, SN addition or PSCell change) involves the first network node and / or the second network node.

[0164] In one embodiment, the third node may be a node or entity that is outside the OAM or SMO system and thus separate therefrom but is responsible for collecting data.

[0165] In another embodiment related to the above embodiment, the third node (pre-)configures the first node with a specific structure of the output feedback identifier, and understands that the specific structure will be followed / used during all possible interactions and AI / ML related processes between the first node, the second node and the third node. The first node can fill in a portion of the output feedback identifier itself, and when interacting with one or more second nodes, the second node will fill in the rest of the output feedback identifier. In a related example, as described below, after (pre-)configuring the first node with the output feedback identifier, the OAM does not need to send a specific request in order to receive (back) the output feedback identifier.

[0166] In another embodiment, the third node may (pre)configure the first node and one or more second nodes with multiple output feedback identifiers and conditions for when to use them. These conditions may include signaling conditions, load / traffic conditions, resource status of the node, etc. Depending on the conditions, the first node and the second node may determine the appropriate output feedback identifier to be used. The first node may then signal the output feedback identifier to be used to the second node, and the second node may respond. In the case where the second node can use the proposed output feedback identifier, the second node confirms this in its response. Otherwise, the second node may reject with an appropriate reason value and instead suggest another output feedback identifier that can be used.

[0167] In another embodiment, the identifier is broadcast / sent to all relevant nodes, i.e., all nodes that are (potentially) affected by the output generated at the first node (or, for example, an action taken by the first node based on the output generated at the first node). Before reporting information including feedback related to the output, the second node will replace its parameterized identifier with the most recently received identifier.

[0168] Step 615 shows the signal and / or use of the output. In this step, the output is used and / or signaled, for example, the output generated by the AI / ML model (trained or updated at the third node and deployed and executed at the first node according to the example considered) is used at the first node (in step 611). In the specific example described herein, the action is selected and applied by the first node considering or based on the output of the AI / ML model. When the output is signaled or the action is applied, the first node may request one or more second nodes to send the output feedback identifier to the third node. For example, as part of the handover preparation triggered by the AI / ML model, the source RAN node (first node) may send the output feedback identifier associated with the initiated action to the target RAN node (second node) and / or to the UE via the source RAN node, and request the target RAN node and / or UE to propagate the output feedback identifier to the third node. Here, the first / source RAN node may send the output feedback identifier to the second / target RAN node as part of the handover preparation signaling in the handover request message. This will implicitly (or explicitly through a separate indication) request the target RAN node to send the output feedback identifier to the third node when the action is completed or failed. After determining that the handover is successful / failed, the second RAN node may send an output feedback identifier to the third node.

[0169] In the case where the third node has (pre-)configured the first RAN node with a structure for outputting a feedback identifier, when the second RAN node receives an identifier from the first RAN node and a part of the identifier is empty, the second RAN node will know that it needs to complete the identifier and then send the identifier to the third node. In this case, there may also be different versions of the identifier, such as a shorter and a longer version. In this case, the RAN nodes can exchange the versions of the identifier they support / use when Xn is established. In this way, the first RAN node will know which identifier the second RAN node supports and act accordingly. In another example, the first node can send the identifier it uses to the second node together with the NG-RAN node configuration update message. It is also possible that during the AI / ML process, the first node decides to change the version of the identifier used, for example, if the signaling conditions deteriorate and / or the UE cannot support the full identifier, it needs to start using a truncated form. In this case, the first node notifies the second node of the change using the NG-RAN node configuration update message.

[0170] In one variant, during the handover execution phase, the UE receives an RRC message (e.g., an RRC reconfiguration message) from a first (source) RAN node, the message including an outgoing feedback identifier or another identifier that is associated with the outgoing feedback identifier and encoded with a reduced number of bits to optimize interference on the air interface, which the UE may return to a second (target) RAN node when the handover is complete and the UE is connected to the target RAN node or later. The outgoing feedback identifier (or a corresponding shortened version of the identifier) ​​may be derived only by the source RAN node, or may be derived only by the target RAN node (and sent to the UE via the source RAN node as part of a handover command prepared by the target RAN node and included in the RRC reconfiguration message), or both.

[0171] Steps 619, 621, 623 and 625 relate to the collection of data related to the output. In these steps, the relevant performance metrics / indicators are monitored and the required information is collected according to the 3GPP standard and / or the feedback / data request from the third node, as described above. If part of the required information needs to be collected and reported by (e.g., can only be) the UE, the second node can configure the UE accordingly in step 617. As part of this configuration signaling step / process, or in a separate signaling step / process, the second node can also signal the output feedback identifier to the UE. The UE can then collect the required data related to the output (in step 623) and signal / report the data to the second node in step 625 (possibly time-delayed (or even at a separate request)). In conjunction with the data, the UE can also signal (back) the output feedback identifier to the second node together or separately. In a variant of the method, there is no second RAN node. In this case, the data collection step is performed between the first node and the UE, as described below. In one example, data collection from the UE may be performed using a Minimization of Drive Testing (MDT) / Tracking framework for collecting training data, input data, and feedback information for AI / ML models. 3GPP Technical Report (TR) 37.817 was produced as a result of the Study Item (SI) "Enhancements to Data Collection for NR and EN-DC" defined in 3GPP RP-201620, and MDT has been specified for both LTE and NR in TS 37.320.

[0172] At steps 627 and 629, the information is reported to the third node together with the output feedback identifier. After monitoring the relevant performance metrics / indicators and collecting the required information according to the 3GPP standard and / or the feedback / data request from the third node as described above, the first node and the one or more second nodes each signal the output feedback identifier issued by the first node to the third node. The first node and the one or more second nodes may signal the output feedback identifier to the third node before, during or after reporting the information including the feedback related to the output, i.e., they may signal the output feedback identifier to the third node together with other information or separately.

[0173] At 631, the third node or the AI / ML model training function and / or the AI / ML model reasoning function located / hosted therein uses the output feedback identifier received from the first node and the one or more second nodes, in combination with the information including the feedback, to identify the information and associate it with the output generated in step 611 and sent / signaled in step 615. Thus, the third node or the AI / ML model training function and / or the AI / ML model reasoning function located / hosted therein can merge / fuse all information related to the specific output generated at the first node using a specific ML model and version (i.e., at a specific time / region).

[0174] In some embodiments, steps 631 and 633 are optional. At step 631, the third node uses the feedback. In this step, it is assumed that the third node or the AI / ML model training function located / hosted therein merges / fused all required information including feedback related to the output. It can then use this information to evaluate ML model performance and / or retrain / update the ML model (deployed in step 601). For the latter, as described above, it can use the merged / fused information to generate one or more new training data / experiences in the form of tuples (s, a, r, s') in the case of RL, or in the form of tuples (i, m) in the case of SL. These can then be used to retrain / update the ML model, such as an RL agent, to improve ML model performance, thereby enabling the first node to produce better outputs, such as selecting better actions, i.e., further optimize or improve RAN operations and processes.

[0175] Step 633 updates the AI / ML model. In some embodiments, step 633 is largely similar to step 601. It is assumed that the third node, or more precisely, the AI / ML model training function and / or the AI / ML model inference function located / hosted therein, uses the received information including feedback to retrain / update the ML model deployed in step 601. The third node may decide to signal or deploy the updated ML model or the new ML model to the first node so that the first node can use the updated ML model or the new ML model. This enables the first node to produce better outputs, such as selecting better actions, i.e., further optimize or improve RAN operations and procedures.

[0176] In one embodiment, the third node may also assign and signal to the first node a new prefix / suffix for the output feedback identifier, or alternatively or additionally, another part of the output feedback identifier, which is from then on issued by the first node for each action taken by the first node using the updated ML model (or new ML model).

[0177] Variant - does not involve / affect the second RAN node

[0178] Figure 7 A flow chart according to some embodiments is shown. Figure 7 A variation is shown in which the AI / ML model or the output of the AI / ML model does not involve or affect (e.g., the action selected and implemented by the first node (702) based on the AI / ML model or the output of the AI / ML model does not involve or affect) a second RAN node (e.g., a neighboring gNB). In this case, the type of second node involved or affected by the AI / ML model or the output of the AI / ML model is a UE (704). Therefore, Figure 6 Step 613 shown in is not applicable, i.e. skipped. In addition, if the UE is configured to collect and report data related to the output in step 617, it will / must signal / report such data to the first node (702) in step 625. Figure 6 The other steps described apply accordingly, including the description regarding the third network node (706), but not involving the second RAN node.

[0179] Variant - Initial feedback / data request sent to first RAN node only

[0180] Figure 8 A flow chart according to some embodiments is shown. Figure 8 A variant is depicted in which the initial feedback / data request is sent only from the third node (806) to the first node (802) and therefore not to the second node (804). In this case, in step 613, the first node forwards the feedback / data request to all relevant second (RAN) nodes (804). In one embodiment, the feedback / data request is simply forwarded / relayed by the first node as received from the third node. In another embodiment, the first node sends its own feedback / data request to the relevant second RAN nodes. In this case, the first node may have (slightly) modified the request received from the third node, or created a new (different) request based on the request received from the third node. In the latter case, the first node may use a different data request mechanism / procedure, such as designed / intended for inter-RAN node signaling, such as via Xn. In either case, the first node signals an outgoing feedback identifier to the relevant second RAN node in step 613 or optionally in step 615. The outgoing feedback identifier may be signaled together with the feedback / data request or may be signaled separately. All other aspects described above apply accordingly, and the other steps described above apply similarly.

[0181] Fig.13A flow chart according to some embodiments is shown. Fig.13 yes Figure 8 , wherein the first node 1302 signals to the relevant second (RAN) node 1304 in step 613 that the collected information should be sent to the first node 1302, such as Fig.13 In one embodiment, the third node 1306 explicitly signals in the feedback / data request of step 603 that the first node 1302 should collect all relevant data before signaling the relevant data to the third node in step 627. Optionally, in the signaling of step 603, the request for collecting information including feedback from the second (RAN) node may be implicit. Fig.13 The other steps in apply accordingly, but taking into account that instead of sending to the third node 1306, or in addition to sending to the third node 1308, the second (RAN) node 1304 signals data related to the output and the output feedback identifier to the first node 1302.

[0182] Model prediction error example

[0183] In some embodiments, a node derives a prediction of a particular parameter, metric, or measurable quantity via an AI / ML-supported process (e.g., via an AI / ML model), or receives such a prediction from another node, and may calculate a prediction error in a manner independent of the type of AI / ML model or AI / ML model implementation used to perform the prediction. This calculation is performed by comparing the prediction to a corresponding ground truth (e.g., a measurement of the predicted parameter, metric, or quantity). This means that the "ground truth" refers to the actual / measured value of the predicted parameter, metric, or quantity at the time or period for which the prediction is assumed to be valid.

[0184] An example of an AI / ML model implementation-independent manner of providing AI / ML model performance feedback in the form of model prediction error is the following equation (1):

[0185] (1)

[0186] Where P_i refers to the predicted output value for a specific parameter (referred to as p above); M_i refers to the ground truth for P_i (referred to as m above); TotNumOutputs is the total number of predicted output values ​​P_i considered when calculating the model prediction error, where TotNumOutputs ≥ 1.

[0187] To achieve the above, the node that derives the predictions (e.g., an NG-RAN node hosting the AI / ML model inference function, or a node that receives such predictions from another node) needs to obtain the ground truth corresponding to these predictions, for example by implementing or requesting relevant / appropriate measurements. It is assumed here that the model prediction error can be determined only for predictions of measurable quantities. Those skilled in the art will recognize that the method can be applied to any type of AI / ML model inference output, as long as the difference between the AI / ML output and the ground truth can be quantified.

[0188] As an example of the latter case, for a mobility case, an output or action (such as "the best target cell is X") may be a prediction of the target cell to which the UE will switch during a conditional handover (CHO) process. In this case, there are several candidate target cells, and there may be a candidate target cell that is the best handover target cell within a specific time interval. However, after a certain time, for example, as the traffic / load in the cell changes, the same handover target cell may no longer be the best handover target cell. Then, the best handover target cell may be a different cell. In this example, the source node prepares candidate target cells X, Y, and Z for CHO (i.e., the action is "prepare"). After CHO is completed or fails, the source node may indicate that candidate target cell X is used (e.g., defined as |P i -M i |=0) or is not used (for example, defined as |P i -M i |=1) is compared to the number of times candidate target cells X were prepared due to prediction or TotNumOutputs. Thus, in the case of performing predictions on actions, this quantifiable comparison can be used to derive the model prediction error.

[0189] Model prediction error enables performance evaluation and comparison of different AI / ML models during or after training (if they were trained in the same environment), as well as performance monitoring of AI / ML models during inference. For example, if the target environment experiences measurable drift, such as data and / or concept drift, AI / ML model decay (i.e., AI / ML model performance degradation) can be detected as model prediction error increases.

[0190] Fig. 9A method 900 according to some embodiments is shown. The method 900 is a computer-implemented method implemented by a first network node in a radio access network (RAN). Step s901 of the method comprises: obtaining an output from a machine learning (ML) model. Step s903 of the method comprises: obtaining an output feedback identifier for the output, wherein the output feedback identifier uniquely identifies the output. Step s905 of the method comprises: generating a first message, wherein the first message comprises the output feedback identifier. Step s907 of the method comprises: sending the first message comprising the output feedback identifier to a third network node.

[0191] Fig.10 A method 1000 according to some embodiments is shown. The method 1000 is a computer-implemented method implemented by a second network node. Step s1001 of the method comprises: receiving a first message generated by a first network node, the first message comprising an output from a machine learning (ML) model. Step s1003 of the method comprises: obtaining an output feedback identifier associated with the output. Step s1005 of the method comprises: collecting data associated with the output or the execution of the output. Step s1007 of the method comprises: generating a second message, the second message comprising the collected data and the output feedback identifier. Step s1009 of the method comprises: sending the second message to the first network node or a third network node.

[0192] Fig.11 A method 1100 according to some embodiments is shown. The method 1100 is a computer-implemented method performed by a third network node. Step s1101 of the method comprises: receiving an output feedback identifier associated with an output from a machine learning (ML) model from a first network node or a second network node. Step s1103 of the method comprises: receiving data associated with the output from the first network node or the second network node. Step s1105 of the method comprises: associating the data with the output from the ML model using the output feedback identifier.

[0193] Fig.12 1 is a block diagram of a network node 1200 according to some embodiments. In some embodiments, the computing device 1200 may include one or more of the components of a network node (such as the first, second, and / or third network nodes described herein). Fig.12As shown, the network node may include: a processing circuit (PC) 1202, which may include one or more processors (P) 1255 (e.g., one or more general-purpose microprocessors and / or one or more other processors, such as application-specific integrated circuits (ASICs), field programmable gate arrays (FPGAs), etc.); a communication circuit 1248, including a transmitter (Tx) 1245 and a receiver (Rx) 1247, for enabling the device to send and receive data over a network 1210 (e.g., wirelessly send / receive data); and a local storage unit (also referred to as a "data storage system") 1208, which may include one or more non-volatile storage devices and / or one or more volatile storage devices. In an embodiment in which the PC 1202 includes a programmable processor, a computer program product (CPP) 1241 may be provided. The CPP 1241 includes a computer-readable medium (CRM) 1242, which stores a computer program (CP) 1243 including computer-readable instructions (CRI) 1244. The CRM 1242 may be a non-transitory computer readable medium, such as a magnetic medium (e.g., a hard disk), an optical medium, a storage device (e.g., a random access memory, a flash memory), etc. In some embodiments, the CRI 1244 of the computer program 1243 is configured such that when executed by the PC 1202, the CRI causes the device to implement the steps described herein (e.g., the steps described with reference to the flowchart). In other embodiments, the device may be configured to implement the steps described herein without requiring code. That is, for example, the PC 1202 may include only one or more ASICs. Thus, the features of the embodiments described herein may be implemented in hardware and / or software.

[0194] Although various embodiments are described herein, it should be understood that they are provided as examples only and not as limitations. Therefore, the breadth and scope of the present disclosure should not be limited by any of the above-described embodiments. In addition, unless otherwise specified herein or clearly contradictory to the context, any combination of the above-mentioned elements in all possible variations thereof is included in the present disclosure.

[0195] Furthermore, although the processes described above and shown in the accompanying drawings are shown as a series of steps, this is for illustration only. Therefore, it is contemplated that some steps may be added, some steps may be omitted, the order of the steps may be rearranged, and some steps may be performed in parallel.

[0196] References

[0197] [1] 3GPP TR 37.817 (v17.0.0): “Study on enhancements to data collection for NR and EN-DC” (Release 17).

[0198] Abbreviations

[0199] 3GPP Third Generation Partnership Project

[0200] 5G Fifth Generation

[0201] 5GC 5G Core Network

[0202] 5GS Fifth Generation System

[0203] AI

[0204] AMF Access and Mobility Management Function

[0205] AR Augmented Reality

[0206] AS Access Layer

[0207] ASN.1 Abstract Syntax Notation One

[0208] AT Note

[0209] CGI Cell Global Identifier

[0210] CN Core Network

[0211] CP Control Plane

[0212] CU Central Unit

[0213] CU-CP Central Unit Control Panel

[0214] CU-UP Central Unit User Plane

[0215] DASH Dynamic Adaptive Streaming over HTTP

[0216] DC Dual Connection

[0217] DL Downlink

[0218] DNS Domain Name System

[0219] DU Distributed Unit

[0220] E-CGI E-UTRAN CGI

[0221] EN E-UTRAN-NR

[0222] eNB Evolved Node B / E-UTRAN Node B

[0223] en-gNB is a gNB that acts as a secondary node in an EN-DC scenario (i.e., in a DC scenario where the eNB is the primary node and the gNB is the secondary node).

[0224] EPC Evolved Packet Core

[0225] EPS Evolved Packet System

[0226] E-UTRA Evolved UTRA

[0227] E-UTRAN / EUTRAN Evolved UTRAN

[0228] gNB NR radio base station

[0229] HSS Home Subscriber Server

[0230] HTTP Hypertext Transfer Protocol

[0231] IAB Integrated Access and Backhaul

[0232] ID Identifier / Identification

[0233] IE Information Elements

[0234] LTE Long Term Evolution

[0235] MAC Media Access Control

[0236] MCC Mobile Country Code

[0237] MCE Measurement Collection Entity / Measurement Collector Entity

[0238] MDT Minimization of driving tests

[0239] ML Machine Learning

[0240] MME Mobility Management Entity

[0241] MNC Mobile Network Code

[0242] MTSI Multimedia Telephony Services for IMS

[0243] N3IWF Non-3GPP Interworking Function

[0244] NG Next Generation

[0245] NG Interface between NG-RAN and 5GC.

[0246] NGAP NG Application Protocol

[0247] NG-RAN NG Radio Access Network

[0248] NID Network Identifier

[0249] NR New Radio

[0250] NWDAF network data analysis function

[0251] O&M Operation & Maintenance

[0252] OAM Operation and Maintenance

[0253] PDCP Packet Data Convergence Protocol

[0254] PDU Protocol Data Unit

[0255] PLMN Public Land Mobile Network

[0256] QMC QoE measurement collection

[0257] QoE Quality of Experience

[0258] RAN Radio Access Network

[0259] RAT Radio Access Technology

[0260] RL

[0261] RLC Radio Link Control

[0262] RNC Radio Network Controller

[0263] RRC Radio Resource Control

[0264] RVQoE RAN Visible QoE

[0265] S1 Interface between RAN and CN in LTE.

[0266] S1AP S1 Application Protocol

[0267] SL Supervised Learning

[0268] SMO Service Management and Orchestration

[0269] S-NSSAI Single Network Slice Selection Auxiliary Information

[0270] SRB Signalling Radio Bearer

[0271] TA Tracking Area

[0272] TCE Trace Collection Entity / Trace Collector Entity

[0273] TNGF Trusted Non-3GPP Gateway Function

[0274] TWIF Trusted WLAN Intercommunication Function

[0275] UDM Unified Data Management

[0276] UE User Equipment

[0277] UMTS Universal Mobile Telecommunications System

[0278] URI Uniform Resource Identifier

[0279] URL Uniform Resource Locator Uniform Resource Locator

[0280] UTRA Universal Terrestrial Radio Access

[0281] UTRAN Universal Terrestrial Radio Access Network

[0282] WLAN Wireless Local Area Network

[0283] Xn The interface between two gNBs in NR.

[0284] XnAP Xn Application Protocol

Claims

1. A computer-implemented method (900) performed by a first network node (202, 302, 402, 502, 602, 702, 802, 1302) in a radio access network RAN, the method comprising: obtaining (s901) an output from a machine learning ML model; obtaining (s903) an output feedback identifier for the output, wherein the output feedback identifier uniquely identifies the output; generating (s905) a first message, wherein the first message includes the output feedback identifier; and The first message including the output feedback identifier is sent (s907) to a third network node (206, 306, 406, 506, 606, 706, 806, 1306).

2. The method according to claim 1, further comprising: Data related to the output from the ML model is collected, wherein the first message also includes the collected data.

3. The method according to any one of claims 1 to 2, further comprising: identifying a second network node (204, 208, 304, 308, 404, 408, 504, 604A, 604B, 704, 804, 808, 1304, 1308) that receives the output, is involved in the output, or is affected by the output; generating a second message including the output feedback identifier; and The second message including the output feedback identifier is sent to the second network node.

4. The method according to claim 3, wherein: The second message is a handover request or a NG-RAN node configuration update.

5. The method according to any one of claims 3 to 4, wherein: The output includes an action to be performed by at least one of the first network node or the second network node.

6. The method according to any one of claims 1 to 5, wherein: The output feedback identifier is a parameterization of one or more outputs generated by the ML model.

7. The method according to any one of claims 1 to 5, wherein: The output feedback identifier is a value assigned from a predefined or configured range of values.

8. The method according to any one of claims 1 to 7, wherein: The output feedback identifier also includes one or more of the following: instructions related to the ML model, an indication related to a use case to which the output of the ML model corresponds, an indication related to said output generated at said first node, an indication related to a second output previously generated at said first network node or a different network node, an indication of the time at which said output was generated, an indication relating to a region in the RAN for which the output is generated, an indication related to the first network node, an indication of one or more network nodes that receive, are involved in, or are affected by the output, or A slice in the RAN corresponding to the output.

9. The method according to any one of claims 1 to 8, further comprising: A third message generated by the third network node is received, wherein the third message includes a request to collect and provide data associated with the output of the ML model.

10. The method according to claim 9, wherein: The third message includes information indicating the output feedback identifier or a configuration for determining the output feedback identifier, and wherein obtaining the output feedback identifier further includes: The output feedback identifier is determined based on the information.

11. The method according to any one of claims 1 to 10, wherein: The ML model is executed at the first network node or a different network node.

12. The method according to any one of claims 1 to 11, wherein: The output from the ML model comprises a predicted parameter, metric or quantity in the RAN.

13. The method according to claim 12, further comprising: obtaining, from a second network node, a first measurement related to the parameter, metric or quantity; as well as A model prediction error of the ML model is determined based on the first measurement.

14. The method according to claim 13, further comprising: A second measurement related to the parameter, metric or quantity is obtained at the first network node, wherein determining the model prediction error of the ML model is based on the first measurement, the second measurement, or a combination of the first measurement and the second measurement.

15. The method according to claim 12, further comprising: A model prediction error of the ML model based on a measurement related to the parameter, metric or quantity is obtained from the second network node.

16. A computer-implemented method (1000) performed by a second network node (204, 208, 304, 308, 404, 408, 504, 604A, 604B, 704, 804, 1304), the method comprising: Receiving (s1001) a first message generated by a first network node (202, 302, 402, 502, 602, 702, 802, 1302), the first message comprising an output from a machine learning (ML) model; obtaining (s1003) an output feedback identifier associated with the output; collecting (s1005) data related to the output or the execution of the output; generating (s1007) a second message, the second message comprising the collected data and the output feedback identifier; as well as The second message is sent (s1009) to the first network node and / or the third network node (206, 306, 406, 506, 606, 706, 806, 1306).

17. The method according to claim 16, further comprising: A third message generated by the third network node is received, wherein the third message includes a request to collect and provide data associated with the output of the ML model.

18. The method according to claim 17, wherein: The third message includes information indicating the output feedback identifier or a configuration for determining the output feedback identifier, and wherein obtaining the output feedback identifier further includes: The output feedback identifier is determined based on the information.

19. The method according to any one of claims 16 to 18, wherein: The ML model is executed at the first network node or a different network node.

20. The method according to any one of claims 16 to 19, wherein: The output relates to an action to be performed by the second network node or a different network node.

21. The method according to any one of claims 16 to 20, wherein: The output from the ML model comprises a predicted parameter, metric or quantity in the RAN.

22. The method according to claim 21, further comprising: collecting measurements related to said parameters, metrics or quantities; as well as The measurements are sent to the first network node or the third network node.

23. The method according to claim 21, further comprising: collecting a first measurement related to the parameter, metric, or quantity; as well as A model prediction error of the ML model is determined based on the first measurement.

24. The method according to claim 23, further comprising: A second measurement related to the parameter, metric, or quantity is obtained, and wherein determining the model prediction error of the ML model is based on the first measurement, the second measurement, or a combination of the first measurement and the second measurement.

25. The method according to any one of claims 23 to 24, further comprising: generating a fourth message, the fourth message comprising the model prediction error of the ML model; as well as The fourth message is sent to the first network node or the third network node.

26. The method according to any one of claims 16 to 25, wherein: The first message is a handover request or a NG-RAN node configuration update.

27. A computer-implemented method (1100) performed by a third network node (206, 306, 406, 506, 606, 706, 806, 1306), the method comprising: receiving (s1101) an output feedback identifier associated with an output from a machine learning (ML) model from a first network node (202, 302, 402, 502, 602, 702, 802, 1302) or a second network node (204, 208, 304, 308, 404, 408, 504, 604A, 604B, 704, 804, 1304); receiving (s1103) data related to the output from the first network node or the second network node; as well as The data is associated with the output from the ML model using the output feedback identifier (s1105).

28. The method according to claim 27, further comprising: aggregating the data with additional data to form a set of aggregated data associated with the ML model; as well as The ML model is trained, monitored, evaluated, or updated using the set of aggregated data.

29. The method according to any one of claims 27-28, wherein: The output feedback identifier and the data are received in a single message from the first network node or the second network node.

30. The method according to any one of claims 27 to 29, further comprising: generating a message, wherein the generated message includes a request to collect and provide data associated with the output of the ML model; and The generated message is sent to at least one of the first network node or the second network node.

31. The method according to claim 30, wherein: The generated message includes information indicating one or more of the following: the output feedback identifier, for determining the configuration of the output feedback identifier, or An indication of what data should be collected.

32. The method according to any one of claims 27 to 31, wherein: The ML model is executed at the first network node or a different network node.

33. The method of claim 32, wherein: The different network node is the third network node.

34. The method according to any one of claims 27 to 33, wherein: The output relates to an action to be performed by the first network node or the second network node.

35. The method according to any one of claims 27 to 34, wherein: The third network node is included in an Operation, Administration and Maintenance OAM or Service, Management and Orchestration SMO system.

36. The method according to any one of claims 27 to 35, further comprising: The output feedback identifier is received from both the first network node and the second network node.

37. The method of claim 36, further comprising: Data related to the output is received from both the first network node and the second network node.

38. The method according to any one of claims 27 to 37, wherein: The output from the ML model comprises a predicted parameter, metric or quantity in the RAN.

39. The method of claim 38, further comprising: obtaining measurements related to said parameters, measures or quantities; as well as A model prediction error of the ML model is determined based on the measurements.

40. The method of claim 38, further comprising: A model prediction error of the ML model based on a measurement related to the parameter, metric or quantity is obtained from the second network node or the first network node.

41. A network node (1200) adapted to implement the method of any one of claims 1-40.

42. A computer program (1243) comprising instructions (1244) which, when executed by a processing circuit (1202) of a network node (1200), cause the network node to implement the method of any one of claims 1-40.

43. A carrier comprising a computer program according to claim 42, wherein: The carrier is one of an electronic signal, an optical signal, a radio signal and a computer-readable storage medium (1242).

Images