Synchronization of quality based on network digital twinning

Abstract

A method includes receiving, from a service consumer of a network digital twinning, a request to provide an indication of a quality of the network digital twinning; determining the quality of the network digital twinning; in response to receiving the request, an indication of the determined quality is provided to the service consumer, where the quality indicates how accurately the network digital twin represents the real network.

Description

TECHNICAL FIELD

[0001] The present disclosure relates to the quality of network digital twinning. Abbreviations 3GPP: Third Generation Partnership Project 5G / 6G / 7G: Fifth, Sixth, Seventh Generation AI: Artificial Intelligence AIML: Artificial Intelligence - Machine Learning AnLF: Analytics Logic Function ES: Energy Saving gNB: Next Generation (5G) Node B HO: Handover ID: Identifier KPI: Key Performance Indicator MDA: Management Data Analytics MDAS: Management Data Analytics Service ML: Machine Learning MLOps: Machine Learning Operations MM: Mobility Management MnS: Management Service NDT: Network Digital Twin NWDAF: Network Data Analytics Function OAM: Operation and Maintenance PRB: Physical Resource Block QoNDT: Quality of Network Digital Twin RRC: Radio Resource Control RSRP: Reference Signal Received Power RSRQ: Reference Signal Received Quality SINR: Signal to Interference and Noise Ratio SON: Self-Organizing Network UE: User Equipment BACKGROUND

[0002] Digital twinning is a virtual instance of a physical system (the twin) that is continuously updated with performance, maintenance, and health status data of the physical system throughout its entire life cycle. Such updates are often referred to as synchronization between the NDT and the physical / real system. Besides importing update information from the real system into the NDT, the opposite synchronization direction (from the NDT to the real system) is also envisaged.

[0003] Digital twin network: a digital twin used in a network environment. This is also referred to as a digital twin for a network or a network digital twin (NDT). The ANDT can be used to answer “what-if” questions to estimate the impact of certain measures (e.g., (re)configurations) on the network, to make recommendations, and to collect feedback. SUMMARY

[0004] It is an object to improve the prior art.

[0005] According to a first aspect, there is provided an apparatus comprising: one or more processors, and a memory storing instructions that, when executed by the one or more processors, cause the apparatus to perform: receiving, from a service consumer of a network digital twin, a request for an indication of a quality of the network digital twin; determining the quality of the network digital twin; in response to receiving the request, providing, to the service consumer, an indication of the determined quality, wherein the quality indicates how accurately the network digital twin represents a real network.

[0006] The quality can be represented by at least one of: an input drift indicating a difference between first data used to implement the network digital twin and corresponding data measured in the real network; an average of the input drift over a first time period; an output drift indicating a difference between second data generated by the network digital twin and corresponding data measured in the real network; or an average of the output drift over a second time period.

[0007] The instructions, when executed by the one or more processors, can further cause the apparatus to perform: receiving a specification of at least one of: the first data, the second data, the first time period, or the second time period; performing the determining by determining at least one of the input drift or the average input drift for a specified first data or for a specified first time period, respectively, or by determining at least one of the output drift or the average output drift for a specified second data or for a specified second time period, respectively.

[0008] The instructions, when executed by the one or more processors, can further cause the apparatus to perform: receiving, from the service consumer, a first requirement for the quality; checking whether the determined quality meets the first requirement; in response to checking that the determined quality does not meet the first requirement, synchronizing from the real network to the network digital twin; determining the quality after the synchronisation; providing an indication of the determined quality such that the quality after the synchronisation is provided.

[0009] The instructions, when executed by the one or more processors, can further cause the apparatus to, in response to checking that the determined quality does not meet the first requirement, perform at least one of: determining a time instance at which the synchronisation from the real network to the network digital twin should start; determining a time instance at which the synchronisation from the real network to the network digital twin should end; determining a trigger event for initiating the synchronisation from the real network to the network digital twin; determining data in the network digital twin that will be updated by the synchronisation from the real network to the network digital twin; or determining data of the real network that will be used for the synchronisation from the real network to the network digital twin.

[0010] The request for providing an indication of the quality of the network digital twin can comprise a request to perform a synchronisation from the network digital twin to the real network; and the instructions, when executed by the one or more processors, can further cause the apparatus to perform: receiving a second requirement for the quality from the service consumer; checking whether the determined quality meets the second requirement; in response to checking that the determined quality meets the second requirement, synchronising from the network digital twin to the real network in accordance with the request to perform the synchronisation; determining the quality after the synchronisation; providing an indication of the determined quality such that the quality after the synchronisation is provided.

[0011] The instructions, when executed by the one or more processors, can further cause the apparatus to, in response to checking that the determined quality meets the second requirement, perform at least one of: determining one or more actions that will be performed by the synchronisation from the network digital twin to the real network; determining a time instance at which the synchronisation from the network digital twin to the real network is performed; or determining a trigger event and stopping the synchronisation from the network digital twin to the real network in response to detecting the trigger event.

[0012] The instructions, when executed by the one or more processors, can further cause the apparatus to perform: in response to checking that the determined quality does not meet the second requirement, inhibiting the synchronisation from the network digital twin to the real network in accordance with the request to perform the synchronisation.

[0013] According to a second aspect, there is provided an apparatus comprising: one or more processors, and memory storing instructions that, when executed by the one or more processors, cause the apparatus to perform: requesting a service producer of a network digital twin to provide an indication of a quality of the network digital twin; receiving the indication of the quality in response to the requesting, the quality indicating how accurately the network digital twin represents a real network.

[0014] The quality can be indicated by at least one of: an input drift indicating a difference between first data used to implement the network digital twin and corresponding data measured in the real network; an average of the input drift over a first time period; an output drift indicating a difference between second data generated by the network digital twin and corresponding data measured in the real network; or an average of the output drift over a second time period.

[0015] The instructions, when executed by the one or more processors, can further cause the apparatus to perform: specifying, to the service producer, at least one of the first data, the second data, the first time period, or the second time period.

[0016] The instructions, when executed by the one or more processors, can further cause the apparatus to perform: informing the service producer of a requirement for the quality.

[0017] The instructions, when executed by the one or more processors, can further cause the apparatus to perform: requesting the service producer to perform a synchronization from the network digital twin to the real network.

[0018] The instructions, when executed by the one or more processors, can further cause the apparatus to perform: checking whether the quality meets a criterion; determining an action on the real network based on an analysis of assumptions of the network digital twin; in response to checking that the quality meets the criterion, instructing a consumer of a service to perform the determined action.

[0019] The instructions, when executed by the one or more processors, can further cause the apparatus to perform: in response to determining that the quality does not meet the criterion, refraining from determining the action on the real network; or in response to determining that the quality does not satisfy the criterion, prohibiting a consumer of the instruction management service from performing the determined action on the real network.

[0020] The action can relate to at least one of mobility management in the real network or energy saving in the real network.

[0021] According to a third aspect, there is provided a method comprising: receiving, from a service consumer of a network digital twin, a request for an indication of a quality of the network digital twin; determining a quality of the network digital twin; in response to receiving the request, providing the indication of the determined quality to the service consumer, wherein the quality indicates how accurately the network digital twin represents the real network.

[0022] The quality can be indicated by at least one of: an input drift indicating a difference between first data used to implement the network digital twin and corresponding data measured in the real network; an average of the input drift over a first time period; an output drift indicating a difference between second data generated by the network digital twin and corresponding data measured in the real network; or an average of the output drift over a second time period.

[0023] The method can further comprise: receiving a specification of at least one of: the first data, the second data, the first time period, or the second time period; wherein The determining can comprise determining at least one of the input drift or the average input drift for the specified first data or for the specified first time period, respectively, or determining at least one of the output drift or the average output drift for the specified second data or for the specified second time period, respectively.

[0024] The method can further comprise: receiving, from the service consumer, a first requirement for the quality; checking whether the determined quality satisfies the first requirement; in response to checking that the determined quality does not satisfy the first requirement, synchronizing from the real network to the network digital twin; determining the quality after the synchronizing; wherein the indication of the determined quality is provided such that the quality after the synchronizing is provided.

[0025] The method can further comprise at least one of the following in response to checking that the determined quality does not satisfy the first requirement: - determining a time instance at which synchronization from the real network to the network digital twin should be started; - determining a time instance at which synchronization from the real network to the network digital twin should be ended; - determining a trigger event for initiating synchronization from the real network to the network digital twin; - determining data in the network digital twin that is to be updated by synchronization from the real network to the network digital twin; or - determining data of the real network that is to be used for synchronization from the real network to the network digital twin.

[0026] The request for an indication of the quality of the network digital twin can comprise a request to perform synchronization from the network digital twin to the real network; and the method can further comprise: receiving a second requirement for the quality from the service consumer; checking whether the determined quality meets the second requirement; in response to checking that the determined quality meets the second requirement, synchronizing from the network digital twin to the real network in accordance with the request to perform synchronization; determining the quality after the synchronization; wherein the indication of the determined quality is provided such that the quality after the synchronization is provided.

[0027] The method can further comprise at least one of the following in response to checking that the determined quality meets the second requirement: - determining one or more actions that are to be performed by synchronization from the network digital twin to the real network; - determining a time instance at which to perform synchronization from the network digital twin to the real network; or - determining a trigger event and stopping synchronization from the network digital twin to the real network in response to detecting the trigger event.

[0028] The method can further comprise: in response to checking that the determined quality does not meet the second requirement, inhibiting synchronization from the network digital twin to the real network in accordance with the request to perform synchronization.

[0029] According to a fourth aspect, a method is provided, the method comprising: one or more processors, and a memory storing instructions that, when executed by the one or more processors, cause the method to perform: requesting a service producer of a network digital twin to provide an indication of a quality of the network digital twin; receiving the indication of the quality in response to the request, wherein the quality indicates how accurately the network digital twin represents a real network.

[0030] The quality can be represented by at least one of: an input drift indicating a difference between the first data used to implement the network digital twin and corresponding data measured in the real network; an average of the input drift over the first time period; an output drift indicating a difference between the second data generated by the network digital twin and corresponding data measured in the real network; or an average of the output drift over the second time period.

[0031] The method can further comprise: specifying to the service producer at least one of the first data, the second data, the first time period, or the second time period.

[0032] The method can further comprise informing the service producer about the requirement for the quality.

[0033] The method can further comprise: requesting the service producer to perform a synchronization from the network digital twin to the real network.

[0034] The method can further comprise: checking whether the quality meets a criterion; determining an action on the real network based on the analysis of the assumptions of the network digital twin; instructing a consumer of the management service to perform the determined action on the real network in response to checking that the quality meets the criterion.

[0035] The method can further comprise: inhibiting determining an action on the real network in response to determining that the quality does not meet the criterion; or inhibiting instructing a consumer of the management service to perform the determined action on the real network in response to determining that the quality does not meet the criterion.

[0036] The action can relate to at least one of mobility management in the real network or energy saving in the real network.

[0037] Each of the methods of the third aspect or the fourth aspect can be a synchronized method.

[0038] According to a fifth aspect, there is provided a computer program product comprising a set of instructions which, when executed on an apparatus, is configured to cause the apparatus to perform a method according to any of the third aspect or the fourth aspect. The computer program product can be embodied as a non-transitory computer-readable medium, or directly loadable into a computer.

[0039] According to some example embodiments, at least one of the following advantages can be achieved: the quality of the NDT (i.e., how accurately the NDT reflects the real system); the synchronization schedule can be optimized such that synchronization from the real system to the NDT can be performed when needed, while unnecessary synchronization from the real system to the NDT can be avoided. synchronization from the NDT to the real system can be performed only when actions and recommendations derived from the NDT have a high confidence to accurately predict the behavior of the real system.

[0040] It should be appreciated that any of the above modifications can be applied to the respective aspect to which it refers, either individually or in combination, unless it is explicitly stated that the alternative is excluded. BRIEF DESCRIPTION OF DRAWINGS

[0041] Further details, features, objects, and advantages will be apparent from the following detailed description of the preferred example embodiments taken in conjunction with the accompanying drawings, in which:

[0042] Figure 1 A message sequence chart is shown in accordance with some example embodiments;

[0043] Figure 2 A message sequence chart is shown in accordance with some example embodiments;

[0044] Figure 3 A message sequence chart is shown in accordance with some example embodiments;

[0045] Figure 4 A message sequence chart is shown in accordance with some example embodiments;

[0046] Figure 5 A message sequence chart is shown in accordance with some example embodiments;

[0047] Figure 6 A message sequence chart is shown in accordance with some example embodiments;

[0048] Figure 7 A message sequence chart is shown in accordance with some example embodiments;

[0049] Figure 8 An apparatus is shown in accordance with example embodiments;

[0050] Figure 9 A method is shown in accordance with example embodiments;

[0051] Figure 10 An apparatus is shown in accordance with example embodiments;

[0052] Figure 11 A method is shown in accordance with example embodiments; and

[0053] Figure 12An apparatus according to example embodiments is shown. DETAILED DESCRIPTION

[0054] In the following certain example embodiments are described in more detail, wherein features of the example embodiments can, if not stated otherwise, be freely combined with each other. It is however expressly understood that the description of certain example embodiments is given by way of example only and that this is by no means intended to restrict the disclosure to the disclosed details.

[0055] Furthermore, it is understood that an apparatus is configured to perform a corresponding method, although in some cases not every element of the method is described with respect to the apparatus, or vice versa.

[0056] A network digital twin should be as close as possible to the real system it represents, e.g. a single network function, a set of network functions, or an entire network, etc. However, creating an identical copy of an object of a real system with all its dynamics is often not simple.

[0057] A network digital twin can be implemented using different types of network data, e.g. historical data, real-time data, near real-time data, or any combination thereof. For example, the term historical data can refer to data collected in the past and that can be leveraged to build an NDT model, the term (near) real-time can refer to data collected from a real mobile network, i.e. almost real-time data. Thus, depending on the data used to implement the NDT, the frequency by which the NDT is synchronized with the real-world object it represents, and the complexity and computational cost of the tools used to build the NDT, the difference between the NDT and the real-world object it represents can vary greatly. Correspondingly, the measure of “goodness” of a recommendation or decision (e.g. network reconfiguration) derived in the NDT can depend not only on the properties of the decision logic applied in the NDT, but also on the quality / reliability of the data used to implement the NDT. Thus, such quality / reliability of the data used by a consumer to implement the NDT needs to be assessed, monitored, and communicated to the NDT consumer.

[0058] Some example embodiments: quantifying and monitoring the difference between the NDT and the real system it represents, providing information to an authorized consumer about the difference between the NDT and the real system it represents, performing NDT management based on the difference between the NDT and the real system it represents, o e.g. performing updates / synchronization between the NDT and the real system (in one or both directions between the NDT and the real system).

[0059] Some example embodiments provide metrics for quality of network digital twin (NDT), i.e., QoNDT. Such metrics reflect the fidelity / reliability / accuracy of the NDT with respect to the real system it is supposed to represent. This is a measure of the difference between the NDT and the ground truth in a precise and quantifiable manner.

[0060] It differentiates between the KPI(s) of the real system, which are the KPIs / measurements / performance measures collected for a given real system (e.g., a network function, a set of network functions, or an entire network), and the digital twin KPI(s), which represent the KPIs / measurements / performance measures generated by the NDT for the given real system. Using this differentiation, QoNDT is represented by means of: Input drift: the delta / drift of the data used to implement the NDT compared to the measured data of the real system. The input drift can be given in terms of: o a time instance. o a single input data instance (e.g., a particular KPI / measure) used to implement the NDT, o a set of input data instances (e.g., a particular set of most critical KPIs / measurements) o any combination of the above Average input drift: the average delta / drift of the data used to implement the NDT compared to the measured data of the real system over a given time period. The time period to derive the average input drift can be: o a time window specified by the consumer that requires the average input drift o a time window specified by the producer o a time period since the last synchronization of the NDT based on data from the real system, which is the default value if no other value is specified by the consumer of the producer Output drift: the delta / drift of the data generated by the NDT compared to the measured data of the real system, i.e., the drift between the digital twin KPIs and the real system KPIs. The output drift can be given in terms of: o a time instance o a single digital twin KPI data instance (e.g., a particular KPI / measure) generated by the NDT o a set of digital twin KPI data instances (e.g., a particular set of most critical KPIs / measurements generated by the NDT) o any combination of the above Average output drift: the average drift between the digital twin KPIs and the real system KPIs over a given time period. The time period to derive the average output drift can be: o Time window specified by the consumer requiring average output drift o Time window specified by the producer o Time period since last synchronization based on NDT from data from the physical system, which is the default value if no other value is specified by the producer to the consumer.

[0061] Some example embodiments allow NDT service consumers (sometimes denoted as "consumers" for brevity) to monitor the QoNDT of the NDT. In particular, they allow the NDT service consumers to express their requirements for the monitoring in terms of the metrics that should be exposed and the conditions of the indicators that should be notified. Correspondingly, they allow the NDT service producers (sometimes denoted as "producers" for brevity) to report the current QoNDT to the consumers based on the monitoring requirements of the respective consumers.

[0062] Some example embodiments allow QoNDT driven synchronization between the NDT and the physical system it represents. This can include one or both of the following synchronizations: o Real system to NDT: Determine the optimal schedule for performing updates of the NDT by data from the real system (synchronization in the direction from the real system to the NDT) to achieve the desired input and output QoNDT Determine the need to perform an update, e.g., due to QoNDT degradation Determine the time instance to perform an update, e.g., based on a trend of QoNDT degradation Determine, based on data from the physical system, the KPIs / metrics / performance measures that should be updated at the NDT o NDT to real system: Determine the optimal schedule for performing updates from the NDT to the real system, i.e., determine the time instances in which synchronization to the real system is desired. For example, for digital twin KPIs, a time instance can be determined in which the output drift is minimal. This value of the digital twin KPI should be used as input for decision making / recommendations, or based on the NDT output, which actions / recommendations can be performed to the real system

[0063] In the following, reference is made to Figures 1 to 7 Several methods and procedures according to some example embodiments are explained in more detail. QoNDT monitoring and reporting

[0064] Figure 1 QoNDT monitoring and reporting is shown. The NDT service consumer can be a management entity, e.g., a SON function, or an MDA service or a NWDAF-AnLF. The NDT service producer can be part of the OAM system of the network. The actions are as follows:

[0065] 1 : The NDT service consumer issues a request to monitor the performance of a specific NDT instance indicated by NDTJD. In the monitoring request, the consumer can indicate which metrics should be provided, e.g. input / output drift and / or their average. Optionally, the consumer can indicate the time window in which the average should be computed. If not specified, a default value can be used by the producer.

[0066] 2: The NDT service producer provides a report on the QoNDT metrics requested in action 1. Optionally, the producer can provide information on the time window used to compute the average. If not specified, a default value is assumed. Synchronization between real system and NDT

[0067] Based on the QoNDT requirements specified by the consumer and the monitored actual QoNDT of the NDT instance, synchronization between the real system and the NDT can be performed. There are two possible directions of synchronization between the NDT and the real system: a) synchronization from the real system to the NDT: This synchronization typically occurs if the QoNDT falls below the expected level indicated by the consumer. This indicates that the input / output drift is larger than expected and the data used to implement the NDT, or the deviation of the data produced by the NDT compared to the real system, is larger than expected. In this case, new (up-to-date) data can be taken from the real system and used to update the NDT. b) synchronization from the NDT to the real system: This synchronization typically occurs if the QoNDT meets the expected level indicated by the consumer. This indicates that the input / output drift is lower than requested and the data used to implement the NDT, or the deviation of the data produced by the NDT compared to the real system, is smaller than allowed. In this case, the results of the NDT can reflect the behavior of the real system with high confidence and, therefore, it is expected that it will be used for actions and recommendations in the real system.

[0068] The expected level for synchronization from the real system to the NDT and for synchronization from the NDT to the real system can be the same or different from each other. For example, in some example embodiments, to ensure high confidence of actions and recommendations, the expected level for QoNDT for synchronization from the NDT to the real system can be higher than the expected level for QoNDT triggering synchronization from the real system to the NDT. Synchronization from the real system to the NDT

[0069] Figure 2 Actions for synchronization from the real system to the NDT according to some example embodiments are shown.

[0070] 1 : The NDT service consumer can express requirements for the desired QoNDT of the NDT instance to the producer. The request can include information about the desired input / output drift and / or its average value.

[0071] 2: Based on the request received by the consumer and the actually monitored QoNDT, the producer determines the level of satisfaction of the QoNDT requirements. If the actually monitored QoNDT is lower than the consumer required QoNDT, the synchronization from the real system to the NDT will be implemented. If the actually monitored QoNDT is higher than the consumer required QoNDT, the synchronization from the real system to the NDT can be omitted.

[0072] The NDT service producer determines whether an NDT update (by performing a synchronization with the real system) can be recommended based on the QoNDT requirements. An existing degradation of the QoNDT below the consumer specified requirements, or a negative trend in the monitored QoNDT can be a trigger which determines that an NDT update is necessary.

[0073] 3: In case the synchronization from the real system to the NDT should be implemented, the producer will decide on the actual synchronization details and schedule. This can be achieved based on the actually monitored drift or trends in the drift between the NDT and the real system (e.g. increasing drift over time). The synchronization details and schedule can include: Time instances at which the synchronization between the NDT and the real system should start Time instances at which the synchronization between the NDT and the real system should end Trigger events to initiate the synchronization between the NDT and the real system, e.g. a reconfiguration of the real network can be a trigger to synchronize the NDT Which input data should be updated based on real system KPIs / measurements / performance measures. There can be cases where the drift is caused by one input data or a subset of input data of the NDT and not by all data. Which type of real system data should be used for the update, e.g. o Real-time data, o Near real-time data (with a specified maximum lag compared to real-time data), or o Historical data (with a specified maximum lag compared to real-time data) etc.

[0074] 4: From the real system, synchronization to the NDT is performed based on the schedule determined in action 3. This includes getting real system data as specified in the schedule, e.g. certain KPIs / measurements / performance measures obtained by near real-time measurements of the real system, and using it as input for the implementation of the NDT. Typically, during the synchronization time (between the time instance when the synchronization should start and the time instance when the synchronization should end), the output of the NDT should not be used, i.e. synchronization from the NDT to the real system should not be implemented.

[0075] 5: After performing the synchronization, the producer can report the new QoNDT metrics to the consumer Synchronization from NDT to real system

[0076] Figure 3 An action of synchronization from NDT to real system is shown according to some example embodiments.

[0077] 1 : The NDT service consumer requests the NDT service producer to perform the synchronization from NDT to real system. The request is made under the condition that the QoNDT requirements of the NDT instance are fulfilled. In particular, the request can include information about the expected input / output drift and / or its average value.

[0078] 2: Based on the request received by the consumer and the actually monitored QoNDT, the producer determines the level of fulfillment of the QoNDT requirements. Depending on the level of fulfillment, if the actually monitored QoNDT is higher than the QoNDT required by the consumer, the synchronization from NDT to real system can be implemented. Otherwise, if the QoNDT does not fulfill the requirements, the synchronization from NDT to real system can not be implemented.

[0079] In the example of Figure 3 , the NDT service producer determines that the output quality from the NDT is satisfactory and that it can be used with high confidence for actions or recommendations on the real system. The currently monitored QoNDT is higher than the requirements specified by the consumer, or a positive trend in the monitored QoNDT can be a trigger which determines that the output of the NDT can be used for actions and recommendations on the real system.

[0080] 3: In case the QoNDT fulfills the consumer requirements, the output from the NDT can be used to derive actions and recommendations on the real system (“synchronization from NDT to real system”). The producer will decide on the actual synchronization details and schedule. This can be achieved based on the actual monitored drift or trends in the drift between NDT and real system (e.g. decreasing drift over time). The synchronization details and schedule can include: which operations on the real system can be performed based on the NDT output, e.g. which parameters can be reconfigured and how in case of a reconfiguration which of the recommendations for the real system can be implemented, e.g. which energy saving strategy can be applied, or which cells can be switched off for energy saving purposes where the actions or recommendations can be applied to the time instance of the real system triggering events for stopping the synchronization between NDT and real system, e.g. a detected reconfiguration of the real network, which is not recommended as a result of the NDT etc.

[0081] 4: From NDT to real system synchronization is performed based on the schedule determined in action 3. This can include a reconfiguration of the real system based on the actions and recommendations derived from the NDT.

[0082] 5: After the synchronization from NDT to real system is performed, the producer can report new QoNDT metrics to the consumer.

[0083] Reference Figures 4 to 7 Two specific use cases according to some example embodiments are explained in more detail. Example of use case specific NDT - mobility management

[0084] One of the common uses of network digital twinning is the so-called “what-if” type of analysis and network optimization, applying / training ML solutions in the operational network and in network management automation. One example use case is mobility management. In this use case, “what-if” experiments can be applied to HO parameter settings in the network digital twin. Thus, the NDT provides a digital copy of the gNB configuration and environment model, where ML is applied to decide on the best HO decision. The ML model runs in such NDT build and provides inference results with a certain accuracy / confidence. The inference result can be, for example, a recommendation for the best target cell to perform HO for optimizing mobility management. The ML model tests the proposed settings in the NDT and in case it leads to better performance, the corresponding configuration (e.g. HO settings performing HO to the recommended target cell) can be applied in the real network.

[0085] However, when applying the ML inference results to the actual / real network, the operator can not only consider the ML KPIs (accuracy / confidence), but also the performance KPIs of the NDT (i.e. QoNDT), where the “what-if” analysis has been performed. In other words, the consumer of such ML inference needs to know how “close” the NDT actually is to the real network to be able to apply the results obtained in such NDT with a certain confidence level.

[0086] Figure 4 and Figure 5 It is shown how NDT can be applied to the mobility management use case.Figure 4 Positive example is shown where the "hypothesis" analysis of NDT is eventually provided to the network management system to apply them on the network, while Figure 5 Negative example is shown where the "hypothesis" analysis is not performed due to QoNDT being lower than expected.

[0087] The NDT service consumer (e.g. MDA (Management Data Analytics) assisted mobility management (MM) service (see 3GPP TS 28.104)) takes into account different input parameters to provide recommendations to the MDA service consumer (e.g. network management system) of optimal handover parameters. This MDA service can leverage NDT as a source of required input parameters such as performance measurements (e.g. consumed physical and virtual resources) and UE measurements (MDT data) such as RSRP, RSRQ, SINR on serving and neighbor cells, UE location information, etc. Figure 4 The actions shown are as follows:

[0088] 1 : As MDA assisted MM relies on NDT data to produce its output, the MDA assisted MM service can request the monitoring of the QoNDT of NDT (i.e. the MDA assisted MM plays the role of NDT service consumer). The MDA service can specify in its request which metrics should be reported, e.g. input and output drift of RSRP and SINR.

[0089] 2: The NDT service producer measures input and output drift with respect to the requested KPIs and provides a report. Input drift is measured by assessing the data distribution drift of the actual input data of the ML model compared to the data used during the last synchronization between the NDT and the real system. One possible method to measure output drift, and thus the NDT accuracy, is to compare the predictions / forecasts provided by the NDT with the real data collected from the real network: the NDT models / virtualizes the network behavior (or the specific aspect / entity / network element the NDT is capturing). This means that, starting from the actual network state, the NDT will be able to forecast the future state. If at the same time no configuration updates are performed, the producer can measure the error between the forecast and the real data collected when this is collected. This provides a measure of the NDT output drift. In the mobility management use case, the NDT can model the UE interference and RSRP measured by the network cells in a certain network area. Thus, the NDT can forecast the expected future UE interference and RSRP for all cells and, if the network configuration does not change at the same time (e.g., if no cells are turned on / off), the error with respect to the real data collected when this is collected can be assessed. This provides a measure of the NDT accuracy and can be repeated periodically. Of course, during the measure of the NDT accuracy, configuration changes should not be applied, otherwise new forecasts should be obtained from the NDT to update the new network state.

[0090] In Figure 4 the example, the QoNDT meets the requirements set by the NDT service consumer on the QoNDT.

[0091] 3: Based on the predictions / forecasts provided by the NDT, the MDA service can provide a suggestion of the optimal handover parameters. In addition, being able to monitor and quantify the NDT accuracy, the MDA service can associate a confidence level to its suggestion.

[0092] 4. The consumer's suggestion of handover parameters is provided to the MDA service consumer, e.g., the network management system, together with an assigned confidence value (based on the NDT accuracy).

[0093] Figure 5 The same use case as Figure 4 is shown, but for the case where the QoNDT is not good enough. Figure 5 Actions 1 and 2 in Figure 4 are the same as those in . However, since the report indicates that the QoNDT is not good enough, the NDT service consumer does not perform the "what-if" analysis to derive the best HO cell (action 3). The confidence level of such "what-if" analysis would be too low.

[0094] In Figure 4In the example of Figure 1, after receiving a report from the NDT service provider indicating that the QoNDT is good enough, action 3 ("Derive best HO target cell", "Hypothesis" analysis) is performed. In some example embodiments, action 3 can instead start before the QoNDT report arrives at the NDT service consumer. However, in such example embodiments, the recommendations are only provided to the network management system (action 4) after receiving a report indicating that the QoNDT is good enough. If the reported QoNDT is not good enough, no recommendations are provided to the network management system. Example of use case specific NDT - Energy Saving

[0095] Another use case where NDT can be applied for optimization decisions is energy saving. Figure 6 and Figure 7 NDT applied to the energy saving use case is illustrated. Figure 6 A positive example is illustrated, where the "Hypothesis" analysis for NDT is eventually provided to the network management system to apply them on the network, while Figure 7 A negative example is illustrated, where the "Hypothesis" analysis is not performed due to QoNDT being lower than expected.

[0096] The NDT service consumer (e.g. MDA (Management Data Analytics) assisted Energy Saving (ES) service (see 3GPP TS 28.104)) takes into account different input parameters to provide recommendations for optimal energy saving decisions, such as: Suggesting a cell to enter energy saving state. Suggesting a cell to take over traffic of a closed cell Time instances and load thresholds to enter and terminate energy saving state

[0097] This MDA service can leverage NDT as a source of required input parameters, such as performance measurements (e.g. traffic load variations: PRB utilization; RRC connection number; ) and UE measurements (MDT data) such as RSRP, UE location information, etc. The actions are as follows:

[0098] 1: As the MDA assisted ES relies on NDT data to produce the output, the MDA assisted ES service requests the monitoring of the QoNDT of NDT. The MDA service can specify in its request which metrics should be reported, e.g. input and output drift of traffic load (PRB utilization; RRC connection number) and UE location.

[0099] 2. NDT service producers measure and report input and output drift for requested KPIs. Input drift is measured by evaluating the data distribution drift of the actual input data of the ML model compared to the data used during the most recent synchronization between the NDT and the real system. Similar to mobility management use cases, NDT can predict expected future traffic load and UE location, and, if the network configuration does not change simultaneously (e.g., no cells are turned on / off), the error relative to the collected real data can be evaluated when real data is collected.

[0100] exist Figure 6 In the example, the reported QoNDT is good enough.

[0101] 3. Based on the forecasts / predictions provided by NDT, the MDA service offers recommendations (e.g., cells entering energy-saving mode, cells taking over services from closed cells, time instances of entering and terminating energy-saving mode, and / or load thresholds). Furthermore, by monitoring and quantifying the accuracy of NDT, the MDA service can correlate confidence levels with its recommendations.

[0102] 4: Consumers’ energy-saving recommendations, along with the assigned confidence values ​​(based on NDT accuracy), are provided to MDA service consumers (e.g., network management systems).

[0103] Figure 7 It shows the relationship with Figure 6 Same use case, but for situations where QoNDT is not good enough. Figure 7 Actions 1 and 2 in the middle Figure 6 The actions are the same as those in the previous section. However, because the reporting indicates that QoNDT is not good enough, NDT service consumers do not perform "what-if" analyses to derive energy-saving decisions (Action 3). The confidence level of such "what-if" analyses would be too low.

[0104] exist Figure 6 In the example, action 3 ("deriving the optimal energy-saving decision," "hypothesis" analysis) is performed after receiving a report from the NDT service provider indicating that QoNDT is good enough. In some example embodiments, instead, action 3 may begin before the QoNDT report reaches the NDT service consumer. However, in such example embodiments, a decision (action 4) is provided to the network management system only after receiving a report indicating that QoNDT is good enough. If the reported QoNDT is not good enough, no decision is provided to the network management system.

[0105] Figure 8 An apparatus according to an example embodiment is shown. The apparatus may be an NDT producer or a component thereof. Figure 9 A method according to an example embodiment is shown. Figure 8 The device can performFigure 9 The method, but not limited to this method. Figure 9 The method can be derived from Figure 8 The device performs the action, but is not limited to the device performing the action.

[0106] The device includes a receiving component 110, a determining component 120, and a providing component 130. The receiving component 110, the determining component 120, and the providing component 130 can be a receiving component, a determining component, and a providing component, respectively. The receiving component 110, the determining component 120, and the providing component 130 can be a receiver, a determining unit, and a providing unit, respectively. The receiving component 110, the determining component 120, and the providing component 130 can be a receiving processor, a determining processor, and a providing processor, respectively.

[0107] The receiving component 110 receives a request from a service consumer of the network digital twin for an indication of the quality of the provided network digital twin (S110). The quality indication is how accurately the network digital twin represents the real network. The quality can be QoNDT.

[0108] The component 120 used for determination determines the quality of the network digital twin (S120). The component used for determination may determine the quality in response to receiving a request in S110, or independently of such a request (e.g., periodically).

[0109] In response to receiving the request in S110, the component 130 for providing the service provides the service consumer with an indication of the determined quality (S130).

[0110] Figure 10 An apparatus according to an example embodiment is shown. The apparatus may be an NDT consumer or an element thereof. Figure 11 A method according to an example embodiment is shown. Figure 10 The device can perform Figure 11 The method, but not limited to this method. Figure 11 The method can be derived from Figure 10 The device performs the action, but is not limited to the device performing the action.

[0111] The device includes a requesting component 210 and a receiving component 220. The requesting component 210 and the receiving component 220 can be a requesting component and a receiving component, respectively. The requesting component 210 and the receiving component 220 can be a requester and a receiver, respectively. The requesting component 210 and the receiving component 220 can be a request processor and a receiving processor, respectively.

[0112] Component 210 requests the service provider of the network digital twin to provide an indication of the quality of the network digital twin (S210). The quality indication is how accurately the network digital twin represents the real network. The quality can be QoNDT.

[0113] The receiving component 220 receives an indication of the quality in response to the request (S220).

[0114] Figure 12 An apparatus according to an example embodiment is shown. The apparatus includes at least one processor 810 and at least one memory 820 storing instructions that, when executed by the at least one processor 810, cause the apparatus to perform at least one method according to at least one of the following figures and related descriptions: Figure 9 or Figure 11 .

[0115] Some example embodiments are explained in relation to 5G (NR). However, other example embodiments can be used for other 3GPP generations, such as 4G, 6G, 7G, etc. Some example embodiments can be applied to non-3GPP communication networks (such as wired networks) or networks different from communication networks, such as power grids.

[0116] UE is an example of a terminal. Other examples are MTC devices. Each terminal can be implemented as a smartphone, mobile phone, laptop, sensor device, etc.

[0117] A message can be sent from one entity to another in one or more messages. Each of these messages can include additional (different) information.

[0118] The names of network elements, network functions, protocols, and methods are based on the current standard. In other versions or other technologies, the names of these network elements and / or network functions and / or protocols and / or methods may differ, as long as they provide the corresponding functionality. Correspondingly, the same applies to the terminal.

[0119] Unless otherwise stated or explicitly stated from the context, different statements by two entities indicate that they perform different functions. This does not necessarily mean they are based on different hardware. That is, each entity described in this specification may be based on different hardware, or some or all of the entities may be based on the same hardware. This does not necessarily mean they are based on different software. That is, each entity described in this specification may be based on different software, or some or all of the entities may be based on the same software. Each entity described in this specification may be deployed in the cloud.

[0120] Based on the above description, it should be apparent that the exemplary embodiments provide, for example, an NDT service consumer or a component thereof, implementing the same apparatus, methods for controlling and / or operating the same, and controlling and / or operating the same (multiple) computer programs(s), and a medium carrying such (multiple) computer programs(s) and forming (multiple) computer program(s) products(s).

[0121] Implementations of any of the blocks, devices, systems, techniques, or methods described above include, as a non-limiting example, implementations of hardware, software, firmware, dedicated circuitry or logic, general-purpose hardware or controllers or other computing devices, or combinations thereof. Each entity described in this specification may be represented in the cloud.

[0122] It should be understood that the above description represents what are currently considered preferred exemplary embodiments. However, it should be noted that the description of preferred exemplary embodiments is given by way of example only, and various modifications may be made without departing from the scope of this disclosure as defined in the appended claims.

[0123] The terms “first X” and “second X” include the following options: “first X” is the same as “second X” and “first X” is different from “second X”, unless otherwise specified. As used herein, “at least one of the following: ” and “at least one of ” and similar wording, wherein the list of two or more elements is connected by “and” or “or”, indicates at least any one of the elements, or at least any two or more of the elements, or at least all of the elements.

Claims

1. An apparatus comprising: one or more processors, and memory storing instructions that, when executed by the one or more processors, cause the apparatus to perform: receiving, from a service consumer of a network digital twin, a request for an indication of a quality of the network digital twin; determining the quality of the network digital twin; in response to receiving the request, providing the indication of the determined quality to the service consumer, wherein the quality indicates how accurately the network digital twin represents a real network.

2. The apparatus of claim 1, wherein the quality is indicated by at least one of: an input drift indicating a difference between first data used to implement the network digital twin and corresponding data measured in the real network; an average of the input drift over a first time period; an output drift indicating a difference between second data generated by the network digital twin and corresponding data measured in the real network; or an average of the output drift over a second time period.

3. The apparatus of claim 2, wherein the instructions, when executed by the one or more processors, further cause the apparatus to perform: receiving a specification of at least one of: the first data, the second data, the first time period, or the second time period; performing the determining by determining the at least one of the input drift or the average input drift for a specified first data or for a specified first time period, respectively, or by determining the at least one of the output drift or the average output drift for a specified second data or for a specified second time period, respectively.

4. The apparatus of any one of claims 1 to 3, wherein the instructions, when executed by the one or more processors, further cause the apparatus to perform: receiving, from the service consumer, a first requirement for the quality; checking whether the determined quality meets the first requirement; in response to checking that the determined quality does not meet the first requirement, synchronizing from the real network to the network digital twin; determining the quality after the synchronizing; providing the indication of the determined quality such that the quality after the synchronizing is provided.

5. The apparatus of claim 4, wherein the instructions, when executed by the one or more processors, further cause the apparatus to perform, in response to checking that the determined quality does not meet the first requirement, at least one of: determining a time instance at which the synchronizing from the real network to the network digital twin should start; determining a time instance at which the synchronizing from the real network to the network digital twin should end; determining a trigger event for initiating the synchronizing from the real network to the network digital twin; determining data in the network digital twin that is to be updated by the synchronizing from the real network to the network digital twin; or determining data of the real network that is to be used for the synchronizing from the real network to the network digital twin. ​ 6. The apparatus of any one of claims 1-5, wherein the request for the indication of the quality of providing the network digital twin comprises: a request to perform a synchronization from the network digital twin to the real network; and the instructions, when executed by the one or more processors, further cause the apparatus to perform: receiving, from the service consumer, a second requirement for the quality; checking whether the determined quality meets the second requirement; in response to checking that the determined quality meets the second requirement, synchronizing from the network digital twin to the real network in accordance with the request to perform the synchronization; determining the quality after the synchronization; providing the indication of the determined quality so that the quality after the synchronization is provided.

7. The apparatus of claim 6, wherein the instructions, when executed by the one or more processors, further cause the apparatus to, in response to checking that the determined quality meets the second requirement, perform at least one of: determining one or more actions to be performed by the synchronization from the network digital twin to the real network; determining a time instance to perform the synchronization from the network digital twin to the real network; or determining a trigger event and stopping the synchronization from the network digital twin to the real network in response to detecting the trigger event.

8. The apparatus of claim 7, wherein the instructions, when executed by the one or more processors, further cause the apparatus to perform: in response to checking that the determined quality does not meet the second requirement, inhibiting the synchronization from the network digital twin to the real network in accordance with the request to perform the synchronization.

9. An apparatus comprising: one or more processors, and memory storing instructions that, when executed by the one or more processors, cause the apparatus to perform: requesting a service producer of a network digital twin to provide an indication of a quality of the network digital twin; receiving the indication of the quality in response to the request, wherein the quality indicates how accurately the network digital twin represents a real network.

10. The apparatus of claim 9, wherein the quality is indicated by at least one of: an input drift indicating a difference between first data used to implement the network digital twin and corresponding data measured in the real network; an average of the input drift over a first time period; an output drift indicating a difference between second data generated by the network digital twin and corresponding data measured in the real network; or an average of the output drift over a second time period.

11. The apparatus of claim 10, wherein the instructions, when executed by the one or more processors, further cause the apparatus to perform: specifying the at least one of the first data, the second data, the first time period, or the second time period to the service producer.

12. The apparatus of any one of claims 9 to 11, wherein the instructions, when executed by the one or more processors, further cause the apparatus to perform: informing the service producer of a requirement for the quality.

13. The apparatus of claim 12, wherein the instructions, when executed by the one or more processors, further cause the apparatus to perform: requesting the service producer to perform synchronization from the network digital twin to the real network.

14. The apparatus of any one of claims 9 to 13, wherein the instructions, when executed by the one or more processors, further cause the apparatus to perform: checking whether the quality meets a criterion; determining an action on the real network based on the hypothetical analysis of the network digital twin; in response to checking that the quality meets the criterion, instructing a consumer of a management service to perform the determined action.

15. The apparatus of claim 14, wherein the instructions, when executed by the one or more processors, further cause the apparatus to perform at least one of: in response to determining that the quality does not meet the criterion, refraining from determining the action on the real network; or in response to determining that the quality does not meet the criterion, refraining from instructing the consumer of the management service to perform the determined action on the real network.

16. The apparatus of any one of claims 14 and 15, wherein the action relates to at least one of mobility management in the real network or energy saving in the real network.

17. A method comprising: from a service consumer of a network digital twin, receiving a request for an indication of a quality of providing the network digital twin; determining the quality of the network digital twin; in response to receiving the request, providing the indication of the determined quality to the service consumer, wherein the quality indicates how accurately the network digital twin represents a real network.

18. A method comprising: requesting a service producer of a network digital twin to provide an indication of a quality of the network digital twin; receiving the indication of the quality in response to the request, wherein the quality indicates how accurately the network digital twin represents a real network.

19. A computer program product comprising a set of instructions which, when executed on an apparatus, is configured to cause the apparatus to perform the method of any one of claims 17 and 18.

20. The computer program product of claim 19, embodied as a computer-readable medium or directly loadable into a computer.

Images