End-to-end network monitoring

Abstract

To enable end-to-end performance monitoring, different solutions are disclosed for obtaining end-to-end values for different key performance metrics. For example, when key performance indicator data is received from a plurality of different domains, the received key performance indicator data may be adapted as a time series data stream. Then, for each domain, the value of the key performance indicator may be extracted from the time series data stream. End-to-end values of the key performance metrics may be determined based on the extracted values of the key performance metrics in the different domains. The end-to-end value may be used for end-to-end performance monitoring of the network.

Description

Technical Field

[0001] Various example embodiments relate to a communication system. Background Art

[0002] Communication systems are constantly evolving. 5G, 5G-Advanced, and future wireless networks or network generations are designed to support multiple services, use cases, and industrial verticals. With the rise of 5G and the upcoming 6G and future developments deploying services in virtualized and disaggregated multi-vendor environments operating in the cloud, network infrastructure has become more diverse and distributed, resulting in a wide range of network components, domains, and their interactions to be monitored. Enhancements such as those that make end-to-end monitoring easy are desired. Summary of the invention

[0003] The independent claim defines the scope and different embodiments are defined by the dependent claims.

[0004] According to one aspect, a device is provided, which includes at least one processor and at least one memory, wherein the at least one memory stores instructions, which, when executed by the at least one processor, cause the device to at least: receive key performance indicator data from multiple different domains; adapt the received key performance indicator data to a time series data stream; extract a value of the key performance indicator from the time series data stream for each domain; and based on the extracted key performance indicator values ​​in different domains, determine an end-to-end value for the key performance indicator for each key performance indicator using a preset rule for the key performance indicator, wherein the end-to-end value can be used for end-to-end performance monitoring of a network.

[0005] In an embodiment combinable with any aspect and any other embodiment, the at least one memory and the instructions are configured to utilize the at least one processor to cause the apparatus to further: determine, for each domain, a delivery type of key performance indicator data; when the delivery type of the domain is batch or data stream, subscribe to the key performance indicator data from the domain to receive the key performance indicator data; and when the delivery type of the domain is request-based delivery, periodically poll the domain to receive the key performance indicator data.

[0006] In an embodiment combinable with any aspect and any other embodiment, the at least one memory and instructions are configured to, together with the at least one processor, cause the apparatus to further: access report configuration information; for each report, based on the report configuration information, determine at least a key performance indicator, the end-to-end value of which will be included in the report; for each report, generate a report by at least including the determined end-to-end value of the key performance indicator; and send the report.

[0007] In an embodiment combinable with any aspect and any other embodiment, the at least one memory and instructions are configured to, together with the at least one processor, cause the device to also: for each report, determine, based on the report configuration information, whether to include any value of the extracted key performance indicator into the report; and when the extracted value is determined to be included, include the value into the report when generating the report.

[0008] In an embodiment that may be combined with any aspect and any other embodiment, the reporting configuration information includes multiple reporting configurations, and the reporting configurations are for each predetermined use case for end-to-end values, and the predetermined use cases include: a use case for performance management, a use case for data analysis, a use case for security management, a use case for network configuration management, or a use case for event-based management.

[0009] In an embodiment combinable with any aspect and any other embodiment, the at least one memory and instructions are configured to, together with the at least one processor, cause the apparatus to further: determine, for each report recipient, whether the reception type is a batch or a stream; and for each report, send the report to a recipient whose reception type is a batch using batch delivery, and send the report to a recipient whose reception type is a stream using streaming transmission.

[0010] In an embodiment combinable with any aspect and any other embodiment, the at least one memory and instructions are configured to, together with the at least one processor, cause the device to also: receive a subscription for a report from a report recipient, the subscription defining one or more use cases to be reported; and update the report configuration information accordingly.

[0011] In an embodiment combinable with any aspect and any other embodiment, the at least one memory and instructions are configured to, together with the at least one processor, cause the apparatus to further: store at least the end-to-end value to a data warehouse; and access at least data including the end-to-end value in the data warehouse when generating a report.

[0012] In an embodiment combinable with any aspect and any other embodiment, the at least one memory and the instructions are configured to, with the at least one processor, cause the apparatus to further: store, for each domain, the extracted key performance indicator values ​​in a data warehouse.

[0013] In an embodiment combinable with any aspect and any other embodiment, the at least one memory and the instructions are configured to, with the at least one processor, cause the apparatus to further: determine, based on a preset rule, a domain whose value is to be stored from a plurality of different domains.

[0014] In an embodiment combinable with any aspect and any other embodiment, the at least one memory and the instructions are configured to, together with the at least one processor, cause the apparatus to further: receive a request for key performance indicators from an orchestration entity of a network; determine domains included in the network; for each domain, obtain a key performance indicator whose value is determined in the domain; and in response to the request, send information of the key performance indicator.

[0015] In an embodiment combinable with any aspect and any other embodiment, the at least one memory and instructions are configured to, together with the at least one processor, cause the apparatus to further: repeatedly determine domains included in the network and obtain key performance indicators; and upon detecting a change in one or more key performance indicators in a domain, or a change in multiple domains, send change information indicating the change to an orchestration entity.

[0016] In an embodiment combinable with any aspect and any other embodiment, the at least one memory and instructions are configured to, together with the at least one processor, cause the apparatus to further: receive one or more end-to-end key performance service models from an orchestration entity of the network; and determine one or more preset rules for key performance indicators from the one or more end-to-end key performance service models.

[0017] In an embodiment combinable with any aspect and any other embodiment, the at least one memory and the instructions are configured to, together with the at least one processor, cause the apparatus to further: update the model and the corresponding one or more preset rules according to an instruction to update the end-to-end key performance service model when the instruction to update the end-to-end key performance service model is received from the orchestration entity of the network; and delete the model and the corresponding one or more preset rules according to the instruction to delete the end-to-end key performance service model when the instruction to delete the end-to-end key performance service model is received from the orchestration entity of the network.

[0018] According to one aspect, a system is provided, the system comprising at least one processor and at least one memory storing instructions which, when executed by the at least one processor, cause the system to at least: receive key performance indicator data from a plurality of different domains; adapt the received key performance indicator data to a time series data stream; extract a value of the key performance indicator from the time series data stream for each domain; and based on the extracted key performance indicator values ​​in different domains, for each key performance indicator, determine an end-to-end value for the key performance indicator using a preset rule for the key performance indicator; and perform end-to-end performance monitoring of a network using at least the end-to-end value.

[0019] In an embodiment combinable with any aspect and any other embodiment, the at least one memory and instructions are configured to, together with the at least one processor, cause the apparatus to further: access report configuration information; for each report, based on the report configuration information, determine at least a key performance indicator, the end-to-end value of which will be included in the report; for each report, generate a report by at least including the determined end-to-end value of the key performance indicator; and send the report.

[0020] In an embodiment combinable with any aspect and any other embodiment, the at least one memory and the instructions are configured to, together with the at least one processor, cause the apparatus to further: receive a request for key performance indicators from an orchestration entity of a network; determine domains included in the network; for each domain, obtain a key performance indicator whose value is determined in the domain; and in response to the request, send information of the key performance indicator.

[0021] According to one aspect, a method is provided, comprising: receiving key performance indicator data from multiple different domains; adapting the received key performance indicator data to a time series data stream; extracting a value of the key performance indicator from the time series data stream for each domain; and based on the extracted key performance indicator values ​​in different domains, for each key performance indicator, using preset rules for the key performance indicator to determine an end-to-end value for the key performance indicator, which end-to-end value can be used for end-to-end performance monitoring of the network.

[0022] According to one aspect, a computer-readable medium is provided, which includes instructions that, when executed by an apparatus, cause the apparatus to at least perform: receiving key performance indicator data from multiple different domains; adapting the received key performance indicator data to a time series data stream; extracting a value of the key performance indicator from the time series data stream for each domain; and based on the extracted key performance indicator values ​​in different domains, for each key performance indicator, using preset rules for the key performance indicator to determine an end-to-end value for the key performance indicator, which end-to-end value can be used for end-to-end performance monitoring of the network.

[0023] According to one aspect, a non-transitory computer-readable medium is provided, which includes instructions that, when executed by an apparatus, cause the apparatus to at least perform: receiving key performance indicator data from a plurality of different domains; adapting the received key performance indicator data to a time series data stream; extracting a value of the key performance indicator from the time series data stream for each domain; and determining an end-to-end value for the key performance indicator for each key performance indicator based on the extracted key performance indicator values ​​in different domains using preset rules for the key performance indicator, the end-to-end value being usable for end-to-end performance monitoring of a network.

[0024] According to one aspect, a computer program is provided, which includes instructions that, when executed by an apparatus, cause the apparatus to at least perform: receiving key performance indicator data from multiple different domains; adapting the received key performance indicator data to a time series data stream; extracting a value of the key performance indicator from the time series data stream for each domain; and based on the extracted key performance indicator values ​​in different domains, for each key performance indicator, determining an end-to-end value for the key performance indicator using preset rules for the key performance indicator, which end-to-end value can be used for end-to-end performance monitoring of the network. BRIEF DESCRIPTION OF THE DRAWINGS

[0025] Embodiments are described below by way of example only with reference to the accompanying drawings, in which:

[0026] Figure 1 An exemplary high-level structure is illustrated;

[0027] Figure 2 is a flow chart illustrating example functionality;

[0028] Figure 3 is a flow chart illustrating example functionality;

[0029] Figure 4 An example of information exchange is illustrated;

[0030] Figure 5 An example of information exchange is illustrated;

[0031] Figure 6 is a high-level example of an end-to-end monitoring integration structure component;

[0032] Figure 7 is a schematic block diagram; and

[0033] Figure 8 is a schematic block diagram. DETAILED DESCRIPTION

[0034] The following embodiments are examples. Although the specification may mention "one", "a kind of" or "some" embodiments in several positions, this does not necessarily mean that each such mention refers to the same embodiment, or the feature is only applicable to a single embodiment. The individual features of different embodiments may also be combined to provide other embodiments. In addition, the words "comprise" and "comprising" should be understood as not limiting the described embodiments to only being composed of those features already mentioned, and such embodiments may also include features / structures not specifically mentioned. In addition, although the terms including ordinal numbers, such as "first", "second", etc., may be used to describe various elements, structural elements are not limited by these terms. These terms are only used to distinguish the purpose of an element from other elements. For example, without departing from the scope of the present disclosure, the first signal may be referred to as the second signal, and similarly, the second signal may also be referred to as the first signal.

[0035] 5G (Fifth Generation), 5G-Advanced and Beyond Future wireless networks are designed to support multiple services, use cases and industrial verticals, such as driverless mobility with fully autonomous connected vehicles, other vehicle-to-everything (V2X) services, or smart environments, such as smart industry, smart grid or smart city, to name a few examples. In order to provide various services with different requirements, such as enhanced mobile broadband, ultra-reliable low-latency communication, massive machine-type communication, wireless networks are envisioned to adopt network slicing, flexible decentralized and / or distributed computing systems and ubiquitous computing, with local spectrum licensing, spectrum sharing, infrastructure sharing, and intelligent automated management supported by mobile edge computing, artificial intelligence, such as machine learning, tools, cloudification, short packet communication and blockchain technology. For example, in network slicing, multiple independent and dedicated network slice instances can be created within the same infrastructure to run services with different requirements for latency, reliability, throughput and mobility.

[0036] The key features of 6G (sixth generation) are expected to include intelligent connected management and control capabilities, programmability, integrated sensing and communication, reduced energy consumption, trusted infrastructure, scalability and affordability. In addition to these, 6G is also intended to cover new use cases that integrate positioning and sensing capabilities into the system definition to unify the user experience across the physical and digital worlds.

[0037] Figure 1 The diagram illustrates an exemplary high-level network architecture, showing only some functional entities, all of which are logical units and may be implemented differently from what is shown. Figure 1 The connections shown in are logical connections; the actual physical connections may be different.

[0038] refer to Figure 1, a system 100 providing tools for end-to-end assurance includes one or more wireless networks 110 , an end-to-end monitoring platform 120 , and a network orchestration platform 130 .

[0039] The wireless network 110 includes a device component 101 for device functions in one or more device domains, an access network component 102 for access network functions in one or more access network domains and / or in one or more cloud domains, a transport network component 103 for transport network functions in one or more access network domains and / or in one or more cloud domains, a core network component 104 for core network functions in one or more core network domains 104 and / or in one or more cloud domains, and a data network component 105 for data network functions in a data network domain.

[0040] Device component 101 may be any electrical device or apparatus that can be connected to an access network. A non-limiting list of examples of device component 101 includes user devices, smartphones, IoT devices, industrial IoT devices, consumer IoT devices, personal devices, wearable devices (such as smart watches, smart rings), electronic health-related devices, medical monitoring devices, sensors (such as pressure sensors, humidity sensors, thermometers, motion sensors, actuators, accelerometers, etc.), surveillance cameras, vehicles, automated guided vehicles, autonomous connected vehicles, etc.

[0041] The access network may be any type of access network capable of providing access to device components and delivering services between the device and a data network. The access network may be a wireless access network, such as a cellular access network, such as an advanced 5G network or a new radio network, a non-terrestrial network, a traditional cellular radio access network, such as a Long Term Evolution Advanced (LTE Advanced, LTE-A) network or a 3G network, or a non-cellular access network, such as a wireless local area network. To provide access, the access network includes means such as access devices as access network components 102. There are a variety of access devices, including access devices that provide wireless access, such as different types of base stations, such as eNBs, gNBs, split gNBs, transmission reception points, network-controlled repeaters, nodes operatively coupled to one or more remote radio heads, satellites, donor nodes in integrated access and backhaul (IAB), fixed IAB nodes, mobile IAB nodes mounted on vehicles, etc. At least some of the devices in the access network may provide an abstraction platform to separate the abstraction of network functions from processing hardware.

[0042] The transport network component 103 forms one or more transport networks between the access network and the core network, wherein the transport network provides connectivity services. The transport network and the transport network component 103 can use any technology, such as optical-based technology and / or Internet protocol-based technology, to enable the radio access network and the core network to deliver network services between devices and data networks. A non-limiting list of examples of the transport network component 103 includes access routers, metropolitan edge gateways, metropolitan aggregation edge gateways, metropolitan backbone routers, and backbone routers toward the core network.

[0043] The core network components 104 form one or more core networks. The core network can be based on a non-independent core network, such as an evolved packet core network, or an independent access network, such as a 5G core network. However, it should be understood that the core network and the core network components 104 can use any technology to enable network services to be delivered between devices and data networks.

[0044] The data network may be any network, such as the Internet, an intranet, a wide area network, etc. Different remote monitoring and / or data collection services for different use cases may be achieved via the data network and the data network component 105 .

[0045] Different domains can manage their resources autonomously and collect different types of key performance indicators KPIs. For example, there may be access KPIs 112, core KPIs 114, transport KPIs 113, and cloud KPIs 115. Non-limiting examples of access KPIs 112 include mobile access KPIs and fixed access KPIs. Access KPIs may include 4G / 5G / 6G radio KPIs, slice KPIs, vendor-specific KPIs, infrastructure system KPIs, packet data network KPIs, wireless local area network KPIs, and the like. Transport KPIs 113 may include KPIs for optical transport and / or Internet Protocol (IP)-based transport. Non-limiting examples of transport KPIs include reconfigurable optical add / drop multiplexer KPIs, optical transport module KPIs, optical channel KPIs, physical impairment KPIs, layer 2 / layer 3 virtual private network KPIs, multi-protocol label switching KPIs, segment routing KPIs, slice KPIs, open configuration KPIs, vendor-specific KPIs, infrastructure system KPIs, and the like. The core KPIs 114 naturally depend on the type of core network. For example, the core KPIs 114 of a 5G core may include user plane function KPIs, network function computing resource KPIs, slice KPIs, vendor-specific KPIs, infrastructure system KPIs, etc. The cloud KPIs 115 may include Prometheus KPIs, Jaeger KPIs, application KPIs, slice KPIs, vendor-specific KPIs, infrastructure system KPIs, etc. It should be understood that even in Figure 1In the example of , cloud KPI 115 is collected from core network components in one or more cloud domains, and cloud KPI can also be collected from access network components and transport network components in the cloud domains accordingly. In other words, there are many different domain-specific KPIs that can represent the performance of domain-specific services in a vendor-specific format.

[0046] The end-to-end monitoring platform 120 includes one or more end-to-end monitoring integrated structure (E2EMIF) entities 106. The end-to-end monitoring integrated structure entity 106 provides one or more programmable abstractions for network monitoring. The end-to-end monitoring integrated structure entity 106 can be implemented in one or more devices. Examples of end-to-end monitoring or its sub-entities or corresponding functions will be described in more detail below. Depending on the implementation, the end-to-end monitoring integrated structure 106 may include different components or sub-entities for providing specific functions, and the functions performed by a component (sub-entity) may be different in different implementations. In an implementation, the end-to-end monitoring integrated structure 106 may include an end-to-end KPI value determination component (E2E KPI determiner) to determine the end-to-end value of one or more key performance indicators. In addition, non-limiting examples of components include end-to-end key performance indicator service model, end-to-end key performance indicator service model lifecycle management, end-to-end key performance indicator subscription management, end-to-end key performance indicator reporting, end-to-end key performance indicator extraction-load-conversion-processing, end-to-end key performance indicator service producer, domain adapter management, communication enabler between different components and / or underlying domains (domain systems). In other words, end-to-end monitoring enabled by one or more end-to-end monitoring integration structure (E2E MIF) entities 106 provides a comprehensive approach to ensure the performance, functionality and security of the entire network infrastructure from an end-to-end perspective. Therefore, it can be said that end-to-end monitoring is critical to the network management task.

[0047] The end-to-end monitoring provided by the end-to-end monitoring platform 120 can be part of end-to-end network assurance. End-to-end monitoring can even be considered a key enabler of end-to-end network assurance by observing the performance of end-to-end systems from one domain to another and analyzing the end-to-end network performance from the data source to its destination. This gives important insights into how a network or system operates across different services, nodes, and connections, providing complete visibility into network functionality. In addition, by monitoring the performance of the network and identifying potential problems before they become serious, end-to-end monitoring as part of end-to-end network assurance can help prevent network downtime, data breaches, and other security incidents.

[0048] The network orchestration platform 130 includes different use case applications 107 for different purposes, such as use cases for performance management, use cases for data analysis, use cases for security management, use cases for network configuration management, and / or use cases for event-based management. Use cases can be used for service / slice implementation, automation, assurance, root cause analysis, artificial intelligence / machine learning, etc. Use case applications (even use case applications for the same purpose) can use different key performance indicators. For example, an end-to-end network automation solution can be improved faster and more accurately, for example, by generating and providing batch data sets of end-to-end key performance indicators for training and validating machine learning models, or by generating and providing streams of end-to-end key performance indicators for machine learning model reasoning. A non-limiting list of machine learning use cases includes anomaly detection based on machine learning, key performance indicator prediction, soft fault prediction, etc. In addition, when the machine learning model is improved, for example, by changing the key performance indicators used (for example, by adding new key performance indicators or deleting key performance indicators), the end-to-end monitoring integration structure 106 can quickly adapt to the use cases based on the machine learning model. Different reporting, root cause and / or service impact analysis use cases can leverage the unified inventory / data source provided by one or more end-to-end monitoring integration structures 106 to feed different assurance processes, thereby enabling end-to-end performance monitoring and rapid troubleshooting by combining and leveraging any data in the system without defining in the orchestration platform how to obtain end-to-end key performance indicators or corresponding data.

[0049] Furthermore, end-to-end monitoring integration fabric 106 may be based on centralized data aggregation / processing, which reduces data processing costs and can provide end-to-end network monitoring as a service to use cases in orchestration platform 107 .

[0050] Figure 2 Illustrated are example functionalities of an end-to-end monitoring integration fabric entity or its components depending on the implementation and component(s) included in the end-to-end monitoring integration fabric entity.

[0051] Reference Figure 2, an entity receives (block 201) key performance indicator data from at least a plurality of different domains. In one implementation, when data may be received from or delivered by domains of different delivery types, the receiving may include, for each domain, determining a delivery type of the key performance indicator data, and then performing the receiving based on the delivery type. For example, when the delivery type of a domain is batch (batch delivery), key performance indicator data may be subscribed from the domain to receive the key performance indicator data. Correspondingly, when the delivery type of a domain is data stream, key performance indicator data may be subscribed from the domain to receive the key performance indicator data. When the delivery type of a domain is request-based delivery, the domain may be periodically polled to receive the key performance indicator data.

[0052] Regardless of the type of transmission, the received key performance indicator data is adapted (block 202) to a time series data stream. In a time series data stream, data collected over a period of time is in a data stream according to the time at which it was collected. For example, batch data or polled data can be adapted to a time series data stream by forwarding data segments from the oldest data to the newest data. Naturally, if the received data stream is already a time series data stream, no adaptation occurs.

[0053] Then, for each domain, the entity extracts (box 203) the value of the key performance indicator from the time series data stream. In other words, domain-specific values ​​of the key performance indicator are extracted. Then, for each key performance indicator, the end-to-end value of the key performance indicator is determined using one or more preset rules for the key performance indicator (box 204). The preset rules may be to add the values ​​to each other, or to provide different weights to values ​​from different domains and then calculate the sum of the weighted values, just to mention a couple of non-limiting examples of what the preset rules may be. The determined end-to-end value is based on the values ​​of the extracted key performance indicators in different domains, combined according to one or more preset rules for the key performance indicators. It should be understood that different key performance indicators may share the same one or more preset rules, or have their own one or more preset rules. In Figure 2 At the end of the processing, there are multiple end-to-end values ​​for multiple key performance indicators, one value for each key performance indicator, and the use case may use different combinations of values ​​depending on the key performance indicators used in the corresponding use case when performing end-to-end performance monitoring of the network.

[0054] Figure 2 The process can be continuous, or repeated at regular intervals.

[0055] Figure 3 Another example functionality of the end-to-end monitoring integration structure entity or its components is illustrated, depending on the implementation and components included in the end-to-end monitoring integration structure entity. Figure 3In the example above, it is assumed that the end-to-end values ​​of the key performance indicators are available. In other words, Figure 2 The process is already underway Figure 3 A further assumption made is that the report configuration information exists and is accessible to the entity.

[0056] Reference Figure 3 , the entity accesses (block 301) reporting configuration information. The reporting configuration information may be stored in a separate data warehouse or data lake, which may be located on any cloud accessible to the entity. The reporting configuration information includes reporting configurations for use cases using end-to-end values ​​of one or more key performance indicators. Thus, there are multiple reporting configurations, one for each predetermined use case of the end-to-end value. A non-limiting list of examples of predetermined use cases includes a use case for performance management, a use case for data analytics, a use case for security management, a use case for network configuration management, and / or a use case for event-based management.

[0057] Then, for each report, based on the report configuration information, at least a key performance indicator whose end-to-end value is to be included in the report is determined (block 302). Further, in some implementations, for each report, based on the report configuration information, it is determined (block 302) whether each report includes any of the extracted values ​​of the key performance indicator (i.e., whether it includes a domain-specific value). Then, for each report, a report is generated (block 303) by including at least the end-to-end value of the key performance indicator determined to be included in the report. In some implementations, generating (block 303) the report further includes including the extracted value in the report when the value is determined to be included. Then, the generated report is sent (block 304). Then, the recipient uses the report, or more precisely, uses at least one or more end-to-end values ​​of the one or more key performance indicators, to perform end-to-end network performance monitoring according to the use case.

[0058] It should be understood that the process can be repeated to generate and send reports using information about the report content determined earlier. In other words, blocks 303 and 304 can be repeated, while blocks 301 and 302 can be performed once or less frequently.

[0059] Figure 4 A highly simplified example of information exchange related to management of key performance models for end-to-end monitoring of a network is illustrated.

[0060] refer to Figure 4, the orchestration entity of the network sends a request for the key performance indicators (message 4-1) to the end-to-end monitoring integration structure entity E2E MIF. For example, a developer / modeler of an operator can define a new use case that requires at least one or more end-to-end values ​​of key performance indicators via the orchestration entity. It can be said that in order to design an end-to-end key performance service model, the developer / modeler can request the key performance indicators supported by the domain in each domain in the network via the orchestration entity by entering the corresponding user input, even if the domain is unknown.

[0061] When the end-to-end monitoring integration fabric entity receives a request for key performance indicators, it determines (block 4-2) the domains included in the network and, for each domain, obtains (for clarity, in Figure 4 In other words, the end-to-end monitoring integration fabric entity may retrieve or request key performance indicators from a domain of the network, which domain may include, for example, one or more access network domains, and one or more core network domains. Then, in a response to the request (message 4-4), the end-to-end monitoring integration fabric entity sends information on the key performance indicators. The end-to-end monitoring integration fabric entity, or its domain adapter component, may also expose a public set of monitoring execution application programming interfaces (APIs) (e.g., create, update, delete, and get) to enable the attributes of the monitoring execution to be domain-specific.

[0062] The key performance indicators can be displayed or otherwise output to, for example, a developer who then designs and defines one or more end-to-end key performance models (service models). Once multiple models are ready, or a model is ready, user input is received ( Figure 4 (not shown in the figure), so that (multiple) models are sent (messages 4-5) from the orchestration entity to the end-to-end monitoring integration structure entity. The end-to-end key performance model thus created is abstract and may contain a list of end-to-end key performance indicators whose values ​​are obtained, calculated, and transformed from domain key performance indicators. The end-to-end value of the end-to-end key performance indicator may indicate whether the service / slice service level agreement is met. Different models may present various parameters with different granularity and frequency. The end-to-end value may be calculated based on the value of one or more key performance indicators, which may be different, from one domain, or from different domains. For example, the model may define that the latency of an end-to-end slice will be calculated by sub-slice latency measurements from the network domain.

[0063] When the end-to-end monitoring integration structure entity receives (message 4-5) one or more end-to-end key performance service models from the orchestration entity of the network, the end-to-end monitoring integration structure entity determines (block 4-6) one or more preset rules of key performance indicators from the one or more end-to-end key performance service models. Message 4-5 may be an upload message, resulting in the one or more end-to-end key performance service models being stored in a data warehouse, for example, stored in a data warehouse indicated in the upload message.

[0064] Then, the end-to-end monitoring integration structure entity determines (block 4-7) the end-to-end value of the key performance indicator according to the rules, as shown in the above reference Figure 2 Said, and / or referred to below Figure 5 As described above, and reporting (box 4-7) end-to-end values ​​according to the rules, as described above with reference to Figure 3 Said, and / or referred to below Figure 5 described.

[0065] In the illustrated example, the end-to-end monitoring integration fabric entity repeatedly (message 4-8) determines the domains included in the network and obtains key performance indicators, and upon detecting a change in one or more key performance indicators in one domain (block 4-9) and / or a change in multiple domains, sends change information indicating the change to the orchestration entity (message 4-10).

[0066] Changes, such as new or changed KPIs and / or domains, can be displayed ( Figure 4 4-10), or is otherwise output to a developer, for example, who then updates one or more end-to-end key performance models, or creates a new model. In the example shown, for purposes of description, it is assumed that one or more existing models are updated. Creating / adding a new model follows the process described using messages 4-5 and boxes 4-6. Once the update to one or more models is ready, the corresponding user input received causes the update to the model to be sent (message 4-11) from the orchestration entity to the end-to-end monitoring integration structure entity. Message 4-11 can be an instruction to update one or more end-to-end key performance service models. It should be understood that the update may also be sent for other reasons, i.e., no information about the change is received (no message 4-10), such as the need for a change based on a use case.

[0067] When the end-to-end monitoring integration structure entity receives (message 4-11) an instruction to update one or more end-to-end key performance service models from the orchestration entity of the network, the end-to-end monitoring integration structure entity updates (box 4-6) one or more end-to-end key performance service models and corresponding one or more preset rules according to the content of the received instruction.

[0068] In addition, Figure 4In the example illustrated in FIG. 4 , an instruction to delete one or more end-to-end key performance service models (message 4-13) is sent from the orchestration entity (message 4-11) to the end-to-end monitoring integration structure entity. When an instruction to delete one or more end-to-end key performance service models is received from the orchestration entity of the network, one or more models and corresponding one or more preset rules (block 4-14) are deleted according to the instruction. It should be understood that the instruction to delete one or more models may be due to a change in the domain, i.e., due to the information received in message 4-10.

[0069] Using the above information exchange, end-to-end key performance indicators for implementations of different use cases can be designed. It should be understood that a use case implementation can use one or more end-to-end key performance indicator service models.

[0070] The process can be that after the designer has designed the end-to-end key performance service model, the designed end-to-end key performance service model can be loaded into the end-to-end monitoring integration structure via the REST API provided by the end-to-end key performance service model management. The end-to-end key performance service model management can decompose the end-to-end key performance indicators into specific domain key performance indicators and generate service producer specifications, including stream and batch mechanism reports. In addition, depending on the use case implementation method, the end-to-end key performance service model management can subscribe to streams and / or polling and / or batch mechanisms. The end-to-end key performance service model management can also generate specifications describing how to extract domain key performance indicators and convert them into end-to-end key performance indicators. The end-to-end key performance service model management can generate and return a service consumer SDK for developers to use to implement use cases. The service consumer SDK is a general component with a specific end-to-end key performance indicator service model plug-in, which can be installed as needed. A similar process can be applied when updating an existing end-to-end key performance indicator service model.

[0071] Figure 5 A highly simplified example of information exchange related to end-to-end key performance indicator reporting is illustrated. In the example shown, "consumer" means a report receiver. A report receiver can be a plugin, such as a service consumer SDK, or a corresponding tool or orchestration entity that a developer can use to implement one or more use cases to perform end-to-end monitoring.

[0072] exist Figure 5 In the illustrated example, it is assumed that the end-to-end monitoring integrated structure entity E2EMIF includes (block 5-0), or has access to one or more end-to-end key performance service models, or at least accesses, for example, according to Figure 5The example defines one or more corresponding preset rules and report configuration information (report configuration). The report configuration information may include multiple report configurations, for example, a report configuration for each predetermined use case of the end-to-end value.

[0073] refer to Figure 5 For each domain, the E2E MIF determines (block 5-1) the delivery type of the key performance indicator data used by the domain. When the delivery type of the domain is batch or data stream, the E2E MIF subscribes (message 5-2) to the key performance indicator data from the domain and receives (message 5-3) the key performance indicator data from the domain based on the subscription. When the delivery type of the domain is request-based delivery, the E2E MIF periodically polls (message 5-4) the domain to receive the key performance indicator data.

[0074] The E2E MIF uses the received key performance indicator data to determine (block 5-5) an end-to-end value of an end-to-end key performance indicator, for example as described above in conjunction with Figure 2 In addition, in the example shown, at least the end-to-end values ​​of the key performance indicators are stored (block 5-5) to the data warehouse. Depending on the implementation, the E2E MIF may, for example, store (block 5-5) for each domain the values ​​of the key performance indicators from all or a subset of domains extracted from the data flow from the domain to the data warehouse according to the corresponding end-to-end key performance service indicator model. Alternatively or additionally, the E2E MIF may determine, from a plurality of different domains, the domain whose values ​​are to be stored based on a preset rule.

[0075] When it is time to report at least the end-to-end value to one or more use cases, the E2E MIF accesses (block 5-6) data in the data warehouse including at least the end-to-end value when generating (block 5-6) the report, e.g., as described above in conjunction with Figure 3 As described. When the same report is subscribed, the E2E MIF can generate a report that will be sent to multiple consumers (report receivers). Then, in the example shown, the E2E MIF determines (box 5-6) whether the reception (rx) type is batch or stream for each report receiver (for each consumer), and for each report, sends the report to consumers (receivers) whose reception type is batch using batch delivery (message 5-7) and sends the report to consumers (receivers) whose reception type is stream using streaming (message 5-8). Then, the consumer uses (box 5-9) the report according to the monitoring use case, such as end-to-end values ​​for key performance indicators.

[0076] In addition, Figure 5In the illustrated example, the developer subscribes to a report, which causes the consumer (report receiver) to send (message 5-10) a subscription to the report, which defines one or more use cases to be reported. It should be understood that message 5-10 may also be a cancellation of a subscription, or contain an update to an existing subscription. The E2E MIF correspondingly updates (block 5-11) the report configuration information, i.e., according to the content of message 5-10.

[0077] Figure 6 A non-limiting high-level example of an end-to-end monitoring integration architecture component is illustrated. It should be understood that for clarity of description, a single component is used and that the end-to-end monitoring integration architecture may include multiple corresponding components. Figure 6 The modular architecture of the examples provides building blocks and flexibility in various deployment scenarios.

[0078] refer to Figure 6 The end-to-end monitoring integration structure 106 includes: an end-to-end key performance indicator extraction, loading and transformation E2E KPI ELT component 610; an end-to-end key performance indicator report generator component 620; an end-to-end key performance indicator model manager component 630; and a data warehouse 640.

[0079] exist Figure 6 In the example of , the E2E KPI ELT component 610 includes an adapter 611 and an end-to-end key performance indicator determiner 612. The adapter 611 is configured to receive data from different data sources (domains) in data streams 601, or in batches 602, or by polling 603, and adapt the received data to one or more time-series data streams 604 that the end-to-end key performance indicator determiner 612 is configured to receive. In other words, the adapter 611, which can be called a domain adapter, enables communication between the end-to-end monitoring integration structure 106 and the underlying domain system.

[0080] The end-to-end key performance indicator determiner 612 is configured to perform an end-to-end key performance indicator extraction, loading and transformation EK-ELT processing operation, which is a data integration process. The data integration process first extracts domain key performance indicators from different time series data streams 604 of domain key performance indicator data, and loads the key performance indicator KPI data into the data warehouse 640, for example, for each domain, the KPI is loaded into the table / collection of the domain itself. In addition, the end-to-end key performance indicator determiner 612 is configured to transform the domain key performance indicator data into predefined end-to-end key performance indicator KPI data, for example, into end-to-end KPI values, and load the end-to-end KPI data into the data warehouse 640. It can be said that for each model, the end-to-end key performance indicator determiner integrates data from multiple domains into a single end-to-end key performance indicator service model representation.

[0081] Even if not Figure 6 As illustrated in FIG, the end-to-end key performance indicator determiner 612 may include one or more user interfaces, such as a graphical user interface, which allows a user (e.g., a developer) to modify / change any part of the process, such as a data conversion process, by changing the formula or preset rules of the end-to-end key performance indicator.

[0082] exist Figure 6 In the illustrated example, the service generator component 620 includes an end-to-end key performance indicator report generator 621, a stream generator (sp) 622, and a batch generator (bp) 623. The end-to-end key performance indicator report generator 621 is configured to generate a report including one or more end-to-end values ​​of at least one key performance indicator for each subscription (subscription report), such as described above in conjunction with Figure 3 As described, and the report is delivered as a stream 605 through a stream generator 622, or as a batch sent through a batch generator 623. The end-to-end key performance indicator report generator 621 can be configured to generate a report for a subscription with the same report content according to the subscribed delivery type, which will be used to deliver the same report through the stream generator 622 and / or through the batch generator 623. In other words, the service generator component 620 is configured to deliver at least the end-to-end value of the key performance indicator to the use case application according to the corresponding subscription to be used for further processing and operation, for example, for end-to-end monitoring. Since the report supports both streaming and batch delivery methods, a wide range of use cases can be supported.

[0083] exist Figure 6 In the illustrated example, the end-to-end key performance indicator model manager component 630 is configured to manage the life cycle of each end-to-end key performance indicator service model, including model creation, uploading the model to the data warehouse 640, updating the model, and deletion of the model. In addition, in the illustrated example, the end-to-end key performance indicator model manager component 630 is configured to manage subscription requests for reports from use case applications, including processing subscriptions to determine, for example, the content of the report to be generated and its delivery type for each subscription, uploading the processed subscription to the data warehouse 640, updating the subscription, and deleting the subscription.

[0084] Although not in Figure 6, but there may be a separate domain adapter management component configured to interact with the adapter 611 and the end-to-end key performance indicator model manager component 630, or the adapter 611 and / or the end-to-end key performance indicator model manager component 630 may be configured to perform at least a portion of the corresponding functions. The domain adapter management component may be configured to convert the end-to-end key performance indicator monitoring requirements into one or more specific monitoring methods provided by the network domain to be implemented via the corresponding domain adapter.

[0085] The data warehouse 640 can be a single, consistent data warehouse or data lake that at least the end-to-end key performance indicator determiner 612, the end-to-end key performance indicator report generator 621, and the end-to-end key performance indicator model manager component 630 can access to upload (store) or download (retrieve, obtain) data.

[0086] As can be seen from the above examples, the described framework provides a tool that simplifies the complexity of the underlying network infrastructure and enables the design and development of a wide variety of use case applications to address network orchestration needs, each of which potentially requires different end-to-end key performance metrics through the abstraction and programmability provided by the end-to-end key performance integration fabric entity.

[0087] Above through Figures 1 to 6 The described boxes, related functions, and information exchanges (messages, signaling) are not in absolute time order, and some of them can be executed simultaneously or in an order different from the given order. Other functions can also be executed between them or within them, and other information can be sent and / or other rules are applied. Some boxes or parts of boxes or one or more pieces of information can also be omitted or replaced by corresponding boxes or parts of boxes or one or more pieces of information.

[0088] Figure 7 An apparatus 701 is illustrated which may be configured to perform any of the functions of the apparatus described herein. The apparatus 701 may be, or be included in, or include a platform entity or component, examples of which are listed above. Figure 8 The diagram shows that it can be achieved Figure 7 A device with distributed functionality as shown in the diagram.

[0089] The apparatus 701 may include one or more communication control circuit systems 720 (such as at least one processor) and at least one memory 730, the at least one memory 730 including one or more algorithms 731 (such as computer program code (software)), wherein the at least one memory and the computer program code (software) are configured to, together with the at least one processor, cause the apparatus to perform the above-referenced Figures 1 to 6The at least one memory 730 may also include at least one database 732.

[0090] refer to Figure 7 , the one or more communication control circuits 720 of the device 701 include at least an end-to-end (E2E) circuit system 721, which is configured to perform related functions related to the end-to-end key performance indicators, such as extracting the value of the key performance indicator, determining the end-to-end value, or performing reporting on the use case application, or managing the model. To this end, the end-to-end circuit system 721 of the device 701 is configured to use one or more separate circuit systems to perform, for example, Figures 1 to 6 At least some of the functionality described.

[0091] refer to Figure 7 , the memory 730 can be implemented using any suitable data storage technology, such as semiconductor-based memory devices, flash memory, magnetic memory devices and systems, optical memory devices and systems, fixed memory, and removable memory. In addition, for example, the memory can provide a data repository.

[0092] refer to Figure 7 , the device 701 may also include different interfaces 710, such as one or more communication interfaces (TX / RX), including hardware and / or software for implementing communication connections according to one or more communication protocols. One or more communication interfaces 710 may enable connection to different domains (such as the Internet), and / or to a core network of a wireless communication network, and / or to a radio access network, and / or to other devices within the device range. One or more communication interfaces 710 may provide communication capabilities to the device to communicate in a cellular communication system, and enable communication with different domains, network components, or mobile devices (such as terminal equipment or user equipment). One or more communication interfaces 710 may include standard known components controlled by a corresponding control unit, such as amplifiers, filters, frequency converters, (de) modulators, and encoder / decoder circuit systems, and possible one or more antennas.

[0093] In one embodiment, Figure 8 As shown in Figure 7 At least some of the functions of the apparatus may be shared between two physically separate devices forming an operational entity. Thus, it can be seen that the apparatus depicts an operational entity comprising one or more physically separate devices for performing at least some of the described processes. Figure 8The device may include a control unit CU 820, or a remote control unit, such as a host computer or a server computer, which is operably coupled (e.g., via a wireless or wired connection) to a remote distributed unit DU 822 located in the device, or in a remote head of the device. In one embodiment, at least some of the processes may be performed by the CU 820. In one embodiment, the execution of at least some of the processes may be shared between the DU 822 and the CU 820.

[0094] Similar to Figure 7 , Figure 8 The apparatus may include one or more communication control circuit systems (CNTL) 720 (such as at least one processor), and at least one memory (MEM) 730, including one or more algorithms (PROG) 731, such as computer program code (software), wherein the at least one memory and the computer program code (software) are configured to, together with the at least one processor, cause the apparatus to perform, for example, by means of Figures 1 to 6 Any of the exemplary functions described.

[0095] In an embodiment, the CU 820 may generate a virtual network through which the CU 820 communicates with the DU 822. In general, virtual networking may involve the process of combining hardware and software network resources and network functions into a single software-based management entity (virtual network). Network virtualization may involve platform virtualization, often in combination with resource virtualization. Network virtualization may be classified as combining many networks or portions of networks into an external virtual network to a server computer or host computer (e.g., to a CU). The goal of external network virtualization is to optimize network sharing. Another type is an internal virtual network, which provides network-like functionality to software containers on a single system.

[0096] In an embodiment, the virtual network may provide a flexible distribution of operations between the DU and the CU. In fact, any digital signal processing task may be performed in either the DU or the CU, and the boundaries at which responsibilities are transferred between the DU and the CU may be selected according to the implementation.

[0097] As used in this application, the term "circuitry" may refer to one or more or all of the following: (a) hardware circuit implementations only, such as implementations in analog and / or digital circuitry only; and (b) combinations of hardware circuitry and software (and / or firmware), such as (as applicable): (i) combinations of analog and / or digital hardware circuitry and software / firmware, and (ii) any portion of a hardware processor with software, including a digital signal processor, software, and memory, which work together to enable an apparatus (such as a terminal device or access node) to perform various functions; and (c) hardware circuitry and processors, such as microprocessors or portions of microprocessors, which require software (e.g., firmware) for operation, but when the software is not required for operation, the software may not be present. This definition of "circuitry" applies to all uses of the term in this application (including any claims). As another example, as used in this application, the term "circuitry" also covers implementations of hardware circuitry or processor (or multiple processors), or a portion of a hardware circuitry or processor and its (or their) accompanying software and / or firmware. The term "circuitry" also covers (for example, and if applicable to a particular claim element) baseband integrated circuits for an access node or terminal device or other computing or network equipment.

[0098] In one embodiment, in combination Figures 1 to 6 At least some of the processes described may be performed by an apparatus including corresponding components for performing at least some of the processes described. Some example components for performing these processes may include at least one of the following: a detector, a processor (including dual-core and multi-core processors), a digital signal processor, a controller, a receiver, a transmitter, an encoder, a decoder, a memory, a RAM, a ROM, software, firmware, a display, a user interface, a display circuit system, a user interface circuit system, a user interface software, a display software, a circuit, an antenna, an antenna circuit system, and a circuit system. In one embodiment, at least one processor, a memory, and a computer program code form a processing component or include one or more computer program code portions for performing the processing according to Figures 1 to 6 Any one of the embodiments or its operations to perform one or more operations.

[0099] According to an embodiment, a device is provided, comprising: a component for receiving key performance indicator data from multiple different domains; a component for adapting the received key performance indicator data to a time series data stream; a component for extracting the value of the key performance indicator from the time series data stream for each domain; and a component for determining an end-to-end value for the key performance indicator for each key performance indicator based on the extracted key performance indicator values ​​in different domains using preset rules for the key performance indicator, the end-to-end value being suitable for end-to-end performance monitoring of the network.

[0100] According to an embodiment, a system is provided, comprising: a component for receiving key performance indicator data from multiple different domains; a component for adapting the received key performance indicator data to a timing data stream; a component for extracting the value of the key performance indicator from the timing data stream for each domain; and a component for determining an end-to-end value for the key performance indicator for each key performance indicator using preset rules for the key performance indicator based on the extracted key performance indicator values ​​in different domains, and a component for performing end-to-end performance monitoring of a network using at least the end-to-end value.

[0101] The described embodiments and examples may also be implemented in the form of a computer process defined by a computer program or part thereof. Figures 1 to 6 The embodiment of the described function can be realized by executing at least a part of the computer program including the corresponding instruction. The computer program can be provided as a computer-readable medium including the program instructions stored thereon, or provided as a non-transient computer-readable medium including the program instructions stored thereon. The computer program can be in source code form, object code form, or some intermediate form, and it can be stored in a certain carrier, which can be any entity or device capable of carrying the program. For example, the computer program can be stored on a computer program distribution medium readable by a computer or processor. The computer program medium can be, for example but not limited to, a recording medium, a computer memory, a read-only memory, an electric carrier signal, a telecommunication signal, and a software distribution package. The computer program medium can be a non-transient medium. As used herein, the term "non-transient" is a restriction on the medium itself (that is, tangible, not a signal), rather than a restriction on data storage persistence (for example, RAM to ROM). The coding of the software for executing the illustrated and described embodiments is completely within the scope of those of ordinary skill in the art.

[0102] Although the embodiments are described above with reference to the examples according to the accompanying drawings, it is obvious that the embodiments are not limited thereto, but can be modified in various ways within the scope of the claims. Therefore, all words and expressions should be interpreted broadly, and they are intended to illustrate rather than limit the embodiments. It is obvious to those skilled in the art that, with the advancement of technology, the concepts of the present invention can be implemented in various ways within the scope of the independent claims. In addition, it is obvious to those skilled in the art that the described embodiments can, but are not required to be combined with other embodiments in various ways within the scope of the independent claims.

Claims

1. An apparatus for network monitoring, comprising at least one processor and at least one memory, wherein the at least one memory stores instructions, and when the instructions are executed by the at least one processor, the apparatus at least: Receive key performance indicator data from multiple different domains; Adapting the received key performance indicator data into a time series data stream; For each domain, extracting a value of a key performance indicator from the time series data stream; as well as Based on the extracted values ​​of the key performance indicators in the different domains, for each key performance indicator, an end-to-end value for the key performance indicator is determined using a preset rule for the key performance indicator, and the end-to-end value can be used for end-to-end performance monitoring of the network.

2. The apparatus of claim 1, wherein the at least one memory and the instructions are configured to, with the at least one processor, cause the apparatus to further: For each domain, determining a delivery type of the key performance indicator data; When the delivery type of the domain is batch or data stream, subscribing to the key performance indicator data from the domain to receive the key performance indicator data; and When the delivery type of a domain is request-based delivery, the domain is periodically polled to receive the key performance indicator data.

3. The apparatus according to claim 1 or 2, wherein the at least one memory and the instructions are configured to, together with the at least one processor, cause the apparatus to further: Access report configuration information; For each report, based on the report configuration information, determining at least a key performance indicator, an end-to-end value of the key performance indicator to be included in the report; generating, for each report, a report by including at least the determined end-to-end value of the key performance indicator; as well as The report is sent.

4. The apparatus of claim 3, wherein the at least one memory and the instructions are configured to, with the at least one processor, cause the apparatus to further: For each report, determining, based on the report configuration information, whether to include any of the extracted values ​​of the key performance indicators into the report; and When the extracted value is determined to be included, the value is included in the report when the report is generated.

5. The apparatus according to claim 3 or 4, wherein the report configuration information comprises a plurality of report configurations, the report configuration being for each predetermined use case for the end-to-end value, the predetermined use case comprising: Use cases for performance management, use cases for data analytics, use cases for security management, use cases for network configuration management, or use cases for event-based management.

6. The apparatus according to claim 3 or 4, wherein the at least one memory and the instructions are configured to, together with the at least one processor, cause the apparatus to further: For each report recipient, determining whether the receipt type is batch or stream; and For each report, use batch delivery to send the report to recipients with a receive type of batch, and use streaming to send the report to recipients with a receive type of stream.

7. The apparatus according to claim 3 or 4, wherein the at least one memory and the instructions are configured to, together with the at least one processor, cause the apparatus to further: receiving a subscription for a report from a report recipient, the subscription defining one or more use cases to be reported; and The report configuration information is updated accordingly.

8. The apparatus according to claim 3 or 4, wherein the at least one memory and the instructions are configured to, together with the at least one processor, cause the apparatus to further: storing at least the end-to-end value in a data warehouse; and At least data including the end-to-end values ​​in the data warehouse is accessed in generating the report.

9. The apparatus of claim 8, wherein the at least one memory and the instructions are configured to, with the at least one processor, cause the apparatus to further: For each domain, the extracted value of the key performance indicator is stored in the data warehouse.

10. The apparatus of claim 9, wherein the at least one memory and the instructions are configured to, with the at least one processor, cause the apparatus to further: Based on a preset rule, a field whose value is to be stored is determined from among the plurality of the different fields.

11. The apparatus according to claim 1 or 2, wherein the at least one memory and the instructions are configured to, together with the at least one processor, cause the apparatus to further: receiving a request for a key performance indicator from an orchestration entity of the network; determining domains included in the network; For each domain, obtaining a key performance indicator whose value is determined in the domain; as well as In response to the request, information of the key performance indicator is sent.

12. The apparatus of claim 11, wherein the at least one memory and the instructions are configured to, with the at least one processor, cause the apparatus to further: repeatedly determining domains included in the network and obtaining the key performance indicators; and Upon detecting a change in one or more key performance indicators in a domain, or a change in a plurality of domains, change information indicative of the change is sent to the orchestration entity.

13. The apparatus according to claim 11 or 12, wherein the at least one memory and the instructions are configured to, together with the at least one processor, cause the apparatus to further: receiving one or more end-to-end key performance service models from the orchestration entity of the network; and One or more preset rules for the key performance indicators are determined from the one or more end-to-end key performance service models.

14. The apparatus of claim 13, wherein the at least one memory and the instructions are configured to, with the at least one processor, cause the apparatus to further: When an instruction to update an end-to-end key performance service model is received from the orchestration entity of the network, updating the model and corresponding one or more preset rules according to the instruction; and When an instruction to delete an end-to-end critical performance service model is received from the orchestration entity of the network, the model and the corresponding one or more preset rules are deleted according to the instruction.

15. A system for network monitoring, comprising at least one processor and at least one memory, wherein the at least one memory stores instructions, which, when executed by the at least one processor, cause the system to at least: Receive key performance indicator data from multiple different domains; Adapting the received key performance indicator data into a time series data stream; For each domain, extracting a value of a key performance indicator from the time series data stream; as well as Based on the extracted values ​​of the key performance indicators in the different domains, for each key performance indicator, determine an end-to-end value for the key performance indicator using a preset rule for the key performance indicator; as well as Using at least the end-to-end value, end-to-end performance monitoring of the network is performed.

16. The system of claim 15, wherein the at least one memory and the instructions are configured to, with the at least one processor, cause the apparatus to further: Access report configuration information; For each report, based on the report configuration information, determining at least a key performance indicator, an end-to-end value of the key performance indicator to be included in the report; generating, for each report, a report by including at least the determined end-to-end value of the key performance indicator; as well as The report is sent.

17. The system of claim 15 or 16, wherein the at least one memory and the instructions are configured to, together with the at least one processor, cause the apparatus to further: receiving a request for a key performance indicator from an orchestration entity of the network; determining domains included in the network; For each domain, obtaining a key performance indicator whose value is determined in the domain; as well as In response to the request, information of the key performance indicator is sent.

18. A method for network monitoring, comprising: Receive key performance indicator data from multiple different domains; Adapting the received key performance indicator data into a time series data stream; For each domain, extracting a value of a key performance indicator from the time series data stream; as well as Based on the extracted values ​​of the key performance indicators in the different domains, for each key performance indicator, an end-to-end value for the key performance indicator is determined using a preset rule for the key performance indicator, and the end-to-end value can be used for end-to-end performance monitoring of the network.

19. A computer-readable medium comprising instructions which, when executed by a device, cause the device to at least: Receive key performance indicator data from multiple different domains; Adapting the received key performance indicator data into a time series data stream; For each domain, extracting a value of a key performance indicator from the time series data stream; as well as Based on the extracted values ​​of the key performance indicators in the different domains, for each key performance indicator, an end-to-end value for the key performance indicator is determined using a preset rule for the key performance indicator, and the end-to-end value can be used for end-to-end performance monitoring of the network.

20. The computer readable medium of claim 19, wherein the computer readable medium is a non-transitory computer readable medium.

Images