Improved accuracy of analytics in wireless communication networks

Abstract

A data analysis function including a processor and a receiver is provided. The processor is arranged to generate analysis data for the analysis service using the at least one data source. The receiver is arranged to receive events related to an analysis service. The processor is further arranged to determine a rating of the at least one data source, the rating based on the supplemental data.

Description

Technical Field

[0001] The subject matter disclosed herein generally relates to the field of achieving improved accuracy of analysis in wireless communication networks.This document defines a data analysis function, a method in a data analysis function, a data storage entity, and a method in a data storage entity. Background Art

[0002] In 3GPP, data analysis services are provided by the Network Data Analysis Function (NWDAF) (see 3GPPTS23.288v17.5.0) and are intended to support network data analysis services in the 5G core network. Such analysis can collect data from other network functions (NFs), application functions (AFs), or operations and maintenance (OAMs), and can be exposed to third parties and / or AFs to provide statistics and predictions related to the operation of the wireless communication network. Such statistics and predictions can be related to slice load levels, observed service experience, NF load, network performance, UE-related analysis (mobility, communication), user data congestion, quality of service (QoS) sustainability, data network (DN) performance, etc. In addition, in 3GPP SA5 (3GPPTS28.533v17.2.0), the Management Data Analysis Service (MDAS) provides data analysis for the network. MDAS can be deployed at different levels, for example, at the domain level (e.g., Radio Access Network (RAN), Core Network (CN), Network Slice Subnet), or in a centralized manner (e.g., at the Public Land Mobile Network (PLMN) level). The goal of MDAS is to optimize the management plane (at the network / domain level, at the slice / slice subnet level) by performing analysis based on network management data. Such services can be exposed to third parties / MDAS service consumers to provide PM analysis, FM analysis, network slice instance (NSI) / network slice subnet instance (NSSI) analysis, optionally exposed to recommended appropriate management actions, such as resource scaling, admission control, load balancing of services, etc. Additional analysis capabilities in 3GPP are discussed in 3GPP SA6 (3GPP TR 23.700-36v0.4.0), where Application Data Analysis Enabling Service (ADAES) is defined to perform application layer and edge / cloud analysis outside the 3GPP domain.

[0003] In 3GPP TR 23.700-81v0.3.0 (Title: Enabling Study for 5G Network Automation - Phase 3), one of the key issues discussed is Key Issue #1: How to improve the correctness of NWDAF analysis. The correctness of predictions is usually associated with accuracy, which represents the most prominent key performance indicator (KPI) for rating machine learning (ML) models. However, accuracy can be undermined by drift related to the mismatch between training data and inference data. Therefore, ensuring accuracy is of vital importance. Incorrect predictions may be due to the fact that the accuracy of the ML model during inference is lower than the accuracy of the same ML model during training. This may occur if the training dataset differs significantly from the input data that the ML model is fed during inference in terms of distribution, range, and characteristics. Summary of the invention

[0004] The data analysis functions defined herein tend to provide improved analysis data. This is achieved by facilitating the detection of the correctness of the analysis data and the correction of the analysis data. Improved data analysis tends to be provided as a result of the rating of the data sources used for analysis. The rating can be used as a criterion for selecting which sources to collect data from, thereby improving the quality of the analysis service.

[0005] Disclosed herein is a program for providing improved accuracy of analysis in a wireless communication network. The program may be implemented by a data analysis function, a method in a data analysis function, a data storage entity, and a method in a data storage entity.

[0006] Therefore, a data analysis function is provided, comprising a processor and a receiver. The processor is arranged to generate analysis data for an analysis service using at least one data source. The receiver is arranged to receive events related to the analysis service. The processor is also arranged to determine a rating of at least one data source, the rating being based on supplementary data.

[0007] A method in a data analysis function is also provided, the method comprising: generating analysis data for an analysis service using at least one data source; receiving an event related to the analysis service; and determining a rating of at least one data source in response to receiving the event, the rating being based on supplemental data.

[0008] There is also provided a data storage entity comprising a receiver and a memory. The receiver is arranged to receive a rating of at least one data source. The memory is arranged to store the rating of at least one data source.

[0009] A method in a data storage entity is also provided, the method comprising: receiving a rating of at least one data source; and storing the rating of at least one data source. BRIEF DESCRIPTION OF THE DRAWINGS

[0010] In order to describe the manner in which the advantages and features of the present disclosure can be obtained, the description of the present disclosure is presented by reference to certain devices and methods shown in the accompanying drawings. Each of these drawings depicts only certain aspects of the present disclosure and therefore should not be considered as limiting the scope of the present disclosure. For the sake of clarity, the drawings may have been simplified and are not necessarily drawn to scale.

[0011] A method and apparatus for providing improved accuracy of analysis in a wireless communication network will now be described, by way of example only, with reference to the accompanying drawings, in which:

[0012] Figure 1 Embodiments of a wireless communication system for providing improved accuracy of analysis in a wireless communication network are described;

[0013] Figure 2 depicts a user equipment device that may be used to implement the methods described herein;

[0014] Figure 3 depicts further details of a network node that may be used to implement the methods described herein;

[0015] Figure 4 The methodologies in the data analysis function are illustrated;

[0016] Figure 5 illustrates a method in a data storage entity;

[0017] Figure 6 The solution presented in this article is illustrated;

[0018] Figure 7 illustrates the implementation at the ADRF for ANLF-based rating and storage; and

[0019] Figure 8 An alternative implementation is illustrated which uses another network function, in particular a trusted rating logic function, for performing data source rating. DETAILED DESCRIPTION

[0020] Those skilled in the art will appreciate that aspects of the present disclosure may be embodied as systems, devices, methods, or program products. Therefore, the arrangements described herein may be implemented in full hardware form, full software form (including firmware, resident software, microcode, etc.), or in a combination of software and hardware aspects.

[0021] For example, the disclosed methods and apparatus may be implemented as a hardware circuit including a custom very large scale integrated (VLSI) circuit or gate array, an off-the-shelf semiconductor (such as a logic chip, a transistor, or other discrete components). The disclosed methods and apparatus may also be implemented in a programmable hardware device, such as a field programmable gate array, a programmable array logic, a programmable logic device, etc. As another example, the disclosed methods and apparatus may include one or more physical or logical blocks of executable code, which may be organized, for example, as an object, a procedure, or a function.

[0022] In addition, the method and apparatus may take the form of a program product embodied in one or more computer-readable storage devices storing machine-readable code, computer-readable code, and / or program code, hereinafter referred to as code. The storage device may be tangible, non-transitory, and / or non-transmitting. The storage device may not embody a signal. In some arrangements, the storage device employs only a signal for accessing the code.

[0023] Any combination of one or more computer readable media may be utilized. The computer readable medium may be a computer readable storage medium. The computer readable storage medium may be a storage device storing code. The storage device may be, for example, but not limited to, an electronic, magnetic, optical, electromagnetic, infrared, holographic, micromechanical, or semiconductor system, apparatus, or device, or any suitable combination of the foregoing.

[0024] More specific examples of storage devices (a non-exhaustive list) include: an electrical connection with one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above. In the context of this document, a computer-readable storage medium may be any tangible medium that can contain or store a program for use by or in conjunction with an instruction execution system, apparatus, or device.

[0025] Throughout this specification, references to examples or similar language of a particular method or device mean that the specific features, structures, or characteristics described in conjunction with the example are included in at least one implementation of the methods and devices described herein. Therefore, unless otherwise expressly provided, references to examples or similar language features of a particular method or device may, but not necessarily, all refer to the same example, but mean "one or more but not all examples". Unless otherwise expressly provided, the terms "including", "comprising", "having" and their variants mean "including but not limited to". Unless otherwise expressly provided, an enumerated list of items does not imply that any or all items are mutually exclusive. Unless otherwise expressly provided, the terms "one", "a kind of" and "the" also refer to "one or more".

[0026] As used herein, a list with the conjunction "and / or" includes any single item in the list or a combination of items in the list. For example, a list of A, B, and / or C includes only A, only B, only C, a combination of A and B, a combination of B and C, a combination of A and C, or a combination of A, B, and C. As used herein, a list using the term "one or more of ..." includes any single item in the list or a combination of items in the list. For example, one or more of A, B, and C includes only A, only B, only C, a combination of A and B, a combination of B and C, a combination of A and C, or a combination of A, B, and C. As used herein, a list using the term "one of ..." includes one and only one of any single item in the list. For example, "one of A, B, and C" includes only A, only B, or only C, and does not include a combination of A, B, and C. As used herein, "a member selected from a group including A, B, and C" includes one and only one of A, B, or C, and does not include a combination of A, B, and C. As used herein, “a member selected from the group consisting of A, B, and C and combinations thereof” includes only A, only B, only C, a combination of A and B, a combination of B and C, a combination of A and C, or a combination of A, B, and C.

[0027] In addition, the features, structures, or characteristics described herein may be combined in any suitable manner. In the following description, many specific details are provided, such as examples of programming, software modules, user selections, network transactions, database queries, database structures, hardware modules, hardware circuits, hardware chips, etc., to provide a comprehensive understanding of the present disclosure. However, those skilled in the relevant art will recognize that the disclosed methods and apparatus may be practiced without one or more of the specific details, or may be practiced using other methods, components, materials, etc. In other instances, well-known structures, materials, or operations are not shown or described in detail to avoid confusing aspects of the present disclosure.

[0028] The following is a description of various aspects of the disclosed method and apparatus with reference to schematic flow charts and / or schematic block diagrams of methods, apparatuses, systems, and program products. It will be understood that each block of the schematic flow charts and / or schematic block diagrams, and the combination of blocks in the schematic flow charts and / or schematic block diagrams, can be implemented by code. The code can be provided to a processor of a general-purpose computer, a special-purpose computer, or other programmable data processing device to produce a machine, so that instructions executed by a processor of a computer or other programmable data processing device are created to implement the components of the functions / actions specified in the schematic flow charts and / or schematic block diagrams.

[0029] The code may also be stored in a storage device that can instruct a computer, other programmable data processing apparatus, or other device to operate in a specific manner so that the instructions stored in the storage device produce a manufactured product that includes instructions for implementing the functions / actions specified in the schematic flowchart and / or schematic block diagram.

[0030] The code may also be loaded onto a computer, other programmable data processing apparatus, or other device so that a series of operating steps are executed on the computer, other programmable apparatus, or other device to produce a computer-implemented process, so that the code executed on the computer or other programmable apparatus provides a process for implementing the functions / actions specified in the schematic flowchart and / or schematic block diagram.

[0031] The schematic flow chart and / or schematic block diagram in the figure illustrate the possible architecture, function and operation of the device, system, method and program product.In this regard, each block in the schematic flow chart and / or schematic block diagram can represent a module, segment, or part of code, which includes one or more executable instructions for implementing the code of (multiple) specified logical functions.

[0032] It should also be noted that in some alternative implementations, the functions shown in the blocks may not appear in the order shown in the figures. For example, in fact, two blocks shown in succession may be executed substantially simultaneously, or the blocks may sometimes be executed in reverse order, depending on the functions involved. Other steps and methods that are equivalent in function, logic, or effect to one or more blocks or portions thereof of the illustrated drawings may be envisioned.

[0033] The description of elements in each figure may refer to elements of subsequent figures. The same reference numerals refer to the same elements in all figures.

[0034] Figure 1An embodiment of a wireless communication system 100 for providing improved accuracy of analysis in a wireless communication network is depicted. In one embodiment, the wireless communication system 100 includes a remote unit 102 and a network unit 104. Although Figure 1 1 and 104, a specific number of remote units 102 and network units 104 are depicted, but those skilled in the art will recognize that any number of remote units 102 and network units 104 may be included in the wireless communication system 100. The remote unit 102 may be a user equipment device 200, an analysis consumer 710, an analysis consumer 810, or a data source 840 as described herein. The network unit 104 may be a network node 300, an analysis consumer 710, an analysis consumer 810, a NWDAF 720, a NWDAF 730, a NWDAF 825, a data source 740, a data source 840, an ARDF 850, or a TRLF 860 as described herein.

[0035] In one embodiment, the remote unit 102 may include a computing device such as a desktop computer, a laptop computer, a personal digital assistant (PDA), a tablet computer, a smart phone, a smart TV (e.g., a TV connected to the Internet), a set-top box, a game console, a security system (including a security camera), a car computer, a network device (e.g., a router, a switch, a modem), an aircraft, a drone, etc. In some embodiments, the remote unit 102 includes a wearable device such as a smart watch, a fitness band, an optical head-mounted display, etc. In addition, the remote unit 102 may be referred to as a subscriber unit, a mobile station, a mobile station, a user, a terminal, a mobile terminal, a fixed terminal, a subscriber station, a UE, a user terminal, a device, or other terms used in the art. The remote unit 102 may communicate directly with one or more of the network units 104 via UL communication signals. In some embodiments, the remote unit 102 may communicate directly with other remote units 102 via side link communications.

[0036] The network elements 104 may be distributed over a geographical area. In some embodiments, the network elements 104 may also be referred to as access points, access terminals, base stations, base stations, node Bs, eNBs, gNBs, home node Bs, relay nodes, devices, core networks, air servers, radio access nodes, APs, NRs, network entities, access and mobility management functions (AMFs), unified data management functions (UDMs), unified databases (UDRs), UDM / UDRs, policy control functions (PCFs), radio access networks (RANs), network slice selection functions (NSSFs), operations, maintenance, and management (OAMs), session management functions (SMFs), user plane functions (UPFs), application functions, authentication server functions (AUSFs), security anchor functions (SEAFs), trusted non-3GPP gateway functions (TNGFs), application functions, service enabling architecture layer (SEAL) functions, vertical application enabling servers, edge enabling servers, edge configuration servers, mobile edge computing platform functions, mobile edge computing applications, application data analysis enabling servers, SEAL data delivery servers, middleware entities, network slice capability management servers, or any other term used in the art. The network elements 104 are generally part of a radio access network that includes one or more controllers communicatively coupled to one or more corresponding network elements 104. The radio access network is generally communicatively coupled to one or more core networks, which may be coupled to other networks, such as the Internet and a public switched telephone network, etc. These and other elements of the radio access network and core network are not illustrated, but are generally known to those of ordinary skill in the art.

[0037] In one implementation, the wireless communication system 100 complies with the New Radio (NR) protocol standardized in 3GPP, wherein the network unit 104 transmits on the downlink (DL) using an orthogonal frequency division multiplexing (OFDM) modulation scheme, and the remote unit 102 transmits on the uplink (UL) using a single carrier frequency division multiple access (SC-FDMA) scheme or an OFDM scheme. However, more generally, the wireless communication system 100 may implement some other open or proprietary communication protocol, such as WiMAX, IEEE 802.11 variants, GSM, GPRS, UMTS, LTE variants, CDMA2000, ZigBee, Sigfoxx, etc. The present disclosure is not intended to be limited to any particular wireless communication system architecture or protocol implementation.

[0038] The network unit 104 can serve multiple remote units 102 within a service area (eg, a cell or cell sector) via wireless communication links. The network unit 104 sends DL communication signals to serve the remote units 102 in the time, frequency, and / or spatial domains.

[0039] In 3GPP, data analysis services are provided by the Network Data Analysis Function (NWDAF) (see 3GPPTS23.288v17.5.0) and are intended to support network data analysis services in the 5G core network. Such analysis can collect data from other network functions (NFs), application functions (AFs), or operations and maintenance (OAMs), and can be exposed to third parties and / or AFs to provide statistics and predictions related to the operation of the wireless communication network. Such statistics and predictions can be related to slice load levels, observed service experience, NF load, network performance, UE-related analysis (mobility, communication), user data congestion, quality of service (QoS) sustainability, data network (DN) performance, etc. In addition, in 3GPP SA5 (3GPPTS28.533v17.2.0), the Management Data Analysis Service (MDAS) provides data analysis for the network. MDAS can be deployed at different levels, for example, at the domain level (e.g., Radio Access Network (RAN), Core Network (CN), Network Slice Subnet), or in a centralized manner (e.g., at the Public Land Mobile Network (PLMN) level). The goal of MDAS is to optimize the management plane (at the network / domain level, at the slice / slice subnet level) by performing analysis based on network management data. Such services can be exposed to third parties / MDAS service consumers to provide PM analysis, FM analysis, network slice instance (NSI) / network slice subnet instance (NSSI) analysis, optionally exposed to recommended appropriate management actions, such as resource scaling, admission control, traffic load balancing, etc. Additional analysis capabilities in 3GPP are discussed in 3GPP SA6 (3GPP TR 23.700-36v0.4.0), where Application Data Analysis Enabling Service (ADAES) is defined to perform application layer and edge / cloud analysis outside the 3GPP domain.

[0040] In 3GPP TR 23.700-81v0.3.0 (Title: Enabling Study for 5G Network Automation - Phase 3), one of the key issues discussed is Key Issue #1: How to improve the correctness of NWDAF analysis. The correctness of predictions is usually associated with accuracy, which represents the most prominent key performance indicator (KPI) for rating machine learning (ML) models. However, accuracy can be corrupted by drift related to the mismatch between training data and inference data. Therefore, ensuring accuracy is of vital importance. Incorrect predictions may be due to the fact that the accuracy of the ML model during inference is lower than the accuracy of the same ML model during training. This may occur if the training dataset differs significantly from the input data that the ML model is fed during inference in terms of distribution, range, and characteristics.

[0041] A variety of solutions have been proposed for this critical issue. 3GPP TR 23.700-81v0.3.0 lists 60 such solutions, numbered as solutions #1 to #60. These solutions can be subdivided into solutions that propose NWDAF to determine the accuracy of analysis, solutions that analyze consumers receive feedback from NWDAF, and solutions that improve the accuracy of analysis.

[0042] For the first category of solutions, where NWDAF determines the analysis accuracy, the solutions can be further subdivided into the following subcategories.

[0043] Solutions that propose NWDAF (ANLF) determine the accuracy of analysis (Solutions: 1, 3, 6, 28, 29, 32). Solutions can be further subdivided into solutions that compare the analysis output with real-time data (i.e., solutions 3, 6, 28, 29 based on "ground truth"), and solutions where ANLF uses multiple ML models to determine the analysis output (aggregating ML models) (Solutions 1, 32). Some solutions propose that MTLF subscribe to ANLF for monitoring the performance of ML models.

[0044] The solution of NWDAF(MTLF) is proposed to determine the analysis accuracy by determining the ML model drift (Solutions: 4, 36).

[0045] A solution is proposed that utilizes both ANLF and MTLF to determine the analysis accuracy (Solution 5, 30).

[0046] Where reference is made herein to NWDAF, it should be understood that this may refer to either the NWDAF Analysis Logic Function (ANLF) or the NWDAF Model Training Logic Function (MTLF). Whether the ANLF or the MTLF determines analysis accuracy, there are advantages and disadvantages. For example, if the ANLF compares analysis output to live data in addition to reasoning, the processing and storage impact on the ANLF increases, as well as for the MTLF case, the MTLF needs to track the data used to train the ML model and compare that data to live data.

[0047] For the second category of solutions, where the analytics consumer determines the analytics accuracy, the solutions can be further broken down as follows.

[0048] A solution (Solution 31) in which a consumer compares multiple analysis outputs to all possible benchmark true solutions from the multiple analysis outputs.

[0049] A solution where NWDAF ANLF registers the accuracy of the operation with NRF (Solution 34).

[0050] For the third category of solutions, NWDAF determines whether the analysis requires correction (e.g., by updating the ML model). This solution can be further subdivided as follows.

[0051] Introduce new NFs to monitor changes in the network environment that require updating the ML model (Solutions 7, 35).

[0052] A new analysis ID is introduced to divide the network into sub-areas with similar data statistics (data statistics refers to data distribution, i.e., information about the values ​​or intervals of data, such as network load, network utilization, service usage, UE behavior, etc.), and monitor potential deviations that require triggering retraining at the NWDAF MTLF.

[0053] The analytics consumer provides feedback that indicates that the analytics output has a negative impact on the expected performance beyond a predefined threshold limit (Solution 3).

[0054] If the previous analysis output is inaccurate, the NWDAF ANLF reports the error to the customer and corrects the analysis (Solution 33).

[0055] The data generator provides weighting factors to assist NWDAF in determining how to use the input data (Solution 2).

[0056] Current solutions that enable NWDAF to determine possible corrections for drift assume that all data sources are trustworthy. Such an assumption is reasonable in cases where NF and / or MnS belong to the same operator and / or provide reliable data sources. The accuracy of the analysis output may be reduced for various reasons (data source, data itself, environment), but none of the existing solutions provide detailed information on how to ensure the correctness of the analysis when the problem is the data source itself. Furthermore, the above solutions fail to check how the accuracy of the possible analysis output is mapped or associated to the data source reliability.

[0057] It should be noted that there are data sources used to provide analysis inputs that are not on the premises of the network operator to easily control and check for errors and security. For these types of sources, the quality of the data may be questionable. Significant changes in the data distribution or if there is a significant deviation between the prediction and the baseline real data can indicate reduced data quality.

[0058] An example of such a data source (from 3GPP TS23.288v17.5.0) is as follows:

[0059] Service experience analysis;

[0060] NF load;

[0061] DN performance analysis;

[0062] UE related analysis; and / or

[0063] User data congestion analysis

[0064] The service experience analysis may use inputs from the AF related to the location of the application (represented by the DNAI). The service experience analysis may use service experience measurements from the AF, which refer to the quality of experience (QoE) per service flow established in the SLA and during onboarding. Such measurements may be, for example, the mean opinion score (MOS) or video MOS specified in ITU-TP.1203.3, or may be a customized MOS for any type of service, including services not related to video or voice. The service experience analysis may use QoE metrics from the UE observed by the UE (via the AF). Such QoE metrics and measurements are described in TS26.114, TS26.247, TS26.118, TS26.346, TS26.512, or ASP-specific QoE metrics agreed with the MNO in the SLA may be used. The service experience analysis may use performance data from the AF and from the OAM, or indeed other inputs from the OAM, NF.

[0065] The NF load may require: inputs via OAM such as MDT input data for UEs; per-UE attributes (route, speed, direction, arrival time) to be collected and processed by the AF; and AF input data to the NWDAF for collective behavior of UEs.

[0066] DN performance analysis may require input from the AF such as performance data, for example, average packet delay, average loss rate, and throughput.

[0067] UE related analysis may include expected UE behavior parameters specified in 3GPP TS23.502v17.5.0, service data related to UE mobility from AF.

[0068] User data congestion analysis may include measurements collected from a user plane function (UPF) or from an AF or from OAM related to user data congestion control.

[0069] In all these examples, the NWDAF may receive inputs from one or more similar data sources (i.e., similar component data sources that may complement or even provide the same data), some of which may be related to AF / server measurements or UE-related data. For some of these inputs, similar data (e.g., performance data) may be derived from the application layer or from the network stack on the UPF or DN side or from the UE's own applications via application layer signaling (UE-AF-NWDAF or via ADAEC-ADAES AF-NWDAF).

[0070] In such a case, possible drift may be due to problems with the data source, and such problems may not be in the control of the mobile network operator (MNO) to check whether the data source itself (which may be trusted or untrusted) provides correct inferred data.

[0071] Mechanisms are provided herein that can detect possible drifts on the NWDAF side due to data source abnormal / unreliable behavior (ie, from data sources that are not in the operator's control).

[0072] Additionally, mechanisms are provided that allow for dynamic reaction to detection of drift caused by anomalous data sources to ensure minimal impact on the accuracy of analytical services.

[0073] The solution presented herein provides a complementary solution in 3GPP TR 23.700-81 key issue #1, which is related to how to detect and improve the correctness of NWDAF analysis. The solution presented herein enables the rating of data sources. Such ratings can be based on (i) local estimation / calculation between predicted data and benchmark real data, (ii) analysis of consumer feedback, or (iii) provided by AF in the form of weights. Therefore, NWDAF generates a rating related to the data source profile / reputation, which can be used as a criterion for selecting which sources to collect data from. When selecting an appropriate data source, NWDAF can also use expected confidence as a criterion, that is, associating the result with the input data source. The solution presented here is more suitable for analysis from UE (via AF) or from AF using input, which AF cannot be as credible as OAM and NF (such as the analysis service exemplified above). It should be noted that the granularity of the rating can be provided by analysis ID or by analysis event ID or by analysis service area or by the data statistics range associated with the analysis ID.

[0074] Figure 2A user equipment device 200 is depicted that can be used to implement the methods described herein. The user equipment device 200 is used to implement one or more of the solutions described herein. The user equipment device 200 conforms to one or more of the user equipment devices described in the embodiments herein. The user equipment device 200 can be a remote unit 102, an analysis consumer 710, an analysis consumer 810, or a data source 840 as described herein. The user equipment device 200 includes a processor 205, a memory 210, an input device 215, an output device 220, and a transceiver 225.

[0075] The input device 215 and the output device 220 may be combined into a single device, such as a touch screen. In some implementations, the user equipment apparatus 200 does not include any input device 215 and / or output device 220. The user equipment apparatus 200 may include one or more of the following: a processor 205, a memory 210, and a transceiver 225, and may not include an input device 215 and / or an output device 220.

[0076] As depicted, the transceiver 225 includes at least one transmitter 230 and at least one receiver 235. The transceiver 225 can communicate with one or more cells (or wireless coverage areas) supported by one or more base units. The transceiver 225 can be operable on an unlicensed spectrum. In addition, the transceiver 225 can include multiple UE panels supporting one or more beams. Additionally, the transceiver 225 can support at least one network interface 240 and / or application interface 245. (Multiple) application interface 245 can support one or more APIs. (Multiple) network interface 240 can support 3GPP reference points such as Uu, N1, PC5, etc. As understood by those of ordinary skill in the art, other network interfaces 240 can be supported.

[0077] The processor 205 may include any known controller capable of executing computer-readable instructions and / or capable of performing logical operations. For example, the processor 205 may be a microcontroller, a microprocessor, a central processing unit (CPU), a graphics processing unit (GPU), an auxiliary processing unit, a field programmable gate array (FPGA), or a similar programmable controller. The processor 205 may execute instructions stored in the memory 210 to perform the methods and routines described herein. The processor 205 is communicatively coupled to the memory 210, the input device 215, the output device 220, and the transceiver 225.

[0078] Processor 205 may control user equipment device 200 to implement user equipment device behaviors described herein. Processor 205 may include an application processor (also referred to as a main processor) that manages application domain and operating system (OS) functions and a baseband processor (also referred to as a baseband radio processor) that manages radio functions.

[0079] The memory 210 may be a computer-readable storage medium. The memory 210 may include a volatile computer storage medium. For example, the memory 210 may include a RAM including a dynamic RAM (DRAM), a synchronous dynamic RAM (SDRAM), and / or a static RAM (SRAM). The memory 210 may include a non-volatile computer storage medium. For example, the memory 210 may include a hard drive, a flash memory, or any other suitable non-volatile computer storage device. The memory 210 may include both a volatile computer storage medium and a non-volatile computer storage medium.

[0080] The memory 210 may store data related to implementing the traffic class field as described herein. The memory 210 may also store program code and related data, such as an operating system or other controller algorithms operating on the device 200.

[0081] Input device 215 may include any known computer input device, including a touchpad, buttons, keyboard, stylus, microphone, etc. Input device 215 may be integrated with output device 220, for example, as a touch screen or similar touch-sensitive display. Input device 215 may include a touch screen so that text can be entered using a virtual keyboard displayed on the touch screen and / or by handwriting on the touch screen. Input device 215 may include two or more different devices, such as a keyboard and a touchpad.

[0082] Output device 220 may be designed to output visual, auditory, and / or tactile signals. Output device 220 may include an electronically controllable display or display device capable of outputting visual data to a user. For example, output device 220 may include, but is not limited to, a liquid crystal display (LCD), a light emitting diode (LED) display, an organic LED (OLED) display, a projector, or a similar display device capable of outputting images, text, etc. to a user. As another non-limiting example, output device 220 may include a wearable display that is separate from the rest of user device apparatus 200 but is communicatively coupled to the rest, such as a smart watch, smart glasses, a head-up display, etc. In addition, output device 220 may be a component of a smart phone, a personal digital assistant, a television, a desktop computer, a notebook (laptop) computer, a personal computer, a vehicle dashboard, etc.

[0083] Output device 220 may include one or more speakers for producing sound. For example, output device 220 may produce an audible alarm or notification (e.g., a beep or buzzer). Output device 220 may include one or more tactile devices for producing vibration, motion, or other tactile feedback. All or part of output device 220 may be integrated with input device 215. For example, input device 215 and output device 220 may form a touch screen or similar touch-sensitive display. Output device 220 may be located near input device 215.

[0084] The transceiver 225 communicates with one or more network functions of the mobile communication network via one or more access networks. The transceiver 225 operates under the control of the processor 205 to send messages, data, and other signals, and also receives messages, data, and other signals. For example, the processor 205 can selectively activate the transceiver 225 (or part thereof) at a specific time to send and receive messages.

[0085] The transceiver 225 includes at least one transmitter 230 and at least one receiver 235. One or more transmitters 230 can be used to provide uplink communication signals to a base unit of a wireless communication network. Similarly, one or more receivers 235 can be used to receive downlink communication signals from the base unit. Although only one transmitter 230 and one receiver 235 are shown, the user equipment device 200 can have any suitable number of transmitters 230 and receivers 235. In addition, (multiple) transmitters 230 and (multiple) receivers 235 can be any suitable type of transmitter and receiver. The transceiver 225 may include a first transmitter / receiver pair for communicating with a mobile communication network through a licensed radio spectrum, and a second transmitter / receiver pair for communicating with a mobile communication network through an unlicensed radio spectrum.

[0086] A first transmitter / receiver pair that can be used to communicate with a mobile communication network via a licensed radio spectrum and a second transmitter / receiver pair that can be used to communicate with a mobile communication network via an unlicensed radio spectrum can be combined into a single transceiver unit, such as a single chip that performs functions for use with both the licensed radio spectrum and the unlicensed radio spectrum. The first transmitter / receiver pair and the second transmitter / receiver pair can share one or more hardware components. For example, some of the transceivers 225, transmitters 230, and receivers 235 can be implemented as physically separate components that access shared hardware resources and / or software resources, such as, for example, a network interface 240.

[0087] One or more transmitters 230 and / or one or more receivers 235 can be implemented and / or integrated into a single hardware component, such as a multi-transceiver chip, a system on chip, an application specific integrated circuit (ASIC), or other types of hardware components. One or more transmitters 230 and / or one or more receivers 235 can be implemented and / or integrated into a multi-chip module. Other components (such as a network interface 240 or other hardware components / circuits) can be integrated into a single chip with any number of transmitters 230 and / or receivers 235. Transmitters 230 and receivers 235 can be logically configured as transceivers 225 using one or more common control signals, or can be configured as modular transmitters 230 and receivers 235 implemented in the same hardware chip or multi-chip module.

[0088] Figure 3 Depicted are further details of a network node 300 that may be used to implement the methods described herein. The network node 300 may be a network element 104, an analytics consumer 710, an analytics consumer 810, a NWDAF 720, a NWDAF 730, a NWDAF 825, a data source 740, a data source 840, an ARDF 850, or a TRLF 860 as described herein. The network node 300 includes a processor 305, a memory 310, an input device 315, an output device 320, and a transceiver 325.

[0089] The input device 315 and the output device 320 may be combined into a single device, such as a touch screen. In some implementations, the network node 300 does not include any input device 315 and / or output device 320. The network node 300 may include one or more of the following: a processor 305, a memory 310, and a transceiver 325, and may not include an input device 315 and / or an output device 320.

[0090] As depicted, the transceiver 325 includes at least one transmitter 330 and at least one receiver 335. Here, the transceiver 325 communicates with one or more remote units 200. Additionally, the transceiver 325 can support at least one network interface 340 and / or an application interface 345. The (multiple) application interface 345 can support one or more APIs. The (multiple) network interface 340 can support 3GPP reference points such as Uu, N1, N2, and N3. As understood by those of ordinary skill in the art, other network interfaces 340 can be supported.

[0091] The processor 305 may include any known controller capable of executing computer-readable instructions and / or capable of performing logical operations. For example, the processor 305 may be a microcontroller, a microprocessor, a CPU, a GPU, an auxiliary processing unit, an FPGA, or a similar programmable controller. The processor 305 may execute instructions stored in the memory 310 to perform the methods and routines described herein. The processor 305 is communicatively coupled to the memory 310, the input device 315, the output device 320, and the transceiver 325.

[0092] The memory 310 may be a computer-readable storage medium. The memory 310 may include a volatile computer storage medium. For example, the memory 310 may include a RAM including a dynamic RAM (DRAM), a synchronous dynamic RAM (SDRAM), and / or a static RAM (SRAM). The memory 310 may include a non-volatile computer storage medium. For example, the memory 310 may include a hard drive, a flash memory, or any other suitable non-volatile computer storage device. The memory 310 may include both a volatile computer storage medium and a non-volatile computer storage medium.

[0093] The memory 310 may store data related to establishing a multipath unicast link and / or mobile operation. For example, as described herein, the memory 310 may store parameters, configurations, resource allocations, policies, etc. The memory 310 may also store program codes and related data, such as an operating system or other controller algorithms operating on the network node 300.

[0094] Input device 315 may include any known computer input device, including a touchpad, buttons, keyboard, stylus, microphone, etc. Input device 315 may be integrated with output device 320, for example, as a touch screen or similar touch-sensitive display. Input device 315 may include a touch screen so that text can be entered using a virtual keyboard displayed on the touch screen and / or by handwriting on the touch screen. Input device 315 may include two or more different devices, such as a keyboard and a touchpad.

[0095] Output device 320 may be designed to output visual, auditory, and / or tactile signals. Output device 320 may include an electronically controllable display or display device capable of outputting visual data to a user. For example, output device 320 may include, but is not limited to, an LCD display, an LED display, an OLED display, a projector, or a similar display device capable of outputting images, text, etc. to a user. As another non-limiting example, output device 320 may include a wearable display that is separate from the rest of network node 300 but is communicatively coupled to the rest, such as a smart watch, smart glasses, a head-up display, etc. In addition, output device 320 may be a component of a smart phone, a personal digital assistant, a television, a desktop computer, a notebook (laptop) computer, a personal computer, a vehicle dashboard, etc.

[0096] Output device 320 may include one or more speakers for producing sound. For example, output device 320 may produce an audible alarm or notification (e.g., a beep or buzzer). Output device 320 may include one or more tactile devices for producing vibration, motion, or other tactile feedback. All or part of output device 320 may be integrated with input device 315. For example, input device 315 and output device 320 may form a touch screen or similar touch-sensitive display. Output device 320 may be located near input device 315.

[0097] The transceiver 325 includes at least one transmitter 330 and at least one receiver 335. One or more transmitters 330 can be used to communicate with the UE, as described herein. Similarly, one or more receivers 335 can be used to communicate with network functions in the PLMN and / or RAN, as described herein. Although only one transmitter 330 and one receiver 335 are illustrated, the network node 300 can have any suitable number of transmitters 330 and receivers 335. In addition, the (multiple) transmitters 330 and (multiple) receivers 335 can be any suitable type of transmitter and receiver.

[0098] Therefore, a data analysis function is provided, comprising a processor and a receiver. The processor is arranged to generate analysis data for an analysis service using at least one data source. The receiver is arranged to receive events related to the analysis service. The processor is also arranged to determine a rating of at least one data source, the rating being based on supplementary data.

[0099] The data analysis functions defined herein tend to provide improved analysis data. This is achieved by facilitating the detection of the correctness of the analysis data and the correction of the analysis data. Improved data analysis tends to be provided as a result of the rating of the data sources used for analysis. The rating can be used as a criterion for selecting which sources to collect data from, thereby improving the quality of the analysis service.

[0100] The data analysis function may be located at OAM, MDAS, or an application layer (such as ADAES). An event may include an evaluation of an analysis service. The event may include an evaluation by analysis ID, by analysis event ID, by analysis service area, or by a data statistical range associated with the analysis ID. The evaluation may include a level of accuracy, performance, or correctness. The analysis service may be used by a consumer. The consumer may be another analysis function (e.g., MTLF, ANLF). Events associated with the analysis service may be received from a consumer. Supplementary data may include a previous rating of at least one data source. Supplementary data may include a historical rating of at least one data source.

[0101] The processor may be further arranged to identify a rating of at least one data source as being below a predetermined threshold and to trigger corrective action for the at least one data source having a rating below the predetermined threshold.

[0102] The corrective action may include: requesting supplemental data from one or more additional data sources; updating the specified accuracy of the analysis report; updating a mapping of one or more data sources to the analysis service; or some combination thereof.

[0103] The event may be received by at least one of: an analytics consumer, an analytics producer, a NWDAF, a NWDAFANLF, a NWDAF MTLF, an AF, or any combination thereof.

[0104] The at least one data source may include a UE, an AF, or a network element. The network element may be operated by a network operator different from the operator of the data analysis function.

[0105] The processor may also be arranged to verify the data from the data source by comparing the data from the data source with the supplementary source, determining the rating of the data source. The supplementary data may be provided by another data source. The supplementary source may be a source that generates and / or provides the supplementary data. An example of a supplementary source may be a network function, a management function, another UE, or another AF. The supplementary source may provide data of the same data type as the original data. The data type may include any of real-time data, network data, user data, and / or granularity. The supplementary source may provide data of different types but focusing on the same parameters. Such supplementary data may include QoS data from a wireless communication network, QoS data from an application server, or QoS data from a UE. Another data source may be a data source of the same type. Another data source may be a data source for the same analysis service, service area, or analysis event.

[0106] Determining the rating of at least one data source may include obtaining a data source contribution weight. The data analysis function may also be arranged to store the rating in a data storage entity. The data storage entity may be an ADRF.

[0107] The transmitter may be further arranged to send the data source rating to at least one network node. The at least one network node may include a consumer. The transmitter may be arranged to send the data source rating to at least one application entity. The transmitter may be arranged to send the data source rating to at least one of: an analysis consumer, an analysis producer, a NWDAF, an ANLF, a MTLF, an AF, and a UE.

[0108] Where the receiver is arranged to receive events related to the evaluation of the analysis service from a particular node, then the sender may also be arranged to send the data source rating to the particular node. In the alternative, the data analysis function reports the data source rating to a node other than the particular node from which the events related to the evaluation of the analysis service are received.

[0109] The transmitter may be further arranged to send a report of the corrective action to at least one network node.The at least one network node may comprise a consumer.

[0110] The determined rating may include a confidence level. The rating may also be based on a previous data source rating. The previous data source rating may be retrieved from a data storage entity. The previous rating of at least one data source may include a historical rating of at least one data source.

[0111] Figure 4 The method 400 in a data analysis function is illustrated. The method 400 includes: generating 410 analysis data for an analysis service using at least one data source; receiving 420 an event related to the analysis service; and in response to receiving the event, determining 430 a rating of at least one data source, the rating based on supplemental data.

[0112] The event may include an evaluation of the analysis service. The event may include an evaluation by analysis ID or by analysis event ID or by analysis service area or by a statistical range of data associated with the analysis ID. The rating may include a level of accuracy, performance, or correctness.

[0113] There is also provided a data storage entity comprising a receiver and a memory. The receiver is arranged to receive a rating of at least one data source. The memory is arranged to store the rating of at least one data source.

[0114] The data storage entities described herein tend to provide improved analytical data. This is achieved by facilitating the collection of ratings of data sources used for analysis. The ratings can be used by analytical service providers as criteria for selecting which sources to collect data from, thereby improving the quality of analytical services.

[0115] The data storage entity may further comprise a transmitter arranged to transmit the rating of the at least one data source to the data analysis function.The stored previous rating of the at least one data source may comprise a historical rating of the at least one data source.

[0116] Figure 5 A method 500 in a data storage entity is illustrated. The method 500 comprises receiving 510 a rating of at least one data source, and storing 520 a rating of at least one data source.

[0117] The method may further include sending the rating of the at least one data source to the data analysis function.The stored previous rating of the at least one data source may include a historical rating of the at least one data source.

[0118] Figure 6 The solution presented in this article is illustrated by way of the following high-level steps.

[0119] At 681, the analytics consumer requests and consumes analytics services from the data analytics function (eg, NWDAF ANLF). This can be accomplished in a variety of ways, two example options are presented below.

[0120] Option 1: The data analysis function requests and receives a post-analysis evaluation (accuracy level, or alternatively, predicted success / failure) of the consumed service.

[0121] Option 2: Instead of getting an evaluation from the consumer, it is also possible that an existing solution (MTLF evaluates the correctness of the data, or ANLF evaluates the correctness of the analysis, based on existing technology) can be used to make a local evaluation based on the correlation of the predicted data with the benchmark real data, or optionally trigger a check on the data source rating. For example, if an error is found based on other solutions, the ANLF / MTLF will flag the wrong data to the rating function.

[0122] At 682, the data analysis function (or another function, such as TRLF) processes the evaluation and identifies data sources that deviate statistically significantly from previous data (eg, based on source ID).

[0123] At 683, the analysis function (ANLF or MTLF) or another function marks problematic data sources that are used as input in case of bad data / bad ratings. Such ratings may result in a down-weighting or even blacklisting of data sources with ratings below a predetermined threshold. The predetermined threshold may be set by the MNO depending on the desired QoS or QoE.

[0124] At 684, the analysis function (ANLF or MTLF) or another function may tag and then provide a rating or weight associated with the data source ID, indicating the accuracy of the input data. In another use case, the rating or weight may also be stored in the ADRF, or may indicate whether data from a particular data source should be stored in the ADRF at all (e.g., data from data sources with a rating below a certain predetermined threshold should not be stored; the predetermined threshold may be set by the MNO depending on the desired QoS or QoE).

[0125] Some sources may have a predefined rating (e.g., OAM data sources) such that only some other sources are rated according to the proposed method. Such other sources may include at least one of an AF, a UE, a RAN node, and a UPF. Furthermore, in case multiple data sources contribute to the analysis of the rating (such as from a specific AF), then the contribution weight of each of these data sources used by the AF may be taken as input to rank at least one of the multiple data sources.

[0126] The data analysis function discovers possible alternative sources and requests and receives supplemental data for the required input at 685. If the data distribution deviates from the alternative source, the rating should be downgraded.

[0127] At 686, the data analysis function (or TRLF or ADRF) stores all ratings for the data source based on the deviation (assuming input from multiple requests). Such ratings may be quantified (e.g., low, medium, high) or expressed as a percentage value (% accuracy) or delta offset compared to a threshold, or in any other format (based on pre-configuration from the MNO). If the rating is below a predetermined threshold, the corresponding data may not be stored but discarded.

[0128] At 687, the new analysis request arrives at the analysis function associated with the target analysis service (based on the analysis ID).

[0129] At 688, the data analysis function retrieves and checks the ratings of the data sources for the analysis service. For example: in the case where the MTLF needs training data to train the ML model, the MTLF queries the rating function to find the data source (or historical data from ADRF) or new data.

[0130] At 689, if there are data sources that would reduce the confidence of the prediction or data sources that are below a threshold rating, the analysis function will do one of the following:

[0131] If more than one source can provide similar data, the highest rated source is selected

[0132] Request supplementary data from other available data sources, and use the supplementary data for verification of data from low-rated data sources. Such supplementary data may be beneficial for certain analysis services (e.g., for analysis applicable to any UE, such as region-based analysis), and the determination at the NWDAF to use the supplementary data also takes into account the complexity of obtaining the data and the freshness of the data, etc. Alternatively or additionally, such supplementary data may be offline data or real-time data. Such supplementary data may be beneficial for certain analysis services, and the determination at the NWDAF to use the supplementary data also takes into account the complexity of obtaining the data and the freshness of the data, etc.

[0133] Modify the confidence level of the analysis based on the accuracy / correctness rating of one or more data sources involved.

[0134] At 690, if the required confidence level is achieved and / or the data is verified, the analysis function sends the analysis output to the consumer.

[0135] Figure 7 An implementation for ANLF-based rating and storage at ADRF is illustrated. Figure 7 An analysis consumer 710, NWDAF ANLF 720, MWDAF MTLF 730, at least one data source 740, and ADRF 750 are shown. The implementation shows enhancement at ANLF 720 and storage of ratings at ADRF 750. The process can be initiated in various ways, Figure 7 Two such options are given as examples.

[0136] Option 1, at 781a, after the analytics consumer 710 consumes the analytics service with analytics ID="xx", the NWDAF / ANLF 720 requests feedback / evaluation of the analytics service (good or bad, experience level, predicted success or failure, and possible reasons) from the consumer.

[0137] At 781b, the NWDAF / ANLF 720 receives feedback as requested. (rating, success / failure, reason). Such steps may be performed multiple times for multiple consumers for a given service.

[0138] Option 2: At 782a, the NWDAF MTLF 730 evaluates the ML model correctness (using a conventional approach) or evaluates the performance of the analytical model, and based on this (e.g., checking the performance based on a predefined threshold), the NWDAF MTLF 730 may decide to notify the ANLF 720 of the correctness.

[0139] At 782b, the NWDAF / ANLF 720 receives a notification from the MTLF 730 indicating possible low performance or low correctness, and optionally requests the ANLF 720 to further check the inference data / data source for correctness.

[0140] In step 782a / 782b, it is mentioned that MTLF 730 evaluates the correctness (correlation of prediction with reference truth), but correlation of prediction with reference truth may also be performed by ANLF 720. Therefore, step 782a / 782b may have different signaling based on which node performs the evaluation.

[0141] At 783, NWDAF / ANLF 720 conditionally (i.e., if necessary) requests and receives additional data from different data sources 740 (if available) to verify data source quality or data source correctness. Such data can be, for example, performance data from OAM to supplement AF, or data from UPF to supplement AF.

[0142] At 784a, where the data deviates from the supplemental data (or where step 783 is not implemented), the NWDAF / ANLF 720 updates the rating for the source, which was automatically changed based on the analysis feedback in step 782b.

[0143] At 784b, ANLF 720 may also optionally notify MTLF 730 of the data source rating (if MTLF 730 has not yet been involved in the rating process) in case the data source rating is also used to train the ML model. Such notification may be used by MTLF 730 to exclude the data source from training or mark the data source as untrustworthy.

[0144] At 785, NWDAF ANLF 720 stores the rating to ADRF 750 or any other repository function (or the rating may also be stored within NWDAF 720). Alternatively, NWDAF ANLF 720 may determine not to store the data at ADRF 750 if the rating of the data source is very low.

[0145] At 786 , a new analytics request arrives from analytics consumer 710 for the analytics service with analytics ID=“xx”.

[0146] At 787 , the NWDAF / ANLF 720 retrieves the ratings for the data source corresponding to the analysis ID.

[0147] At 788, if the rating of one or more data sources is below a threshold (preset), the NWDAF ANLF 720 triggers an action among: selection of an alternative data source with the highest rating; or requiring supplementary data from other available data sources and using the supplementary data to verify data from low-rated data sources.

[0148] At 789, if a new data source or additional data sources are needed, the NWDAF ANLF 720 subscribes to the new data source 740 and requests / receives new data. Such data requests may take the form of subscriptions.

[0149] At 790, NWDAF ANLF 720 obtains the analysis data and checks whether the confidence level is above the requested threshold. The derivation of the threshold also takes into account the rating of the data source (or the aggregated rating of the data source based on the individual ratings). NWDAF ANLF 720 can also verify / compare data from different sources on the same parameters.

[0150] At 791 , NWDAF ANLF 720 provides analysis output to analysis consumer 710 .

[0151] Figure 8 An alternative implementation is illustrated that uses a Trusted Rating Logic Function (TRLF) with another NF for performing data source rating. Here, another NF (e.g., similar to the TRLF) performs the rating of the data source instead of the ANLF. Figure 8 An analytics consumer 810 , a NWDAF 825 , at least one data source 840 , and a TRLF 860 are shown.

[0152] The process begins with the analysis consumer 810 discovering the NWDAF 825, which is Figure 8 In addition, the mapping table of analysis services and data sources / inputs is already known at NWDAF 825. This mapping table can be obtained from, for example, OAM ( Figure 8 825. Typically, fixed ratings are provided for data sources that are part of the network operator's premises, as such data sources are within the control of the network operator and can therefore be trusted to be accurate.

[0153] At 881 , the analytics consumer 810 requests analytics services from the selected NWDAF 825 , also specifying its consumer ID, which includes the NF (instance or collection) ID and the vendor ID.

[0154] At 882, the NWDAF 825 generates a token that can be used to rate the data source corresponding to the analysis service.

[0155] At 883, the NWDAF 825 sends to the TRLF 860 information about the consumer ID, the analysis ID, information about the ML model used to generate the analysis (if any), its own NWDAF 820 (instance or collection) ID, the data source ID and address, and / or a mapping table related to the analysis service and the token generated for the data source rating. In this way, the TRLF 860 can associate the rating from the consumer 810 with the analysis service provided by the NWDAF 825, and in the case where the analysis service is based on the ML model, implicitly associate the rating from the consumer 810 with the ML model and the data source used to generate the ML model. Here, instead of obtaining the analysis rating from the consumer 810, there is an alternative / complementary option, which is based on the local rating at the NWDAF (MTLF or ANLF) 825.

[0156] At 884 , TRLF 860 sends an acknowledgement to NWDAF 825 .

[0157] At 885 , NWDAF 825 sends the analytics response to analytics consumer 810 along with the generated token, which is used to allow only authenticated consumers (ie, only consumers who actually consume the service) to evaluate the analytics service.

[0158] If the analytics consumer 810 subscribes to the analytics service, the token is valid for the entire subscription duration, and the consumer 810 can update its rating by sending another Ntrlf_AnalyticsRating request. Once the subscription is terminated, the NWDAF 825 should inform the TRLF 860 of this so that the consumer 810 can only provide the final rating, after which the token is revoked.

[0159] At 886, the analysis consumer 810 evaluates the performance of the analysis service using the metrics obtained by the NRF during the discovery procedure. Such an evaluation metric can be the experience of the analysis service (e.g., poor, average, good), or can be a predicted success or failure. In the case of a predicted failure, the consumer 810 can also specify the reason for the failure (to indicate whether the failure is from the consumer side or from the analysis service).

[0160] At 887, analytics consumer 810 sends its rating to TRLF 860 via the Ntrlf_AnalyticsRating service. The request also includes the consumer ID of analytics consumer 810 and the received token.

[0161] At 888, if the tokens match and the analytics consumer 810 is not a model producer, the TRLF 860 processes the ratings and maps to the data sources.

[0162] At 889 , the TRLF 860 requests and receives supplemental data from the different data sources 840 to verify the data sources 840 .

[0163] At 890, the rating of data source 840 is changed (either automatically based on the validation of step 889 or based on the analysis rating), then TRLF 860 converts the new rating into an update to the rating of data source 840 for which the rating may have changed. TRLF 860 stores the ratings by analysis ID and by data source and by consumer ID.

[0164] At 891, if the rating has dropped from a predefined threshold, the TRLF 860 sends a notification regarding an update of the rating for the data source to the NWDAF 825. For example, such a threshold may be defined as being below an average.

[0165] At 892, NWDAF 825 checks the rating of the data source with the low rating and performs one of the following: marking; changing the mapping; or sending an alarm to OAM. Specifically, NWDAF 825 may mark the data source to request supplementary data from other available data sources in additional analysis requests, and use the supplementary data for verification of data from the low-rated data source. Alternatively, if more than one source can provide similar data, NWDAF 825 may update the mapping table to remove the data source or change the priority of the data source. Still alternatively, if the rating is very low (or incorrect data is provided multiple times), NWDAF 825 may send an alarm to OAM ( Figure 8 An alert is sent (not shown) to indicate that a data source may be blacklisted.

[0166] It should be noted that the process described herein can also be applied to data analysis functions at the OAM (MDAS specified in 3GPP SA5) or application layer (ADAES defined in 3GPP SA6). In addition, the rating can also be a weight associated with the data source, which can assist the ANLF in performing the selection with consideration of the target confidence.

[0167] The problem solved by some of the arrangements described herein is how to detect possible drift and relate the drift to a specific data source and the abnormal / unreliable behavior therein. In addition, some of the arrangements described herein define how the NWDAF should react dynamically. For example, the NWDAF may select an alternative data source or a complementary data source to ensure minimal impact on the accuracy of the analytical service it provides.

[0168] Certain arrangements defined herein define how to detect precision mismatches at ML model inferences and ensure correctness of analysis by enabling rating of data sources. NWDAF generates data source ratings / weights / profiles / reputations, which are used as criteria for selecting how and from which data sources to collect data. Such a solution is able to capture possible drifts at ML model inferences due to data sources. The solution is particularly useful for data sources that are UEs (via AF) or AFs, which are not as trusted as OAMs and NFs (analysis services as exemplified above).

[0169] The solution presented herein uses alternative / supplemental data sources to provide input for an analytical service to provide verification of the accuracy of a data source and a rating of the data source.

[0170] A solution is provided in which ANLF / NWDAF evaluates the performance of data sources, rates or assigns weights to data sources, and stores the ratings at ADRF. ANLF / NWDAF can request to supplement data sources to improve NWDAF correctness. A solution is also provided in which TRLF performs the rating of data sources instead of ANLF.

[0171] Therefore, a method for detecting the accuracy of data for network analysis at a data analysis function is provided, the method comprising: obtaining an evaluation of an analysis service; determining a rating of one or more data sources for the analysis service based on the evaluation of the analysis service output; identifying low-rated data sources for the analysis service and / or analysis events; and triggering correctness improvement actions based on the ratings of the data sources.

[0172] Evaluation of analytical services can be provided from the customer or NWDAF [ANLF / MTLF] or from the AF.

[0173] The data source may be a UE, an AF, or may be outside the control of the operator of the wireless communication network.

[0174] The actions may include one or more of: requesting additional data from one or more additional data sources; adapting the accuracy of the analysis based on the ratings; and / or updating a mapping of one or more data sources to the analysis service based on the identified low reliability data sources.

[0175] The method may also include validating the data by comparing data from one or more data sources of the same type and targeting the same analytics service or services or analytics events.

[0176] The method may also include obtaining a data source contribution weight before determining the data source rating.

[0177] The method may further include storing the data source rating to a repository function. The repository function may be an ADRF.

[0178] The method may also include sending the data source rating and / or events related to the correctness improvement action to at least one further network node and / or consumer.Such sending may be performed after the action is triggered.

[0179] Data source ratings may be associated with expected confidence levels and / or predefined confidence levels.

[0180] It should be noted that the above methods and apparatus illustrate rather than limit the present invention, and that those skilled in the art will be able to design many alternative arrangements without departing from the scope of the appended claims. The word "comprising" does not exclude the presence of other elements or steps than those listed in the claims, "a" or "an" does not exclude a plurality, and a single processor or other unit may fulfil the functions of multiple units described in the claims. Any reference signs in the claims should not be construed as limiting their scope.

[0181] Furthermore, although examples have been given in the context of specific communication standards, these examples are not intended to limit the communication standards to which the disclosed methods and apparatus may be applied. For example, although specific examples have been given in the context of 3GPP, the principles disclosed herein may also be applied to another wireless communication system, and indeed to any communication system that uses routing rules.

[0182] The method may also be embodied in an instruction set stored on a computer-readable medium, which, when loaded into a computer processor, a digital signal processor (DSP), etc., causes the processor to perform the above method.

[0183] The described methods and apparatus may be practiced in other specific forms. The described methods and apparatus are considered in all respects to be illustrative only and not restrictive. Therefore, the scope of the present invention is indicated by the appended claims rather than by the foregoing description. All changes that fall within the meaning and equivalent range of the claims should be included within the scope of the claims.

[0184] The following abbreviations are relevant to the areas of this document: AF, application function; NF, network function; NWDAF, network data analysis function; OAM, operations and maintenance; UE, user equipment; MDAS, management domain analysis service; ADAES, application data analytics enabling service / server; ANLF, analysis logic function; MTLF, model training logic function; DNAI, data network access identifier; MOS, mean opinion score; MDT, drive test minimization; ADAEC, application data analytics enabling client; TRLF, trust rating logic function; and ML, machine learning.

Claims

1. A data analysis function, comprising: a processor arranged to generate analytical data for an analytical service using at least one data source; a receiver arranged to receive events related to said analytical service; The processor is further arranged to determine a rating of the at least one data source, the rating being based on the supplementary data.

2. The data analysis function according to claim 1, wherein: The processor is further arranged to identify the rating of the at least one data source as being below a predetermined threshold; The processor is further arranged to trigger corrective action for the at least one data source having a rating below the predetermined threshold.

3. The data analysis function of claim 2, wherein the corrective action comprises: requesting supplemental data from one or more additional data sources; Update the specified accuracy of analytical reports; updating a mapping of the one or more data sources to the analysis service; or Some combination of the above.

4. A data analysis function according to any of the preceding claims, wherein the event is received by at least one of: an analysis consumer, an analysis producer, a network data analysis function, a network data analysis function analysis logic function, a network data analysis function model training logic function, an application function, or any combination of the above.

5. A data analysis function according to any of the preceding claims, wherein the at least one data source comprises a user device, an application function, or a network element.

6. A data analysis function according to any one of the preceding claims, wherein the processor is further arranged to: verify data from the data source by comparing the data from the data source with a supplementary source, and determine a rating of the data source.

7. The data analysis function of any of the preceding claims, wherein determining the ranking of the at least one data source comprises obtaining a data source contribution weight.

8. The data analysis function of any of the preceding claims, further comprising storing the rating in a data storage entity.

9. A data analysis function according to any of the preceding claims, wherein the transmitter is further arranged to transmit the data source rating to at least one network node.

10. A data analysis function according to claim 2 and any one of claims 3 to 9 as dependent on claim 2, wherein the transmitter is further arranged to send a report of the corrective action to at least one network node.

11. A data analysis function according to any of the preceding claims, wherein the determined rating includes a confidence level.

12. A data analysis function according to any one of the preceding claims, wherein the rating is also based on previous data source ratings.

13. A method in a data analysis function, the method comprising: generating analytical data for analytical services using at least one data source; receiving events related to the analysis service; In response to receiving the event, a rating of the at least one data source is determined, the rating being based on the supplemental data.

14. A data storage entity, comprising: a receiver arranged to receive a rating of at least one data source, and A memory arranged to store the rating of the at least one data source.

15. The data storage entity according to claim 14, further comprising a transmitter for transmitting the rating of the at least one data source to a data analysis function.

16. A method in a data storage entity, the method comprising: receiving a rating of at least one data source, and The rating of the at least one data source is stored.

17. The method of claim 16, further comprising sending the rating of the at least one data source to a data analysis function.

Images