Rating accuracy for analysis in wireless communication network

Abstract

Accordingly, a data analysis function comprising a receiver and a processor is provided. The receiver is arranged to receive analysis feedback information for a particular analysis from an analysis consumer. The processor is arranged to: identify an affected network function based on analyzing the feedback information; determining whether the affected network function is used as a data source for a particular analysis; rating the network function as a data source by comparing the prediction with the source data; and determining an accuracy of the particular analysis based on the rating of the network functions used as the data source for the particular analysis.

Description

Technical Field

[0001] The subject matter disclosed herein relates generally to the field of enabling rating accuracy of analytics in wireless communication networks.This document defines data analysis functions and methods in data analysis functions. Background Art

[0002] In 3GPP, data analysis services are provided by the Network Data Analysis Function (NWDAF) (see 3GPPTS23.288v17.5.0) and are intended to support network data analysis services in the 5G core network. Such analysis can collect data from other network functions (NFs), application functions (AFs), or operations and maintenance (OAMs), and can be exposed to third parties and / or AFs to provide statistics and predictions related to the operation of the wireless communication network. Such statistics and predictions can be related to slice load levels, observed service experience, NF load, network performance, UE-related analysis (mobility, communication), user data congestion, quality of service (QoS) sustainability, data network (DN) performance, etc. In addition, in 3GPP SA5 (3GPPTS28.533v17.2.0), the Management Data Analysis Service (MDAS) provides data analysis for the network. MDAS can be deployed at different levels, such as at the domain level (e.g., Radio Access Network (RAN), Core Network (CN), Network Slice Subnet) or in a centralized manner (e.g., at the Public Land Mobile Network (PLMN) level). The goal of MDAS is to optimize the management plane (at the network / domain level, slice / slice subnet level) by performing analysis based on network management data. Such services can be exposed to third parties / MDAS service consumers to provide PM analysis, FM analysis, network slice instance (NSI) / network slice subnet instance (NSSI) analysis, and optionally recommend appropriate management actions such as resource scaling, admission control, traffic load balancing, etc. Additional analysis capabilities in 3GPP are discussed in 3GPP SA6 (3GPP TR 23.700-36v0.4.0), which defines the Application Data Analysis Support Service (ADAES) for performing application layer and edge / cloud analysis outside the 3GPP domain.

[0003] In 3GPP TR 23.700-81v0.3.0 (Title: 5G Network Automation Enabler Study - Phase 3), one of the key issues discussed is Key Issue #1: How to improve the correctness of NWDAF analysis. The correctness of predictions is usually associated with accuracy, which represents the most prominent key performance indicator (KPI) for rating machine learning (ML) models. However, drift associated with the mismatch between training data and inference data can undermine accuracy. Therefore, ensuring accuracy is critical. Incorrect predictions may be due to the fact that the accuracy of the ML model during inference may be lower than the accuracy of the same ML model during training. Summary of the invention

[0004] The data analysis capabilities defined in this article tend to provide improved analytical data. This is achieved by facilitating the rating of data sources used in the analysis. Improved data analysis is often the result of rating the data sources used for analysis. The rating can be used as a criterion for selecting which sources to collect data from, thereby improving the quality of the analytical service.

[0005] Disclosed herein is a process for rating the accuracy of analysis in a wireless communication network. The process may be implemented by a data analysis function and a method in the data analysis function.

[0006] Therefore, a data analysis function is provided, comprising a receiver and a processor. The receiver is arranged to: receive analysis feedback information about a particular analysis from an analysis consumer. The processor is arranged to: identify an affected network function based on the analysis feedback information; determine whether the affected network function is used as a data source for the particular analysis; rate the network function as a data source by comparing the prediction to the source data; and determine the accuracy of the particular analysis based on the rating of the network function used as a data source for the particular analysis.

[0007] A method in a data analysis function is also provided. The method includes: receiving analysis feedback information about a specific analysis, the analysis feedback information received from an analysis consumer; and identifying an affected network function based on the analysis feedback information. The method also includes: determining whether the affected network function is used as a data source for the specific analysis; rating the network function as a data source by comparing the prediction to the source data; and determining the accuracy of the specific analysis based on the rating of the network function used as the data source for the specific analysis.

[0008] Rating the network functions defined herein allows the data analysis function to adapt to changes in the wireless communication network, thereby compensating for 'model drift' that can adversely affect the accuracy of the analysis over time. In addition, such a data analysis function can identify data sources affected by erroneous or unstable AnLF analysis and limit their impact on future analysis outputs by controlling their impact on (multiple) AnLF and (multiple) MTLF. Such control can be facilitated by rating the data sources used by erroneous or unstable AnLF analysis. BRIEF DESCRIPTION OF THE DRAWINGS

[0009] In order to describe the manner in which the advantages and features of the present disclosure can be obtained, the present disclosure is described by reference to certain devices and methods shown in the accompanying drawings. Each of these drawings depicts only certain aspects of the present disclosure and therefore should not be considered as limiting the scope thereof. For the sake of clarity, the drawings may have been simplified and are not necessarily drawn to scale.

[0010] A method and apparatus for evaluating analysis accuracy in a wireless communication network will now be described, by way of example only, with reference to the accompanying drawings, in which:

[0011] Figure 1 An embodiment of a wireless communication system for evaluating the accuracy of analytics in a wireless communication network is described;

[0012] Figure 2 depicts a user equipment device that may be used to implement the methods described herein;

[0013] Figure 3 depicts further details of a network node that may be used to implement the methods described herein;

[0014] Figure 4 An example wireless communication system is illustrated;

[0015] Figure 5 The architecture of a wireless communication system is illustrated;

[0016] Figure 6 illustrates methods that may be performed by the data analysis functions described herein;

[0017] Figure 7 illustrates a process for notifying other potentially affected AnLF(s) operating in the same area as the data analysis function; and

[0018] Figure 8 Illustration of methods in the data analysis function. DETAILED DESCRIPTION

[0019] Those skilled in the art will appreciate that aspects of the present disclosure may be embodied as systems, devices, methods or program products. Therefore, the arrangements described herein may be implemented in full hardware form, full software form (including firmware, resident software, microcode, etc.), or in a combination of software and hardware aspects.

[0020] For example, the disclosed methods and apparatus may be implemented as hardware circuits, including custom very large scale integrated (VLSI) circuits or gate arrays, off-the-shelf semiconductors such as logic chips, transistors, or other discrete components. The disclosed methods and apparatus may also be implemented in programmable hardware devices, such as field programmable gate arrays, programmable array logic, programmable logic devices, etc. As another example, the disclosed methods and apparatus may include one or more physical or logical blocks of executable code, which may be organized, for example, as objects, procedures, or functions.

[0021] In addition, the method and apparatus may take the form of a program product embodied in one or more computer-readable storage devices storing machine-readable code, computer-readable code, and / or program code, hereinafter referred to as code. The storage device may be tangible, non-transitory, and / or non-transmitting. The storage device may not embody a signal. In some arrangements, the storage device employs only a signal for accessing the code.

[0022] Any combination of one or more computer readable media may be used. The computer readable medium may be a computer readable storage medium. The computer readable storage medium may be a storage device storing code. The storage device may be, for example, but not limited to, an electronic, magnetic, optical, electromagnetic, infrared, holographic, micromechanical, or semiconductor system, device or apparatus, or any suitable combination of the above.

[0023] More specific examples of storage devices (a non-exhaustive list) include the following: an electrical connection with one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above. In the context of this document, a computer-readable storage medium may be any tangible medium that can contain or store a program for use by or in connection with an instruction execution system, apparatus, or device.

[0024] Throughout this specification, references to examples or similar language of a particular method or device indicate that a particular feature, structure, or characteristic described in conjunction with the example is included in at least one implementation of the method and device described herein. Therefore, unless otherwise expressly provided, references to examples or similar language features of a particular method or device may, but not necessarily, all refer to the same example, but mean "one or more but not all examples". Unless otherwise expressly provided, the terms "include", "comprise", "have", and their variants mean "including but not limited to". Unless otherwise expressly provided, a list of items listed does not mean that any or all items are mutually exclusive. Unless otherwise expressly provided, the terms "one", "an", and "the" also refer to "one or more".

[0025] As used herein, a list with the conjunction "and / or" includes any single item in the list or a combination of items in the list. For example, a list of A, B, and / or C includes only A, only B, only C, a combination of A and B, a combination of B and C, a combination of A and C, or a combination of A, B, and C. As used herein, a list using the term "one or more of ..." includes any single item in the list or a combination of items in the list. For example, one or more of A, B, and C includes only A, only B, only C, a combination of A and B, a combination of B and C, a combination of A and C, or a combination of A, B, and C. As used herein, a list using the term "one of ..." includes one and only one of any single item in the list. For example, "one of A, B, and C" includes only A, only B, or only C, and does not include a combination of A, B, and C. As used herein, "a member selected from a group consisting of A, B, and C" includes one and only one of A, B, or C, and does not include a combination of A, B, and C. As used herein, “a member selected from the group consisting of A, B and C and combinations thereof” includes only A, only B, only C, a combination of A and B, a combination of B and C, a combination of A and C, or a combination of A, B and C.

[0026] In addition, the features, structures or characteristics described herein may be combined in any suitable manner. In the following description, many specific details are provided, such as examples of programming, software modules, user selections, network transactions, database queries, database structures, hardware modules, hardware circuits, hardware chips, etc., to provide a comprehensive understanding of the present disclosure. However, those skilled in the relevant art will recognize that the disclosed methods and apparatus may be practiced without one or more of the specific details, or may be practiced using other methods, components, materials, etc. In other cases, well-known structures, materials or operations are not shown or described in detail to avoid confusing aspects of the present disclosure.

[0027] The following is a description of various aspects of the disclosed method and apparatus with reference to schematic flow charts and / or schematic block diagrams of methods, apparatuses, systems, and program products. It will be understood that each frame of the schematic flow chart and / or schematic block diagram, and the combination of frames in the schematic flow chart and / or schematic block diagram can be implemented by code. The code can be provided to a processor of a general-purpose computer, a special-purpose computer, or other programmable data processing device to produce a machine, so that instructions executed by a processor of a computer or other programmable data processing device can create a component for implementing the function / action specified in the schematic flow chart and / or schematic block diagram.

[0028] The code may also be stored in a storage device that can direct a computer, other programmable data processing apparatus, or other device to work in a specific manner so that the instructions stored in the storage device produce an article of manufacture that includes instructions for implementing the functions / actions specified in the schematic flowchart and / or schematic block diagram.

[0029] The code may also be loaded onto a computer, other programmable data processing apparatus, or other device so that a series of operating steps are executed on the computer, other programmable apparatus, or other device, thereby producing a computer-implemented process, so that the code executed on the computer or other programmable apparatus provides a process for implementing the functions / actions specified in the schematic flowchart and / or schematic block diagram.

[0030] The schematic flow charts and / or schematic block diagrams in the figure illustrate the possible architectures, functions and operations of the devices, systems, methods and program products. In this regard, each box in the schematic flow charts and / or schematic block diagrams can represent a module, segment, or portion of a code, which includes one or more executable instructions for implementing the code of (multiple) specified logical functions.

[0031] It should also be noted that in some alternative implementations, the functions shown in the blocks may not appear in the order shown in the figures. For example, two blocks shown in succession may, in fact, be executed substantially simultaneously, or the blocks may sometimes be executed in the reverse order, depending on the functions involved. Other steps and methods may be envisioned that are equivalent in function, logic or effect to one or more blocks or portions thereof of the illustrated figures.

[0032] The description of an element in each figure may refer to the elements of the subsequent figures. The same numerals represent the same elements throughout the drawings.

[0033] Figure 1 An embodiment of a wireless communication system 100 for evaluating the accuracy of analysis in a wireless communication network is depicted. In one embodiment, the wireless communication system 100 includes a remote unit 102 and a network unit 104. Although Figure 1A specific number of remote units 102 and network units 104 are depicted in FIG. 1 , but one skilled in the art will recognize that any number of remote units 102 and network units 104 may be included in the wireless communication system 100.

[0034] In one embodiment, the remote unit 102 may include a computing device such as a desktop computer, a laptop computer, a personal digital assistant (PDA), a tablet computer, a smart phone, a smart TV (e.g., a TV connected to the Internet), a set-top box, a game console, a security system (including a security camera), a vehicle computer, a network device (e.g., a router, a switch, a modem), an aircraft, a drone, etc. In some embodiments, the remote unit 102 includes a wearable device such as a smart watch, a fitness band, an optical head-mounted display, etc. In addition, the remote unit 102 may be referred to as a subscriber unit, a mobile device, a mobile station, a user, a terminal, a mobile terminal, a fixed terminal, a subscriber station, a UE, a user terminal, a device, or other terms used in the art. The remote unit 102 may communicate directly with one or more of the network units 104 via UL communication signals. In some embodiments, the remote unit 102 may communicate directly with other remote units 102 via sidelink communications.

[0035] The network elements 104 may be distributed over a geographical area. In some embodiments, the network elements 104 may also be referred to as access points, access terminals, base stations, base stations, node Bs, eNBs, gNBs, home node Bs, relay nodes, devices, core networks, air servers, radio access nodes, APs, NRs, network entities, access and mobility management functions (AMFs), unified data repositories (UDRs), UDM / UDRs, policy control functions (PCFs), location management functions (LMFs), radio access networks (RANs), network slice selection functions (NSSFs), operations, maintenance, and management (OAMs), session management functions (SMFs), user plane functions (UPFs), application functions, authentication server functions (AUSFs), security anchor functions (SEAFs), trusted non-3GPP gateway functions (TNGFs), application functions, service enablement architecture layer (SEAL) functions, vertical application enablement servers, edge enablement servers, border configuration servers, mobile edge computing platform functions, mobile edge computing applications, application data analysis enablement servers, SEAL data delivery servers, middleware entities, network slice capability management servers, or any other term used in the art. The network elements 104 are generally part of a radio access network that includes one or more controllers communicatively coupled to one or more corresponding network elements 104. The radio access network is generally communicatively coupled to one or more core networks, which may be coupled to other networks, such as the Internet and a public switched telephone network, etc. These and other elements of the radio access network and core network are not illustrated, but are generally known to those of ordinary skill in the art.

[0036] In one implementation, the wireless communication system 100 complies with the New Radio (NR) protocol standardized in 3GPP, wherein the network unit 104 transmits on the downlink (DL) using an orthogonal frequency division multiplexing (OFDM) modulation scheme, and the remote unit 102 transmits on the uplink (UL) using a single carrier frequency division multiple access (SC-FDMA) scheme or an OFDM scheme. More generally, however, the wireless communication system 100 may implement some other open or proprietary communication protocol, such as WiMAX, IEEE 802.11 variants, GSM, GPRS, UMTS, LTE variants, CDMA2000, ZigBee, Sigfox, etc. The present disclosure is not intended to be limited to the implementation of any particular wireless communication system architecture or protocol.

[0037] The network unit 104 can serve multiple remote units 102 within a service area (eg, a cell or cell sector) via wireless communication links. The network unit 104 sends DL communication signals to serve the remote units 102 in the time, frequency, and / or spatial domains.

[0038] Figure 2 A user equipment device 200 is depicted that can be used to implement the methods described herein. The user equipment device 200 is used to implement one or more of the solutions described herein. The user equipment device 200 conforms to one or more of the user equipment devices described in the embodiments herein. In particular, the user equipment device 200 may include a remote unit 102 or a UE 404 as described herein. The user equipment device 200 includes: a processor 205, a memory 210, an input device 215, an output device 220, and a transceiver 225.

[0039] The input device 215 and the output device 220 may be combined into a single device, such as a touch screen. In some implementations, the user equipment apparatus 200 does not include any input device 215 and / or output device 220. The user equipment apparatus 200 may include one or more of the following: a processor 205, a memory 210, and a transceiver 225, and may not include an input device 215 and / or an output device 220.

[0040] As depicted, the transceiver 225 includes at least one transmitter 230 and at least one receiver 235. The transceiver 225 can communicate with one or more cells (or wireless coverage areas) supported by one or more base units. The transceiver 225 can be operable on an unlicensed spectrum. In addition, the transceiver 225 can include multiple UE panels supporting one or more beams. In addition, the transceiver 225 can support at least one network interface 240 and / or application interface 245. (Multiple) application interface 245 can support one or more APIs. (Multiple) network interface 240 can support 3GPP reference points such as Uu, N1, PC5, etc. As understood by those of ordinary skill in the art, other network interfaces 240 can be supported.

[0041] The processor 205 may include any known controller capable of executing computer-readable instructions and / or capable of performing logical operations. For example, the processor 205 may be a microcontroller, a microprocessor, a central processing unit (CPU), a graphics processing unit (GPU), an auxiliary processing unit, a field programmable gate array (FPGA), or a similar programmable controller. The processor 205 may execute instructions stored in the memory 210 to perform the methods and routines described herein. The processor 205 is communicatively coupled to the memory 210, the input device 215, the output device 220, and the transceiver 225.

[0042] Processor 205 may control user equipment device 200 to implement the user equipment device behaviors described herein. Processor 205 may include an application processor (also referred to as a main processor) that manages application domain and operating system (OS) functions and a baseband processor (also referred to as a baseband radio processor) that manages radio functions.

[0043] The memory 210 may be a computer-readable storage medium. The memory 210 may include a volatile computer storage medium. For example, the memory 210 may include a RAM, including a dynamic RAM (DRAM), a synchronous dynamic RAM (SDRAM), and / or a static RAM (SRAM). The memory 210 may include a non-volatile computer storage medium. For example, the memory 210 may include a hard drive, a flash memory, or any other suitable non-volatile computer storage device. The memory 210 may include both a volatile computer storage medium and a non-volatile computer storage medium.

[0044] The memory 210 may store data related to implementing the traffic class field as described herein. The memory 210 may also store program code and related data, such as an operating system or other controller algorithms operating on the device 200.

[0045] Input device 215 may include any known computer input device, including a touchpad, buttons, keyboard, stylus, microphone, etc. Input device 215 may be integrated with output device 220, for example, as a touch screen or similar touch-sensitive display. Input device 215 may include a touch screen so that text can be entered using a virtual keyboard displayed on the touch screen and / or by handwriting on the touch screen. Input device 215 may include two or more different devices, such as a keyboard and a touchpad.

[0046] The output device 220 may be designed to output visual signals, auditory signals, and / or tactile signals. The output device 220 may include an electronically controllable display or display device capable of outputting visual data to a user. For example, the output device 220 may include, but is not limited to, a liquid crystal display (LCD), a light emitting diode (LED) display, an organic LED (OLED) display, a projector, or a similar display device capable of outputting images, text, etc. to a user. As another non-limiting example, the output device 220 may include a wearable display that is separate from the rest of the user device apparatus 200 but communicatively coupled to the rest, such as a smart watch, smart glasses, a head-up display, etc. In addition, the output device 220 may be a component of a smart phone, a personal digital assistant, a television, a desktop computer, a notebook (laptop) computer, a personal computer, a vehicle dashboard, etc.

[0047] Output device 220 may include one or more speakers for producing sound. For example, output device 220 may produce an audible alarm or notification (e.g., a beep or buzzer). Output device 220 may include one or more tactile devices for producing vibration, motion, or other tactile feedback. All or part of output device 220 may be integrated with input device 215. For example, input device 215 and output device 220 may form a touch screen or similar touch-sensitive display. Output device 220 may be located near input device 215.

[0048] The transceiver 225 communicates with one or more network functions of the mobile communication network via one or more access networks. The transceiver 225 operates under the control of the processor 205 to send messages, data, and other signals, and also receives messages, data, and other signals. For example, the processor 205 can selectively activate the transceiver 225 (or part thereof) at a specific time to send and receive messages.

[0049] The transceiver 225 includes at least one transmitter 230 and at least one receiver 235. One or more transmitters 230 can be used to provide uplink communication signals to the base station unit of the wireless communication network. Similarly, one or more receivers 235 can be used to receive downlink communication signals from the base station unit. Although only one transmitter 230 and one receiver 235 are shown, the user equipment device 200 can have any suitable number of transmitters 230 and receivers 235. In addition, (multiple) transmitters 230 and (multiple) receivers 235 can be any suitable type of transmitter and receiver. The transceiver 225 may include a first transmitter / receiver pair for communicating with a mobile communication network on a licensed radio spectrum, and a second transmitter / receiver pair for communicating with a mobile communication network on an unlicensed radio spectrum.

[0050] A first transmitter / receiver pair that can be used to communicate with a mobile communication network on a licensed radio spectrum and a second transmitter / receiver pair that can be used to communicate with a mobile communication network on an unlicensed radio spectrum can be combined into a single transceiver unit, such as a single chip that performs functions for use with both the licensed radio spectrum and the unlicensed radio spectrum. The first transmitter / receiver pair and the second transmitter / receiver pair can share one or more hardware components. For example, some transceivers 225, transmitters 230, and receivers 235 can be implemented as physically separate components that access shared hardware resources and / or software resources, such as a network interface 240.

[0051] One or more transmitters 230 and / or one or more receivers 235 may be implemented and / or integrated into a single hardware component, such as a multi-transceiver chip, a system on chip, an application specific integrated circuit (ASIC), or other types of hardware components. One or more transmitters 230 and / or one or more receivers 235 may be implemented and / or integrated into a multi-chip module. Other components such as a network interface 240 or other hardware components / circuits may be integrated into a single chip with any number of transmitters 230 and / or receivers 235. Transmitters 230 and receivers 235 may be logically configured as transceivers 225 using one or more common control signals, or may be configured as modular transmitters 230 and receivers 235 implemented in the same hardware chip or multi-chip module.

[0052] Figure 3 Depicted are further details of a network node 300 that may be used to implement the methods described herein. The network node 300 may be an implementation of an entity in a wireless communication network (e.g., one or more of the wireless communication networks described herein). The network node 300 may include a data analysis function, NWDAF 550, 650, or AnLF NWDAF 410, 750 as described herein. The network node 300 includes: a processor 305, a memory 310, an input device 315, an output device 320, and a transceiver 325.

[0053] The input device 315 and the output device 320 may be combined into a single device, such as a touch screen. In some implementations, the network node 300 does not include any input device 315 and / or output device 320. The network node 300 may include one or more of a processor 305, a memory 310, and a transceiver 325, and may not include an input device 315 and / or an output device 320.

[0054] As depicted, the transceiver 325 includes at least one transmitter 330 and at least one receiver 335. Here, the transceiver 325 communicates with one or more remote units 102. In addition, the transceiver 325 can support at least one network interface 340 and / or an application interface 345. The (multiple) application interface 345 can support one or more APIs. The (multiple) network interface 340 can support 3GPP reference points such as Uu, N1, N2, and N3. As will be appreciated by one of ordinary skill in the art, other network interfaces 340 may be supported.

[0055] The processor 305 may include any known controller capable of executing computer-readable instructions and / or capable of performing logical operations. For example, the processor 305 may be a microcontroller, a microprocessor, a CPU, a GPU, an auxiliary processing unit, an FPGA, or a similar programmable controller. The processor 305 may execute instructions stored in the memory 310 to perform the methods and routines described herein. The processor 305 is communicatively coupled to the memory 310, the input device 315, the output device 320, and the transceiver 325.

[0056] The memory 310 may be a computer-readable storage medium. The memory 310 may include a volatile computer storage medium. For example, the memory 310 may include a RAM, including a dynamic RAM (DRAM), a synchronous dynamic RAM (SDRAM), and / or a static RAM (SRAM). The memory 310 may include a non-volatile computer storage medium. For example, the memory 310 may include a hard drive, a flash memory, or any other suitable non-volatile computer storage device. The memory 310 may include both a volatile computer storage medium and a non-volatile computer storage medium.

[0057] The memory 310 may store data related to establishing a multipath unicast link and / or mobile operation. For example, as described herein, the memory 310 may store parameters, configurations, resource allocations, policies, etc. The memory 310 may also store program code and related data, such as an operating system or other controller algorithm operating on the network node 300.

[0058] Input device 315 may include any known computer input device, including a touchpad, buttons, keyboard, stylus, microphone, etc. Input device 315 may be integrated with output device 320, for example, as a touch screen or similar touch-sensitive display. Input device 315 may include a touch screen so that text can be entered using a virtual keyboard displayed on the touch screen and / or by handwriting on the touch screen. Input device 315 may include two or more different devices, such as a keyboard and a touchpad.

[0059] Output device 320 can be designed to output visual signals, auditory signals, and / or tactile signals. Output device 320 may include an electronically controllable display or display device capable of outputting visual data to a user. For example, output device 320 may include, but is not limited to, an LCD display, an LED display, an OLED display, a projector, or a similar display device capable of outputting images, text, etc. to a user. As another non-limiting example, output device 320 may include a wearable display that is separate from the rest of network node 300 but communicatively coupled to the rest, such as a smart watch, smart glasses, a head-up display, etc. In addition, output device 320 may be a component of a smart phone, a personal digital assistant, a television, a desktop computer, a notebook (laptop) computer, a personal computer, a vehicle dashboard, etc.

[0060] Output device 320 may include one or more speakers for producing sound. For example, output device 320 may produce an audible alarm or notification (e.g., a beep or buzzer). Output device 320 may include one or more tactile devices for producing vibration, motion, or other tactile feedback. All or part of output device 320 may be integrated with input device 315. For example, input device 315 and output device 320 may form a touch screen or similar touch-sensitive display. Output device 320 may be located near input device 315.

[0061] The transceiver 325 includes at least one transmitter 330 and at least one receiver 335. One or more transmitters 330 can be used to communicate with the UE, as described herein. Similarly, one or more receivers 335 can be used to communicate with network functions in the PLMN and / or RAN, as described herein. Although only one transmitter 330 and one receiver 335 are shown, the network node 300 can have any suitable number of transmitters 330 and receivers 335. In addition, the (multiple) transmitters 330 and (multiple) receivers 335 can be any suitable type of transmitter and receiver.

[0062] When a consumer requests analysis from the NWDAF, the consumer may include a target area and / or (multiple) target objects (e.g., network functions "NF") and / or a target UE or a group of (multiple) UEs in the request. The NWDAF performs analysis by collecting data and / or events from one or more network functions to ensure that the data comes from the requested target area or involves (multiple) target UEs. The above analysis is performed taking into account a "snapshot" of the UE's location based on the AMF's determination that the UE enters or leaves a specific area of ​​interest at the granularity of (multiple) tracking areas or (multiple) cell IDs.

[0063] In the current Release 16 and Release 17 3GPP architecture, the NWDAF (defined in 3GPP TS 23.288 v17.2.0) provides analysis outputs to one or more analysis consumer NFs or AFs or OAMs based on data collected from one or more data producer NFs and / or AFs and / or OAMs.

[0064] Figure 4 An example wireless communication system 400 is illustrated. The system 100 includes: a NWDAF analysis logic function (ANLF) 410, a NWDAF model training logic function (MTLF) 412, a plurality of data producer network functions, in this example an application function (AF) 420, a 5G network function 422, and an operation, administration and maintenance (OAM) 424. The wireless communication system 400 also includes a data collection coordination function (DCCF) 440 and a plurality of analysis consumer network functions, in this example including: an application function 430, a 5G network function 432, and an OAM 434. In the current Release 16 and Release 17 3GPP architecture, the NWDAF 410, 412 (defined in 3GPP Technical Specification 23.288 v17.2.0) provides analysis outputs to one or more of the analysis consumers NF 430, NF 432, and NF 434 based on data collected from one or more data producers NF 420, NF 422, and NF 424. At least a portion of the collected data may be collected via the DCCF 440. Analysis output may be derived by the NWDAF 410, 412 using the trained ML model.

[0065] As part of the work for Release 18, one goal is to improve the accuracy of the analyses of the NWDAF. The accuracy of the analyses achieved using a trained ML model may deteriorate over time. One reason for this is the case of ML model drift. The ML model is trained using data collected from one or more network functions AF and OAM. The collected data may become inaccurate or invalid over time, resulting in a "drift" in the accuracy of the analyses performed using such collected data. For example, a cause of such drift may be that the network operator changes the resources of the network functions that are the source of the analysis. Due to such drift, the model may become unstable periodically and the predictions provided thereby may be accurate initially, but over time, i.e. after a period of time has passed, may become erroneous.

[0066] Solutions are disclosed in 3GPP TR 23.700-81 to allow NWDAF (AnLF or MTLF) to determine the accuracy of ML models or analytical predictions. Some solutions propose that MTLF determines the accuracy of ML models by comparing historical data with predictions and real data (where real is real-time data). Some solutions propose that AnLF determines the accuracy of analytical predictions by comparing predictions with real data. Some solutions propose that analytical consumers provide feedback information about the accuracy of analysis to AnLF or MTLF. One solution proposes that the analytical consumer provides an indication that the action taken will have a significant impact on the network (which will be a trigger for NWDAF to start monitoring the accuracy of analytical predictions). Another solution proposes to use the predicted output of the analysis ID to measure the impact of NF decisions. The impact can be calculated based on the change of relevant KPIs of NF after the decision is executed based on the predicted output of the analysis ID. In particular, the impact can be characterized by analysis ID level information (e.g., a real number between [-1,1]). If the level information is below a limit, a cooldown time associated with the use of the analysis ID is introduced to suspend erroneous and / or unstable decisions and allow time to refresh the output. Finally, some solutions propose that NWDAF rates data sources by evaluating data quality. If NWDAF determines that a data source does not provide accurate data, NWDAF may discard data from the data source.

[0067] PCT / EP2022 / 062706 (SMM920210250-WO-PCT) is an international patent application that describes a method in which the MTLF determines ML model drift by considering feedback information received from analytics consumers. The feedback information may include the affected NF. If the NF ID corresponds to a data producer NF, the MTLF uses this information to determine if there is ML model drift.

[0068] PCT / EP2022 / 075414 (SMM920220076-WO-PCT) is an international patent application that describes a method for rating data sources to ensure analysis correctness, and a solution for detecting and improving the correctness of NWDAF analysis by enabling rating of data sources (i.e., profiles / reputations). This rating can be based on (i) local estimation / calculation between predicted and real data, (ii) analysis of consumer feedback, or (iii) provided by the AF in the form of weights. The solution is particularly suitable for analysis that obtains input from a UE (via an AF) or from an AF that cannot be a trusted data source like the OAM and NF inside the wireless communication network.

[0069] Consumer feedback can be based on the performance of the analysis and can serve as a trigger for data source investigation and rating by NWDAF, which can collect supplemental data if possible. Data source investigation can change the confidence of subsequent predictions made using the same analysis.

[0070] A solution is provided herein that provides information about possible feedback that an analysis consumer may provide, and possible actions that the NWDAF AnLF and / or NWDAF MTLF may perform upon receipt of said feedback. Unlike other approaches, the approach presented herein is intended to identify data sources affected by erroneous or unstable AnLF analysis, and to limit their impact on future analysis outputs by controlling their impact on the AnLF and MTLF. Such control may be facilitated by rating the data sources used by erroneous or unstable AnLF analysis.

[0071] The data analysis function described herein (which may be the NWDAF) can be implemented by comparing historical data collected from the analytical data repository function "ADRF" or other data sources, real data, and comparing the results with those based on Figure 5 The accuracy of the analytical predictions is determined by comparing the analytical predictions with those of the architecture shown in . Figure 5 A wireless communication system 500 including a 5G core 520, an ADRF 530, an OAM 540, and an NWDAF 550 is illustrated. The 5G core 520 includes an access and mobility management function (AMF) 522, a session management function (SMF) 524, a user plane function (UPF) 526, and a network repository function (NRF) 528. In operation, the NWDAF 550 receives data from any one of the 5G core, the ADRF 530, and the OAM 540. The NWDAF 550 performs analysis on the received data. The analysis may include specific analysis operations.

[0072] The analytics consumer may provide the following information as analytics feedback: NF identification (NF ID) of the NF affected by the action, the region affected by the action, and / or identification of any management service producers (MnS producers) affected by the action.

[0073] If the consumer is a PCF, examples of NFs affected by the action may include the NF ID of the SMF, where the PCC rules including the updated QoS rules are provided. Alternatively, if the consumer is an SMF, the NFs affected by the action may include the NF ID of the UPF in which the traffic is routed (for example, when the SMF decides to relocate the UPF). If the consumer is an AF, the NFs affected by the action may include the DNAI in which the traffic of one or more UEs will be routed. If the consumer is an NSSF, the NFs affected by the action may include new slices instantiated. If the consumer is an NSSF or an AMF, the NFs affected by the action may include refresh decisions related to existing ongoing slices.

[0074] The affected area may be based on network topology, service area, and / or geographic area.

[0075] If the consumer is an MDA MnS producer analysis type (e.g., network slice load, or fault management, or service experience, etc.), the affected MnS producer ID may include an identifier of the MF, a slice identifier, etc. If the consumer is an orchestrator, such as a communication service management function (CSMF), and / or a network slice management function (NSMF), and / or a network slice subnet management function (NSSMF), or any other orchestrator not limited to supporting slice capabilities (e.g., edge computing), the affected MnS producer ID may include affected managed objects, i.e., managed entities, subnets, etc.

[0076] When NWDAF receives feedback information, NWDAF can take the following actions:

[0077] Start rating data sources by comparing data collected in the past (from ADRF) with new data.

[0078] When the data of these NFs are used for inference or model training, the historical data of the network functions / service areas indicated in the feedback information are discarded.

[0079] Correlate the performance metrics of the affected data sources with their performance impact on future analysis IDs, and introduce weights to control their impact on inference or model training.

[0080] Notify other AnLF analysis IDs operating in the same target region of data sources that are negatively impacted by a misprediction.

[0081] Estimate the duration associated with the impact of erroneous AnLF predictions.

[0082] • Estimate the amount of new data (ie, data volume) required to recover from the effects of an unreliable AnLF prediction, and / or the time the AnLF needs to collect new data from the affected data sources.

[0083] Notification of data sources negatively affected by incorrect predictions to other AnLF analysis IDs operating in the same target area can be: performed directly between AnLFs in the same target area with the assistance of an NRF that allows AnLFs to identify other AnLFs in the same area; in this way, since a certain AnLF does not know the exact data sources used by other AnLFs, it can notify everyone, i.e., a brute force approach; or processed via (multiple) corresponding MTLFs operating in the same target area with the assistance of an NRF that allows AnLFs to identify other AnLFs in the same area; using MTLFs can reduce signaling costs for certain deployment scenarios because the MTLFs know the (multiple) data sources used by a specific AnLF; or a component for introducing a pause or cooldown duration for a data source affected by incorrect predictions.

[0084] The estimated duration associated with the impact of an erroneous AnLF prediction may be: the duration associated with the prediction, for example, when the SMF selects a suboptimal UPF due to receiving erroneous information regarding the expected load; or the estimated time for the introduction of a domino effect (longer than the duration of the erroneous prediction) related to the performance degradation impact of the prediction on a network area or slice, and may be estimated when encountering too many involved network objects and their interrelationships.

[0085] Figure 6 The method 600 that can be performed by the data analysis function described in this article is illustrated. The data analysis function may include NWDAF. The data analysis function uses analysis feedback information to improve the accuracy of analysis prediction. The method 600 is performed between the consumer 630, NWDAF 650, rating function 652, ADRF 640, and data source 620. For example, the data source 620 may include 5G core network functions, OAM, and / or application functions. The functions of ADRF 640 described in this example can be performed by OAM instead.

[0086] At 671 , NWDAF 650 provides the requested analysis to consumer 630 .

[0087] At 672, the consumer 630 determines an action based on the received analysis. For example, if the analysis indicates that the load at the UPF function is high, the consumer (in this case, the SMF) can select a UPF with a lower load.

[0088] At 673, the consumer 630 sends a feedback notification to the NWDAF 650. The feedback notification may include the NF ID and / or the service area. The consumer 630 reports the action taken to the NWDAF 650. An example of an action is that the consumer 630 reports the selected UPF and the load of the selected UPF.

[0089] At 674 , NWDAF 650 determines all affected NFs based on the feedback information provided by analytics consumer 630 .

[0090] For example, if the consumer 630 includes the NF ID of the SMF, the NWDAF 650 determines the UPF served by the SMF by retrieving the information from the UDM (not shown). For another example, if the consumer 650 includes a UE list, the NWDAF 650 determines the NF serving the listed UEs.

[0091] The NWDAF 650 may determine whether the affected NF is used as a data source for model training or analytical reasoning by referring to one or more analysis IDs.

[0092] In a first alternative, at 675a, the NWDAF 650 may decide to discard the data when the NF determines in step 674 that the data is used as a data source in analytical reasoning or model training.

[0093] In a second alternative, at 675b, the NWDAF may decide to rank the data source by sending a request to the rating function 652. Based on the rating, a weight may be introduced for each affected source to limit its impact on future analysis outputs. The rating request may include the NF ID and / or service area. The rating function 652 may be collocated with the NWDAF 650.

[0094] At 676, the rating function 652 within the NWDAF 650 may retrieve historical data from the ADRF 640. At 677, the rating function 652 subscribes to real-time data from the data source 620. At 678, the rating function 652 rates the data source by comparing the historical data to the real-time data. The real-time data may include current data received from the data source. The real-time data from the data source may include data recently received from the data source.

[0095] At 679, information regarding the rating of the data source may be provided by the rating function 652 within the NWDAF 650. In one embodiment, the information may be an error indication indicating "data drift" from historical data. "Data drift" may include differences between trends predicted from historical data and real-time data.

[0096] The NWDAF 650 uses this information to determine whether the analysis accuracy is affected at 680. The NWDAF 650 may decide to retrain the ML model or adjust its analysis prediction accuracy based on the reported errors.

[0097] Figure 7 A process 700 is illustrated for notifying any other potentially affected AnLF(s) operating in the same region as a data analytics function. The process 700 occurs in a system that includes an analytics consumer 730, a data analytics function that includes an NWDAF AnLF 750, an NWDAF MTLF 760, an NRF 720, and at least one NWDAF AnLF in an affected region 726. The process 700 contemplates two different variants: (i) 'AnLF-based notification', where the NWDAFAnLF 750 directly notifies the other potentially affected AnLF(s) 726, and (ii) 'MTLF-based notification', where the NWDAF AnLF 750 notifies the corresponding NWDAF MTLF 760, which in turn notifies the selected NWDAF AnLF(s) 726 that use the same erroneous data source.

[0098] At 771 , NWDAF AnLF 750 provides the requested analysis to consumer 730 .

[0099] At 772, consumer 730 may, as described above (in conjunction with Figure 6 ) determines an action based on the analysis received.

[0100] At 773, consumer 730 is as described above (combined with Figure 6 ) Report the actions taken to NWDAF AnLF 750.

[0101] At 774, NWDAF AnLF 750 combines the above Figure 6 Steps 674 to 680 determine the accuracy of the analysis.

[0102] At 775, the NWDAF AnLF 750 optionally determines the affected geographic area taking into account the erroneous data source. If the AnLF cannot determine the affected geographic area, it may use the IP address of the erroneous data source(s) and send a request to the NRF 720.

[0103] In the first of the two alternatives, option (i) 'AnLF-based notification', the NWDAFAnLF 750 directly contacts other potentially affected AnLFs to notify them of the source of the erroneous data.

[0104] At 776a, the NWDAF AnLF 750 requests from the NRF 720 other NWDAF AnLF 726 operating in the geographic area containing the erroneous data source(s).

[0105] At 777a, the NWDAF AnLF 750 receives a list of potentially affected AnLF(s) operating in the indicated geographic area.

[0106] At 778a, the NWDAF AnLF 750 may then notify other NWDAF AnLFs 726 by sending a message listing the erroneous data sources (including associated ratings and / or suggested weights). The NWDAF AnLF 750 may identify other affected NWDAF AnLFs 726 based on the associated analysis ID (since the analysis ID has a predetermined list of input data sources). Otherwise, if this is not possible, the NWDAF AnLF 750 notifies all AnLFs in the wireless communication network in a brute force manner.

[0107] In the second of the two alternatives, option (ii) 'MTLF-based notification', the NWDAF AnLF 750 indirectly contacts other potentially affected NWDAF AnLFs 726 by notifying the corresponding NWDAF MTLF to notify them of the erroneous data source.

[0108] At 776b, NWDAF AnLF 750 sends a request to NRF 720 for the identification of corresponding MTLF(s) operating in the geographic area containing the erroneous data source.

[0109] At 777b, the NWDAF AnLF 750 receives a list of potentially affected MTLF(s) operating in the indicated geographic area.

[0110] At 778b1, the NWDAF AnLF 750 notifies the listed MTLF(s) by sending a message listing the erroneous data sources (including associated ratings and / or suggested weights). The NWDAF AnLF 750 can determine the affected MTLF(s) based on the associated analysis ID (since the analysis ID has a predetermined list of input data sources). Otherwise, if this is not possible, it notifies all MTLF(s) in the wireless communication network (which can be considered a brute force approach).

[0111] At 778b2, the NWDAF MTLF 760 notifies the affected NWDAF AnLF 726 by sending a message listing the erroneous data sources, including the indicated ratings and / or suggested weights. The NWDAF MTLF 760 may determine the affected NWDAF AnLF(s) 726 based on training provided considering the input data source list.

[0112] Following option (i) or (ii), at 779, NWDAF AnLF 750 may decide to discard data and / or apply weights to limit the impact of the indicated data sources.

[0113] In an alternative, when the NWDAF AnLF 750 receives the feedback information, the NWDAF AnLF 750 may indicate where the NWDAF AnLF 750 obtained the training model and the affected NF to the NWDAF MTLF 760. If the NWDAF MTLF 760 uses data to train the ML model from the affected NF, the NWDAF MTLF 760 uses the information to determine the accuracy of the ML model.

[0114] In another alternative, when the NWDAF AnLF 750 receives feedback information, and the NWDAF AnLF 750 has rated the data source according to the feedback information, the NWDAF AnLF 750 may indicate where the NWDAF AnLF 750 obtained the training model and the ratings of the NFs to the NWDAF MTLF 760. If the NWDAF MTLF 760 uses the data to train ML models from these NFs, the NWDAF MTLF 760 uses this information to determine the accuracy of the ML model.

[0115] Therefore, a data analysis function is provided, comprising a receiver and a processor. The receiver is arranged to: receive analysis feedback information about a particular analysis from an analysis consumer. The processor is arranged to: identify an affected network function based on the analysis feedback information; determine whether the affected network function is used as a data source for the particular analysis; rate the network function as a data source by comparing the prediction to the source data; and determine the accuracy of the particular analysis based on the rating of the network function used as a data source for the particular analysis.

[0116] Ranking network capabilities as defined herein allows the data analysis function to adapt to changes in the wireless communication network, thereby compensating for 'model drift' that can adversely affect analysis accuracy over time.

[0117] The analysis may include: model training, analytical reasoning and / or trained machine learning models. The source data may include data received from a source. The source data may include real data. The source data may include historical data and / or real-time data. The data analysis function may include NWDAF or AnLF NWDAF. Identifying the affected network function based on the analysis feedback information may include identifying the affected service area, and then identifying the affected network function as a network function serving the affected service area.

[0118] The analysis feedback information may include a network function identification, and the processor may be further arranged to: identify the affected network function as a network function having the network function identification, and / or a network function served by the network function having the network function identification.

[0119] The analyzed feedback information may include: an affected service area of ​​the wireless communication network.

[0120] The processor may also be arranged to identify the affected network function by sending a request for information about network functions served by and / or serving the affected network function. For example, the data analysis function may identify the affected network function by sending information of the affected network function to the UDM and receiving in response the network function serving the affected network function.

[0121] If, when analyzing the feedback information, the service area is included, the processor can identify the affected network function as a network function serving the indicated service area.

[0122] The processor may identify the affected network functions by sending a request for information related to network functions served by and / or serving the indicated service area.

[0123] The processor may also be arranged to rank the network function as a data source by comparing the prediction derived from the particular analysis with the source data. The processor may rank the network function as a data source by comparing the historical data with the source data. The source data may include real-time data.

[0124] The processor may also be arranged to discard data received from the affected network functions to perform analytical reasoning or train a machine learning model based on the ratings of the affected network functions.

[0125] The processor may be further arranged to provide a rating of the network function as a data source. The rating may be provided as an error indication. The error indication may include a reported error. The error indication may indicate data drift. The data drift may include deviation from a trend set by historical data.

[0126] The specific analysis may include a machine learning model, and the processor may also be arranged to: determine that a network function used as a data source for training the machine learning model is affected based on a rating of the affected network function, and the processor may also be arranged to: retrain the machine learning model by collecting new data from the affected network function.

[0127] The specific analysis may include a machine learning model, and the processor may be further arranged to determine that the network function used as a data source for the machine learning model or reasoning is affected based on the rating of the network function, and the processor may be further arranged to: adjust the accuracy of the analysis prediction based on the reported rating. The NWDAF may be an AnLF that uses a machine learning model and data from a data source to provide analytical reasoning, or an MTLF that trains a machine learning model based on data from a data source.

[0128] The rating may include: an error percentage indicating the degree of data distribution drift. The rating may include: an error percentage indicating the degree of data distribution drift.

[0129] The data analysis function may further comprise a transmitter arranged to indicate the rated data sources to other data analysis functions.

[0130] The data analysis function may further comprise a transmitter. The transmitter may be arranged to send a request to the network repository function for identification and rating information of other data analysis functions operating in a geographic area containing the affected data source. The receiver may further be arranged to receive a list of potentially affected data analysis functions. The transmitter may further be arranged to send a message to the potentially affected data analysis function indicating the affected network functions and including the ratings of the affected network functions.

[0131] Alternatively, the data analysis function may send a message to all other data analysis functions in the wireless communication network indicating the affected network functions and including the ratings of the affected network functions.

[0132] Other data analysis functions operating in the geographic region may include: NWDAF analysis logic function (ANFF), or NWDAF model training logic function (MTLF). In the case where the data analysis function sends the message to the NWDAF model training logic function (MTLF), the NWDAF model training logic function may forward the message to the associated NWDAF analysis logic function (ANFF).

[0133] Figure 8A method 800 in a data analysis function is illustrated. The method 800 includes receiving 810 analysis feedback information about a particular analysis, the analysis feedback information being received from an analysis consumer; and identifying 820 an affected network function based on the analysis feedback information. The method 800 also includes determining 830 whether the affected network function is used as a data source for the particular analysis; rating 840 the network function as a data source by comparing the prediction to the source data; and determining 850 the accuracy of the particular analysis based on the rating of the network function used as a data source for the particular analysis.

[0134] In some embodiments, the method 800 may be performed by a processor that executes program code, such as a microcontroller, a microprocessor, a CPU, a GPU, an auxiliary processing unit, an FPGA, etc.

[0135] Ranking network capabilities as defined herein allows the data analysis function to adapt to changes in the wireless communication network, thereby compensating for 'model drift' that can adversely affect analysis accuracy over time.

[0136] The analysis may include: model training, analytical reasoning and / or trained machine learning models. The source data may include data received from a source. The source data may include real data. The source data may include historical data and / or real-time data. The data analysis function may include NWDAF or AnLF NWDAF. Identifying the affected network function based on the analysis feedback information may include identifying the affected service area, and then identifying the affected network function as a network function serving the affected service area.

[0137] The analysis feedback information may include a network function identifier, and the method may further include: identifying the affected network function as a network function having the network function identifier and / or a network function served by a network function having the network function identifier. The analysis feedback information may include: an affected service area of ​​the wireless communication network.

[0138] Identifying the affected network function may include sending a request for information about network functions served by and / or serving the affected network function. For example, the data analysis function may identify the affected network function by sending information about the affected network function to the UDM and receiving in response the network function serving the affected network function.

[0139] The analyzing feedback information may include a service area, and the method further comprises identifying the affected network function as a network function serving the indicated service area.

[0140] Identifying the affected network functions may include sending a request for information related to network functions served by and / or serving the indicated service area.

[0141] The method may also include: rating the network function as a data source by comparing the prediction derived from the specific analysis with the source data. Rating the network function as a data source may include comparing historical data with the source data. The source data may include real-time data. The method may also include: discarding data received from the affected network function to perform analytical reasoning or train a machine learning model based on the rating of the affected network function.

[0142] The method may also include providing a rating of the network function as a data source. The rating may be provided as an error indication. The error indication may include a reported error. The error indication may indicate data drift. Data drift may include: deviation from a trend set by historical data.

[0143] The specific analysis may include a machine learning model, and the method may also include: determining that a network function used as a data source for training the machine learning model is affected based on the ratings of the affected network functions, and retraining the machine learning model by collecting new data from the affected network functions.

[0144] The specific analysis may include a machine learning model, and the method may also include: determining that the network function used as a data source for the machine learning model or reasoning is affected based on the rating of the network function, and adjusting the analysis prediction accuracy based on the reported rating. The NWDAF can be an AnLF that uses a machine learning model and data from a data source to provide analytical reasoning, or it can be an MTLF that trains a machine learning model based on data from a data source.

[0145] The rating may include: an error percentage indicating the degree of data distribution drift. The rating may include: an error percentage indicating the degree of data distribution drift.

[0146] The method may also include indicating the rated data sources to other data analysis functions.

[0147] The method may also include: sending a request to the network repository function for identification and rating information of other data analysis functions operating in the geographic area containing the affected data source; receiving a list of potentially affected data analysis functions; and sending a message to the potentially affected data analysis functions, the message indicating the affected network functions and including the ratings of the affected network functions.

[0148] Alternatively, the data analysis function may send a message to all other data analysis functions in the wireless communication network indicating the affected network functions and including the ratings of the affected network functions.

[0149] Other data analysis functions operating in the geographic region may include: NWDAF analysis logic function (ANFF), or NWDAF model training logic function (MTLF). In the case where the data analysis function sends the message to the NWDAF model training logic function (MTLF), the NWDAF model training logic function may forward the message to the associated NWDAF analysis logic function (ANFF).

[0150] As part of the work for Release 18, one goal was to improve the accuracy of the analyses performed by NWDAF. One problem with existing analyses is that the accuracy of analyses performed using trained ML models can degrade over time. One reason for this is the case of ML model drift. ML models are trained using data collected from one or more network functions. The collected data becomes inaccurate or invalid over time (for example, when a network operator changes the resources of a network function), which causes the accuracy of analyses performed using such models to "drift." Due to this drift, the models continue to become unstable and the predictions continue to be wrong over time.

[0151] A solution is provided herein, in which NWDAF determines whether the actions taken by the analytics consumer affect the NF used as a data source for analytics generation or model training. NWDAF uses the feedback provided by the analytics consumer to determine the affected NF. NWDAF then starts rating the NF to determine whether its data distribution has changed that may affect the accuracy of the ML model and / or analytics. If NWDAF determines that the NF is affected, NWDAF can introduce weights to control its impact on inference or model training.

[0152] A solution is disclosed in 3GPP TR 23.700-81 to allow the NWDAF (AnLF or MTLF) to determine the accuracy of the ML model or analytical prediction.

[0153] Some solutions propose that MTLF determines the accuracy of the ML model by comparing historical data with forecasts and true data (where true is actually real-time data).

[0154] Some solutions propose that AnLF determines the accuracy of analytical predictions by comparing the predictions with real data.

[0155] Some solutions propose that the analytics consumer provides feedback to the AnLF or MTLF about the accuracy of the analytics. Some solutions propose (the solution proposed by Lenovo) that the analytics consumer provides an indication that the action taken will have a significant impact on the network (which would be a trigger for the NWDAF to start monitoring the accuracy of the analytics predictions).

[0156] Finally, some solutions propose that NWDAF rates data sources by assessing data quality. If NWDAF determines that a data source does not provide accurate data, NWDAF can discard data from the data source.

[0157] According to the present disclosure, NWDAF can use analysis feedback information from consumers to trigger data source rating. NWDAF can use feedback information to trigger data source rating. In addition, AnLF can notify other AnLF / MTLF of affected NFs based on consumer feedback.

[0158] Therefore, a NWDAF is provided that receives analysis feedback information from analysis consumers and: based on the analysis feedback information, determines an affected NF; determines whether the affected NF is used as a data source for model training or analysis reasoning; rates the data source by comparing historical data with real data; and based on the rating of the data source, determines the accuracy of the analysis / trained ML model.

[0159] The feedback information may include the affected service area of ​​the network.

[0160] Feedback information may include affected network functionality from the actions taken.

[0161] The NWDAF may determine the affected NF by sending information of the affected NF to the NF (UDM), and receive a NF serving the affected NF in response.

[0162] The NWDAF may indicate the rated data sources to other analysis NFs.

[0163] It should be noted that the above methods and apparatus illustrate rather than limit the present invention, and that those skilled in the art will be able to design many alternative arrangements without departing from the scope of the appended claims. The word "comprising" does not exclude the presence of other elements or steps than those listed in the claims, "a" or "an" does not exclude a plurality, and a single processor or other unit may fulfil the functions of several units described in the claims. Any reference signs in the claims should not be construed as limiting their scope.

[0164] Furthermore, although examples have been given in the context of specific communication standards, these examples are not intended to limit the communication standards to which the disclosed methods and apparatus may be applied. For example, although specific examples have been given in the context of 3GPP, the principles disclosed herein may also be applied to another wireless communication system, in fact any communication system that uses routing rules.

[0165] The method may also be embodied in a set of instructions stored on a computer-readable medium, which when loaded into a computer processor, a digital signal processor (DSP), etc., causes the processor to perform the above-described method.

[0166] The described methods and apparatus may be practiced in other specific forms. The described methods and apparatus are considered in all respects to be illustrative only and not restrictive. Therefore, the scope of the present invention is indicated by the appended claims rather than by the foregoing description. All changes within the meaning and equivalent range of the claims should be included within their scope.

[0167] The following abbreviations are relevant to the areas covered by this document: CSMF, communication service management function; machine learning; NF, network function; NWDAF, network data analysis function; NRF, network repository function; NSMF, network slice management function; NSSMF, network slice subnet management function; OAM, operation and maintenance; UE, user equipment; MTLF, model training logic function; AnLF, analytical reasoning logic function.

Claims

1. A data analysis function, comprising: a receiver arranged to receive analysis feedback information from an analysis consumer regarding a particular analysis; The processor is arranged as: Based on the analysis feedback information, identifying affected network functions; determining whether the affected network function is used as a data source for the particular analysis; rating said network functionality as a data source by comparing the predictions to the source data; as well as Based on the ratings of the network functions used as data sources for the particular analysis, the accuracy of the particular analysis is determined.

2. The data analysis function according to claim 1, wherein the analysis feedback information includes a network function identifier, and the processor is further arranged to: identify the affected network function as the network function having the network function identifier, and / or the network function served by the network function having the network function identifier.

3. A data analysis function according to claim 1 or 2, wherein the processor identifies the affected network function by sending a request for information related to network functions served by and / or serving the affected network function.

4. The data analysis function of claim 1, wherein when the analysis feedback information includes a service area, the processor is capable of identifying the affected network function as the network function serving the indicated service area.

5. A data analysis function according to claim 1 or 4, wherein the processor identifies the affected network functions by sending a request for information related to network functions served by and / or serving the indicated service area.

6. A data analysis function according to any preceding claim, wherein the processor is further arranged to: rank the network function as a data source by comparing predictions derived from the specific analysis with source data.

7. A data analysis function according to any of the preceding claims, wherein the processor is also arranged to: discard data received from the affected network function to perform analytical reasoning or train a machine learning model based on the rating of the affected network function.

8. A data analysis function according to any one of the preceding claims, wherein the processor is further arranged to provide a rating of the network function as a data source.

9. A data analysis function according to any one of the preceding claims, wherein the specific analysis includes a machine learning model, and the processor is further arranged to: determine that a network function used as a data source for training the machine learning model is affected based on the rating of the affected network function, and the processor is further arranged to: retrain the machine learning model by collecting new data from the affected network function.

10. A data analysis function according to any one of claims 1 to 8, wherein the specific analysis includes a machine learning model, and the processor is further arranged to: determine, based on the rating of the network function, whether the network function used as a data source for the machine learning model or inference is affected, and the processor is further arranged to: adjust the analysis prediction accuracy based on the reported rating.

11. The data analysis function of any preceding claim, wherein the rating comprises: The error percentage that indicates a shift in the data distribution.

12. A data analysis function according to any preceding claim, further comprising a transmitter arranged to indicate the rated data sources to other data analysis functions.

13. The data analysis function according to any one of the preceding claims, further comprising: a sender arranged to: send a request to the network repository function for identification and rating information of other data analysis functions operating in a geographic area containing said affected data source; The receiver is further arranged to: receive a list of potentially affected data analysis functions; The transmitter is further arranged to send a message to the potentially affected data analysis function, the message indicating the affected network function and comprising a rating of the affected network function.

14. A method in a data analysis function, the method comprising: receiving analysis feedback information regarding a particular analysis, the analysis feedback information being received from an analysis consumer; Based on the analysis feedback information, identifying affected network functions; determining whether the affected network function is used as a data source for the particular analysis; rating said network functionality as a data source by comparing the predictions to the source data; as well as Based on the ratings of the network functions used as data sources for the particular analysis, the accuracy of the particular analysis is determined.

15. The method according to claim 14, wherein when the analysis feedback information includes a network function identifier, the affected network function is identified as the network function having the network function identifier and / or the network function served by the network function having the network function identifier.

16. The method of claim 14 or 15, wherein identifying the affected network functions comprises: A request is sent for information related to network functions served by and / or serving the affected network function.

17. The method according to claim 14, wherein the analysis feedback information includes a service area, and the method further comprises: An affected network function is identified as the network function serving the indicated service area.

18. The method of claim 14 or 17, wherein identifying the affected network functions comprises: A request is sent for information related to network functions served by and / or serving the indicated service area.

19. The method according to any one of claims 14 to 18, further comprising: The network functionality is ranked as a data source by comparing the predictions derived from the particular analysis to the source data.

Images