Methods and apparatus relating to analytics in wireless communication network
By comparing training data with real-time data, determining and retraining the ML model, the problem of NWDAF analysis accuracy deteriorating over time is solved, the stability and accuracy of analysis predictions are improved, and the network changes are adapted.
Patent Information
- Application Number
- CN202280093843.3
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Priority Date
- 2022-03-24
- Filing Date
- 2022-05-10
- Publication Date
- 2024-11-08
AI Technical Summary
In 3GPP wireless communication networks, the analysis accuracy provided by NWDAF deteriorates over time, causing ML models to drift, affecting the accuracy of analysis predictions.
Ensure the validity and accuracy of the data by receiving feedback from the analytical consumer side, comparing training data with real-time data to determine the accuracy of the ML model, and retraining the ML model when the model accuracy is below the threshold.
It improves the analysis accuracy provided by NWDAF, prevents ML model drift, maintains the stability and accuracy of analysis prediction, and adapts to network changes.
Smart Images

Figure CN118923092A_ABST
Abstract
Description
Technical Field
[0001] The subject matter disclosed herein relates generally to analytics in wireless communication networks. Background Art
[0002] In 3GPP wireless communication networks, analytics may be provided by a Network Data Analysis Function (NWDAF). The NWDAF derives analytics using data / events collected from one or more network entities and provides the derived analytics to one or more analytics consumers.
[0003] NWDAF can derive analytics by using machine learning (ML) models on the collected data / events. Summary of the invention
[0004] There is a need to improve the accuracy of the analysis provided by NWDAF.
[0005] Disclosed herein is a procedure for improving the accuracy of the analysis provided by NWDAF.
[0006] A method is provided, comprising: determining the accuracy of an ML model that needs to be identified based on feedback received from an analysis consumer, the ML model being used to derive an analysis for an analysis ID; receiving a first data set from a first network function, wherein the first data set was collected in the past and is used to train the ML model; receiving a second data set corresponding to the analysis ID from a second network function, wherein the second data set is real-time data; and determining the accuracy of the ML model by comparing the first data set and the second data set.
[0007] Provided is an apparatus comprising a processor configured to: determine, based on feedback received from an analytics consumer, the accuracy of a need-identified ML model used to derive analytics for an analytics ID; receive a first data set from a first network function, wherein the first data set was collected in the past and is used to train the ML model; receive a second data set corresponding to the analytics ID from a second network function, wherein the second data set is real-time data; and determine the accuracy of the ML model by comparing the first data set with the second data set.
[0008] A method is provided, comprising: receiving a plurality of machine learning (ML) models for deriving an analysis for an analysis ID; receiving a request for an analysis for the analysis ID; determining a plurality of analysis outputs for the requested analysis using the plurality of ML models associated with the requested analysis; and determining an accuracy of each of the plurality of ML models based on the plurality of analysis outputs. BRIEF DESCRIPTION OF THE DRAWINGS
[0009] In order to describe the manner in which the advantages and features of the present disclosure can be obtained, the description of the present disclosure is presented by reference to certain devices and methods illustrated in the accompanying drawings. Each of these drawings depicts only certain aspects of the present disclosure and should not be considered as limiting the scope thereof. The drawings may have been simplified for clarity and are not necessarily drawn to scale.
[0010] A method and apparatus for improving analytical accuracy will now be described, by way of example only, with reference to the accompanying drawings, in which:
[0011] Figure 1 An example wireless communication system is illustrated;
[0012] Figure 2 A user equipment device is illustrated;
[0013] Figure 3 The network nodes are illustrated;
[0014] Figure 4 Methods used to improve analytical accuracy are illustrated;
[0015] Figure 5 Another method for improving the accuracy of analysis is illustrated;
[0016] Figure 6 A wireless communication system for improving analysis accuracy is illustrated;
[0017] Figure 7 Pictured Figure 6 communications between entities within a wireless communications system;
[0018] Figure 8 Another wireless communication system for improving analysis accuracy is illustrated; and
[0019] Fig. 9 Communication between entities according to yet another embodiment is illustrated. DETAILED DESCRIPTION
[0020] As will be appreciated by those skilled in the art, various aspects of the present disclosure may be implemented as a system, device, method or program product. Therefore, the arrangements described herein may be implemented in a complete hardware form, a complete software form (including firmware, resident software, microcode, etc.), or a combination of software and hardware aspects.
[0021] For example, the disclosed methods and apparatus may be implemented as hardware circuits including custom very large scale integration ("VLSI") circuits or gate arrays, off-the-shelf semiconductors such as logic chips, transistors or other discrete components. The disclosed methods and apparatus may also be implemented in programmable hardware devices such as field programmable gate arrays, programmable array logic, programmable logic devices, etc. As another example, the disclosed methods and apparatus may include one or more physical or logical blocks of executable code, which may be organized, for example, as an object, procedure, or function.
[0022] In addition, the method and apparatus may take the form of a program product implemented in one or more computer-readable storage devices that store machine-readable code, computer-readable code, and / or program code (hereinafter referred to as code). The storage device may be tangible, non-transient, and / or non-transmitting. The storage device may not implement signals. In some arrangements, the storage device only uses signals to access the code.
[0023] Any combination of one or more computer readable media may be utilized. The computer readable medium may be a computer readable storage medium. The computer readable storage medium may be a storage device storing code. For example, the storage device may be, but is not limited to, an electronic, magnetic, optical, electromagnetic, infrared, holographic, micromechanical or semiconductor system, apparatus or device, or any suitable combination of the foregoing.
[0024] More specific examples of storage devices (a non-exhaustive list) would include the following: an electrical connection having one or more wires, a portable computer diskette, a hard disk, a random access memory ("RAM"), a read-only memory ("ROM"), an erasable programmable read-only memory ("EPROM" or flash memory), a portable compact disk read-only memory ("CD-ROM"), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing. In the context of this document, a computer-readable storage medium may be any tangible medium that can contain or store a program for use by or in conjunction with an instruction execution system, apparatus, or device.
[0025] References to examples of specific methods or devices throughout the specification or similar language mean that the specific features, structures, or characteristics described in conjunction with the examples are included in at least one implementation of the methods and devices described herein. Therefore, unless otherwise expressly specified, references to features of examples of specific methods or devices or similar language may, but do not necessarily, all refer to the same examples, but rather represent "one or more but not all examples." The terms "including," "comprising," "having," and variations thereof mean "including but not limited to," unless otherwise expressly specified. An enumerated list of items does not mean that any or all items are mutually exclusive, unless otherwise expressly specified. The terms "a," "an," and "the" also refer to "one or more," unless otherwise expressly specified.
[0026] As used herein, a list with the conjunction "and / or" includes any single item in the list or a combination of items in the list. For example, a list of A, B, and / or C includes only A, only B, only C, a combination of A and B, a combination of B and C, a combination of A and C, or a combination of A, B, and C. As used herein, a list using the term "one or more of..." includes any single item in the list or a combination of items in the list. For example, one or more of A, B, and C includes only A, only B, only C, a combination of A and B, a combination of B and C, a combination of A and C, or a combination of A, B, and C. As used herein, a list using the term "one of..." includes only one of any single item in the list. For example, "one of A, B, and C" includes only A, only B, or only C, and does not include a combination of A, B, and C. As used herein, "a member selected from the group consisting of A, B, and C" includes only one of A, B, or C, and does not include a combination of A, B, and C. As used herein, “a member selected from the group consisting of A, B, and C, and combinations thereof” includes only A, only B, only C, a combination of A and B, a combination of B and C, a combination of A and C, or a combination of A, B, and C.
[0027] In addition, the described features, structures or characteristics described herein may be combined in any suitable manner. In the following description, many specific details are provided, such as examples of programming, software modules, user selections, network transactions, database queries, database structures, hardware modules, hardware circuits, hardware chips, etc., to provide a comprehensive understanding of the present disclosure. However, those skilled in the relevant art will recognize that the disclosed methods and apparatus may be practiced without one or more of the specific details or using other methods, components, materials, etc. In other instances, well-known structures, materials, or operations are not shown or described in detail to avoid confusing various aspects of the present disclosure.
[0028] Various aspects of the disclosed method and apparatus are described below with reference to schematic flow charts and / or schematic block diagrams of the method, apparatus, system, and program product. It is to be understood that each frame of the schematic flow chart and / or schematic block diagram and the combination of frames in the schematic flow chart and / or schematic block diagram can be implemented by code. The code can be provided to a processor of a general-purpose computer, a special-purpose computer, or other programmable data processing device to produce a machine, so that instructions executed by the processor of the computer or other programmable data processing device create components for implementing the functions / actions specified in the schematic flow chart and / or schematic block diagram.
[0029] The code may also be stored in a storage device that can instruct a computer, other programmable data processing apparatus, or other device to operate in a specific manner, so that the instructions stored in the storage device produce an article of manufacture that includes instructions for implementing the functions / actions specified in the schematic flowchart and / or schematic block diagram.
[0030] The code may also be loaded onto a computer, other programmable data processing apparatus, or other device so that a series of operating steps are executed on the computer, other programmable apparatus, or other device to produce a computer-implemented process, so that the code executed on the computer or other programmable apparatus provides a process for implementing the functions / actions specified in the schematic flowchart and / or schematic block diagram.
[0031] The schematic flow charts and / or schematic block diagrams in the accompanying drawings illustrate the architecture, functionality and operation of possible implementations of devices, systems, methods and program products. In this regard, each box in the schematic flow charts and / or schematic block diagrams can represent a module, segment or code portion, which includes one or more executable instructions of a code for implementing (multiple) specified logical functions.
[0032] It should also be noted that in some alternative implementations, the functions mentioned in the blocks may not occur in the order mentioned in the drawings. For example, two blocks shown in succession may, in fact, be executed substantially concurrently, or the blocks may sometimes be executed in reverse order, depending on the functionality involved. Other steps and methods may be envisioned that are equivalent in function, logic, or effect to one or more blocks, or portions thereof, of the illustrated drawings.
[0033] The description of the elements in each figure may refer to the elements of the process diagram. Like numbers refer to like elements in all figures.
[0034] Currently, when an analysis consumer requests analysis from the NWDAF, the analysis consumer can include a target area and / or a target user equipment (UE) or group of (multiple) UEs in the request. The NWDAF derives the analysis by collecting data / events from one or more network functions (NFs), ensuring that the data comes from the requested target area or involves (multiple) target UEs. The analysis is derived based on the static location of the UE taken into account by the Access and Mobility Function (AMF), which determines the UE's entry or exit into a specific area of interest at the granularity of (multiple) tracking areas or (multiple) cell IDs.
[0035] Figure 1 An example wireless communication system 100 is illustrated. The system 100 includes multiple NWDAFs 110, multiple data producer NFs 120, multiple analysis consumer NFs 130, and a data collection coordination function (DCCF). In the current Release 16 and Release 17 3GPP architecture, the NWDAF 110 (defined in 3GPP TS 23.288 v17.2.0) provides analysis output to one or more analysis consumer NFs 130 based on data collected from one or more data producer NFs 120. At least a portion of the collected data may be collected via the DCCF 140. The analysis output is derived by the NWDAF 110 using a trained ML model.
[0036] As part of the Release 18 work, one goal is to improve the analytical accuracy of the analytics provided by the NWDAF 110. One problem is that the accuracy of analytics using a trained ML model may deteriorate over time. One reason for this scenario is the case of ML model drift. The ML model is trained using data collected from one or more NFs. Over time, the collected data becomes inaccurate or invalid (for example when the network operator changes the resources of the NF), which causes a "drift" in the accuracy of the analytics provided by the ML model. Due to this drift, the ML model remains unstable and the predictions made by the ML model become wrong over time.
[0037] This application proposes a solution to this problem by providing a way for NWDAF to determine the accuracy of the ML model and decide actions accordingly.
[0038] Figure 2 A user equipment device 200 is depicted which may be used to implement the methods described herein. The user equipment device 200 is used to implement one or more of the solutions described above. The user equipment device 200 comprises a processor 205, a memory 210, an input device 215, an output device 220 and a transceiver 225.
[0039] The input device 215 and the output device 220 may be combined into a single device, such as a touch screen. In some implementations, the user equipment apparatus 200 does not include any input device 215 and / or output device 220. The user equipment apparatus 200 may include one or more of the following: a processor 205, a memory 210, and a transceiver 225, and may not include an input device 215 and / or an output device 220.
[0040] As depicted, the transceiver 225 includes at least one transmitter 230 and at least one receiver 235. The transceiver 225 can communicate with one or more cells (or wireless coverage areas) supported by one or more base units. The transceiver 225 can be operable on an unlicensed spectrum. Moreover, the transceiver 225 can include multiple UE panels supporting one or more beams. Additionally, the transceiver 225 can support at least one network interface 240 and / or application interface 245. (Multiple) application interface 245 can support one or more APIs. (Multiple) network interface 240 can support 3GPP reference points such as Uu, N1, PC5, etc. Other network interfaces 240 can be supported, as understood by those of ordinary skill in the art.
[0041] The processor 205 may include any known controller capable of executing computer-readable instructions and / or capable of performing logical operations. For example, the processor 205 may be a microcontroller, a microprocessor, a central processing unit (“CPU”), a graphics processing unit (“GPU”), an auxiliary processing unit, a field programmable gate array (“FPGA”), or a similar programmable controller. The processor 205 may execute instructions stored in the memory 210 to perform the methods and routines described herein. The processor 205 is communicatively coupled to the memory 210, the input device 215, the output device 220, and the transceiver 225.
[0042] Processor 205 may control user equipment device 200 to implement the above UE behaviors. Processor 205 may include an application processor (also referred to as a "main processor") that manages application domain and operating system ("OS") functions and a baseband processor (also referred to as a "baseband radio processor") that manages radio functions.
[0043] Memory 210 may be a computer-readable storage medium. Memory 210 may include volatile computer storage media. For example, memory 210 may include RAM, including dynamic RAM ("DRAM"), synchronous dynamic RAM ("SDRAM"), and / or static RAM ("SRAM"). Memory 210 may include non-volatile computer storage media. For example, memory 210 may include a hard drive, flash memory, or any other suitable non-volatile computer storage device. Memory 210 may include volatile and non-volatile computer storage media.
[0044] The memory 210 may store data related to implementing the above-mentioned service class fields. The memory 210 may also store program codes and related data, such as an operating system or other controller algorithms operating on the device 200.
[0045] Input device 215 may include any known computer input device, including a touchpad, buttons, keyboard, stylus, microphone, etc. Input device 215 may be integrated with output device 220, such as a touch screen or similar touch-sensitive display. Input device 215 may include a touch screen so that text can be entered using a virtual keyboard displayed on the touch screen and / or by handwriting on the touch screen. Input device 215 may include two or more different devices, such as a keyboard and a touchpad.
[0046] The output device 220 may be designed to output visual signals, audible signals, and / or tactile signals. The output device 220 may include an electronically controllable display or display device capable of outputting visual data to a user. For example, the output device 220 may include, but is not limited to, a liquid crystal display ("LCD"), a light emitting diode ("LED") display, an organic LED ("OLED") display, a projector, or a similar display device capable of outputting images, text, etc. to a user. As another non-limiting example, the output device 220 may include a wearable display that is separate from the rest of the user device apparatus 200 but communicatively coupled to the rest, such as a smart watch, smart glasses, a head-up display, etc. Further, the output device 220 may be a component of a smart phone, a personal digital assistant, a television, a desktop computer, a notebook (laptop) computer, a personal computer, a vehicle dashboard, etc.
[0047] Output device 220 may include one or more speakers for producing sound. For example, output device 220 may produce an audible alarm or notification (e.g., a beep or a chime). Output device 220 may include one or more tactile devices for producing vibration, motion, or other tactile feedback. All or part of output device 220 may be integrated with input device 215. For example, input device 215 and output device 220 may form a touch screen or similar touch-sensitive display. Output device 220 may be located near input device 215.
[0048] The transceiver 225 communicates with one or more network functions of the mobile communication network via one or more access networks. The transceiver 225 operates under the control of the processor 205 to send messages, data and other signals, and also receive messages, data and other signals. For example, the processor 205 can selectively activate the transceiver 225 (or part thereof) at a specific time to send and receive messages.
[0049] The transceiver 225 includes at least one transmitter 230 and at least one receiver 235. One or more transmitters 230 can be used to provide UL communication signals to the base unit of the wireless communication network. Similarly, one or more receivers 235 can be used to receive DL communication signals from the base unit. Although only one transmitter 230 and one receiver 235 are illustrated, the user equipment device 200 can have any suitable number of transmitters 230 and receivers 235. Further, (multiple) transmitters 230 and (multiple) receivers 235 can be any suitable type of transmitter and receiver. The transceiver 225 may include a first transmitter / receiver pair for communicating with a mobile communication network via a licensed radio spectrum and a second transmitter / receiver pair for communicating with a mobile communication network via an unlicensed radio spectrum.
[0050] A first transmitter / receiver pair that can be used to communicate with a mobile communication network via a licensed radio spectrum and a second transmitter / receiver pair that can be used to communicate with a mobile communication network via an unlicensed radio spectrum can be combined into a single transceiver unit, such as a single chip that performs functions for use with licensed and unlicensed radio spectrums. The first transmitter / receiver pair and the second transmitter / receiver pair can share one or more hardware components. For example, some of the transceivers 225, transmitters 230, and receivers 235 can be implemented as physically separated components that access shared hardware resources and / or software resources (such as, for example, a network interface 240).
[0051] One or more transmitters 230 and / or one or more receivers 235 may be implemented and / or integrated into a single hardware component, such as a multi-transceiver chip, a system on a chip, an application specific integrated circuit ("ASIC"), or other type of hardware component. One or more transmitters 230 and / or one or more receivers 235 may be implemented and / or integrated into a multi-chip module. Other components such as a network interface 240 or other hardware components / circuits may be integrated into a single chip with any number of transmitters 230 and / or receivers 235. Transmitters 230 and receivers 235 may be logically configured as transceivers 225 using one or more common control signals, or configured as modular transmitters 230 and receivers 235 implemented in the same hardware chip or multi-chip module.
[0052] Figure 3 Further details of a network node 300 that can be used to implement the methods described herein are depicted. The network node 300 may be one implementation of an entity in a wireless communication network. The network node 300 includes a processor 305, a memory 310, an input device 315, an output device 320, and a transceiver 325.
[0053] The input device 315 and the output device 320 may be combined into a single device, such as a touch screen. In some implementations, the network node 300 does not include any input device 315 and / or output device 320. The network node 300 may include one or more of the following: a processor 305, a memory 310, and a transceiver 325, and may not include an input device 315 and / or an output device 320.
[0054] As depicted, the transceiver 325 includes at least one transmitter 330 and at least one receiver 335. Here, the transceiver 325 communicates with one or more remote units 200. Additionally, the transceiver 325 can support at least one network interface 340 and / or an application interface 345. The application interface(s) 345 can support one or more APIs. The network interface(s) 340 can support 3GPP reference points such as Uu, N1, N2, and N3. Other network interfaces 340 can be supported, as understood by those of ordinary skill in the art.
[0055] The processor 305 may include any known controller capable of executing computer-readable instructions and / or capable of performing logical operations. For example, the processor 305 may be a microcontroller, a microprocessor, a CPU, a GPU, an auxiliary processing unit, an FPGA, or a similar programmable controller. The processor 305 may execute instructions stored in the memory 310 to perform the methods and routines described herein. The processor 305 is communicatively coupled to the memory 310, the input device 315, the output device 320, and the transceiver 325.
[0056] Memory 310 may be a computer-readable storage medium. Memory 310 may include volatile computer storage media. For example, memory 310 may include RAM, including dynamic RAM ("DRAM"), synchronous dynamic RAM ("SDRAM"), and / or static RAM ("SRAM"). Memory 310 may include non-volatile computer storage media. For example, memory 310 may include a hard drive, flash memory, or any other suitable non-volatile computer storage device. Memory 310 may include volatile and non-volatile computer storage media.
[0057] The memory 310 may store data related to establishing a multipath unicast link and / or mobile operation. For example, the memory 310 may store the above parameters, configurations, resource assignments, policies, etc. The memory 310 may also store program codes and related data, such as an operating system or other controller algorithms operating on the network node 300.
[0058] Input device 315 may include any known computer input device, including a touch pad, buttons, keyboard, stylus, microphone, etc. Input device 315 may be integrated with output device 320, such as a touch screen or similar touch-sensitive display. Input device 315 may include a touch screen so that text can be entered using a virtual keyboard displayed on the touch screen and / or by handwriting on the touch screen. Input device 315 may include two or more different devices, such as a keyboard and a touch pad.
[0059] Output device 320 can be designed to output visual signals, audible signals and / or tactile signals. Output device 320 may include an electronically controllable display or display device capable of outputting visual data to a user. For example, output device 320 may include, but is not limited to, an LCD display, an LED display, an OLED display, a projector, or a similar display device capable of outputting images, text, etc. to a user. As another non-limiting example, output device 320 may include a wearable display that is separate from the rest of network node 300 but communicatively coupled to the rest, such as a smart watch, smart glasses, a head-up display, etc. Further, output device 320 may be a component of a smart phone, a personal digital assistant, a television, a desktop computer, a notebook (laptop) computer, a personal computer, a vehicle dashboard, etc.
[0060] Output device 320 may include one or more speakers for producing sound. For example, output device 320 may produce an audible alarm or notification (e.g., a beep or a chime). Output device 320 may include one or more tactile devices for producing vibration, motion, or other tactile feedback. All or part of output device 320 may be integrated with input device 315. For example, input device 315 and output device 320 may form a touch screen or similar touch-sensitive display. Output device 320 may be located near input device 315.
[0061] The transceiver 325 includes at least one transmitter 330 and at least one receiver 335. One or more transmitters 330 may be used to communicate with a UE, as described herein. Similarly, one or more receivers 335 may be used to communicate with a network function in a PLMN and / or RAN, as described herein. Although only one transmitter 330 and one receiver 335 are illustrated, the network node 300 may have any suitable number of transmitters 330 and receivers 335. Further, the transmitter(s) 330 and the receiver(s) 335 may be any suitable type of transmitter and receiver.
[0062] Figure 4A method 400 for improving the accuracy of analysis is illustrated. The method 400 includes determining 410 the accuracy of an ML model that needs to be identified based on feedback received from an analysis consumer, the ML model being used to derive an analysis for an analysis ID. The method 400 also includes receiving 420 a first data set from a first network function, wherein the first data set is collected in the past and is used to train the ML model. The method 400 also includes receiving 430 a second data set corresponding to the analysis ID from a second network function, wherein the second data set is real-time data. The method 400 also includes determining 440 the accuracy of the ML model by comparing the first data set with the second data set. The feedback from the analysis consumer may be an action of the consumer based on the analysis derived using the ML model. The method 400 may also include receiving a minimum accuracy of the ML model. Determining 410 that the accuracy of the ML model needs to be identified may be based on the minimum accuracy. The method 400 may also include retraining the ML model in response to determining that the accuracy of the ML model is below a threshold. The method 400 may also include notifying the network function that the accuracy of the ML model is below a threshold. The method 400 may also include providing the retrained ML model to the network function. The method may also include subscribing to the first data set and the second data set. The method may be performed by a NWDAF model training logic function (MTLF). Also provided is a device comprising a processor configured to perform the above method 400. The device may be the above referenced Figure 2 and 3 A user equipment device 200 or a network node 300 is described.
[0063] Figure 5 Another method 500 for improving analysis accuracy is illustrated. The method 500 includes: receiving 510 a plurality of machine learning (ML) models for deriving an analysis for an analysis ID; receiving 520 a request for an analysis for the analysis ID; determining 530 a plurality of analysis outputs for the requested analysis using a plurality of ML models associated with the requested analysis; and determining 540 the accuracy of each of the plurality of ML models based on the plurality of analysis outputs. The method 500 may also include providing the accuracy of the plurality of ML models to a network function. The method 500 may also include providing the plurality of analysis outputs of the requested analysis to an analysis consumer. The method 500 may be performed by a NWDAF analysis inference logic function (AnLF).
[0064] A device comprising a processor configured to execute the above method 500 is also provided. The device may be the device described above with reference to Figure 2 and 3 A user equipment device 200 or a network node 300 is described.
[0065] Figure 6A wireless communication system 600 for improving analysis accuracy according to a first embodiment is illustrated. The system 600 includes a NWDAF 610, a 5G core (5GC) 620, an analysis data repository function (ADRF) 630, and an operation and management function (OAM) 640. The 5GC 620 includes an access and mobility management function (AMF) 622, a session management function (SMF) 624, a user plane function (UPF) 626, and a network repository function (NRF) 628. The usual roles of these functions in the context of a 3GPP network are well understood and will not be described herein.
[0066] In a first embodiment, the NWDAF 610 supporting model training (i.e., NWDAF MTLF) identifies ML model drift or data drift by comparing the data collected for training the ML model with the real-time data. The NWDAF MTLF 610 uses statistical methods (e.g., by using a specific ML model) to detect any drift between the data used to train the ML model and the real-time data, and determines whether the ML model needs to be retrained based on statistical analysis. If the detected model drift exceeds the NWDAF analysis inference logic function (AnLF) ( Figure 6 If the minimum accuracy threshold requested by the ML model is not shown in Figure 1, the NWDAF MTLF 610 may determine that the ML model needs to be retrained. The NWDAF MTLF 610 may determine to start identifying whether there is drift in the ML model based on the actions performed by the analytics consumer that may significantly change the network behavior. The NWDAF MTLF 610 subscribes to the analytics consumer (via the AnLF NWDAF) to receive notifications of the actions taken.
[0067] Each NF that is a consumer of the analysis can determine the action by considering the analysis information received by the NWDAF AnLF. The NF actions based on each analysis supported by the NWDAF AnLF are described in 3GPP TS23.288. If the consumer is a policy control function (PCF), the actions considering the analysis input are defined in 3GPP TS23.503 Section 6.1.1.3.
[0068] If the NWDAF MTLF 610 subscribes to the PCF via the NWDAF AnLF for actions to be notified, each NF may report the actions taken to the NWDAF MTLF 610 via the NWDAF AnLF. For example, if the consumer is the PCF and the PCF has subscribed to service experience analysis (defined in 23.288), the PCF notifies the NWDAF MTLF 610 that the QoS for one or more UEs has been upgraded.
[0069] Alternatively, the analysis consumer may include an indication that the actions taken may typically significantly change the behavior of the NF, slice, or 5GC 620.
[0070] For example, when the analysis consumer is an SMF and the SMF selects a UPF with a smaller load for a protocol data unit (PDU) session, changing the behavior of the NF may occur. In this example, the SMF may notify the NWDAF MTLF 610 via the AnLF NWDAF that the UE's service will be routed to the second UPF, effectively increasing the load of the second UPF. The SMF may include the selected second UPF in the feedback notification. The NWDAF MTLF 610 may then determine whether data drift may occur (whether the NWDAF MTLF 610 uses the NF load information from the second UPF to train the data).
[0071] For example, when the AMF 622 or the Network Slice Selection Function (NSSF) is a consumer of slice load analysis from the NWDAF, the behavior of changing the slice may occur. If the NSSF decides to instantiate a new slice instance for the slice and routes the UE to the new slice instance, the existing network slice will be less loaded, resulting in data drift because the NWDAF MTLF 610 uses historical data from the existing slice instances to train the ML model. In this scenario, the analysis consumer can indicate to the NWDAFMTLF 610 that data drift may occur due to instantiating a new slice instance.
[0072] For example, changing the behavior of the 5GC network 620 may occur when the NF decides on an action that affects the operation of the entire 5GC 620. For example, assigning a PCF with higher QoS to multiple UEs will result in increased load in the RAN and 5GC 620. If the NWDAF MTLF 610 uses data from the RAN or UPF to train the ML model, the NWDAF MTLF 610 may need to retrain the ML model with the updated data. In this scenario, the analytics consumer can indicate the affected NFs (or NF types) where their behavior will change due to the NF action.
[0073] Figure 7 Pictured Figure 6 Communication between entities within a wireless communication system. In step 700, the analysis consumer sends a request for an analysis corresponding to an analysis ID to the AnLF NWDAF (e.g., as defined in 3GPP TS23.288). The analysis ID is included in the request. For example, the analysis consumer may request an analysis of the analysis ID for the NF load. The request may include a minimum accuracy. The minimum accuracy may represent a required confidence level for the requested analysis (as defined in 3GPP TS23.288).
[0074] In step 705, if the AnLF NWDAF does not have an ML model for the requested analysis ID, the AnLF NWDAF sends a request from the MTLF NWDAF for an ML model to derive an analysis for the requested analysis ID. The request may include a minimum accuracy. The request may be a one-time request or a subscription. If the request is a subscription, the request may include a minimum accuracy threshold to allow the MTLF NWDAF to determine when to provide an updated ML model to the AnLF NWDAF.
[0075] AnLF NWDAF may convert the minimum accuracy received in step 700 into a minimum ML model drift accuracy.
[0076] At step 710 , the MTLF NWDAF provides the current ML model for the analysis ID.
[0077] At step 715, based on the minimum accuracy in the request, the MTLF NWDAF determines that the ML model drift needs to be calculated for the provided ML model. The MTLF NWDAF may convert the minimum accuracy received at step 705 into a minimum ML model drift accuracy. Alternatively, the MTLF NWDAF may be configured (e.g., via OAM) according to a universal minimum accuracy drift for the ML model. The MTLF NWDAF may subscribe to data to determine model drift.
[0078] The MTLF NWDAF may already have a trained ML model available. The MTLF NWDAF does not need to wait for a trigger from the AnLF NWDAF to start training the ML model.
[0079] At step 720, the MTLF NWDAF provides the requested ML model to the AnLF NWDAF. When the analysis consumer determines an action based on the analysis provided by the AnLF NWDAF, the MTLF NWDAF may also send a request for feedback. This feedback may be used to determine whether drift detection should be performed on the ML model (see step 755).
[0080] At step 725, AnLF NWDAF derives an analysis using the provided ML model.
[0081] The AnLF NWDAF provides the requested analysis to the analysis consumer at step 730. When the consumer determines an action, the AnLF NWDAF also sends a request to the consumer to provide feedback.
[0082] The consumer may determine an action based on the received analysis at step 735. For example, if the analysis indicates a high load at the UPF function, the consumer (eg, SMF) may select a less loaded UPF.
[0083] At step 740 , based on the request for feedback, the consumer determines an action to report as feedback.
[0084] The consumer reports the action taken to the AnLF NWDAF in a feedback notification at step 750. For example, the consumer may report the selected UPF and the load of the selected UPF. Alternatively, if the consumer is a PCF, the consumer may report an upgrade or downgrade of the applied QoS.
[0085] In step 755, the AnLF NWDAF forwards the action report to the MTLF NWDAF in a feedback notification.
[0086] At step 755, the MTLF NWDAF determines based on the feedback notification whether the action taken by the consumer may trigger drift in the data that will invalidate the current ML model. If the action taken by the consumer may trigger drift in the data that invalidates the current ML model, the MTFL NWDAF determines to evaluate the accuracy of the current ML model.
[0087] At step 760, the MTLF NWDAF subscribes to the historical data used to train the current ML model (or the earliest historical data available if the data used to train the ML model is not available) for the specific analysis ID from the ADRF and / or OAM (if the MTLF does have historical data stored locally). For example, if the ML model is used to analyze NF load, the training data may be NF load data. Various different types of training data that can be used are defined in 3GPP TS23.288.
[0088] In step 765, the MTLF NWDAF subscribes to real-time data (or near real-time data) from the 5GC NF, AF, OAM that is required to derive analysis for the analysis ID (as described in 3GPP TS23.288 for each analysis ID). For example, the real-time data may be NF load data.
[0089] At steps 760 and 765, the MTLF NWDAF can determine where data drift may occur based on feedback received from the analytics consumer / AnLF. For example, when the analytics consumer provides feedback, the feedback can include information about the NFs affected by the action (e.g., selecting a UPF with less load for a business that will increase the load in the selected UPF). Based on the feedback, the MTLF NWDAF can determine which data needs to be checked for data drift and whether such data drift will affect the ML model performance.
[0090] The MTLF NWDAF determines model drift by comparing historical data with real-time data at step 770. The MTLF NWDAF may determine data / model drift using statistical analysis or using AI / ML programs.
[0091] At step 775, if the MTLF NWDAF determines that the model drift exceeds a threshold, the MTLF NWDAF retrains the ML model. Based on step 770, the MTLF NWDAF can know which data contributes to the drift and retrain the ML model, excluding invalid historical data.
[0092] The MTLF NWDAF may notify the AnLF NWDAF that the ML model drift exceeds a threshold at step 780. The notification may notify the AnLF to stop using the ML model to provide analysis. The MTLF NWDAF may also provide an indication of when an updated ML model is available.
[0093] Based on the indication received from the MTLF NWDAF, the AnLF NWDAF may indicate to the analytics consumer a pause in the subscription because the ML model needs to be retrained, at step 785. The AnLF NWDAF may also indicate when new analytics are available.
[0094] In step 790 , the MTLF NWDAF provides the retrained ML model to the AnLF NWDAF.
[0095] In step 795, AnLF notifies the consumer to resume the subscription, indicating that analysis is available. Figure 8 Another wireless communication system 800 for improving analysis accuracy is illustrated. The system 800 includes multiple NWDAFs 810, a model training performance monitoring function (MTPMF) 820, a 5GC 830, an OAM 840, and an application function AF 850. The 5GC 820 includes an access and mobility management function (AMF) 832, a session management function (SMF) 834, a user plane function (UPF) 836, and a network repository function (NRF) 838. The general role of these functions in the context of a 3GPP network is well understood and will not be described herein.
[0096] In this embodiment, the consumer may be a new function, namely, Model Training Performance Monitoring Function (MTPMF) 820, which receives the ML model from one or more NWDAF MTLF 810 and determines data drift or model drift by comparing historical data with near real-time data. The MTPMF 820 may reside in the NWDAF 810 or may be a function of the OAM 840. The NWDAF 810 will provide the trained ML model to the MTPMF 820 and notify the NWDAF 810 when drift is detected and potential data is impacted. The MTPMF 820 will subscribe to each 5GC NF 830, OAM 840 or AF850 that is an analysis consumer to provide feedback (e.g., actions of the NF) to determine whether drift detection should begin (similar to Figure 7 The procedure shown in the figure).
[0097] In an alternative embodiment, the consumer may be a new function, namely, an analysis performance monitoring function (APMF) (not shown), which receives analysis from one or more NWDAFs and determines the accuracy of the analysis by collecting real-time data information. The APMF may reside in the NWDAF, or may be a function of the OAM.
[0098] Fig. 9 Communication between entities according to yet another embodiment is illustrated.
[0099] exist Fig. 9 In an embodiment, an estimate of the accuracy of the ML model is determined by receiving multiple ML models from the MTLF NWDAF and determining the performance of the ML models by comparing the analysis outputs. If there are differences in the analysis outputs between the multiple ML models, the AnLF NWDAF can indicate to the MTLF which ML model needs to be retrained.
[0100] At step 900 , the MTLF NWDAF may train multiple ML models for the same analysis ID.
[0101] The MTLF NWDAF provides multiple ML models to the AnLF NWDAF at step 910. The MTLF NWDAF may optionally subscribe to ML model performance feedback.
[0102] At step 920, the AnLF NWDAF receives a request for an analysis corresponding to the analysis ID from the analysis consumer.
[0103] At step 930, the AnLF NWDAF determines a plurality of analysis outputs based on the plurality of ML models for the requested analysis. Specifically, the AnLF NWDAF determines a corresponding analysis output for the analysis ID from each of the plurality of ML models. The AnLF NWDAF may determine a confidence level for each of the plurality of analysis outputs corresponding to their corresponding ML models.
[0104] The AnLF NWDAF provides the plurality of analysis outputs to the analysis consumer at step 940. The AnLF NWDAF may also provide the determined confidence level to the analysis consumer.
[0105] At step 950 , AnLF NWDAF compares the performance of each ML model supporting the same analysis ID, and identifies the performance of each ML model (eg, confidence level of analysis output, variance between each analysis output, variance of each analysis output).
[0106] The AnLF NWDAF provides the MTLF NWDAF with performance estimates of each ML model at step 960. Optionally, this may only be provided if the MTLF NWDAF has subscribed to receive such feedback.
[0107] At step 970, the MTLF NWDAF determines to retrain one or more of the ML models based on the performance feedback provided by the AnLF NWDAF. For example, the MTLF NWDAF may decide to retrain only the ML model with the lowest accuracy. Alternatively, the MTLF NWDAF may decide to retrain all ML models. The MTLF NWDAF then retrains the one or more ML models it has decided / determined to retrain.
[0108] Therefore, methods and apparatus for providing improved analysis accuracy are provided. The main solution proposes that NWDAF determines ML model drift by comparing data drift between data used to train the ML model and near real-time data. The trigger to start detecting model drift can be based on actions taken by the analysis consumer that significantly change the network behavior.
[0109] In the above embodiments, the NWDAF MTLF and the NWDAF AnLF are separate entities (e.g. Figure 1 However, alternatively, the MTLF and AnLF functions may be supported by a single NWDAF entity that performs the functions of both the MTLF and AnLF described above.
[0110] It should be noted that the methods and devices mentioned above illustrate rather than limit the present invention, and that those skilled in the art will be able to design many alternative arrangements without departing from the scope of the appended claims. The word "comprising" does not exclude the presence of elements or steps other than those listed in the claims, "a" or "an" does not exclude a plurality, and a single processor or other unit may perform the functions of several units listed in the claims. Any reference signs in the claims should not be construed as limiting their scope.
[0111] Further, although examples are given in the context of a specific communication standard, these examples are not intended to limit the communication standards to which the disclosed methods and apparatus may be applied. For example, although specific examples are given in the context of 3GPP, the principles disclosed herein may also be applied to another wireless communication system, and in fact may be applied to any communication system that uses routing rules.
[0112] The method may also be implemented in a set of instructions stored on a computer readable medium, which, when loaded into a computer processor, digital signal processor (DSP) or the like, causes the processor to perform the method described above.
[0113] The described methods and devices may be practiced in other specific forms. The described methods and devices are to be considered in all respects as illustrative only and not restrictive. Therefore, the scope of the present invention is indicated by the appended claims rather than by the foregoing description. All changes that fall within the equivalent meaning and range of the claims are to be included within their scope.
Claims
1. A method comprising: determining, based on feedback received from the analytics consumer, that accuracy of a plurality of ML models to be identified is required, the plurality of ML models used to derive the analytics for the analytics ID; receiving a first data set from a first network function, wherein the first data set was collected in the past and is used to train the ML model; receiving a second data set corresponding to the analysis ID from a second network function, wherein the second data set is real-time data; as well as The accuracy of the ML model is determined by comparing the first data set and the second data set.
2. The method according to claim 1, further comprising: Based on the feedback received from the analytics consumer, it is determined that drift from the first data set has occurred, wherein determining the need to identify the accuracy of the ML model is based on determining that drift from the first data set has occurred.
3. The method of claim 1 or 2, wherein the feedback from the analysis consumer indicates an action taken by the consumer based on the analysis derived using the ML model.
4. The method according to claim 3, wherein the action taken is any of the following: Upgrading the QoS of one or more UEs; Selection of UPF with smaller load for PDU session; Instantiation of a new slice instance of a slice, and routing of the UE to the new slice instance.
5. The method according to any one of the preceding claims, further comprising: A minimum accuracy is received for the ML model.
6. The method of any one of the preceding claims, wherein determining the accuracy required to identify the ML model is based on the minimum accuracy.
7. The method according to any one of the preceding claims, further comprising: In response to determining that the accuracy of the ML model is below a threshold, retraining the ML model.
8. The method according to any one of the preceding claims, further comprising: A network function is notified that the accuracy of the ML model is below the threshold.
9. The method according to claim 8, further comprising: The retrained ML model is provided to the network function.
10. The method according to any one of the preceding claims, further comprising: Subscribe to the first data set and the second data set.
11. A method according to any one of the preceding claims, wherein the method is performed by a NWDAF MTLF.
12. An apparatus comprising a processor, the processor being configured to: determining, based on feedback received from the analytics consumer, that accuracy of a plurality of ML models to be identified is required, the plurality of ML models used to derive the analytics for the analytics ID; receiving a first data set from a first network function, wherein the first data set was collected in the past and is used to train the ML model; receiving a second data set corresponding to the analysis ID from a second network function, wherein the second data set is real-time data; as well as The accuracy of the ML model is determined by comparing the first data set and the second data set.
13. An apparatus according to claim 12, wherein the processor is further configured to: determine that drift from the first data set has occurred based on the feedback received from the analysis consumer, wherein determining the need to identify the accuracy of the ML model is based on determining that drift from the first data set has occurred.
14. The apparatus of claim 12 or 13, wherein the feedback from the analytics consumer indicates an action taken by the consumer based on the analytics derived using the ML model.
15. The apparatus of claim 14, wherein the action taken is any of the following: Upgrading the QoS of one or more UEs; Selection of UPF with smaller load for PDU session; Instantiation of a new slice instance of a slice, and routing of the UE to the new slice instance.
16. The apparatus of any one of claims 12 to 15, wherein the processor is configured to receive a minimum accuracy for the ML model.
17. The apparatus according to any one of claims 12 to 16, wherein the processor is configured to determine the accuracy of the ML model that needs to be identified based on the minimum accuracy.
18. The apparatus of any one of claims 12 to 17, wherein the processor is configured to: in response to determining that the accuracy of the ML model is below a threshold, retrain the ML model.
19. The apparatus of any one of claims 12 to 18, wherein the processor is configured to notify a network function that the accuracy of the ML model is below the threshold.
20. The apparatus of claim 19, wherein the processor is configured to provide the retrained ML model to the network function.
21. The apparatus according to any one of claims 12 to 20, wherein the processor is configured to subscribe to the first data set and the second data set.
22. The device according to any one of claims 12 to 21, wherein the device is a NWDAF MTLF device.
23. A method comprising: receiving a plurality of machine learning (ML) models for deriving an analysis for an analysis ID; receiving a request for an analysis of the analysis ID, and determining a plurality of analysis outputs for the requested analysis using the plurality of ML models associated with the requested analysis; as well as Based on the plurality of analysis outputs, an accuracy of each ML model in the plurality of ML models is determined.
