Method, apparatus and computer program

Abstract

There is provided an apparatus for analyzing a consumer, the apparatus comprising means for providing a request to an analysis producer for a machine learning model capable of predicting a confidence value for each reasoning, and means for receiving the machine learning model from the analysis producer. The apparatus further comprises means for operating the received machine learning model for reasoning to determine at least one measurement confidence value using the machine learning model, and means for performing an action using the at least one measurement confidence value.

Description

Technical Field

[0001] The present application relates to methods, apparatus and computer programs for wireless communication systems. Background Art

[0002] A communication system may be a facility for enabling communication sessions between two or more entities (such as user terminals, base stations / access points, and / or other nodes) by providing carrier waves between the various entities involved in the communication path. For example, a communication system may be provided by means of a communication network and one or more compatible communication devices. A communication session may include, for example, communications for carrying data such as voice, electronic mail (email), text messages, multimedia, and / or content data. Non-limiting examples of the services provided include two-way or multi-way calls, data communications or multimedia services, and access to data network systems (such as the Internet). Summary of the invention

[0003] According to one aspect, a device for an analytics consumer is provided, the device comprising: a component for providing a request to an analytics producer for a machine learning model capable of predicting a confidence value for each inference; a component for receiving the machine learning model from the analytics producer; a component for operating the received machine learning model for inference to determine at least one measurement confidence value using the machine learning model; and a component for performing an action using at least one measurement confidence value.

[0004] In one example, components for operating a received machine learning model for inference include components for operating the machine learning model to determine a plurality of measurement confidence values ​​for each of a plurality of inferences.

[0005] In one example, the apparatus includes means for calculating an average measurement confidence value using a plurality of measurement confidence values.

[0006] In one example, the average confidence value is an arithmetic mean. In one example, the average confidence value is a weighted average.

[0007] In one example, the average confidence value is a running average calculated for each operational inference.

[0008] In one example, the apparatus includes components for receiving, from an analytics producer, an average confidence threshold associated with a machine learning model to be monitored by an analytics consumer.

[0009] In one example, a component for performing an action includes: a component for determining whether at least one measurement confidence value is below a received average confidence threshold; and a component for providing a request to an analysis producer for the analysis producer to perform an operation when it is determined that at least one measurement confidence value is below the received average confidence threshold.

[0010] In one example, means for performing an action includes means for comparing at least one measurement value to the received average confidence threshold, and means for providing a request to the analytics producer for the analytics producer to perform an operation based on the comparison.

[0011] In one example, the apparatus includes means for receiving a result of an operation from an analytics producer, the operation being performed by the analytics producer.

[0012] In one example, the apparatus includes: means for receiving a reporting timer associated with a machine learning model from an analytics producer.

[0013] In one example, means for performing an action includes: means for determining whether a received reporting timer has expired; and means for providing a report to an analytics producer upon determining that the reporting timer has expired, wherein the report includes at least one measurement confidence value.

[0014] In one example, the apparatus includes means for receiving, from an analytics producer, results of an operation performed by the analytics producer, wherein the operation is associated with a report provided by an analytics consumer.

[0015] In one example, the operation includes one of terminating the machine learning model at the analytics consumer, providing a machine learning model that has been retrained by the analytics producer, and providing an additional machine learning model to the analytics consumer.

[0016] In one example, a machine learning model is received from an analytics producer along with an indication of an average of confidences supported by the machine learning model.

[0017] In one example, the requested machine learning model is requested to have an average confidence threshold above a determined value, where the determined value of the average confidence threshold is included in the request by the analytics consumer.

[0018] In one example, a measurement confidence value of at least one measurement confidence value is determined for each inference.

[0019] In one example, the request for a machine learning model is for a machine learning model that has been trained by an analytics producer.

[0020] In one example, the analytics consumer is one of: an edge entity, a network function, a user device, a terminal.

[0021] According to one aspect, a device for an analysis producer is provided, the device comprising: a component for receiving a request from an analysis consumer for a machine learning model capable of predicting a confidence value for each inference; a component for training the machine learning model using a data set; a component for providing the trained machine learning model to the analysis consumer so that the analysis consumer can operate the machine learning model for inference; and a component for performing an operation associated with the machine learning model in response to receiving a message from the analysis consumer.

[0022] In one example, the apparatus includes components for providing an average confidence threshold associated with a machine learning model to be monitored by the analytics consumer to an analytics consumer.

[0023] In one example, a component for performing an operation associated with a machine learning model in response to receiving a message from an analytics consumer includes a component for receiving an additional request from the analytics consumer to perform an operation on an analytics producer associated with an average confidence threshold.

[0024] In one example, the apparatus includes means for performing a requested operation associated with the machine learning model in response to receiving an additional request.

[0025] In one example, the apparatus includes means for providing results of operations performed by an analytics producer to an analytics consumer.

[0026] In one example, the apparatus includes components for providing a reporting timer associated with a machine learning model to an analytics consumer.

[0027] In one example, means for performing an action include means for receiving a report from an analytics consumer when a report timer has expired, wherein the report includes at least one measurement confidence value associated with a machine learning model.

[0028] In one example, the apparatus includes components for determining an appropriate action to perform on a machine learning model using at least one received measurement confidence value in response to receiving a report.

[0029] In one example, the at least one measurement confidence value reported comprises an average measurement confidence value calculated by the analytics consumer.

[0030] In one example, the apparatus includes means for providing to an analytics producer results of operations performed by the analytics producer.

[0031] In one example, the operation includes one of terminating the machine learning model at the analytics consumer, providing a machine learning model that has been retrained by the analytics producer, and providing an additional machine learning model to the analytics consumer.

[0032] In one example, a machine learning model is provided to an analytics consumer along with an indication of an average of the confidences supported by the trained machine learning model.

[0033] In one example, the requested machine learning model is requested to have an average confidence threshold above a first value, where the first value of the average confidence threshold is included in the request by the analytics consumer.

[0034] In one example, the analytics producer is one of: a management data analytics function, a network data analytics function, an edge entity, and an edge cloud.

[0035] According to one aspect, a device is provided, comprising: one or more processors and a memory storing instructions, which when executed by the one or more processors cause the device to perform: providing a request to an analysis producer for a machine learning model capable of predicting a confidence value for each inference; receiving the machine learning model from the analysis producer; operating the received machine learning model for inference to determine at least one measurement confidence value using the machine learning model; and performing an action using the at least one measurement confidence value.

[0036] In one example, operating the received machine learning model for inference includes operating the machine learning model to determine a plurality of measurement confidence values ​​for each of a plurality of inferences.

[0037] In one example, the apparatus is caused to perform: calculating an average measurement confidence value using a plurality of measurement confidence values.

[0038] In one example, the average confidence value is an arithmetic mean. In one example, the average confidence value is a weighted average.

[0039] In one example, the average confidence value is a running average calculated for each operational inference.

[0040] In one example, the apparatus is caused to perform: receiving, from an analytics producer, an average confidence threshold associated with a machine learning model to be monitored by an analytics consumer.

[0041] In one example, performing an action includes determining whether at least one measurement confidence value is below a received average confidence threshold and providing a request to the analysis producer for the analysis producer to perform an operation when determining that at least one measurement confidence value is below the received average confidence threshold.

[0042] In one example, performing an action includes means for comparing at least one measurement value to a received average confidence threshold, and providing a request to an analytics producer for the analytics producer to perform an operation based on the comparison.

[0043] In one example, the apparatus is caused to perform: receiving a result of an operation from an analytics producer, the operation being performed by the analytics producer.

[0044] In one example, the apparatus is caused to perform: receiving, from an analytics producer, a report timer associated with a machine learning model.

[0045] In one example, performing an action includes: determining whether a received reporting timer has expired; and upon determining that the reporting timer has expired, providing a report to an analytics producer, wherein the report includes at least one measurement confidence value.

[0046] In one example, the apparatus is caused to perform: receiving from an analytics producer a result of an operation performed by the analytics producer, wherein the operation is associated with a report provided by an analytics consumer.

[0047] In one example, the operation includes one of terminating the machine learning model at the analytics consumer, providing a machine learning model that has been retrained by the analytics producer, and providing an additional machine learning model to the analytics consumer.

[0048] In one example, a machine learning model is received from an analytics producer along with an indication of an average of confidences supported by the machine learning model.

[0049] In one example, the requested machine learning model is requested to have an average confidence threshold above a determined value, where the determined value of the average confidence threshold is included in the request by the analytics consumer.

[0050] In one example, a measurement confidence value of at least one measurement confidence value is determined for each inference.

[0051] In one example, the request for a machine learning model is for a machine learning model that has been trained by an analytics producer.

[0052] In one example, the analytics consumer is one of: an edge entity, a network function, a user device, a terminal.

[0053] According to one aspect, a device is provided, comprising: one or more processors and a memory storing instructions, which when executed by the one or more processors cause the device to perform: receiving a request from an analysis consumer for a machine learning model capable of predicting a confidence value for each inference; training the machine learning model using a data set; providing the trained machine learning model to the analysis consumer so that the analysis consumer can operate the machine learning model for inference; and performing operations associated with the machine learning model in response to receiving a message from the analysis consumer.

[0054] In one example, the apparatus is caused to perform: providing an average confidence threshold associated with a machine learning model to be monitored by the analytics consumer to an analytics consumer.

[0055] In one example, performing operations associated with the machine learning model includes receiving, from the analytics consumer, an additional request for the analytics producer to perform operations associated with the average confidence threshold.

[0056] In one example, the device is caused to perform: in response to receiving another request, performing a requested operation associated with the machine learning model.

[0057] In one example, the apparatus is caused to perform: providing results of operations performed by the analytics producer to the analytics consumer.

[0058] In one example, the apparatus is caused to perform: providing a reporting timer associated with a machine learning model to an analytics consumer.

[0059] In one example, performing an action includes receiving a report from an analytics consumer when a report timer has expired, wherein the report includes at least one measurement confidence value associated with the machine learning model.

[0060] In one example, the apparatus is caused to perform: in response to receiving the report, using at least one received measurement confidence value to determine an appropriate action to perform on the machine learning model.

[0061] In one example, the at least one measurement confidence value reported comprises an average measurement confidence value calculated by the analytics consumer.

[0062] In one example, the apparatus is caused to perform: providing to the analytics producer a result of an operation performed by the analytics producer.

[0063] In one example, the operation includes one of terminating the machine learning model at the analytics consumer, providing a machine learning model that has been retrained by the analytics producer, and providing an additional machine learning model to the analytics consumer.

[0064] In one example, a machine learning model is provided to an analytics consumer along with an indication of an average of the confidences supported by the trained machine learning model.

[0065] In one example, the requested machine learning model is requested to have an average confidence threshold above a first value, where the first value of the average confidence threshold is included in the request by the analytics consumer.

[0066] In one example, the analytics producer is one of: a management data analytics function, a network data analytics function, an edge entity, and an edge cloud.

[0067] According to one example, a method performed by an analytics consumer is provided, the method comprising: providing a request to an analytics producer for a machine learning model capable of predicting a confidence value for each inference; receiving the machine learning model from the analytics producer; operating the received machine learning model for inference to determine at least one measurement confidence value using the machine learning model; and performing an action using the at least one measurement confidence value.

[0068] In one example, operating the received machine learning model for inference includes operating the machine learning model to determine a plurality of measurement confidence values ​​for each of a plurality of inferences.

[0069] In one example, the method includes calculating an average measurement confidence value using a plurality of measurement confidence values.

[0070] In one example, the average confidence value is an arithmetic mean. In one example, the average confidence value is a weighted average.

[0071] In one example, the average confidence value is a running average calculated for each operational inference.

[0072] In one example, the method includes receiving, from an analytics producer, an average confidence threshold associated with a machine learning model to be monitored by an analytics consumer.

[0073] In one example, performing an action includes determining whether at least one measurement confidence value is below a received average confidence threshold and providing a request to the analysis producer for the analysis producer to perform an operation when determining that at least one measurement confidence value is below the received average confidence threshold.

[0074] In one example, performing an action includes means for comparing at least one measurement value to a received average confidence threshold, and providing a request to an analytics producer for the analytics producer to perform an operation based on the comparison.

[0075] In one example, the method includes receiving a result of an operation from an analytics producer, the operation being performed by the analytics producer.

[0076] In one example, the method includes receiving, from an analytics producer, a reporting timer associated with a machine learning model.

[0077] In one example, performing an action includes: determining whether a received reporting timer has expired; and upon determining that the reporting timer has expired, providing a report to an analytics producer, wherein the report includes at least one measurement confidence value.

[0078] In one example, the method includes receiving, from an analytics producer, results of an operation performed by the analytics producer, wherein the operation is associated with a report provided by an analytics consumer.

[0079] In one example, the operation includes one of terminating the machine learning model at the analytics consumer, providing a machine learning model that has been retrained by the analytics producer, and providing an additional machine learning model to the analytics consumer.

[0080] In one example, a machine learning model is received from an analytics producer along with an indication of an average of confidences supported by the machine learning model.

[0081] In one example, the requested machine learning model is requested to have an average confidence threshold above a determined value, where the determined value of the average confidence threshold is included in the request by the analytics consumer.

[0082] In one example, a measurement confidence value of at least one measurement confidence value is determined for each inference.

[0083] In one example, the request for a machine learning model is for a machine learning model that has been trained by an analytics producer.

[0084] In one example, the analytics consumer is one of: an edge entity, a network function, a user device, a terminal.

[0085] According to one aspect, a method performed by an analytics producer is provided, the method comprising: receiving a request from an analytics consumer for a machine learning model capable of predicting a confidence value for each inference; training the machine learning model using a data set; providing the trained machine learning model to the analytics consumer so that the analytics consumer can operate the machine learning model for inference; and performing an operation associated with the machine learning model in response to receiving a message from the analytics consumer.

[0086] In one example, the method includes providing an average confidence threshold associated with a machine learning model to be monitored by the analytics consumer to an analytics consumer.

[0087] In one example, performing operations associated with the machine learning model includes receiving, from the analytics consumer, an additional request for the analytics producer to perform operations associated with the average confidence threshold.

[0088] In one example, the method includes performing a requested operation associated with the machine learning model in response to receiving an additional request.

[0089] In one example, the method includes providing results of operations performed by an analytics producer to an analytics consumer.

[0090] In one example, the method includes providing a reporting timer associated with a machine learning model to an analytics consumer.

[0091] In one example, performing an action includes receiving a report from an analytics consumer when a report timer has expired, wherein the report includes at least one measurement confidence value associated with the machine learning model.

[0092] In one example, the method includes determining an appropriate action to perform on the machine learning model using at least one received measurement confidence value in response to receiving the report.

[0093] In one example, the at least one measurement confidence value reported comprises an average measurement confidence value calculated by the analytics consumer.

[0094] In one example, the method includes providing to an analytics producer results of operations performed by the analytics producer.

[0095] In one example, the operation includes one of terminating the machine learning model at the analytics consumer, providing a machine learning model that has been retrained by the analytics producer, and providing an additional machine learning model to the analytics consumer.

[0096] In one example, a machine learning model is provided to an analytics consumer along with an indication of an average of the confidences supported by the trained machine learning model.

[0097] In one example, the requested machine learning model is requested to have an average confidence threshold above a first value, where the first value of the average confidence threshold is included in the request by the analytics consumer.

[0098] In one example, the analytics producer is one of: a management data analytics function, a network data analytics function, an edge entity, and an edge cloud.

[0099] According to one aspect, a computer program is provided that includes computer executable instructions that, when executed on one or more processors, perform: providing a request to an analytics producer for a machine learning model capable of predicting a confidence value for each inference; receiving the machine learning model from the analytics producer; operating the received machine learning model for inference to determine at least one measurement confidence value using the machine learning model; and performing an action using the at least one measurement confidence value.

[0100] According to one aspect, a computer program is provided that includes computer executable instructions that, when executed on one or more processors, perform: receiving a request from an analytics consumer for a machine learning model that is capable of predicting a confidence value for each inference; training the machine learning model using a data set; providing the trained machine learning model to the analytics consumer so that the analytics consumer can operate the machine learning model for inference; and, in response to receiving a message from the analytics consumer, performing an operation associated with the machine learning model.

[0101] A computer product stored on a medium may cause an apparatus to perform the methods described herein.

[0102] An electronic device may comprise an apparatus as described herein.

[0103] In the above, various aspects have been described. It should be understood that other aspects can be provided by combining any two or more of the above-mentioned various aspects.

[0104] Various other aspects and further embodiments are also described in the following detailed description and appended claims.

[0105] According to some aspects, the subject matter of the independent claims is provided. Some further aspects are defined in the dependent claims. Embodiments not belonging to the scope of the claims are to be construed as examples that aid in understanding the present disclosure.

[0106] List of abbreviations:

[0107] AF: Application Function

[0108] AMF: Access Management Function

[0109] AN: Access Network

[0110] BS: Base Station

[0111] CN: Core Network

[0112] DL: Downlink

[0113] eNB: eNodeB

[0114] eMBB: enhanced mobile broadband

[0115] gNB: gNodeB

[0116] IIoT: Industrial Internet of Things

[0117] LTE: Long Term Evolution

[0118] NEF: Network Exposure Function

[0119] NG-RAN: Next Generation Radio Access Network

[0120] NF: Network Function

[0121] NR: New Radio

[0122] NRF: Network Repository Function

[0123] NW: Network

[0124] NWDAF: Network Data Analysis Function

[0125] MDAF: Management Data Analysis Function

[0126] ML: Machine Learning

[0127] mMTC: massive machine type communications

[0128] MS: Mobile Station

[0129] PCF: Policy Control Function

[0130] PLMN: Public Land Mobile Network

[0131] RAN: Radio Access Network

[0132] RF: Radio Frequency

[0133] SMF: Session Management Function

[0134] UE: User Equipment

[0135] UDR: Unified Data Repository

[0136] UDM: Unified Data Management

[0137] UL: Uplink

[0138] URLLC: Ultra-Reliable Low-Latency Communications

[0139] UPF: User Plane Function

[0140] 3GPP: Third Generation Partnership Project

[0141] 5G: Fifth Generation

[0142] 5GC: 5G core network

[0143] 5G-AN: 5G Radio Access Network

[0144] 5GS: 5G System BRIEF DESCRIPTION OF THE DRAWINGS

[0145] Embodiments will now be described, by way of example only, with reference to the accompanying drawings, in which:

[0146] Figure 1 A schematic representation of a 5G system is shown;

[0147] Figure 2 A schematic representation of the control arrangement is shown;

[0148] Figure 3 A schematic representation of a terminal is shown;

[0149] Figure 4 A schematic representation showing a system level overview of the analysis at different parts of a communication network;

[0150] Figure 5a , Figure 5b and Figure 5c shows an example signaling diagram between an analytics consumer, an analytics producer, and a data source;

[0151] Figure 6 shows a flow chart of an example method performed by an analytics consumer;

[0152] Figure 7 shows a flow chart of an example method performed by an analysis producer; and

[0153] Figure 8 A schematic representation of a non-volatile memory medium storing instructions that, when executed by a processor, allow the processor to perform Figure 6 and Figure 7 One or more of the steps of the method. DETAILED DESCRIPTION

[0154] Before explaining some examples of the present disclosure in detail, Figures 1 to 3 Certain general principles of wireless communication systems and mobile communication devices are briefly explained to aid understanding of the technology behind the described examples.

[0155] In such Figure 1In the wireless communication system 100 shown, a mobile communication device / terminal or user equipment and / or user equipment (UE) and / or machine type communication device 102 is provided with wireless access via at least one base station (not shown) or similar wireless transmission and / or reception node or point. The communication device is provided with appropriate signal receiving and sending devices to achieve communication, such as access to a communication network or direct communication with other devices. The communication device can access a carrier provided by a station or access point and send and / or receive communications on the carrier.

[0156] In the following, certain examples are explained with reference to a mobile communication device capable of communicating via a wireless cellular system and a mobile communication system serving such a mobile communication device. Figure 1 , Figure 2 and Figure 3 Certain general principles of wireless communication systems, their access systems and mobile communication devices are briefly explained to help understand the technology behind the described examples.

[0157] Figure 1 A schematic representation of a 5G system (5GS) 100 is shown. The 5GS may include a device 102 such as a user equipment or terminal, a 5G access network (5G-AN) 106, a 5G core network (5GC) 104, one or more network functions (NFs), one or more application functions (AFs) 108, and one or more data networks (DNs) 110.

[0158] The 5G-AN 106 may include one or more gNodeB distributed unit functions connected to one or more gNodeB (gNB) centralized unit functions.

[0159] 5GC 104 may include access management function (AMF) 112, session management function (SMF) 114, authentication server function (AUSF) 116, user data management (UDM) 118, user plane function (UPF) 120, network exposure function (NEF) 122 and / or other NFs. Some of the examples shown below may be applicable to 3GPP 5G standards. However, some examples may also be applicable to 4G, 3G and other 3GPP standards.

[0160] In such Figure 1In the communication system shown, a mobile communication device / terminal or user equipment and / or user equipment (UE) and / or machine type communication device is provided with wireless access via at least one base station or similar wireless transmission and / or reception node or point. The terminal is provided with appropriate signal receiving and transmitting means to achieve communication, for example, to enable access to a communication network or to communicate directly with other devices. The communication device can access a carrier provided by a station or access point and send and / or receive communications on the carrier.

[0161] Figure 2 The diagram shows the Figure 1 An example of a control device 200 for the functions of a 5G-AN or 5GC shown. The control device may include at least one random access memory (RAM) 211a, at least one read-only memory (ROM) 211b, at least one processor 212, 213, and an input / output interface 214. At least one processor 212, 213 may be coupled to the RAM 211a and the ROM 211b. At least one processor 212, 213 may be configured to execute appropriate software code 215. The software code 215 may, for example, allow one or more steps to be executed to perform one or more aspects of the present invention. The software code 215 may be stored in the ROM 211b. The control device 200 may be interconnected with another control device 200 that controls another function of a 5G-AN or 5GC. In some examples, each function of a 5G-AN or 5GC includes a control device 200. In an alternative example, two or more functions of a 5G-AN or 5GC may share a control device.

[0162] Figure 3 The terminal 300 (such as Figure 1 The example of a terminal shown in FIG. 300 is a terminal 300. The terminal 300 may be provided by any device capable of sending and receiving radio signals. Non-limiting examples include user equipment, a mobile station (MS), an analytical consumer device or a mobile device (such as a mobile phone or a so-called 'smartphone'), a computer provided with a wireless interface card or other wireless interface facility (e.g., a USB dongle), a personal data assistant (PDA) or tablet computer provided with wireless communication capabilities, a machine type communication (MTC) device, a cellular Internet of Things (CIoT) device, or any combination of these devices, etc. The terminal 300 may provide, for example, communication for carrying data for communication. The communication may be one or more of voice, electronic mail (email), text message, multimedia, data, machine data, etc.

[0163] The terminal 300 may receive signals over the air or radio interface 307 via suitable means for receiving, and may transmit signals via suitable means for transmitting radio signals. Figure 3In the mobile device, the transceiver arrangement is schematically represented by a block 306. The transceiver arrangement 306 may be provided, for example, by means of a radio component and an associated antenna arrangement. The antenna arrangement may be arranged internally or externally to the mobile device.

[0164] The terminal 300 may be provided with at least one processor 301, at least one memory ROM 302a, at least one RAM 302b and other possible components 303 for software and hardware assisted execution of the tasks it is designed to perform, including controlling access to and communication with access systems and other communication devices. The at least one processor 301 is coupled to the RAM 302a and the ROM 302a. The at least one processor 301 may be configured to execute appropriate software code 308. The software code 308 may, for example, allow execution of one or more aspects of the present invention. The software code 308 may be stored in the ROM 302a.

[0165] The processor, storage device and other related control devices may be arranged on an appropriate circuit board and / or in a chipset. This feature is indicated by reference numeral 304. The device may optionally have a user interface, such as a keypad 305, a touch-sensitive screen or touchpad, a combination thereof, etc. Depending on the type of device, one or more of a display, a speaker and a microphone may optionally be provided.

[0166] Enhanced Mobile Broadband (eMBB), Ultra-Reliable Low-Latency Communications (URLLC), and Massive Machine Type Communications (mMTC) are key focus areas for 5G and beyond network systems. Wireless edge transformation is one of the key aspects used to achieve the above focus areas. Different entities in the 5G network chain perform different levels of analysis to help make appropriate decisions or achieve optimization. Figure 4 A schematic representation of a system level overview of the analysis at different parts of a communication network is shown.

[0167] Figure 4 A plurality of devices 401 are shown that are capable of communicating with one or more edge entities 403. The plurality of devices 401 include 'smart' devices such as, for example, user equipment (UE), terminals, and Internet of Things (IoT) devices. The plurality of devices 401 may be connected to the edge entity 403 via a 5G network. The edge entity 403 is connected to a data center (or cloud) 405.

[0168] In some known systems, multiple devices 401 can perform device analysis, edge entity 403 can perform distributed analysis on the network, and data center 405 can perform centralized analysis on the network. Typically, edge entity 403 will have more limited resources for analysis than data center 405.

[0169] To address the key focus areas of 5G and beyond, the network should perform well in various aspects such as throughput, reliability, and capacity. This emphasizes the necessity of making edge entities (such as, for example, edge entity 403) more autonomous and intelligent. Model sharing from cloud 405 to edge entity 403 enables edge entity 403 to perform more efficiently. The performance of any machine learning (ML) model will not be the same over time. The performance of ML models needs to be monitored to determine whether a particular ML model is performing well.

[0170] When all analysis functions are performed at the analysis producer, this results in increased response time to system events. Currently, analysis consumers provide data to analysis producers and request analysis output. The analysis output is further used for reasoning and taking appropriate actions. However, the delay in requests and responses for analysis may not help the system to take action quickly in latency sensitive use cases. This emphasizes the importance of allowing / enabling analysis consumers to use models / ML models for reasoning locally at the analysis consumer. This will save time in responding to system events.

[0171] The performance of a model / ML model can be monitored by 'understanding' the decisions made by the model / ML model over time. To achieve this, information about the model's past decisions and past predictions should be collected. In known systems, this information is explicitly sought out by the analytics producer and then studied. This is done after a large amount of perturbations have been observed by the deployed ML model, which can introduce delays in understanding the performance of the ML model.

[0172] Analytics consumers (e.g., operators, network functions (NFs)) may have two options to receive analytics from analytics producers. In 3GPP SA5, the analytics producer may be the Management Data Analysis Function (MDAF), while in 3GPP SA2, the analytics producer may be the Network Data Analysis Function (NWDAF).

[0173] In the first option, an analytics consumer can subscribe to analytics (predictive / statistical) directly from an analytics producer, where an ML model or ML application corresponding to a particular analytics is being trained and then deployed for inference. An analytics consumer can specify a preferred level of accuracy to which the analytics producer should train the ML model. An analytics consumer can specify a confidence threshold that a particular inference output (inferred by the analytics producer) should meet in a subscription request. ML inference is the process of running data into an ML model / application to compute an output, such as a single numeric score (e.g., a confidence score / value). This process may also be referred to as operating an ML model, or putting an ML model into production.

[0174] In the second option, the analysis consumer can request a trained ML model or ML application corresponding to a specific analysis from the analysis producer. The received ML model is deployed at the analysis consumer for inference. When received at the analysis consumer, the trained ML model can be encrypted, for example using homomorphic encryption. Currently, the analysis consumer cannot specify the expected quality of the average inference prediction, except for the preferred level of accuracy that should be achieved when operating the received ML model using inference data at the analysis consumer. In addition, the analysis consumer cannot use the received ML model to determine the quality degradation of the decisions inferred by the deployed ML model, which may result in the analysis consumer not having enough knowledge to request the analysis producer to retrain and reissue the updated ML model.

[0175] One or more of the following examples are intended to address the above-mentioned issues. In some examples, methods and means are proposed for enabling an analysis consumer (e.g., an edge entity, NF, or UE in a wireless network) to perform analysis and derive an average confidence value for the analysis performed. For example, this enables the analysis consumer to evaluate the decisions made and take appropriate actions when the average confidence of the model is below a certain configured threshold. In the example, a "confidence score" is a number between 0 and 1 that represents the likelihood that the output of the ML model is correct and satisfies the user request (1 represents certainty).

[0176] This can reduce latency in analytical inferences and enable analytical consumers to make faster inferences, thereby enabling more efficient and autonomous systems. Specifically, one or more examples illustrate, i) a method for enabling analytical consumers to request ML models that can derive confidence scores / values ​​for inferences, ii) a method for enabling analytical consumers to request ML models that meet specified accuracy levels and average confidence thresholds, and / or iii) a method for enabling analytical producers to train and share ML models that can derive confidences along with corresponding inferences.

[0177] Furthermore, in addition to sharing ML models, analytics producers can also provide average confidence thresholds to be monitored by analytics consumers. This enables analytics consumers to monitor the performance of ML models and take appropriate actions if the average confidence level is below / above the threshold. In particular, these examples have two different alternative behaviors for the system:

[0178] In a first example alternative, there is a method that enables an analytics producer to specify an average confidence threshold below which an analytics consumer should take appropriate action. Also, a method that enables an analytics consumer to take appropriate action, such as, for example, requesting the analytics producer to provide a new or retrained ML model, or to terminate the ML model, or to notify that the average confidence in the ML model has dropped.

[0179] In a second example alternative, there is a method for enabling an analysis consumer to report an average confidence value to an analysis producer. Based on the value received at the analysis producer, the analysis producer can take appropriate action, for example, the analysis producer can retrain the ML model and share it with the analysis consumer, or terminate the ML model.

[0180] These examples are described in more detail below along with FIG. 5 .

[0181] Figure 5 shows an example signaling diagram between an analytics consumer, an analytics producer, and a data source. Figure 5 has been divided into Figure 5a , Figure 5b and Figure 5c .

[0182] In S501, the analysis consumer provides a request to the analysis producer for a predicted ML model (or ML application) that can predict the confidence of each inference. In some examples, the request also defines an average confidence threshold that the ML model should meet.

[0183] When an analysis consumer requests an ML model that can predict the confidence of each inference. In addition, the analysis consumer can also specify an average confidence threshold that the ML model should meet on the test data set. A unique identifier of the request can be provided in the request. The unique identifier can be a number. The following table provides parameters that can be provided in the request message provided by the analysis consumer to the analysis producer. In the example, one or more of the following parameters in Table 1 can be included in the request message.

[0184]

[0185]

[0186] Table 1: Parameters sent by the analytics consumer to the analytics producer

[0187] In S502, the analysis producer trains the ML model according to the request received from the analysis consumer. When training the ML model, the analysis producer can communicate with the data source as needed. The data source can provide a test data set.

[0188] Although S502 may be implementation specific, an example of an ML model for training a multi-class classification problem that returns the class probability for each data point is as follows. In a commonly used ML library (such as the 'scikit-learn' library), after training an ML classifier using the k-nearest neighbor algorithm, the predict() method can be used to predict the actual label / class on a new set of data. The predict() method accepts a single input argument corresponding to the inference data to be predicted, and returns an array, such as (array([1,3,5,2,1])), which contains the predicted labels / classes of the data points. However, the predict_proba() method returns the label / class probability for each data point, i.e., the predict_proba() method accepts a single input argument corresponding to the inference data (on which the probability is to be calculated) and returns an array of lists (array([[0.1, 0.9, 0.], [0.2, 0.2, 0.6], [1., 0., 0.]]) containing the label / class probabilities for the input data points. In other examples, other suitable techniques or libraries are used to train the ML model so that it can predict the confidence scores as well as the actual labels / classes. For example, this can be achieved by manually adding the confidence value as an additional feature to each input data point (e.g., based on domain expertise or based on missing features) before training the ML model. In other examples, the algorithmic theory of randomness methods (Gammerman, Alex, and Volodya Vovk, "Prediction algorithms and confidence measures based on algorithmic randomness theory" (Theoretical Computer Science 287, no. 1 (2002): 209-217) can be used.

[0189] In S503, when the analysis consumer has requested that the ML model meet the average confidence threshold, the analysis producer will continue to train the ML model with more data from the data source. The analysis producer will train the ML model until the requested average confidence threshold is reached.

[0190] Based on the nature of the model (e.g., classification / regression), different metrics (e.g., accuracy) can be used to measure the performance of the ML model on the test dataset of the data source. In addition to these measurements, the confidence of the prediction can also be measured for each data point in the test dataset. Aggregation is performed on each confidence of each prediction in the test dataset. This aggregation is used to compare with the average confidence requested by the analysis consumer. If the trained ML model cannot achieve the average confidence threshold of the predictions in the test dataset as requested by the analysis consumer, the analysis producer will collect more data from the data source. Examples of data sources include base stations (gNBs), 5GC network functions, and operation, administration, and maintenance (OAM) entities.

[0191] Steps S502 and / or S503 are repeated or looped by the analytics producer until the ML model has been trained to the requirements requested by the analytics consumer.

[0192] In S504, the trained ML model is provided by the analysis producer to the analysis consumer. The trained ML model is able to predict the confidence of each inference. In the example, when the analysis consumer requests in S501, the trained ML model also meets the requested average confidence threshold. In the example, the accuracy value of the trained ML model is provided, which indicates the average accuracy of the trained model on the test data set.

[0193] The ML model may be associated with a unique identifier. Each time a message involving the ML model is sent between an analytics consumer and an analytics producer, the unique identifier may be provided in the message.

[0194] The message including the trained ML model may include one or more of the following parameters related to the message, as shown in Table 2.

[0195]

[0196]

[0197] Table 2: Parameters sent by the analytics producer to the analytics consumer. In the output message from the analytics producer, for parameter 5 in Table 2 above, the confidence value should be provided as a simple average or a weighted average.

[0198] S505 to S511 form steps for a first alternative. In the first alternative, the analytics consumer determines the appropriate action to be performed when operating the ML model. S512 to S517 form steps for a second alternative. In the second alternative, the analytics producer determines the appropriate action after operating the ML model at the analytics consumer.

[0199] In S505, the analysis producer provides a confidence threshold to the analysis consumer. The analysis producer may provide an AverageConfidenceThreshold value. In an example, this value will be lower than the AverageConfidenceValue parameter reported in S504 to be monitored by the analysis consumer. This confidence threshold is used by the analysis consumer to determine a possible drop in the quality of the predictions provided by the ML model.

[0200] In S506, the analysis consumer operates the received ML model for inference. In this way, the ML model is operated by the analysis consumer.

[0201] S507 to S511 are performed at the analysis consumer for each inference.

[0202] In S507, a confidence value is determined using the ML model at the analysis consumer using the ML model. For example, the analysis consumer derives a confidence for each prediction / forecast. In the example, a confidence value is determined / derived for each inference.

[0203] In S508, the analysis consumer performs an action of calculating an average confidence value. The average value may be calculated using all confidence values ​​that have been derived since the ML model was operated / deployed at the analysis consumer. In this manner, the average confidence value is calculated using the confidence values ​​calculated in S507. In one example, the average confidence value is an arithmetic mean. In another example, the average confidence value is a weighted average.

[0204] In S509, the analytics consumer performs an action of comparing the measured confidence value (or average confidence value) with the (average) confidence threshold. The analytics consumer may determine whether the confidence value or average confidence value is above or below the confidence threshold provided to the analytics consumer in S505.

[0205] In S510, when it is determined that the confidence value / average confidence value is lower than the confidence threshold, the analysis consumer provides a request to the analysis producer to perform an operation. In other examples, when the confidence value / average confidence value is higher than the confidence threshold, the analysis consumer may request to perform an operation. In some examples, the requested operation is referred to as a requested action. In an example, the requested operation includes terminating the ML model and requesting a retrained model. In other examples, when the analysis producer and the ML model support reconfiguration, the operation may include tuning / reconfiguring the ML model using one or more supported processes and then retraining the ML model. The following table provides one or more parameters associated with the message provided by the analysis consumer when requesting an operation.

[0206]

[0207] Table 3: Parameters sent by an analytics consumer to an analytics producer when requesting an operation.

[0208] When it is determined that the confidence value / average confidence value has remained above the confidence threshold, the method loops back to S507.

[0209] In S511, in response to receiving the request from the analysis consumer, the analysis producer performs the requested operation on the analysis consumer. In one example, the analysis producer terminates the ML model in the analysis consumer. In another example, the analysis producer retrains the ML model and then provides the retrained ML model to the analysis consumer. The following table provides one or more parameters related to the message provided by the analysis producer.

[0210]

[0211] Table 4: Parameters sent by the analytics producer to the analytics consumer to inform or guide the analytics consumer to take corresponding actions. In case of deploying a new model, the parameters mentioned in step 4 should be used to share the parameters of the new model.

[0212] In S512 (second alternative), the analysis consumer receives a report timer associated with the ML model from the analysis producer. In some examples, the timer may be referred to as a periodic reporting interval. The following table provides parameters associated with this message.

[0213]

[0214] Table 5: Parameters sent by an analytics producer to an analytics consumer when providing a report timer.

[0215] In S513, the analysis consumer operates the received ML model for inference. The analysis consumer may also start the received timer (if not already started).

[0216] In S514, the ML model is used to determine a confidence value at the analysis consumer. The confidence value of each inference is obtained from the deployed ML model.

[0217] When the timer has expired, S515 to S517 are executed.

[0218] In S515, when the timer has expired, the analytics consumer performs an action of determining an average confidence value using the confidence values ​​determined during the reporting interval. In some examples, a single confidence value is determined during the reporting interval.

[0219] The analytics consumer then provides a report including the average confidence value to the analytics producer.

[0220] The following table provides the parameters related to this report.

[0221]

[0222] Table 6: Parameters sent by the analytics consumer to the analytics producer after the timer has expired.

[0223] In response to receiving the report, the analysis producer determines an appropriate operation to be performed associated with the ML model in S516. In some examples, the determined operation is referred to as a determined action.

[0224] In S517, the analysis producer performs the determined operation on the analysis consumer. In the example, the determined operation includes terminating the ML model, retraining the ML model, and providing a new ML model. In the case of providing a new ML model to the analysis consumer, the following table shows the parameters to be provided.

[0225]

[0226] Table 7: Parameters emitted by the analytics producer to the analytics consumer using the new ML model.

[0227] In Tables 1 to 7 above, the "Support Qualifier" field has two options, "O" or "M", indicating "optional" or "mandatory". For the "Type" field, one of the options is "Integer", which indicates that an integer is provided. Another option for this field is "Bool", which indicates a Boolean value (i.e., true or false). It should be understood that the "Parameter Name" field shows an example name of the parameter, and in other examples, other suitable names are used.

[0228] One or more of the above examples enable an analytics consumer to perform analytics using an ML model and derive an average confidence value for the analytics performed. For example, this enables the analytics consumer to evaluate the decisions made and take appropriate action when the average confidence of the model is below a certain configured threshold. This reduces latency in analytics reasoning and enables analytics consumers to reason faster, resulting in more efficient and autonomous systems.

[0229] Furthermore, in a typical ML system, the larger the dataset used to train the ML model, the higher the required accuracy and / or the required confidence, which will result in the use of more computing resources. This is due to the computing resources required for training, data collection, data transmission / storage, data preprocessing, reinforcement model training, etc. In this way, more power will be consumed over longer lead times. In the above example, where the defined average confidence value and / or accuracy is requested, the ML model training can be controlled to reduce over- or under-training (perform the most efficient amount of training). Therefore, one advantage is that the power consumption of the analytics producer is more efficient, which can also improve the lead time for training the ML model.

[0230] Figure 6 An example method flow performed by an apparatus is shown. The apparatus may be included in an analysis consumer. In an example, the analysis consumer may be one of the following: a network function, an edge entity, a terminal, a UE.

[0231] In S601 , the method includes providing a request to an analytics producer for a machine learning model capable of predicting a confidence value for each inference.

[0232] In S603, the method includes receiving a machine learning model from an analytics producer.

[0233] In S605, the method includes operating the received machine learning model for inference to determine at least one measurement confidence value using the machine learning model.

[0234] In S607 , the method includes performing an action using the at least one measurement confidence value.

[0235] Figure 7 An example method flow performed by an apparatus is shown. The apparatus may be included in an analysis generator. In an example, the analysis producer is one of: a cloud entity, an MDAF, an NWDAF, an edge entity, an edge cloud, an edge node,

[0236] In S701 , the method includes receiving a request from an analytics consumer for a machine learning model capable of predicting a confidence value for each inference.

[0237] In S703, the method includes using the data set to train a machine learning model.

[0238] In S705, the method includes providing the trained machine learning model to the analysis consumer, so that the analysis consumer can operate the machine learning model for reasoning.

[0239] In S707, the method includes performing operations associated with the machine learning model in response to receiving a message from an analytics consumer.

[0240] Figure 8 Schematic representations of non-volatile memory media 800a (e.g., a computer compact disk (CD) or digital versatile disk (DVD)) and 800b (e.g., a universal serial bus (USB) memory stick) storing instructions and / or parameters 802 that, when executed by a processor, allow the processor to perform Figure 6 or Figure 7 One or more of the steps of the method.

[0241] It should be noted that while the above describes exemplifying embodiments, there are numerous variations and modifications which may be made to the disclosed solution without departing from the scope of the present invention.

[0242] Therefore, these examples may vary within the scope of the appended claims. In general, some embodiments may be implemented in hardware or dedicated circuits, software, logic, or any combination thereof. For example, some aspects may be implemented in hardware, while other aspects may be implemented in firmware or software that may be executed by a controller, microprocessor, or other computing device, but the embodiments are not limited thereto. Although various embodiments may be illustrated and described as block diagrams, flow charts, or using some other graphical representations, it is well understood that, as non-limiting examples, the blocks, devices, systems, techniques, or methods described herein may be implemented in hardware, software, firmware, dedicated circuits or logic, general hardware or controllers or other computing devices, or some combination thereof.

[0243] These examples may be implemented by computer software stored in a memory and executable by at least one data processor of the entity involved, or by hardware, or by a combination of software and hardware. Further, in this regard, it should be noted that any process may represent program steps, or interconnected logic circuits, blocks and functions, or a combination of program steps and logic circuits, blocks and functions. The software may be stored on physical media, such as memory chips or memory blocks implemented within a processor, magnetic media such as hard disks or floppy disks, and optical media such as DVDs and their data variants CDs.

[0244] The memory may be of any type suitable for the local technical environment and may be implemented using any suitable data storage technology, such as semiconductor-based memory devices, magnetic memory devices and systems, optical memory devices and systems, fixed memory, and removable memory. The data processor may be of any type suitable for the local technical environment and may include, as non-limiting examples, one or more of a general-purpose computer, a special-purpose computer, a microprocessor, a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a gate-level circuit, and a processor based on a multi-core processor architecture.

[0245] Alternatively or additionally, some examples may be implemented using a circuit system. The circuit system may be configured to perform one or more of the previously described functions and / or method steps. The circuit system may be disposed in a base station and / or a communication device.

[0246] As used in this application, the term "circuitry" may refer to one or more or all of the following:

[0247] (a) Pure hardware circuit implementation (such as an implementation using only analog and / or digital circuit systems)

[0248] now);

[0249] (b) A combination of hardware circuits and software, such as:

[0250] (i) a combination of analog and / or digital hardware circuits and software / firmware, and

[0251] (ii) any portion of hardware processor(s) (including digital signal processor(s)) with software, software and memory(s) that work together to cause an apparatus (such as a communication device or base station) to perform the various functions previously described; and

[0252] (c) Hardware circuit(s) and / or processor(s), such as microprocessor(s) or portion(s) of microprocessor(s), that require software (e.g., firmware) for operation, but which may not be present when the software is not required for operation.

[0253] This definition of circuitry applies to all uses of this term in this application, including in any claims. As another example, as used in this application, the term circuitry also covers an implementation of only a hardware circuit or processor (or multiple processors) or a portion of a hardware circuit or processor and its (or their) accompanying software and / or firmware. The term circuitry also covers, for example, an integrated device.

[0254] The foregoing description provides a complete and informative description of some embodiments by way of exemplary and non-limiting examples. However, various modifications and adaptations may become apparent to one skilled in the relevant art in view of the above description when read in conjunction with the accompanying drawings and the appended claims. However, all such and similar modifications to the teachings will still fall within the scope defined in the appended claims.

Claims

1. A device for analyzing a consumer, the device comprising: means for providing a request to the analytics producer for a machine learning model capable of predicting a confidence value for each inference; means for receiving the machine learning model from the analytics producer; means for operating the received machine learning model for inference to determine at least one measurement confidence value using the machine learning model; as well as Means for performing an action using the at least one measurement confidence value.

2. The apparatus of claim 1 , wherein the means for operating the received machine learning model for inference comprises: Means for operating the machine learning model to determine a plurality of measurement confidence values ​​for each of a plurality of inferences.

3. The device according to claim 2, wherein the device comprises: Means for calculating an average measurement confidence value using the plurality of measurement confidence values.

4. The device according to any one of claims 1 to 3, wherein the device comprises: A component for receiving from the analytics producer an average confidence threshold associated with the machine learning model to be monitored by the analytics consumer.

5. The apparatus of claim 4, wherein the means for performing an action comprises: means for determining whether said at least one measurement confidence value is below said received average confidence threshold; as well as Means for providing a request to the analytics producer for the analytics producer to perform an operation upon determining that the at least one measurement confidence value is below the received average confidence threshold.

6. The device according to claim 5, wherein the device comprises: Means for receiving a result of the operation from the analytics producer, the operation being performed by the analytics producer.

7. The device according to any one of claims 1 to 6, wherein the device comprises: Means for receiving, from the analytics producer, a report timer associated with the machine learning model.

8. The apparatus of claim 7, wherein the means for performing an action comprises: means for determining whether the received report timer has expired; as well as Means for providing a report to the analysis producer upon determining that the report timer has expired, wherein the report includes the at least one measurement confidence value.

9. The device according to claim 8, wherein the device comprises: Means for receiving from the analytics producer a result of an operation performed by the analytics producer, wherein the operation is associated with the report provided by the analytics consumer.

10. An apparatus according to any one of claims 5 to 9, wherein the operation comprises one of: terminating the machine learning model at the analysis consumer, providing the machine learning model that has been retrained by the analysis producer, and providing another machine learning model to the analysis consumer.

11. An apparatus according to any one of claims 1 to 10, wherein the machine learning model is received from the analysis producer together with an indication of an average value of confidences supported by the machine learning model.

12. An apparatus according to any one of claims 1 to 11, wherein the machine learning model of the request is requested to have an average confidence threshold above a determined value, wherein the determined value of the average confidence threshold is included in the request by the analysis consumer.

13. An apparatus according to any one of claims 1 to 12, wherein the request for the machine learning model is for the machine learning model that has been trained by the analysis producer.

14. A device for analyzing a producer, the device comprising: means for receiving a request from an analytics consumer for a machine learning model capable of predicting a confidence value for each inference; means for training the machine learning model using the dataset; A component for providing the trained machine learning model to the analysis consumer so that the analysis consumer can operate the machine learning model for reasoning; as well as A component for performing operations associated with the machine learning model in response to receiving a message from the analytics consumer.

15. The apparatus according to claim 14, wherein the apparatus comprises: Components for providing the analytics consumer with an average confidence threshold associated with the machine learning model to be monitored by the analytics consumer.

16. The apparatus of claim 15, wherein the means for performing operations associated with the machine learning model in response to receiving a message from the analytics consumer comprises: Means for receiving, from the analytics consumer, an additional request associated with the average confidence threshold for the analytics producer to perform an operation.

17. The apparatus according to claim 16, wherein the apparatus comprises: A component for performing the requested operation associated with the machine learning model in response to receiving the additional request.

18. The apparatus according to claim 17, wherein the apparatus comprises: Means for providing results of said operations performed by said analytics producer to said analytics consumer.

19. The device according to any one of claims 14 to 18, wherein the device comprises: Means for providing a reporting timer associated with the machine learning model to the analytics consumer.

20. The apparatus of claim 19, wherein the means for performing an action comprises: Means for receiving a report from the analytics consumer when the report timer has expired, wherein the report includes: at least one measurement confidence value associated with the machine learning model.

21. The apparatus according to claim 20, wherein the apparatus comprises: A component for determining an appropriate action to be performed on the machine learning model using the at least one measurement confidence value received in response to receiving the report.

22. The apparatus according to claim 21, wherein the apparatus comprises: Means for providing results of operations performed by the analytics producer to the analytics producer.

23. An apparatus according to any one of claims 14 to 22, wherein the operation comprises one of: terminating the machine learning model at the analysis consumer, providing the machine learning model that has been retrained by the analysis producer, and providing another machine learning model to the analysis consumer.

24. An apparatus according to any one of claims 14 to 23, wherein the machine learning model is provided to the analysis consumer together with an indication of an average of confidences supported by the trained machine learning model.

25. An apparatus according to any one of claims 14 to 24, wherein the machine learning model of the request is requested to have an average confidence threshold above a first value, wherein the first value of the average confidence threshold is included in the request by the analysis consumer.

26. A method performed by an analytics consumer, the method comprising: providing a request to the analytics producer for a machine learning model capable of predicting a confidence value for each inference; receiving the machine learning model from the analytics producer; operating the received machine learning model for inference to determine at least one measurement confidence value using the machine learning model; as well as An action is performed using the at least one measurement confidence value.

27. A method performed by an assay producer, the method comprising: receiving a request from an analytics consumer for a machine learning model capable of predicting a confidence value for each inference; Using the dataset to train the machine learning model; Providing the trained machine learning model to the analysis consumer so that the analysis consumer can operate the machine learning model for reasoning; and In response to receiving a message from the analytics consumer, performing operations associated with the machine learning model.

28. A computer program comprising computer executable instructions which, when executed on one or more processors, perform: providing a request to the analytics producer for a machine learning model capable of predicting a confidence value for each inference; receiving the machine learning model from the analytics producer; operating the received machine learning model for inference to determine at least one measurement confidence value using the machine learning model; and An action is performed using the at least one measurement confidence value.

29. A computer program comprising computer executable instructions which, when executed on one or more processors, perform: receiving a request from an analytics consumer for a machine learning model capable of predicting a confidence value for each inference; Using the dataset to train the machine learning model; Providing the trained machine learning model to the analysis consumer so that the analysis consumer can operate the machine learning model for reasoning; as well as In response to receiving a message from the analytics consumer, performing operations associated with the machine learning model.

Images