Apparatus, method and computer program

Abstract

An apparatus is disclosed comprising means for determining a behavioral requirement policy for a machine learning model for a given use case, means for providing an indication of the behavioral requirement policy to an analysis generator, and means for receiving, from the analysis generator, a performance evaluation metric determined based on the behavioral requirement policy.

Description

Technical Field

[0001] The present application relates to a method, apparatus, system and computer program, specifically but not limited to artificial intelligence (AI) / machine learning (ML) model training. Background Art

[0002] A communication system may be viewed as a facility that enables a communication session between two or more entities (such as user terminals, base stations and / or other nodes) by providing a carrier wave between the various entities involved in the communication path. A communication system may be provided, for example, by means of a communication network and one or more compatible communication devices. A communication session may include, for example, data communications for carrying communications such as voice, video, electronic mail (email), text messaging, multimedia and / or content data. Non-limiting examples of the services provided include two-way or multi-way calls, data communications or multimedia services, and access to a data network system (such as the Internet).

[0003] In a wireless communication system, at least a portion of a communication session between at least two stations takes place over a wireless link. Examples of wireless systems include public land mobile networks (PLMNs), satellite-based communication systems, and different wireless local area networks, such as wireless local area networks (WLANs). Some wireless systems may be divided into cells, and are therefore often referred to as cellular systems.

[0004] A user can access the communication system through an appropriate communication device or terminal. The user's communication device may be referred to as a user equipment (UE) or user device. The communication device is equipped with appropriate signal receiving and sending means to achieve communication, for example, to achieve access to the communication network or to communicate directly with other users. The communication device can access a carrier provided by a site (e.g., a base station of a cell) and transmit and / or receive communications on the carrier.

[0005] Communication systems and associated equipment typically operate according to a given standard or specification, which specifies what the various entities associated with the system are allowed to do and how it is achieved. Communication protocols and / or parameters for the connection are also typically defined. An example of a communication system is UTRAN (3G radio). Other examples of communication systems are the Long Term Evolution (LTE) of the Universal Mobile Telecommunications System (UMTS) radio access technology and the so-called 5G or New Radio (NR) networks. NR is being standardized by the Third Generation Partnership Project (3GPP). Summary of the invention

[0006] In a first aspect, a device is provided, comprising: a device for determining a behavioral requirement strategy for a machine learning model for a given use case, a device for providing an indication of the behavioral requirement strategy to an analysis generator, and a device for receiving a performance evaluation metric determined based on the behavioral requirement strategy from the analysis generator.

[0007] The apparatus may include means for receiving, from the analysis generator, a performance assessment metric value or a list of performance assessment metric values ​​determined based on the behavioral requirements policy.

[0008] The indication of the behaviour requirement policy may include an indication of at least one associated performance assessment metric.

[0009] The apparatus may include means for receiving, from the analytics generator, an indication of at least one behavioral requirement policy associated with the machine learning model for a given use case.

[0010] The apparatus may include means for receiving, in a broadcast message from the analytics generator, an indication of at least one behavioral requirement policy associated with the machine learning model for a given use case.

[0011] The apparatus may include means for providing a request to the analytics generator for the at least one policy associated with the machine learning model for the given use case, and receiving an indication of the at least one policy from the analytics generator in response.

[0012] The performance evaluation metric may include at least one of precision, accuracy, recall, f1 score, mean square error, mean absolute error, and root mean square error.

[0013] In a second aspect, an apparatus is provided, comprising: a device for receiving an indication of a behavioral requirement strategy for a given use case from an analysis consumer; a device for determining a performance evaluation metric based on the indication; and a device for providing the determined performance evaluation metric to the analysis consumer.

[0014] The apparatus may include means for providing to the analytics consumer a performance evaluation metric determined based on the behavioral requirements policy.

[0015] The indication of the behaviour requirement policy may include an indication of at least one associated performance assessment metric.

[0016] The apparatus may include means for providing, to the analytics consumer, an indication of at least one behavioral requirement policy associated with the machine learning model for a given use case.

[0017] The apparatus may include means for providing, in a broadcast message, to the analytics consumer an indication of at least one performance requirement policy associated with the machine learning model for a given use case.

[0018] The apparatus may include means for receiving a request from the analytics consumer for the at least one policy associated with the machine learning model for the given use case, and in response providing an indication of the at least one policy to the analytics consumer.

[0019] The performance evaluation metric may include at least one of precision, accuracy, recall, f1 score, mean square error, mean absolute error, and root mean square error.

[0020] The device may include a device for requesting data for training the machine learning model from at least one data source, receiving data for training the machine learning model from the at least one data source, determining whether the data for training the machine learning model allows the performance evaluation metric to be met, and if so, using the data to train the machine learning model, and if not, requesting further data for training the machine learning model from the at least one data source.

[0021] In a third aspect, a method is provided, comprising determining a behavioral requirement strategy for a machine learning model for a given use case, providing an indication of the behavioral requirement strategy to an analytics generator, and receiving from the analytics generator a performance evaluation metric determined based on the behavioral requirement strategy.

[0022] The method may include receiving, from the analytics generator, a performance assessment metric value or a list of performance assessment metric values ​​determined based on the behavioral requirements policy.

[0023] The indication of the behaviour requirement policy may include an indication of at least one associated performance assessment metric.

[0024] The method may include receiving, from the analytics generator, an indication of at least one behavioral requirement policy associated with the machine learning model for a given use case.

[0025] The method may include receiving, from the analytics generator, an indication of at least one behavioral requirement policy associated with the machine learning model for a given use case in a broadcast message.

[0026] The method may include providing a request to the analytics generator for at least one policy associated with the machine learning model for the given use case and receiving an indication of the at least one policy from the analytics generator in response.

[0027] The performance evaluation metric may include at least one of precision, accuracy, recall, f1 score, mean square error, mean absolute error, and root mean square error.

[0028] In a fourth aspect, a method is provided, comprising receiving an indication of a behavior requirement strategy for a given use case from an analytics consumer, determining a performance evaluation metric based on the indication, and providing the determined performance evaluation metric to the analytics consumer.

[0029] The method may include providing to the analytics consumer a performance evaluation metric determined based on the behavioral requirements policy.

[0030] The indication of the behaviour requirement policy may include an indication of at least one associated performance assessment metric.

[0031] The method may include providing, to the analytics consumer, an indication of at least one behavioral requirement policy associated with the machine learning model for a given use case.

[0032] The method may include providing, in a broadcast message, to the analytics consumer an indication of at least one performance requirement policy associated with the machine learning model for a given use case.

[0033] The method may include receiving a request from the analytics consumer for the at least one policy associated with the machine learning model for the given use case and, in response, providing an indication of the at least one policy to the analytics consumer.

[0034] The performance evaluation metric may include at least one of precision, accuracy, recall, f1 score, mean square error, mean absolute error, and root mean square error.

[0035] The method may include requesting data for training the machine learning model from at least one data source, receiving data for training the machine learning model from the at least one data source, determining whether the data for training the machine learning model allows the performance evaluation metric to be met, and if so, using the data to train the machine learning model, and if not, requesting further data for training the machine learning model from the at least one data source.

[0036] In a fifth aspect, a device is provided, comprising: at least one processor and at least one memory comprising computer program code, wherein the at least one memory and the computer program code are configured to, together with the at least one processor, enable the device to at least: determine a behavioral requirement strategy of a machine learning model for a given use case, provide an indication of the behavioral requirement strategy to an analysis generator; and receive performance evaluation metrics determined based on the behavioral requirement strategy from the analysis generator.

[0037] The apparatus may be configured to receive, from the analysis generator, a performance evaluation metric value or a list of performance evaluation metric values ​​determined based on the behavior requirement policy.

[0038] The indication of the behaviour requirement policy may include an indication of at least one associated performance assessment metric.

[0039] The apparatus may be configured to receive, from the analytics generator, an indication of at least one behavioral requirement policy associated with the machine learning model for a given use case.

[0040] The apparatus may be configured to receive, in a broadcast message from the analytics generator, an indication of at least one behavioral requirement policy associated with the machine learning model for a given use case.

[0041] The apparatus may be configured to provide a request to the analytics generator for at least one policy associated with the machine learning model for the given use case, and in response receive an indication of the at least one policy from the analytics generator.

[0042] The performance evaluation metric may include at least one of precision, accuracy, recall, f1 score, mean square error, mean absolute error, and root mean square error.

[0043] In a sixth aspect, a device is provided, comprising: at least one processor and at least one memory comprising computer program code, wherein the at least one memory and the computer program code are configured to, together with the at least one processor, enable the device to at least: receive an indication of a behavioral requirement strategy for a given use case from an analysis consumer, determine a performance evaluation metric based on the indication, and provide the determined performance evaluation metric to the analysis consumer.

[0044] The apparatus may be configured to provide the analytics consumer with a performance evaluation metric determined based on the behavioral requirement policy.

[0045] The indication of the behaviour requirement policy may include an indication of at least one associated performance assessment metric.

[0046] The apparatus may be configured to provide, to the analytics consumer, an indication of at least one behavioral requirement policy associated with the machine learning model for a given use case.

[0047] The apparatus may be configured to provide, in a broadcast message, to the analytics consumer an indication of at least one performance requirement policy associated with the machine learning model for a given use case.

[0048] The apparatus may be configured to receive a request from the analytics consumer for the at least one policy associated with the machine learning model for the given use case and, in response, provide an indication of the at least one policy to the analytics consumer.

[0049] The performance evaluation metric may include at least one of precision, accuracy, recall, f1 score, mean square error, mean absolute error, and root mean square error.

[0050] The apparatus may be configured to request data for training the machine learning model from at least one data source, receive data for training the machine learning model from the at least one data source, determine whether the data for training the machine learning model allows the performance evaluation metric to be met, and if so, train the machine learning model using the data, and if not, request further data for training the machine learning model from the at least one data source.

[0051] In a seventh aspect, a computer-readable medium comprising program instructions is provided for causing an apparatus to perform at least the following: determining a behavioral requirement strategy for a machine learning model for a given use case; providing an indication of the behavioral requirement strategy to an analysis generator; and receiving performance evaluation metrics determined based on the behavioral requirement strategy from the analysis generator.

[0052] The apparatus may be caused to receive, from the analysis generator, a performance evaluation metric value or a list of performance evaluation metric values ​​determined based on the behavior requirement policy.

[0053] The indication of the behaviour requirement policy may include an indication of at least one associated performance assessment metric.

[0054] The apparatus may be caused to perform receiving, from the analytics generator, an indication of at least one behavioral requirement policy associated with the machine learning model for a given use case.

[0055] The apparatus may be caused to perform receiving, from the analytics generator, an indication of at least one behavioral requirement policy associated with the machine learning model for a given use case in a broadcast message.

[0056] The apparatus may be caused to execute a request to provide the analysis generator with at least one policy associated with the machine learning model for the given use case, and in response receive an indication of the at least one policy from the analysis generator.

[0057] The performance evaluation metric may include at least one of precision, accuracy, recall, f1 score, mean square error, mean absolute error, and root mean square error.

[0058] In an eighth aspect, a computer-readable medium comprising program instructions is provided for causing an apparatus to perform at least the following: receiving an indication of a behavioral requirement strategy for a given use case from an analysis consumer; determining a performance evaluation metric based on the indication; and providing the determined performance evaluation metric to the analysis consumer.

[0059] The apparatus may be caused to perform providing the analytics consumer with a performance evaluation metric determined based on the behavioral requirement policy.

[0060] The indication of the behaviour requirement policy may include an indication of at least one associated performance assessment metric.

[0061] The apparatus may be caused to provide an indication to the analytics consumer of at least one behavioral requirement policy associated with the machine learning model for a given use case.

[0062] The apparatus may be caused to provide, in a broadcast message, to the analytics consumer an indication of at least one performance requirement policy associated with the machine learning model for a given use case.

[0063] The apparatus may be caused to perform receiving a request from the analytics consumer for at least one policy associated with the machine learning model for a given use case and, in response, providing an indication of the at least one policy to the analytics consumer.

[0064] The performance evaluation metric may include at least one of precision, accuracy, recall, f1 score, mean square error, mean absolute error, and root mean square error.

[0065] The device can be caused to request data for training the machine learning model from at least one data source, receive data for training the machine learning model from the at least one data source, determine whether the data for training the machine learning model allows the performance evaluation metric to be met, and if so, use the data to train the machine learning model, and if not, request further data for training the machine learning model from the at least one data source.

[0066] In a ninth aspect, there is provided a non-transitory computer-readable medium comprising program instructions for causing an apparatus to at least perform the method according to the third or fourth aspect.

[0067] In the above, many different embodiments have been described. It should be understood that further embodiments can be provided by combining any two or more of the above embodiments. BRIEF DESCRIPTION OF THE DRAWINGS

[0068] Embodiments will now be described, by way of example only, with reference to the accompanying drawings, in which:

[0069] Figure 1 A schematic diagram of an example 5GS communication system is shown;

[0070] Figure 2 A schematic diagram of an example mobile communication device is shown;

[0071] Figure 3 A schematic diagram of an example control device is shown;

[0072] Figure 4 A flow chart showing a method according to an example embodiment is shown;

[0073] Figure 5 A flow chart showing a method according to an example embodiment is shown;

[0074] Figure 6 A signaling flow according to an example embodiment is shown. DETAILED DESCRIPTION

[0075] Before explaining the examples in detail, refer to Figures 1 to 3 Certain general principles of wireless communication systems and mobile communication devices are briefly explained to aid understanding of the technology on which the described examples are based.

[0076] An example of a suitable communication system is a 5G system (5GS). The network architecture in 5GS may be similar to LTE-Advanced. The base stations of the NR system may be referred to as next generation Node Bs (gNBs). Changes to the network architecture may depend on the need to support various radio technologies and more refined QoS support, as well as some on-demand requirements such as QoS levels to support QoE from the user's perspective. Network-aware services and applications, as well as service- and application-aware networks, may bring changes to the architecture. These are related to the Information-Centric Network (ICN) and User-Centric Content Delivery Network (UC-CDN) approaches. NR may use multiple-input multiple-output (MIMO) antennas, more base stations or nodes than LTE (the so-called small cell concept), including macro sites operating in collaborative operation with smaller stations, and may also employ various radio technologies to achieve better coverage and enhanced data rates.

[0077] 5G networks may utilize network function virtualization (NFV), a network architecture concept that proposes virtualizing network node functions into "building blocks" or entities that can be operatively connected or linked together to provide services. A virtualized network function (VNF) may include one or more virtual machines that use standard or general-purpose servers rather than custom hardware to run computer program code. Cloud computing or data storage may also be utilized. In radio communications, this may mean that node operations are performed at least in part in a server, host, or node coupled to a remote radio head. Node operations may also be distributed among multiple servers, nodes, or hosts. It should also be understood that the allocation of labor between core network operations and base station operations may be different from LTE or even non-existent.

[0078] Figure 1 A schematic diagram of a 5G system (5GS) 100 is shown. The 5GS may include a user equipment (UE) 102 (also referred to as a communication device or terminal), a 5G radio access network (5GRAN) 104, a 5G core network (5GCN) 106, one or more application functions (AFs) 108, and one or more data networks (DNs) 110.

[0079] An example 5G core network (CN) includes functional entities. The 5GCN 106 may include one or more access and mobility management functions (AMFs) 112, one or more session management functions (SMFs) 114, an authentication server function (AUSF) 116, a unified data management (UDM) 118, one or more user plane functions (UPFs) 120, a unified data repository (UDR) 122, and / or a network exposure function (NEF) 124. The UPF is controlled by the SMF (session management function), which receives policies from the PCF (policy control function).

[0080] The CN is connected to the terminal device via a radio access network (RAN). The 5G RAN may include one or more gNodeB (GNB) distributed unit functions connected to one or more gNodeB (GNB) centralized unit functions. The RAN may include one or more access nodes.

[0081] The role of UPF (User Plane Function) is called PSA (Protocol Data Unit (PDU) Session Anchor) and can be responsible for forwarding frames back and forth between DN (Data Network) and the tunnel established on 5G towards the UE that exchanges traffic with the DN.

[0082] Now refer to Figure 2 describing in more detail possible mobile communication devices, Figure 2A schematic partial cross-sectional view of a communication device 200 is shown. Such a communication device is generally referred to as a user equipment (UE) or a terminal. Any device capable of sending and receiving radio signals can provide a suitable mobile communication device. Non-limiting examples include a mobile station (MS) or a mobile device (such as a mobile phone or a so-called "smart phone"), a computer equipped with a wireless interface card or other wireless interface facilities (e.g., a USB dongle), a personal data assistant (PDA) or a tablet computer equipped with a wireless communication function, an IP voice (VoIP) phone, a portable computer, a desktop computer, an image capture terminal device (such as a digital camera), a game terminal device, a music storage and playback device, a vehicle-mounted wireless terminal device, a wireless endpoint, a mobile station, a laptop embedded device (LEE), a laptop mounted device (LME), a smart device, a wireless client device (CPE), or any combination of these or the like. A mobile communication device can provide, for example, data communications for carrying communications such as voice, email (email), text messages, multimedia, etc. Therefore, users can supply and provide a variety of services via their communication devices. Non-limiting examples of these services include two-way or multi-way calls, data communications or multimedia services, or simply access to a data communications network system such as the Internet. Broadcast or multicast data may also be provided to the user. Non-limiting examples of content include downloads, television and radio programs, videos, advertisements, various alerts, and other information.

[0083] The mobile device is typically equipped with at least one data processing entity 201, at least one memory 202 and other possible components 203 for software and hardware assistance to perform the tasks it is designed to perform, including control of access and communication to access systems and other communication devices. Data processing, storage and other related control devices can be arranged on appropriate circuit boards and / or in chipsets. This feature is represented by reference numeral 204. The user can control the operation of the mobile device with the help of an appropriate user interface (such as a keyboard 205, voice commands, a touch screen or touchpad, a combination thereof, etc.). It can also be equipped with a display 208, a speaker and a microphone. In addition, the mobile communication device may include appropriate connectors (wired or wireless) for connecting to other devices and / or connecting external accessories (such as hands-free devices) to the device.

[0084] The mobile device 200 may receive signals over the air or radio interface 207 via appropriate means for receiving, and may transmit signals via appropriate means for transmitting radio signals. Figure 2 In the mobile device, the transceiver arrangement is schematically represented by block 206. The transceiver arrangement 206 may be provided, for example, by means of a radio component and an associated antenna arrangement. The antenna arrangement may be arranged internally or externally to the mobile device.

[0085] Figure 3 An example of a control device 300 of a communication system is shown, for example, coupled to and / or used to control a station (such as a RAN node, such as a base station, eNB or gNB, a relay node or a core network node such as an MME or S-GW or P-GW) of an access system, or a core network function such as an AMF / SMF, or a server or host. The method can be implemented in a single control device or multiple control devices. The control device can be integrated with a node or module of a core network or RAN, or external to a node or module of a core network or RAN. In some embodiments, the base station includes a separate control device unit or module. In other embodiments, the control device can be other network elements, such as a radio network controller or a spectrum controller. In some embodiments, each base station can have such a control device and a control device provided in a radio network controller. The control device 300 can be arranged to provide control for communications in a service area of ​​the system. The control device 300 includes at least one memory 301, at least one data processing unit 302, 303, and an input / output interface 304. Via the interface, the control device can be coupled to a receiver and a transmitter of the base station. The receiver and / or the transmitter may be implemented as a radio front end or a remote radio head.

[0086] AI / ML model training is performed for various optimization and prediction use cases. The training process attempts to obtain the best possible model for a given task. This is achieved based on the various performance evaluation metrics selected for the model during the training process. The hyperparameters of each model are fine-tuned to obtain the best performance evaluation metric score. The choice of performance evaluation metric and the selection of the best score for that performance evaluation metric play a key role in the performance of the trained model for a given use case.

[0087] An example of a performance evaluation metric for an AI / ML classification model is accuracy. Accuracy is a metric that gives the proportion of true predictions out of the total number of observed predictions. However, accuracy is a useful metric only if the dataset is completely symmetrical, i.e., the number of false positives and false negatives is almost the same. Therefore, other performance evaluation metrics need to be considered when evaluating models.

[0088] Precision is a measure of the ratio of correctly predicted positive observations to the total predicted positive observations. This is a useful measure if the cost of false positives (FP) is high.

[0089] Recall is a measure of how many actual positives are captured in the model. It is a useful measure if the cost of missing false negatives (FN) is high.

[0090] Finally, the F-measure is a weighted average of precision and recall, and it is used when the classes are not evenly distributed.

[0091] An example of a performance evaluation metric for the AI / ML regression case is the mean absolute error (MAE). MAE estimates the average magnitude of the error in a set of predictions, regardless of its direction (i.e., the average of the absolute values ​​of the differences between the predictions and the corresponding observations).

[0092] The mean squared error (MSE) measures the average of the squared errors (i.e. the average squared difference between the estimated and actual values).

[0093] The root mean square error (RMSE) is a quadratic scoring rule that measures the average magnitude of the errors (i.e., the differences between the estimated and actual values ​​are each squared and then averaged over the sample). The square root of the average is then estimated. Given that the errors are squared before being averaged, the RMSE adds a relatively high weight to large errors. Therefore, the RMSE is most useful when large errors are undesirable. The RMSE is always greater than or equal to the MAE, and the greater the difference between them, the higher the variance of the individual errors in the sample. If the RMSE is equal to the MAE, then all errors have the same magnitude.

[0094] The R2 score (coefficient of determination) represents the coefficient of how well these values ​​fit compared to the original values. The values ​​from 0 to 1 are interpreted as a percentage. The R2 score measures the proportion of data variation that is taken into account in the model. It is equivalent to the accuracy measure in classification problems.

[0095] A Management Service (MnS) consumer requests AI / ML model training from an MnS producer. When training an AI / ML model, the MnS producer collects the necessary data, trains the AI / ML model corresponding to the requested performance evaluation metric, and then deploys the AI / ML model for inference. The MnS producer notifies or responds to the MnS consumer with information related to AI / ML model training.

[0096] The MnS producer trains the AI / ML model (e.g., by selecting relevant hyperparameters and / or adjusting decision thresholds) to optimize the performance evaluation metric selected by the MnS consumer.

[0097] Different use cases can have different behavioral requirements. For example, for regression / time series techniques, a particular use case may be more sensitive to over-prediction of values, while another use case may be more sensitive to under-prediction of values.

[0098] For example, in virtual network function (VNF) autoscaling, if the goal of the MnS consumer is to minimize cost, then training (or fine-tuning) the AI / ML model to minimize false positives (i.e., to limit over-provisioning of resources, which in turn leads to increased operating costs) would be useful, while if the goal of the MnS consumer is to maximize QoS, then training (or fine-tuning) the AI / ML model to minimize false negatives (i.e., to limit under-provisioning of resources, which in turn leads to degraded QoS performance) would be more appropriate.

[0099] Therefore, MnS consumers expect a way to provide behavioral requirements where trained AI / ML models should adhere to the objectives that depend on VNF autoscaling, such as policy (false positives, low), policy (false negatives, low), policy (false positives, false negatives, balanced)).

[0100] Currently, MnS consumers are unable to provide customized (i.e., use case-specific) behavioral requirements for, e.g., false positives, false negatives, and accuracy for classification problems, or MSE, MAE, RMSE, over-prediction, and under-prediction in regression / time series problems, for MnS producers to satisfy during AI / ML model training.

[0101] Currently, MnS producers cannot choose by themselves (i.e., depending on the use case for which the AI / ML model is trained) the most appropriate performance evaluation metric (e.g., when the cost of false positives is high, precision is a good evaluation and reporting metric, and when the cost of FN is high, recall is a good evaluation and reporting metric) which should be reported to MnS consumers.

[0102] Figure 4 A flow chart of a method according to an example embodiment is shown.The method may be performed at an analysis consumer (eg, an MnS consumer such as a UE or any 3GPP management entity that will consume the AIML training service).

[0103] In S1, the method includes determining a behavioral requirement strategy for a machine learning model for a given use case.

[0104] In S2, the method comprises providing an indication of a behavior requirement policy to an analytics generator.

[0105] In S3, the method includes receiving, from an analytics generator, a performance evaluation metric determined based on the behavioral requirement policy.

[0106] Figure 5 A flow chart of a method according to an example embodiment is shown. The method may be performed at an analysis generator (eg, a MnS producer).

[0107] At T1 , the method includes receiving, from an analytics consumer, an indication of a behavioral requirement strategy for a given use case.

[0108] At T2, the method includes determining a performance assessment metric based on the indication.

[0109] At T3, the method includes providing the determined performance evaluation metric to an analytics consumer.

[0110] The indication of a behavioral requirement policy provided from an analytics consumer to an analytics generator may include an indication of at least one associated performance assessment metric.

[0111] In an example embodiment, the consumer indicates the desired behavior-related requirements to the producer in an AIMLT training request. The indication may capture different possible evaluation metrics for a given training case.

[0112] The approach enables analytics consumers and producers to indicate the preferred and used performance evaluation metrics, respectively, for a given use case.

[0113] The method may include receiving, from an analytics generator, a performance assessment metric value or a list of performance assessment metric values ​​determined based on a performance requirement policy.

[0114] Examples of behavioral requirement policies may include, but are not limited to, the following:

[0115] Classification:

[0116] "False positives, low"

[0117] "False positives, high"

[0118] "False negative, low"

[0119] "False negative, high"

[0120] return:

[0121] "Over-predicted, high"

[0122] “Over-forecast, under-forecast”

[0123] "Under-prediction, high"

[0124] "Under-prediction, low"

[0125] The performance evaluation metric may include, but is not limited to, at least one of precision, accuracy, recall, f1 score, MSE, MAE, and RMSE.

[0126] The approach provides a mechanism that enables MnS consumers to provide customized (in other words, use case specific) behavioral requirements to MnS producers. During AI / ML model training, the MnS producers take those requirements into account.

[0127] MnS producers can determine the most relevant / useful performance evaluation metric (e.g., precision is a good evaluation and reporting metric when false positive cost is high, recall is a good evaluation and reporting metric when FN cost is high) that should be reported to MnS consumers, and subsequently report on them, based on the use case for which they train the AI / ML models and the associated behavioral requirements.

[0128] In the case of classification, some use cases can be sensitive to false positives while others can be sensitive to false negatives. In the case of regression or time series scenarios, some use cases can be sensitive to over-predicted values ​​while others can be sensitive to under-predicted values.

[0129] Other use cases may require a balanced metric for optimal results.

[0130] Analytics generators can leverage information from consumers to choose the right evaluation metric for training and fine-tuning models, and report the chosen performance metric and the resulting model's score.

[0131] The method may include: at an analysis generator, requesting data for training a machine learning model from at least one data source; receiving data for training a machine learning model from at least one data source; determining whether the data for training the machine learning model allows a performance evaluation metric to be satisfied; and if so, using the data to train the machine learning model, and if not, requesting further data for training the machine learning model from at least one data source.

[0132] The method may include receiving, at an analytics consumer from an analytics generator, an indication of at least one behavioral requirement policy associated with a machine learning model for a given use case.

[0133] An indication of at least one behavior requirement policy associated with a machine learning model for a given use case may be provided to an analytics consumer in a broadcast message.

[0134] Alternatively or additionally, the method may include providing a request from the analytics consumer to the analytics generator for at least one policy associated with the machine learning model for a given use case, and in response receiving at the analytics consumer an indication of the at least one policy from the analytics generator.

[0135] Figure 6 The signaling flow between an AI / ML Training (AIMLT) MnS consumer, an AIMLT MnS producer, and a data source is shown according to an example embodiment.

[0136] Steps 1 to 3 are optional.

[0137] In step 1, the AIMLT MnS producer periodically broadcasts the strategy of available models to the AIMLT MnS consumers.

[0138] In step 2, the AIMLT MnS consumer requests available strategies for trained and available ML models for a given use case from the AIMLT MnS producer.

[0139] In step 3, the AIMLT MnS producer responds to the AIMLT MnS consumer using available strategies from available trained ML models for the given use case.

[0140] In step 4, the AIMLT MnS consumer identifies the use case and the appropriate behavior and training requirements as the strategy for the identified use case (alternatively, the consumer may select the strategy indicated by the producer in steps 1 or 3).

[0141] Behavioral requirements can be indicated as sensitivity reasoning for a given use case as a policy. For example, for a classification use case, a behavioral requirement can be policy (“false positives, low”) or policy (“false negatives, low”) or policy (“false positives, false negatives, balanced”). For a regression / time series use case, a behavioral requirement can be policy (“overprediction, low”) or policy (“underprediction, low”).

[0142] In step 5, the AIMLT MnS consumer indicates a request for AIML model training to the AIMLT MnS producer, along with the behavior and training requirements identified in step 4 as policies.

[0143] In step 6, based on the policy indicated by the AIMLT MnS Consumer, the AIMLT MnS Producer requests the appropriate data from the data source.

[0144] In step 7, the data source responds to the AIMLT MnS producer with the requested data.

[0145] The AIMLT MnS Producer evaluates the received data relative to the policy indicated by the AIMLT MnS Consumer in step 8. If the data is not of the required quality to meet the policy indicated by the AIMLT MnS Consumer, the AIMLT MnS Producer requests the data source to provide data with the required quality.

[0146] Steps 6, 7 and 8 are executed in a loop until the AIMLT MnS training producer is provided with the correct quality of data to train the model for the indicated strategy.

[0147] After receiving data of the expected quality that meets the requested strategy, the AIMLT MnS training producer starts the training of the model in step 9. The producer also selects an appropriate evaluation metric based on the requested strategy.

[0148] During training, the indicated strategy also affects the determination of the thresholds for the chosen evaluation metric.

[0149] In step 10, the AIMLT MnS consumer provides a training report indicating the selected evaluation metrics and the corresponding performance scores.

[0150] For classification use cases, the performance evaluation metric can be (but not limited to) "precision", "recall" or "F1 score". For regression / time series use cases, the performance evaluation metric can be (but not limited to) "MSE", "MAE" or "RMSE".

[0151] For example, a use case could be mobility robustness optimization, and the reasoning is a classification problem to identify whether a call handoff from a source to a target gNB will succeed, and the source gNB is resource-rich while the target gNB has constrained resources. In this case, it is acceptable for the source gNB to hold the call longer before handoff, as the target gNB has constrained resources and thus the probability of HO failure is greater. This use case is sensitive to false positives, as the probability of a call drop increases if the model incorrectly decides to perform a HO.

[0152] In this case, the behavioral requirements would indicate that this use case is sensitive to high false positive inference. The AIMLT MnS producer uses these requirements to determine an appropriate evaluation metric, such as the accuracy of training. The strategy selected for this example scenario is strategy (false positives, high).

[0153] For example, in this case, to minimize false positives, the decision boundary should not be 0.5 (as in the case of regular binary classification) but should be greater than 0.5.

[0154] like Figure 5 As shown, in step 8 of the signal flow, the data needs to be adequately represented with false positives so that the training process can focus on minimizing the false positive inference of the model. In the case of insufficient false positive representation, the AIMLT MnS producer will request more data from the data source.

[0155] The apparatus may include: a device for determining a behavioral requirement strategy for a machine learning model for a given use case, a device for providing an indication of the behavioral requirement strategy to an analysis generator, and a device for receiving performance evaluation metrics determined based on the behavioral requirement strategy from the analysis generator.

[0156] Alternatively or additionally, the apparatus may include means for receiving an indication of a behavioral requirement strategy for a given use case from an analytics consumer, means for determining a performance evaluation metric based on the indication, and means for providing the determined performance evaluation metric to the analytics consumer.

[0157] It should be understood that the devices may include or be coupled to other units or modules for transmitting and / or receiving, such as radio components or radio heads. Although the devices have been described as one entity, the different modules and memories may be implemented in one or more physical or logical entities.

[0158] It is worth noting that although some embodiments related to 5G networks have been described, similar principles can be applied to other related networks and communication systems. Therefore, although certain embodiments are described above by way of example with reference to certain example architectures of wireless networks, technologies and standards, the embodiments can be applied to any other suitable form of communication system other than those shown and described herein.

[0159] It is also noted herein that while the above describes example embodiments, there are several variations and modifications which may be made to the disclosed solution without departing from the scope of the present invention.

[0160] In general, various embodiments may be implemented in hardware or dedicated circuits, software, logic, or any combination thereof. Certain aspects of the present application may be implemented in hardware, while other aspects may be implemented in firmware or software that may be executed by a controller, microprocessor, or other computing device, but the present application is not limited thereto. Although various aspects of the present application may be illustrated and described by block diagrams, flow charts, or using some other graphical representations, it is well understood that the blocks, devices, systems, techniques, or methods described herein may be implemented, as non-limiting examples, in hardware, software, firmware, dedicated circuits or logic, general-purpose hardware or controllers or other computing devices, or some combination thereof.

[0161] As used in this application, the term "circuitry" may refer to one or more or all of the following:

[0162] (a) hardware-only circuit implementations (such as implementations in analog and / or digital circuits only); and

[0163] (b) a combination of hardware circuitry and software such as (where applicable):

[0164] (i) a combination of analog and / or digital hardware circuitry and software / firmware, and

[0165] (ii) any portion of a hardware processor and software (including a digital signal processor), software and memory that work together to enable a device such as a mobile phone or server to perform various functions;

[0166] (c) A hardware circuit and / or processor, such as a microprocessor or portion of a microprocessor, that requires software (e.g., firmware) to operate, but the software may not be present when not required to operate.

[0167] This definition of circuitry applies to one or all uses of the term in this application, including in any claims. As a further example, as used in this application, the term "circuitry" also covers an implementation of only a hardware circuit or processor (or multiple processors) or a portion of a hardware circuit or processor and its (or their) accompanying software and / or firmware. For example, if applicable to a particular claim, the term "circuitry" also covers a baseband integrated circuit or processor integrated circuit for a mobile device, or a similar integrated circuit in a server, cellular network device, or other computing or network device.

[0168] Embodiments of the present application may be implemented (such as in a processor entity) by computer software executable by a data processor of a mobile device, or by hardware, or by a combination of software and hardware. Computer software or programs, also referred to as program products, include software routines, applets and / or macros, which may be stored in a data storage medium readable by any device, and they include program instructions for performing specific tasks. A computer program product may include one or more computer executable components that are configured to perform an embodiment when the program is running. One or more computer executable components may be at least one software code or a portion thereof.

[0169] Also in this regard, it should be noted that any block in the logic flow of the Figures may represent program steps, or interconnected logic circuits, blocks and functions, or a combination of program steps and logic circuits, blocks and functions. The software may be stored on such physical media as memory chips, or memory blocks implemented within a processor, magnetic media such as hard disks or floppy disks, and optical media such as DVDs and their data variants, CDs. Physical media are non-transitory media.

[0170] The memory may be of any type suitable for the local technical environment and may be implemented using any suitable data storage technology, such as semiconductor-based storage devices, magnetic storage devices and systems, optical storage devices and systems, fixed memory, and removable memory. The data processor may be of any type suitable for the local technical environment and may include one or more of a general-purpose computer, a special-purpose computer, a microprocessor, a digital signal processor (DSP), an application-specific integrated circuit (ASIC), an FPGA, a gate-level circuit based on a multi-core processor architecture, and a processor, as non-limiting examples.

[0171] Embodiments of the present application may be practiced in various components such as integrated circuit modules. The design of integrated circuits is a highly automated process. Complex and powerful software tools are available to convert logic-level designs into semiconductor circuit designs that can be etched and formed on semiconductor substrates.

[0172] The scope of protection sought by various embodiments of the present application is set forth by the independent claims. Embodiments and features described in this specification that do not fall within the scope of the independent claims (if any) should be interpreted as examples of various embodiments that help understand the content of the present application.

[0173] The foregoing description has provided by way of non-limiting examples a complete and informative description of exemplary embodiments of the present application. However, various modifications and adaptations will be apparent to those skilled in the relevant art when read in conjunction with the accompanying drawings and the appended claims. However, all such and similar modifications to the teachings of the present application will still fall within the scope of the present invention as defined by the appended claims. In fact, there are further embodiments that include a combination of one or more embodiments with any other embodiments previously discussed.

Claims

1. A device comprising: means for determining a behavioral requirement strategy for a machine learning model for a given use case; means for providing said behavior requirement policy indication to an analysis generator; as well as Means for receiving, from the analytics generator, a performance evaluation metric determined based on the behavioral requirements policy.

2. The device according to claim 1, comprising: Means for receiving, from the analysis generator, a performance evaluation metric value or a list of performance evaluation metric values ​​determined based on the behavioral requirements policy.

3. The device according to claim 1 or 2, wherein: The indication of the behavior requirement policy includes an indication of at least one associated performance evaluation metric.

4. The apparatus of any one of claims 1 to 3, comprising means for: An indication of at least one behavioral requirement policy associated with the machine learning model for a given use case is received from the analytics generator.

5. The device as claimed in claim 4, comprising: Means for receiving, in a broadcast message from the analytics generator, an indication of at least one behavioral requirement policy associated with the machine learning model for a given use case.

6. The apparatus of claim 4, comprising means for: A request is provided to the analytics generator for at least one policy associated with the machine learning model for the given use case, and an indication of the at least one policy is received from the analytics generator in response.

7. The device according to any one of claims 1 to 6, wherein: The performance evaluation metric includes at least one of precision, accuracy, recall, f1 score, mean square error, mean absolute error, and root mean square error.

8. A device comprising: means for receiving from an analytics consumer an indication of a behavioral requirement strategy for a given use case; means for determining a performance evaluation metric based on the indication; as well as Means for providing said determined performance assessment metric to said analytics consumer.

9. The device of claim 8, comprising: Means for providing to the analytics consumer a performance evaluation metric determined based on the behavioral requirements policy.

10. The device according to claim 8 or 9, wherein: The indication of the behavior requirement policy includes an indication of at least one associated performance evaluation metric.

11. Apparatus as claimed in any one of claims 8 to 10, comprising means for: An indication of at least one behavioral requirement policy associated with the machine learning model for a given use case is provided to the analytics consumer.

12. The device of claim 11, comprising: Means for providing, in a broadcast message, to the analytics consumer an indication of at least one performance requirement policy associated with the machine learning model for a given use case.

13. The apparatus of claim 11, comprising means for: A request is received from the analytics consumer for the at least one policy associated with the machine learning model for the given use case, and in response, an indication of the at least one policy is provided to the analytics consumer.

14. The apparatus of any one of claims 8 to 13, wherein: The performance evaluation metric includes at least one of precision, accuracy, recall, f1 score, mean square error, mean absolute error, and root mean square error.

15. The apparatus of any one of claims 8 to 14, comprising: means for requesting data for training the machine learning model from at least one data source; receiving data for training the machine learning model from the at least one data source; Determine whether the data used to train the machine learning model allows performance evaluation metrics to be met; as well as If so, the data is used to train the machine learning model, and if not, further data for training the machine learning model is requested from the at least one data source.

16. A method comprising: Determine the behavioral requirements strategy for the machine learning model for a given use case; providing an indication of said behavior requirement strategy to an analytics generator; as well as A performance evaluation metric determined based on the behavioral requirement policy is received from the analytics generator.

17. A method comprising: receiving from an analytics consumer an indication of a behavior requirement policy for a given use case; determining a performance evaluation metric based on the indication; as well as The determined performance evaluation metrics are provided to the analytics consumer.

18. An apparatus comprising: at least one processor and at least one memory including computer program code, the at least one memory and the computer program code being configured, with the at least one processor, to cause the apparatus to at least: Determine the behavioral requirements strategy for the machine learning model for a given use case; Provides instructions to analytics generators on behavioral requirement strategies; as well as A performance evaluation metric determined based on the behavioral requirement policy is received from the analytics generator.

19. An apparatus comprising: at least one processor and at least one memory including computer program code, the at least one memory and the computer program code being configured, with the at least one processor, to cause the apparatus to at least: receiving from an analytics consumer an indication of a behavior requirement policy for a given use case; determining a performance evaluation metric based on the indication; as well as The determined performance evaluation metrics are provided to the analytics consumer.

20. A computer readable medium comprising program instructions for causing an apparatus to at least perform the following: Determine the behavioral requirements strategy for the machine learning model for a given use case; providing an indication of said behavioral requirement policy to an analytics generator; and A performance evaluation metric determined based on the behavioral requirement policy is received from the analytics generator.

21. A computer readable medium comprising program instructions for causing an apparatus to at least perform the following: receiving from an analytics consumer an indication of a behavior requirement policy for a given use case; determining a performance evaluation metric based on the indication; and The determined performance evaluation metrics are provided to the analytics consumer.

Images