Machine learning performance abstraction

Abstract

Abstraction of machine learning performance includes a first device sending a first abstraction request to a second device, the first abstraction request for at least one performance of a machine learning (ML) entity provided by the first device, the machine learning entity for a third device. A first device receives a first abstract report from a second device, the first abstract report including at least one indicator of at least one performance, the at least one indicator being understandable by a third device. In this manner, artificial intelligence (AI) or machine learning performance metrics are defined and abstracted as indicators that are understandable by AI or ML consumers.

Description

Technical Field

[0001] Various example embodiments of the present disclosure relate generally to the field of telecommunications, and in particular to methods, devices, apparatus, and computer-readable storage media for machine learning (ML) performance abstraction. Background Art

[0002] Cognitive Autonomous Networking (CAN) aims to provide intelligence and autonomy in network operations, administration and management (OAM) and network processes to support the increased flexibility and complexity of radio networks. To realize cognitive functions in CAN networks, ML and artificial intelligence (AI) technologies play a key role. ML-enabled network or management functions provide ML-based management services (MnS), such as training or inference MnS, to AI / ML MnS consumers.

[0003] MnS consumers may be interested in the performance of ML applications (e.g., ML Apps) included in ML-enabled functions. In addition, MnS consumers may also want to know the performance achievements of AI / ML applications as measured on different performance metrics (e.g., for accuracy, confidence, speed, resource consumption, etc.). Summary of the invention

[0004] In a first aspect of the present disclosure, a first device is provided. The first device includes: at least one processor; and at least one memory storing instructions, which, when executed by the at least one processor, causes the first device to at least: send a first abstract request to a second device, the first abstract request being for at least one performance of a machine learning entity provided by the first device, the machine learning entity being used for a third device; and receive a first abstract report from the second device, the first abstract report including at least one indicator of at least one performance, the at least one indicator being understandable by the third device.

[0005] In a second aspect of the present disclosure, a second device is provided. The second device includes: at least one processor; and at least one memory storing instructions, which, when executed by the at least one processor, causes the second device to at least: receive an abstract request from one of the first device or the third device, the abstract request being for at least one performance of a machine learning entity provided by the first device, the machine learning entity being used for the third device; determine at least one indicator of the at least one performance, the at least one indicator being understandable by the third device; and send an abstract report to a corresponding one of the first device or the third device, the abstract report including at least one indicator of the at least one performance.

[0006] In a third aspect of the present disclosure, a third device is provided. The third device includes: at least one processor; and at least one memory storing instructions, which, when executed by the at least one processor, causes the third device to at least: send an abstract request to one of the first device or the second device, the abstract request being for at least one performance of a machine learning entity provided by the first device, the machine learning entity being used for the third device; and receive an abstract report from a corresponding one of the first device or the second device, the abstract report including at least one indicator of at least one performance, the at least one indicator being understandable by the third device.

[0007] In a fourth aspect of the present disclosure, a method is provided. The method includes: sending a first abstract request at a first device to a second device, the first abstract request being for at least one performance of a machine learning entity provided by the first device, the machine learning entity being used for a third device; and receiving a first abstract report from the second device, the first abstract report including at least one indicator of the at least one performance, the at least one indicator being understandable by the third device.

[0008] In a fifth aspect of the present disclosure, a method is provided. The method includes: receiving, at a second device, an abstract request from one of a first device or a third device, the abstract request for at least one performance of a machine learning entity provided by the first device, the machine learning entity being used for the third device; determining at least one indicator of the at least one performance, the at least one indicator being understandable by the third device; and sending an abstract report to a corresponding one of the first device or the third device, the abstract report including at least one indicator of the at least one performance.

[0009] In a sixth aspect of the present disclosure, a method is provided. The method includes: sending an abstract request at a third device to one of a first device or a second device, the abstract request being for at least one performance of a machine learning entity provided by the first device, the machine learning entity being used by the third device; and receiving an abstract report from a corresponding one of the first device or the second device, the abstract report including at least one indicator of at least one performance, the at least one indicator being understandable by the third device.

[0010] In a seventh aspect of the present disclosure, a first device is provided. The first device includes: a component for sending a first abstract request to a second device, the first abstract request is for at least one performance of a machine learning entity provided by the first device, the machine learning entity is used for a third device; and a component for receiving a first abstract report from the second device, the first abstract report including at least one indicator of at least one performance, the at least one indicator being understandable by the third device.

[0011] In an eighth aspect of the present disclosure, a second device is provided. The second device includes: a component for receiving an abstract request from one of a first device or a third device, the abstract request being for at least one performance of a machine learning entity provided by the first device, the machine learning entity being used for the third device; a component for determining at least one indicator of the at least one performance, the at least one indicator being understandable by the third device; and a component for sending an abstract report to a corresponding one of the first device or the third device, the abstract report including at least one indicator of the at least one performance.

[0012] In a ninth aspect of the present disclosure, a third device is provided. The third device includes: a component for sending an abstract request to one of the first device or the second device, the abstract request is for at least one performance of a machine learning entity provided by the first device, the machine learning entity is used for the third device; and a component for receiving an abstract report from a corresponding one of the first device or the second device, the abstract report including at least one indicator of at least one performance, the at least one indicator being understandable by the third device.

[0013] In a tenth aspect of the present disclosure, a computer-readable medium is provided, wherein the computer-readable medium includes instructions stored thereon for causing a device to at least execute the method according to the fourth aspect.

[0014] In an eleventh aspect of the present disclosure, a computer-readable medium is provided, wherein the computer-readable medium includes instructions stored thereon for causing a device to at least execute the method according to the fifth aspect.

[0015] In a twelfth aspect of the present disclosure, a computer-readable medium is provided, wherein the computer-readable medium includes instructions stored thereon for causing a device to at least execute the method according to the sixth aspect.

[0016] It should be understood that the invention summary is not intended to identify the key or essential features of the embodiments of the present disclosure, nor is it intended to be used to limit the scope of the present disclosure. Other features of the present disclosure will become readily understood through the following description. BRIEF DESCRIPTION OF THE DRAWINGS

[0017] Some example embodiments will now be described with reference to the accompanying drawings, in which:

[0018] Figure 1 An example communication environment is shown in which example embodiments of the present disclosure may be implemented;

[0019] Figure 2 shows a signaling diagram for communication according to some example embodiments of the present disclosure;

[0020] Figure 3 shows a signaling diagram for communication according to some example embodiments of the present disclosure;

[0021] Figure 4 An example of an information model for ML performance abstraction control according to some example embodiments of the present disclosure is shown;

[0022] Figure 5 A schematic diagram showing an ML performance abstract inheritance relationship according to some example embodiments of the present disclosure;

[0023] Figure 6 A flowchart showing a method implemented at a first device according to some example embodiments of the present disclosure is shown;

[0024] Figure 7 A flowchart showing a method implemented at a second device according to some example embodiments of the present disclosure is shown;

[0025] Figure 8 A flowchart of a method implemented at a third device according to some example embodiments of the present disclosure is shown;

[0026] Fig. 9 shows a simplified block diagram of a device suitable for implementing an example embodiment of the present disclosure; and

[0027] Fig.10 A block diagram of an example computer-readable medium is shown according to some example embodiments of the present disclosure.

[0028] Throughout the drawings, the same or similar reference numerals refer to the same or similar elements. DETAILED DESCRIPTION

[0029] The principles of the present disclosure will now be described with reference to some example embodiments. It should be understood that these embodiments are described only for illustrative purposes and help those skilled in the art to understand and implement the present disclosure without implying any limitation on the scope of the present disclosure. In addition to the methods described below, the embodiments described herein can be implemented in various ways.

[0030] In the following description and claims, unless defined otherwise, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this disclosure belongs.

[0031] References in this disclosure to "one embodiment," "an embodiment," "an example embodiment," etc., indicate that the described embodiment may include a particular feature, structure, or characteristic, but not every embodiment necessarily includes the particular feature, structure, or characteristic. Furthermore, such phrases do not necessarily refer to the same embodiment. Furthermore, when a particular feature, structure, or characteristic is described in conjunction with an embodiment, it is considered to be within the knowledge of those skilled in the art to affect such feature, structure, or characteristic in conjunction with other embodiments (whether or not explicitly described).

[0032] It should be understood that although the terms "first", "second", etc. may be used herein to describe various elements, these elements should not be limited by these terms. These terms are only used to distinguish one element from another element. For example, a first element may be referred to as a second element, and similarly, a second element may be referred to as a first element without departing from the scope of the exemplary embodiments. As used herein, the term "and / or" includes any and all combinations of one or more of the listed terms.

[0033] As used herein, “at least one of: ” and “at least one of ” and similar expressions (wherein a list of two or more elements is connected by “and” or “or”) refer to at least any one of the elements, or at least any two or more of the elements, or at least all of the elements.

[0034] As used herein, unless explicitly stated, performing a step “in response to A” does not mean performing the step immediately after “A” occurs, but may include one or more intermediate steps.

[0035] The terms used herein are only used for the purpose of describing specific embodiments and are not intended to limit the exemplary embodiments. As used herein, the singular forms "a", "an", and "the" are intended to include the plural forms as well, unless the context clearly indicates otherwise. It will be further understood that the terms "include", "comprise", "have", "contain", "include", and / or "comprising" when used herein specify the presence of stated features, elements, and / or components, etc., but do not exclude the presence or addition of one or more other features, elements, components, and / or combinations thereof.

[0036] As used in this application, the term "circuitry" may refer to one, more, or all of the following: (a) hardware circuit implementations only (such as implementations within analog and / or digital circuitry only) and (b) a combination of hardware circuitry and software such as (where applicable): (i) a combination of analog and / or digital hardware circuitry and software / firmware, and (ii) any portion of a hardware processor with software (including a digital signal processor, software and memory that work together to enable a device such as a mobile phone or server to perform various functions) and (c) a hardware circuit and / or processor that requires software (e.g., firmware) for operation, such as a microprocessor or a portion of a microprocessor, but where the software may not be present when not required for operation.

[0037] This definition of circuitry applies to all uses of the term in this application, including any claims. As another example, as used in this application, the term circuitry also covers an implementation of only a hardware circuit or processor (or multiple processors) or a portion of a hardware circuit or processor and its (or their) accompanying software and / or firmware. For example and if applicable to a particular claim element, the term circuitry also covers a baseband integrated circuit or processor integrated circuit for a mobile device, or a similar integrated circuit in a server, cellular network device, or other computing or network device.

[0038] As used herein, the term "communication network" refers to a network that complies with any suitable communication standard, such as New Radio (NR), Long Term Evolution (LTE), Advanced LTE (LTE-A), Wideband Code Division Multiple Access (WCDMA), High Speed ​​Packet Access (HSPA), Narrowband Internet of Things (NB-IoT), etc. In addition, the communication between the terminal equipment and the network equipment in the communication network can be performed according to any suitable generation communication protocol, including but not limited to the first generation (1G), second generation (2G), 2.5G, 2.75G, third generation (3G), fourth generation (4G), 4.5G, fifth generation (5G) communication protocol and / or any other protocol currently known or developed in the future. The embodiments of the present disclosure can be applied to various communication systems. In view of the rapid development of communication, there will certainly be communication technologies and systems of future types that can embody the present disclosure. It should not be considered that the scope of the present disclosure is limited to the aforementioned system.

[0039] As used herein, the term "network device" refers to a node in a communication network, via which a terminal device accesses the network and receives services from it. A network device may refer to a base station (BS) or an access point (AP), for example, a Node B (Node B or NB), an evolved Node B (eNodeB or eNB), a NR NB (also referred to as a gNB), a remote radio unit (RRU), a radio head (RH), a remote radio head (RRH), a repeater, an integrated access and backhaul (IAB) node, a low-power node (such as a femto, a micro), a non-terrestrial network (NTN) or a non-terrestrial network device (such as a satellite network device, a low earth orbit (LEO) satellite and a geosynchronous earth orbit (GEO) satellite, an aircraft network device, etc.), depending on the terminology and technology applied. In some example embodiments, a radio access network (RAN) split architecture includes a centralized unit (CU) and a distributed unit (DU) at an IAB donor node. An IAB node includes a mobile terminal (IAB-MT) portion that behaves like a UE toward a parent node, and a DU portion of an IAB node behaves like a base station toward a base station of a next-hop IAB node.

[0040] The term "terminal device" refers to any terminal device capable of wireless communication. As an example and not limitation, a terminal device may also be referred to as a communication device, a user equipment (UE), a subscriber station (SS), a portable subscriber station, a mobile station (MS), or an access terminal (AT). The terminal device may include, but is not limited to, a mobile phone, a cellular phone, a smart phone, a voice over IP (VoIP) phone, a wireless local loop phone, a tablet computer, a wearable terminal device, a personal digital assistant (PDA), a portable computer, a desktop computer, an image capture terminal device such as a digital camera, a game terminal device, a music storage and playback device, a vehicle-mounted wireless terminal device, a wireless endpoint, a mobile station, a laptop embedded device (LEE), a laptop mounted device (LME), a USB dongle, a smart device, a wireless customer premises equipment (CPE), an Internet of Things (IoT) device, a watch or other wearable device, a head mounted display (HMD), a vehicle, a drone, medical equipment and applications (e.g., remote surgery), industrial equipment and applications (e.g., robots and / or other wireless devices operating in industrial and / or automatic processing chain environments), consumer electronic devices, equipment operating on commercial and / or industrial wireless networks, etc. The terminal device may also correspond to the mobile terminal (MT) part of an IAB node (eg, a relay node).In the following description, the terms "terminal device", "communication device", "terminal", "user equipment" and "UE" may be used interchangeably.

[0041] As used herein, the term "resource", "transmission resource", "resource block", "physical resource block" (PRB), "uplink resource" or "downlink resource" may refer to any resource used to perform communication (e.g., communication between a terminal device and a network device), such as a resource in the time domain, a resource in the frequency domain, a resource in the spatial domain, a resource in the code domain, or any other resource capable of communication. In the following, unless explicitly stated, resources in the frequency domain and the time domain will be used as examples of transmission resources for describing some example embodiments of the present disclosure. Note that the example embodiments of the present disclosure are equally applicable to other resources in other domains.

[0042] As used herein, the term "model" is referred to as the association between an input and an output learned from training data, and therefore a corresponding output can be generated for a given input after training. The generation of the model can be based on machine learning (ML) technology. Machine learning technology can also be referred to as artificial intelligence (AI) technology. Typically, a machine learning model can be constructed that receives input information and makes predictions based on the input information. For example, a classification model can predict the categories of input information in a predetermined number of categories. As used herein, a "model" may also be referred to as a "machine learning model," "learning model," "machine learning network," or "learning network," which are used interchangeably herein.

[0043] In the following, AI / ML MnS consumers may be referred to as MnS consumers, AI / ML consumers, or ML consumers. AI / ML MnS producers that include AI / ML-based functions or AI / ML service instances may be referred to as MnS producers, AI / ML producers, or ML producers.

[0044] As mentioned earlier, AI / ML consumers may be interested in understanding the performance of a given AI / ML service instance. However, AI / ML consumers may not always be able to interpret various metrics about performance key performance indicators (KPIs), such as accuracy, confidence, etc. Therefore, there is a need to provide a means for abstracting the measured metrics and defining them as indicators that can be easily interpreted by any consumer of AI / ML related performance management even without in-depth knowledge of specific AI / ML metrics. Example Environment

[0045] Figure 1 An example communication environment 100 is shown in which example embodiments of the present disclosure may be implemented. The communication environment 100 may be a management system in which multiple entities or devices are involved, including a first device 110, a second device 120, and a third device 130.

[0046] like Figure 1 As shown, the first device 110, the second device 120, and the third device 130 can communicate with each other. The first device 110 acts as a producer of management services or tasks that can be implemented by the ML entity 112. The management service or task can be, for example, an AI / ML training or reasoning service or any other AI / ML service. In the following, the first device 110 is also referred to as an MnS producer 110 or an AI / ML MnS producer 110.

[0047] The ML entity 112 may be an AI / ML model, or contain an AI / ML model or AI / ML-enabled functionality that may be managed as a single composite entity. AI / ML training may refer to the capabilities and associated end-to-end processes that enable an AI / ML training function to train its constituent AI / ML models, such as interacting with external parties to collect and format the data needed to train the AI / ML model. An AI / ML model may be a mathematical function or an artifact that contains a mathematical function and metadata about the mathematical function. In example embodiments, the term "AI / ML entity" may refer to any such variation of an artifact that is capable of making predictions using AI / ML logic.

[0048] An AI / ML-enabled function may be any network-related function that applies AI / ML to achieve the purpose of a network-related function. An AI / ML-enabled function may include one or more AI / ML entities. Examples of network-related functions may include: Management functions, such as management data analysis functions or self-organizing network functions; Core network functions for analysis, such as the Network Data Analysis Function (NWDAF) or core network functions for decision making, such as the Access and Mobility Management Function (AMF); RAN network functions, for example, RAN network functions for automation in the gNB, such as DSON functions, or RAN network functions for call processing, such as medium access control functions.

[0049] In some example embodiments, the AI / ML model or AI / ML-enabled function of the ML entity 112 may have at least one performance, including but not limited to precision, recall, F1 score, accuracy, mean absolute error, root mean square error, confidence, resource consumption, speed, etc. Such performance may be characterized by a corresponding performance metric. In some cases, the third device 130 may not be able to interpret the performance metric.

[0050] The third device 130 may be a consumer of the management service provided by the first device 110. Hereinafter, the third device 130 is also referred to as an MnS consumer 130 or an AI / ML MnS consumer 130. The AI / ML MnS consumer may be a network operator or another management function of a 5G system (5GS) that is interested in the AI / ML entities contained within the AI / ML enabled functions or the AI / ML capabilities supported by the AI / ML enabled functions. In some example embodiments, the third device 130 may desire to have qualified and abstracted performance of the ML entity 112.

[0051] The second device 120 is used to implement the performance abstraction function. In some example embodiments, the second device 120 may be used as an external entity of the first device 110 and the third device 130. Hereinafter, the second device 120 is also referred to as a ML performance abstraction producer 120 or a ML performance abstraction MnS producer 120.

[0052] In some example embodiments, the performance abstraction function may be implemented by an information model for performance abstraction 122, which is also referred to as information model 122 or performance abstraction model 122 hereinafter. Specifically, the second device 120 may obtain an original metric (e.g., a metric value) of at least one performance of the ML entity 112, which is an input to the performance abstraction model 122. The performance abstraction model 122 may derive and output a corresponding performance indicator based on the metric value. Such a performance indicator may be understood by the third device 130, which will be described in detail below.

[0053] Note that, although shown as separate devices, two or more of the first device 110, the second device 120, and the third device 130 may operate at least in part in the same cloud infrastructure. Figure 1 The number of devices and their connections shown in the diagram is for illustrative purposes only and does not imply any limitation. Management network 100 may include any suitable number of devices configured to implement the example embodiments of the present disclosure. Although not shown, it will be understood that one or more additional devices may be located in management network 100.

[0054] The communication in the communication environment 100 may be implemented according to any appropriate communication protocol, including but not limited to cellular communication protocols of the first generation (1G), second generation (2G), third generation (3G), fourth generation (4G), fifth generation (5G), sixth generation (6G), etc., wireless local area network communication protocols such as Institute of Electrical and Electronics Engineers (IEEE) 802.11, and / or any other protocol currently known or developed in the future. In addition, the communication may utilize any appropriate wireless communication technology, including but not limited to: code division multiple access (CDMA), frequency division multiple access (FDMA), time division multiple access (TDMA), frequency division duplex (FDD), time division duplex (TDD), multiple input multiple output (MIMO), orthogonal frequency division multiple access (OFDM), discrete Fourier transform spread OFDM (DFT-s-OFDM) and / or any other technology currently known or developed in the future. Working principles and example signaling for communication

[0055] According to some example embodiments of the present disclosure, a solution for abstraction of ML performance is provided. In this solution, a second device is provided to support ML performance abstraction. Specifically, the second device receives an abstract request for at least one performance of an ML entity provided by a first device. The ML entity is used for a third device. The second device determines at least one indicator of at least one performance that can be understood by the third device. The second device sends an abstract report including at least one indicator of at least one performance. In this way, AI / ML performance can be defined and abstracted into indicators that can be understood by consumers of AI / ML service instances. Therefore, communication between producers and consumers of AI / ML service instances can be improved.

[0056] Hereinafter, exemplary embodiments of the present disclosure will be described in detail with reference to the accompanying drawings.

[0057] Reference now Figure 2 , Figure 2 2 shows a signaling diagram 200 for communication according to some example embodiments of the present disclosure. Figure 2 As shown, the signaling diagram 200 involves the first device 110, the second device 120, and the third device 130. For the purpose of discussion, reference is made to Figure 1 To describe the signaling flow 200. Figure 2 , one first device 110 and one third device 130 are shown, but it will be understood that there may be multiple first devices that perform operations similar to those described below with respect to the first device 110 and multiple third devices that perform operations similar to those described below with respect to the third device 130.

[0058] In process 200, the first device 110 provides at least one AI / ML entity, such as ML entity 112, which completes a given AI / ML-related task, such as training or inferring at least one AI / ML application contained in the entity. In this case, ML entity 112 is used for the third device 130. It should be understood that any AI / ML-related tasks or functions other than training or inference may also be used for the implementation of the example embodiment.

[0059] The first device 110 sends 205 a first abstract request (e.g., represented by MLPerfQualRequest) for at least one performance of the ML entity 112 to the second device 120. The second device 120 is configured to implement an ML performance abstraction function, which is hereinafter represented by MLPerformanceAbstraction. In addition, the performance may be characterized by a corresponding performance metric, which is represented by mlPerformanceMetrics.

[0060] In some example embodiments, a first abstract request may be sent to initiate the definition and abstraction of at least one performance metric of the AI / ML entity. As examples, the AI / ML performance may include, but is not limited to, precision, recall, F1 score, accuracy, mean absolute error, root mean square error, confidence, resource consumption, speed of the ML entity 112.

[0061] The first abstract request may indicate the ML entity 112 for which a performance abstraction is desired. In some example embodiments, the first abstract request may include an identifier of the ML entity 112 and at least one raw metric of performance. As examples, the raw metric may include, but is not limited to, a confusion matrix, precision and recall, F1 score, AU-ROC, etc. Additionally, in some example embodiments, the first abstract request may also include the input and expected output of the ML entity 112 for which a performance abstraction is desired.

[0062] The second device 120 determines 210 at least one indicator of at least one performance understandable by the third device 130. Such an indicator may be denoted hereinafter by mlPerformanceIndex.

[0063] In some example embodiments, the performance abstraction model 122 may perform the definition and abstraction of the AI / ML performance of the AI / ML-enabled functions or ML applications included in the ML entity 112. For example, the performance abstraction model 122 may derive corresponding performance indicators from the received metric values.

[0064] The performance index may represent the performance achieved by the ML entity 112 on a particular performance metric as a number on a predetermined level or ML performance index range, denoted by mlPerformanceIndexRange. The ML performance index range may be predetermined at the first device 110, the second device 120, and the third device 130.

[0065] In some example embodiments, the absolute minimum and maximum performance may be pre-specified. For example, a scale of 11 values ​​(such as a range from 0 to 10) may be used, with the lowest value "0" indicating the worst possible performance, and the highest value "10" indicating the best possible performance. In some cases, a value of "5" may indicate a result similar to pure random guessing, which may indicate that the higher the more confident.

[0066] For each performance metric, the second device 120 may map a specific performance metric value to a predefined mlPerformanceIndexRange to generate a specific mlAbstractPerfIndex value for that performance metric value. This may then be communicated to consumers (eg, the first device 110 and the third device 130).

[0067] In some example embodiments, mlPerformanceIndex may be calculated based on only one performance metric. Additionally or alternatively, in some other example embodiments, a composite index may also be derived for a combination of multiple performance metrics to generate a specific mlAggregatePerfIndex value.

[0068] For each abstract request, the second device 120 may generate an abstract report including corresponding performance indicators of the required performance, the abstract report being denoted by MLAbstractPerfReport.Then, the second device 120 sends 215 a first abstract report including performance indicators of the ML performance.

[0069] In some example embodiments, the first abstract request may be submitted directly to the ML entity 112 that performs AI / ML inference. In this case, the ML entity 112 may have already gone through the performance abstraction process. In this way, the first device 110 may provide the result of the performance abstraction process itself.

[0070] In addition, in some example embodiments, the third device 130 may send 220 a second abstract request for ML performance of the ML entity 112. The second abstract request may include an indication of the ML entity 112 and at least one metric of the ML performance. As an example, the indication of the ML entity 112 may be at least one of: a name of the ML entity 112, an identification (ID) of the ML entity 112, a domain name (DN) of the ML entity 112. For each of the AI / ML-enabled functions or ML applications, the corresponding metric may be indicated by the name, ID, DN of the performance metric.

[0071] In some example embodiments, the second abstract request may be sent before transmitting the first abstract request. In this case, the transmission of the second abstract request causes the first abstract request to be sent from the first device 110 to the second device 120. Alternatively, in some other example embodiments, the second abstract request may be sent after transmitting the first abstract request.

[0072] In response to the second abstract request, the first device 110 may send 225 a second abstract report including a performance indicator of the performance metric.

[0073] In some example embodiments, the third device 130 may request the second device 120 to filter and provide an abstract report of at least one ML-enabled function or ML application that satisfies certain filtering criteria. As an example, the filtering criteria may be a prediction accuracy greater than 95% or any other criteria such as prediction accuracy.

[0074] In some example embodiments, the second device 120 may then publish the performance abstraction results (ie, performance indicators) on the authorization portal.

[0075] According to an example embodiment, ML performance abstraction is implemented in a communication network or management system. Various performance metrics of multiple ML applications or AI / ML-enabled functions can be defined and abstracted in a standardized format (i.e., consumer-understandable performance indicators of ML applications or AI / ML-enabled functions).

[0076] Figure 3 300 is shown for communication according to some example embodiments of the present disclosure. Figure 3 As shown, the signaling diagram 300 involves the first device 110, the second device 120 and the third device 130. For the purpose of discussion, reference is made to Figure 1 To describe the signaling flow 200. Figure 3 , one first device 110 and one third device 130 are shown, but it will be understood that there may be multiple first devices that perform operations similar to those described below with respect to the first device 110 and multiple third devices that perform operations similar to those described below with respect to the third device 130.

[0077] Instead of requesting the MnS producer, the MnS consumer requests an abstract capability from the ML capability abstraction MnS producer in process 300. The third device 130 sends 305 a third abstract request for at least one capability of the ML entity 112 provided by the first device 110. In some example embodiments, the third abstract request may include an indication of the ML entity 112 and a performance metric of the ML capability.

[0078] In some example embodiments, the second device 120 may receive a request for abstract performance of an ML application or an ML-enabled network function (e.g., a first or third abstract request) by using an MLPerformanceAbstractionProvisiningManagement service implemented via CRUD (create, read, update, delete) operations on an MLAbstractPerfRequest object.

[0079] The first abstract request in process 200 or the third abstract request in process 300 may be stated as follows: mLFunctionID: When sent to MLPerformanceAbstraction, the request may indicate the identifier of the specific ML-enabled network function that the consumer wishes to have the abstract performance for. However, this may not be required when sent to the ML-enabled network function itself. • MLAppID: The request may optionally state the identifier of a specific MLApp for which the consumer wishes to have capabilities defined and abstracted. mlPerformanceMetrics: This request can indicate ML-related performance metrics and their values, which will be evaluated by MLPerformanceAbstraction to generate abstract performance metrics.

[0080] The second device 120 determines 310 whether the requested ML performance is known. If the abstract performance is unknown, the second device 120 may send 315 a request for at least one metric value for the ML performance to the first device 110. Thus, in response to the request, the second device 120 may send 320 a response including at least one metric value for the ML performance.

[0081] Then, the second device 120 determines 325 a performance indicator corresponding to the at least one metric value. In case the ML performance is known, the second device 120 may directly determine the corresponding performance indicator without requesting the first device 110.

[0082] In some example embodiments, the second device 120 may calculate mlPerformanceIndex as an abstraction of a performance metric value that fits into a predetermined mlPerformanceIndexRange.

[0083] Additionally or alternatively, in some example embodiments, the second device 120 may interact with other ML performance abstraction MnS producers in evaluating raw metrics of easily understandable performance indicators.

[0084] Then, the second device 120 sends 330 a first abstract report including the performance indicator of the ML performance to the first device 110. Additionally or alternatively, the second device 120 sends 335 a third abstract report including the performance indicator of the ML performance to the third device 130. For the calculated mlPerformanceIndex, the information model 122 may compile an MLAbstractPerfReport including the calculated mlPerformanceIndex, and forward the MLAbstractPerfReport to the consumer (i.e., the function requesting the performance abstraction) to inform the consumer of the result of the performance abstraction.

[0085] In some example embodiments, after reporting the performance indicators, the second device 120 may also publish the abstract performance to some shared publishing space.

[0086] It should be understood that some steps in process 200 or 300 are optional or can be omitted, and the order of the steps is given for illustrative purposes. For example, step 330 can be performed after step 335, or step 330 can be performed in parallel with step 335. Therefore, the embodiments of the present disclosure are not limited in this regard.

[0087] According to an example embodiment of the present disclosure, an AI / ML performance abstraction process is implemented between MnS producers and consumers. The ML performance abstraction process can be executed at deployment time or after (re)training. In addition, the network and automation functions are allowed to interact with the ML performance abstraction function or the AI / ML function to determine the abstract performance of the ML instance. Example calculation of ML abstract performance

[0088] The performance abstraction entity 122 may apply multiple mechanisms to derive ML abstract performance, i.e., convert performance metrics into corresponding indicators understandable to MnS consumers. Due to different kinds of performance metrics with various interpretations, different deriving mechanisms for computing abstract performance may be required. Table 1 shows an example mechanism for computing ML abstract performance.

[0089] It should be understood that the performance metrics, calculations and conversions in Table 1 are given for illustrative purposes without implying any limitation on the scope of the present disclosure, and any other performance metrics and corresponding calculations and conversions are also applicable to the implementation of the example embodiments. Table 1. Example mechanisms for computing ML abstraction performance Example Information Model for ML Abstract Performance

[0090] The information object classes (IOCs) and data types required to implement the ML performance abstraction and the relationship between these IOCs and data types will be discussed in the following description. Figure 4 An example of an information model 400 for ML performance abstraction control according to some example embodiments of the present disclosure is shown. Figure 5 A schematic diagram of an ML performance abstract inheritance relationship 500 according to some example embodiments of the present disclosure is shown.

[0091] like Figure 4 As shown, the IOC "MLPerformanceAbstraction" can represent the properties of MLPerformanceAbstraction. MLPerformanceAbstraction is a managed function that can be instantiated from the MLPerformanceAbstraction information object class and name, which is contained in a subnetwork, ManagedFunction, or ManagmentFunction. MLPerformanceAbstraction is a type of ManagedFunction. That is, MLPerformanceAbstraction is a subclass of ManagedFunction and inherits the capabilities of ManagedFunction. MLPerformanceAbstraction has the ability to compile and deliver reports and notifications about the MLPerformanceAbstraction or its associated MLPerfiQualRequests or the MLPerformanceAbstraction itself. An MLPerformanceAbstraction should be associated with one or more MLAbstractPerfReports. • Each MLPerformanceAbstraction may have properties that specify the reporting characteristics of the MLPerformanceAbstraction (e.g., periodically, upon completion, etc.). MLPerformanceAbstractionMnS producers can also interact with other MLPerformanceAbstractionMnS producers when evaluating input / output or RAW metrics with easily understandable metrics. MLPerformanceAbstraction MnS has an information model for computation and for interactions related to abstract performance values. All received metric values ​​are mapped onto a defined fixed mlPerformanceIndexRange. For example, a range [0, 10] may be used, and all abstract performance values ​​should be within this range, where 0 indicates the lowest / worst possible performance and 10 indicates the best possible performance.

[0092] MLPerformanceAbstraction may be associated with at least one MLApp. For example, the MLApp associated with MLPerformanceAbstraction may be associated via a list of MLAppIdentifers.

[0093] MLPerformanceAbstraction can contain at least one MLPerfiQualRequests.

[0094] Table 2 shows example properties of the MLPerformanceAbstraction IOC: Table 2. Example properties of MLPerformanceAbstraction

[0095] In some example embodiments, the IOC "MLPerfQualRequest" may represent an attribute of MLPerfiQualRequest. For each request to abstract and qualify the performance of a given MLApp, a consumer may create a new MLPerfQualRequest on MLPerformanceAbstraction, i.e., MLPerfQualRequest should be the information object class instantiated for each request to abstract and qualify performance. • Each MLPerfQualRequest identifies (eg, using MLAppID) at least one MLApp that has generated the performance of the requested performance abstraction. MLPerfQualRequest may indicate a source function (eg, as sourceFunctionID) to identify where the request came from. This may be, for example, the DN of the source function. The sourceFunctionID and MLAppID are required so that MLPerformanceAbstraction can associate the exported abstract performance with the corresponding functions, and so that it can report the appropriate abstract performance for each corresponding function in subsequent abstract performance requests. Each MLPerfQualRequest must include the performance metric and the value to be evaluated and converted to an abstract performance.

[0096] Table 3 shows example attributes of the IOC "MLPerf1QualRequest". Table 3. Example properties of MLPerf1QualRequest

[0097] In some example embodiments, the IOC "MLPerfQualRequest" may represent a property of MLAbsractPerfRequest. Abstract performance may be requested from the AI / ML function itself (i.e., the function hosting the MLApp) or from the MLPerformanceAbstraction function.

[0098] For each abstract performance request, a consumer can create a new MLAbstractPerfRequest on MLPerformanceAbstraction or ML function, i.e., MLAbstractPerfRequest is an information object class instantiated for each abstract performance request. • Each MLAbstractPerfRequest identifies (eg, using MLAppID) at least one MLApp whose abstract performance is required. • The MLAbstractPerfRequest to the MLPerformanceAbstraction may identify (eg, using the MLFunctionID) the function for which the abstract performance is desired. MLAbstractPerfRequest may indicate the names of one or more performance metrics for which abstract performance is desired. If this is the case, the report may indicate results only for those stated performance metric names.

[0099] Table 4 shows example properties of the IOC "MLAbstractPerfRequest". Table 4. Example properties of MLAbstractPerfRequest

[0100] In some example embodiments, the data type "MLAbsractPerfReport" may represent an attribute of an MLAbstractPerfReport. MLPerformanceAbstraction may generate one or more MLAbsractPerfReports, and each MLAbsractPerfReport may be associated with one or more MLApps. MLPerformanceAbstraction may provide a report on a given MLAbstractPerfRequest on one or more MLApps. Accordingly, an MLAbsractPerfReport is associated with an instance of an MLAbstractPerfRequest.

[0101] MLAbsractPerfReport can provide abstract performance for each performance metric included in the request as well as the complete set of performance metrics.

[0102] As an example, if performance metrics are stated in the request for abstract performance, MLPerformanceAbstraction can be implemented so that the abstract performance is reported only for those performance metrics for each metric included in the MLAbstractPerfRequest. Otherwise, if the request for abstract performance indicates only MLFunction, MLPerformanceAbstraction can be implemented so that the abstract performance is reported for all performance metrics supported by MLFunction and for the aggregate abstract performance. In principle, at least one of them must be reported, so both mlAggregatePerfIndex and abstractPerfIndices are conditionally mandatory (CM).

[0103] Table 5 shows example properties of the data type MLAbstractPerfReport. Table 5. Example properties of MLAbstractPerfReport

[0104] Table 6 shows example attribute definitions according to an example embodiment of the present disclosure. Table 6. Attribute definitions Example Method

[0105] Figure 6FIG. 6 is a flowchart of an example method 600 implemented at a first device according to some example embodiments of the present disclosure. For discussion purposes, Figure 1 The method 600 is described from the perspective of the first device 110.

[0106] At block 610 , the first device 110 sends a first abstract request to the second device 120 , the first abstract request being for at least one capability of a machine learning entity provided by the first device. The machine learning entity is for the third device 130 .

[0107] At block 820 , the first device 110 receives a first abstract report from the second device 120 , the first abstract report including at least one indicator of at least one performance, the at least one indicator being understandable by the third device 130 .

[0108] In some example embodiments, the first abstract request may include an identification of the machine learning entity and at least one metric of at least one performance.

[0109] In some example embodiments, the first abstract request may be sent by the first device 110 that performs the task of the machine learning entity.

[0110] In some example embodiments, first device 110 may receive a second abstract request including an indication of a machine learning entity and at least one metric for at least one performance from third device 130. First device 110 may send a first abstract request including an indication of a machine learning entity and at least one metric value for at least one metric for at least one performance to second device 120.

[0111] In some example embodiments, the first device 110 may send a second abstract report to the third device 130, the second abstract report including at least one indicator of at least one performance.

[0112] In some example embodiments, the indication of the machine learning entity may include at least one of: a name of the machine learning entity, an identifier of the machine learning entity, a domain name of the machine learning entity.

[0113] In some example embodiments, first device 110 may receive a second abstract request from third device 130, the second abstract request including an indication of the machine learning entity and at least one metric for at least one performance. First device 110 may send a second abstract report to third device 130, the second abstract report including at least one metric for at least one performance.

[0114] In some example embodiments, the at least one performance may include at least one of the following: precision, recall, F1 score, accuracy, mean absolute error, root mean square error, confidence, resource consumption, speed of the machine learning entity.

[0115] In some example embodiments, an indicator range for at least one performance may be predetermined at the first device 110 , the second device 120 , and the third device 130 .

[0116] In some example embodiments, the first device 110 may include a machine learning management service producer or machine-enabled functionality. The second device 120 may include a performance abstraction management service producer that provides performance abstraction. The third device 130 may include a performance abstraction management service consumer.

[0117] Figure 7 FIG. 7 is a flowchart of an example method 700 implemented at a second device according to some example embodiments of the present disclosure. For discussion purposes, Figure 1 Method 700 is described from the perspective of the second device 120.

[0118] At 710 , the second device 120 receives an abstract request from one of the first device 110 or the third device 130 , the abstract request being for at least one capability of a machine learning entity provided by the first device 110 . The machine learning entity is for the third device 130 .

[0119] At 720 , the second device 120 determines at least one indicator of at least one performance. The at least one indicator may be understood by the third device 130 .

[0120] At 730 , the second device 120 sends an abstract report including at least one indicator of at least one performance to a corresponding one of the first device 110 or the third device 130 .

[0121] In some example embodiments, second device 120 may receive a first abstract request from first device 110, the first abstract request including an indication of a machine learning entity and at least one metric value for at least one performance. Second device 120 may determine at least one indicator corresponding to the at least one metric value.

[0122] In some example embodiments, the second device 120 may send a first abstract report including at least one indicator of at least one performance to the first device 110 .

[0123] In some example embodiments, the first abstract request may be received from a first device performing a task of the machine learning entity.

[0124] In some example embodiments, second device 120 may receive a third abstract request from third device 130, the third abstract request including an indication of the machine learning entity and at least one metric for at least one performance. Second device 120 may send a request for at least one metric value for at least one performance to first device 110. Second device 120 may receive a response including at least one metric value for at least one performance from first device 110. Thus, second device 120 may determine at least one indicator corresponding to the at least one metric value.

[0125] In some example embodiments, the second device 120 may send a third abstract report including at least one indicator of at least one performance to the third device 130 .

[0126] In some example embodiments, the indication of the machine learning entity may include at least one of: a name of the machine learning entity, an identifier of the machine learning entity, a domain name of the machine learning entity.

[0127] In some example embodiments, the at least one performance includes at least one of the following: precision, recall, F1 score, accuracy, mean absolute error, root mean square error, confidence, resource consumption, speed of the machine learning entity.

[0128] In some example embodiments, at least one indicator may be determined according to an indicator range for at least one performance predetermined at the first device 110 , the second device 120 , and the third device 130 .

[0129] In some example embodiments, the second device 120 may publish at least one indicator of at least one performance to a portal shared by the first device 110 and the third device 130 .

[0130] In some example embodiments, the first device 110 may include a machine learning management service producer or machine learning enabled functionality. The second device 120 may include a performance abstraction management service producer that provides performance abstraction. The third device 130 may include a performance abstraction management service consumer.

[0131] Figure 8 FIG. 8 is a flowchart of an example method 800 implemented at a third device according to some example embodiments of the present disclosure. For discussion purposes, Figure 1 Method 800 is described from the perspective of a third device 130.

[0132] At box 810, the third device 130 sends an abstract request for at least one performance of a machine learning entity provided by the first device to one of the first device 110 or the second device 120, the machine learning entity being used for the third device 130.

[0133] At block 820 , the second device 120 receives an abstract report from a corresponding one of the first device 110 or the second device 120 , the abstract report including at least one indicator of at least one performance, the at least one indicator being understandable by the third device 130 .

[0134] In some example embodiments, third device 130 may send a second abstract request to first device 110, the second abstract request comprising an indication of a machine learning entity and at least one metric for at least one performance, the sending of the second abstract request causing the first abstract request to be sent from the first device to the second device.

[0135] In some example embodiments, the third device 130 may receive a second abstract report from the first device 110, the second abstract report including at least one indicator of at least one performance.

[0136] In some example embodiments, third device 130 may send a third abstract request to second device 120, the third abstract request including an indication of the machine learning entity and at least one metric for at least one performance. The sending of the third abstract request may cause a request for at least one metric value of the at least one performance sent from second device 120 to first device 110.

[0137] In some example embodiments, the third device 130 may receive a second abstract report from the second device 120, the second abstract report including at least one indicator of at least one performance.

[0138] In some example embodiments, the indication of the machine learning entity includes at least one of: a name of the machine learning entity, an identifier of the machine learning entity, a domain name of the machine learning entity.

[0139] In some example embodiments, the at least one performance comprises at least one of: precision, recall, F1 score, accuracy, mean absolute error, root mean square error, confidence, resource consumption, speed of the machine learning entity.

[0140] In some example embodiments, an indicator range for at least one performance may be predetermined at the first device 110 , the second device 120 , and the third device 130 .

[0141] In some example embodiments, the first device 110 may include a machine learning management service producer or machine learning enabled functionality. The second device 120 may include a performance abstraction management service producer that provides performance abstraction. The third device 130 may include a performance abstraction management service consumer. Example devices, equipment, and media

[0142] In some example embodiments, any of the methods 600 can be performed (e.g., Figure 1 The first device of the first device 110 in the embodiment may include a component for performing the corresponding operation of the method 600. The component may be implemented in any suitable form. For example, the component may be implemented in a circuit or a software module. The first device may be implemented as or included in Figure 1 In the first device 110.

[0143] In some example embodiments, a first device includes: a component for sending a first abstract request to a second device, the first abstract request being for at least one performance of a machine learning entity provided by the first device, the machine learning entity being used for a third device; and a component for receiving a first abstract report from the second device, the first abstract report including at least one indicator of the at least one performance, the at least one indicator being understandable by the third device.

[0144] In some example embodiments, the first abstract request includes an identification of the machine learning entity and at least one metric of at least one performance.

[0145] In some example embodiments, the first abstract request is sent by a first device that performs a task of the machine learning entity.

[0146] In some example embodiments, the component for sending the first abstract request includes: a component for receiving a second abstract request from a third device, the second abstract request including an indication of the machine learning entity and at least one metric for at least one performance; and a component for sending the first abstract request to the second device, the first abstract request including an indication of the machine learning entity and at least one metric value for at least one metric for at least one performance.

[0147] In some example embodiments, the first apparatus further comprises means for sending the third apparatus the second abstract report comprising at least one indicator of at least one performance.

[0148] In some example embodiments, the indication of the machine learning entity includes at least one of: a name of the machine learning entity, an identifier of the machine learning entity, a domain name of the machine learning entity.

[0149] In some example embodiments, the first device further includes: a component for receiving a second abstract request from a third device, the second abstract request including an indication of the machine learning entity and at least one metric for at least one performance; and a component for sending a second abstract report to the third device, the second abstract report including at least one metric for at least one performance.

[0150] In some example embodiments, the at least one performance includes at least one of the following: precision, recall, F1 score, accuracy, mean absolute error, root mean square error, confidence, resource consumption, and speed of the machine learning entity.

[0151] In some example embodiments, an indicator range of at least one performance is predetermined at the first device, the second device, and the third device.

[0152] In some example embodiments, the first device includes a machine learning management service producer or machine learning enabled functionality, the second device includes a performance abstraction management service producer that provides performance abstraction, and the third device includes a performance abstraction management service consumer.

[0153] In some example embodiments, any of the methods 700 can be performed (e.g., Figure 1 The second device of the second device 120 in the embodiment may include a component for performing the corresponding operation of the method 700. The component may be implemented in any suitable form. For example, the component may be implemented in a circuit or a software module. The second device may be implemented as or included in Figure 1 In the second device 120.

[0154] In some example embodiments, the second device includes: a component for receiving an abstract request from one of the first device or the third device, the abstract request being for at least one performance of a machine learning entity provided by the first device, the machine learning entity being used by the third device; a component for determining at least one metric of the at least one performance, the at least one metric being understandable by the third device; and a component for sending an abstract report to a corresponding one of the first device or the third device, the abstract report including at least one metric of the at least one performance.

[0155] In some example embodiments, a component for receiving an abstract request includes: a component for receiving a first abstract request from a first device, the first abstract request including an indication of a machine learning entity and at least one metric for at least one performance; and a component for determining at least one metric corresponding to the at least one metric.

[0156] In some example embodiments, means for sending the abstract report comprises means for sending a first abstract report to the first apparatus, the first abstract report comprising at least one indicator of at least one performance.

[0157] In some example embodiments, the first abstract request is received from a first device that performs a task of the machine learning entity.

[0158] In some example embodiments, the component for receiving an abstract request includes: a component for receiving a third abstract request from a third device, the third abstract request including an indication of a machine learning entity and at least one metric for at least one performance; a component for sending a request for at least one metric value for at least one performance to the first device; a component for receiving a response from the first device including at least one metric value for at least one performance; and a component for determining at least one metric corresponding to the at least one metric value.

[0159] In some example embodiments, the means for sending the abstract report comprises means for sending a third abstract report to a third apparatus, the third abstract report comprising at least one indicator of at least one performance.

[0160] In some example embodiments, the indication of the machine learning entity includes at least one of: a name of the machine learning entity, an identifier of the machine learning entity, a domain name of the machine learning entity.

[0161] In some example embodiments, the at least one performance includes at least one of the following: precision, recall, F1 score, accuracy, mean absolute error, root mean square error, confidence, resource consumption, and speed of the machine learning entity.

[0162] In some example embodiments, the at least one indicator is determined according to an indicator range for at least one performance predetermined at the first device, the second device, and the third device.

[0163] In some example embodiments, the second apparatus further comprises means for publishing at least one indicator of at least one performance to a portal shared by the first apparatus and the third apparatus.

[0164] In some example embodiments, the first device includes a machine learning management service producer or machine learning enabled functionality, the second device includes a performance abstraction management service producer that provides performance abstraction, and the third device includes a performance abstraction management service consumer.

[0165] In some example embodiments, any of the methods 800 can be performed (e.g., Figure 1 The third device of the third device 130 in the method 800 may include a component for performing the corresponding operation of the method 800. The component may be implemented in any suitable form. For example, the component may be implemented in a circuit or a software module. The second device may be implemented as or included in Figure 1 In the third device 130.

[0166] In some example embodiments, the third device further includes: a component for sending an abstract request to one of the first device or the second device, the abstract request being for at least one performance of a machine learning entity provided by the first device, the machine learning entity being used by the third device; and a component for receiving an abstract report from a corresponding one of the first device or the second device, the abstract report including at least one indicator of at least one performance, the at least one indicator being understandable by the third device.

[0167] In some example embodiments, the means for sending an abstract request includes: means for sending a second abstract request to a first device, the second abstract request including an indication of a machine learning entity and at least one metric for at least one performance, the sending of the second abstract request causing the first abstract request to be sent from the first device to the second device.

[0168] In some example embodiments, means for receiving the abstract report comprises means for receiving a second abstract report from the first apparatus, the second abstract report comprising at least one indicator of at least one performance.

[0169] In some example embodiments, the means for sending an abstract request includes: means for sending a third abstract request to the second device, the third abstract request including an indication of the machine learning entity and at least one metric for at least one performance, the sending of the third abstract request causing a request to be sent from the second device to the first device for at least one metric value for the at least one performance.

[0170] In some example embodiments, means for receiving the abstract report comprises means for receiving a second abstract report from the second apparatus, the second abstract report comprising at least one indicator of at least one performance.

[0171] In some example embodiments, the indication of the machine learning entity includes at least one of: a name of the machine learning entity, an identifier of the machine learning entity, a domain name of the machine learning entity.

[0172] In some example embodiments, the at least one performance includes at least one of the following: precision, recall, F1 score, accuracy, mean absolute error, root mean square error, confidence, resource consumption, and speed of the machine learning entity.

[0173] In some example embodiments, an indicator range of at least one performance is predetermined at the first device, the second device, and the third device.

[0174] In some example embodiments, the first device includes a machine learning management service producer or machine learning enabled functionality, the second device includes a performance abstraction management service producer that provides performance abstraction, and the third device includes a performance abstraction management service consumer.

[0175] Fig. 9 900 is a simplified block diagram of a device 900 suitable for implementing an example embodiment of the present disclosure. The device 900 may be provided to implement an electronic device, such as Figure 1 The first device 110, the second device 120 or the third device 130 shown in FIG. As shown in the figure, the device 900 includes one or more processors 910, one or more memories 920 coupled to the processor 910, and one or more communication modules 940 coupled to the processor 910.

[0176] The communication module 940 is used for two-way communication. The communication module 940 has one or more communication interfaces to facilitate communication with one or more other modules or devices. The communication interface may represent any interface necessary for communication with other network elements. In some example embodiments, the communication module 940 may include at least one antenna.

[0177] As non-limiting examples, processor 910 may be of any type suitable for a local technology network, and may include one or more of the following: a general purpose computer, a special purpose computer, a microprocessor, a digital signal processor (DSP), and a processor based on a multi-core processor architecture. Device 900 may have multiple processors, such as application specific integrated circuit chips that are time slaved to a clock synchronized with a master processor.

[0178] The memory 920 may include one or more non-volatile memories and one or more volatile memories. Examples of non-volatile memories include, but are not limited to, read-only memory (ROM) 924, electrically programmable read-only memory (EPROM), flash memory, hard disk, compact disk (CD), digital video disk (DVD), optical disk, laser disk, and other magnetic storage and / or optical storage devices. Examples of volatile memories include, but are not limited to, random access memory (RAM) 922 and other volatile memories that will not persist for the duration of a power outage.

[0179] The computer program 930 includes computer executable instructions executed by the associated processor 910. The instructions of the program 930 may include instructions for performing the operations / actions of some example embodiments of the present disclosure. The program 930 may be stored in a memory (e.g., ROM 924). The processor 910 may perform any suitable actions and processes by loading the program 930 into the RAM 922.

[0180] The exemplary embodiments of the present disclosure may be implemented by means of a program 930, so that the device 900 may execute the Figures 2 to 8 Any process of the present disclosure discussed. The exemplary embodiments of the present disclosure may also be implemented by hardware or a combination of software and hardware.

[0181] In some example embodiments, the program 930 may be tangibly embodied in a computer-readable medium that may be included in the device 900 (such as in the memory 920) or in other storage devices accessible by the device 900. The device 1000 may load the program 930 from the computer-readable medium to the RAM 922 for execution. In some example embodiments, the computer-readable medium may include any type of non-transitory storage medium, such as ROM, EPROM, flash memory, hard disk, CD, DVD, etc. The term "non-transitory" as used herein is a limitation of the medium itself (i.e., tangible, not a signal), not a limitation on data storage persistence (e.g., RAM versus ROM).

[0182] Fig.10 An example of a computer readable medium 1000 is shown, which may be in the form of a CD, DVD or other optical storage disk. The computer readable medium 1000 has a program 930 stored thereon.

[0183] Generally, various embodiments of the present disclosure may be implemented in hardware or dedicated circuits, software, logic, or any combination thereof. Some aspects may be implemented in hardware, and other aspects may be implemented in firmware or software that may be executed by a controller, microprocessor, or other computing device. Although various aspects of the embodiments of the present disclosure are shown and described as block diagrams, flow charts, or using some other graphical representations, it should be understood that, as non-limiting examples, the blocks, devices, systems, techniques, or methods described herein may be implemented in hardware, software, firmware, dedicated circuits or logic, general-purpose hardware or controllers or other computing devices, or some combination thereof.

[0184] Some example embodiments of the present disclosure also provide at least one computer program product tangibly stored on a computer-readable medium (such as a non-transient computer-readable medium). The computer program product includes computer executable instructions, such as those instructions executed in a device on a target physical or virtual processor included in a program module, to perform any method as described above. Typically, a program module includes routines, programs, libraries, objects, classes, components, data structures, etc. that perform specific tasks or implement specific abstract data types. The functions of the program modules can be combined or split between program modules as needed in various embodiments. Machine executable instructions for program modules can be executed in local or distributed devices. In distributed devices, program modules can be located in both local and remote storage media.

[0185] The program code for executing the method of the present disclosure can be written in any combination of one or more programming languages. The program code can be provided to a processor or controller of a general-purpose computer, a special-purpose computer or other programmable data processing device so that the program code enables the functions / operations specified in the flow chart and / or block diagram to be implemented when the processor or controller executes the program code. The program code can be executed entirely on the machine, partially on the machine, as an independent software package, partially on the machine and partially on a remote machine, or completely on a remote machine or server.

[0186] In the context of the present disclosure, computer program codes or related data may be carried by any suitable carrier to enable a device, apparatus or processor to perform various processes and operations as described above. Examples of carriers include signals, computer readable media, etc.

[0187] The computer readable medium may be a computer readable signal medium or a computer readable storage medium. The computer readable medium may include, but is not limited to, an electronic, magnetic, optical, electromagnetic, infrared or semiconductor system, device or apparatus, or any suitable combination of the foregoing. More specific examples of computer readable storage media would include an electrical connection with one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

[0188] In addition, although operations are depicted in a particular order, this should not be understood as requiring such operations to be performed in the particular order shown or in a sequential order, or performing all of the operations shown, to achieve the desired result. In some cases, multitasking and parallel processing may be advantageous. Similarly, although several specific implementation details are included in the above discussion, these should not be interpreted as limitations on the scope of the present disclosure, but as descriptions of features that may be specific to a particular embodiment. Unless explicitly stated, certain features described in the context of a separate embodiment may also be implemented in combination in a single embodiment. On the contrary, unless explicitly stated, the various features described in the context of a single embodiment may also be implemented in multiple embodiments individually or in any suitable sub-combination.

[0189] Although the disclosure has been described in language specific to structural features and / or methodological acts, it should be understood that the disclosure defined in the appended claims is not necessarily limited to the specific features or acts described above. Instead, the specific features and acts described above are disclosed as example forms of implementing the claims.

Claims

1. A first device, comprising: at least one processor; as well as at least one memory storing instructions that, when executed by the at least one processor, cause the first device to at least: Sending a first abstract request to a second device, the first abstract request being for at least one performance of a machine learning entity provided by the first device, the machine learning entity being used for a third device; as well as A first abstract report is received from the second device, wherein the first abstract report includes at least one indicator of the at least one performance, and the at least one indicator can be understood by the third device.

2. The first device of claim 1, wherein the first abstract request comprises an identification of the machine learning entity and at least one metric value for the at least one performance.

3. The first device of claim 1 , wherein the first abstract request is sent by the first device that performs the task of the machine learning entity.

4. The first device of claim 1 , wherein the at least one memory stores instructions that, when executed by the at least one processor, cause the first device to send the first abstract request in the following manner: receiving a second abstract request from the third device, the second abstract request comprising an indication of the machine learning entity and at least one metric for the at least one performance; and Sending the first abstract request to the second device, the first abstract request comprising the indication of the machine learning entity and at least one metric value for the at least one metric of the at least one performance.

5. The first device of claim 4, wherein the at least one memory stores instructions that, when executed by the at least one processor, further cause the first device to: A second abstract report is sent to the third device, the second abstract report comprising the at least one indicator of the at least one performance.

6. The first device of claim 4, wherein the indication of the machine learning entity comprises at least one of: The name of the machine learning entity, an identity of the machine learning entity, The domain name of the machine learning entity.

7. The first device of claim 1 , wherein the at least one memory stores instructions that, when executed by the at least one processor, further cause the first device to: receiving a second abstract request from the third device, the second abstract request comprising an indication of the machine learning entity and at least one metric for the at least one performance; and A second abstract report is sent to the third device, the second abstract report comprising the at least one indicator of the at least one performance.

8. The first device of claim 1, wherein the at least one capability comprises at least one of: The precision, recall, F1 score, accuracy, mean absolute error, root mean square error, confidence, resource consumption, and speed of the machine learning entity. 9 . The first device according to claim 1 , wherein an indicator range for the at least one performance is predetermined at the first device, the second device, and the third device.

10. The first device of claim 1, wherein the first device comprises a machine learning management service producer or machine learning-enabled functionality, the second device comprises a performance abstraction management service producer providing performance abstraction, and the third device comprises a performance abstraction management service consumer.

11. A second device, comprising: at least one processor; as well as at least one memory storing instructions that, when executed by the at least one processor, cause the second device to at least: receiving, from one of the first device or the third device, an abstract request for at least one capability of a machine learning entity provided by the first device, the machine learning entity being for the third device; determining at least one indicator of the at least one performance, the at least one indicator being understandable by the third device; as well as An abstract report is sent to a corresponding one of the first device or the third device, the abstract report comprising the at least one indicator of the at least one performance.

12. The second device according to claim 11, wherein the at least one memory stores instructions, which, when executed by the at least one processor, cause the second device to receive the abstract request in the following manner: receiving a first abstract request from the first device, the first abstract request comprising an indication of the machine learning entity and at least one metric for the at least one performance; and The at least one indicator corresponding to the at least one metric value is determined.

13. The second device of claim 12, wherein the at least one memory stores instructions that, when executed by the at least one processor, cause the second device to send the abstract report in the following manner: A first abstract report is sent to the first device, the first abstract report comprising the at least one indicator of the at least one performance.

14. The second device of claim 12, wherein the first abstract request is received from the first device that performs the task of the machine learning entity.

15. The second device of claim 11, wherein the at least one memory stores instructions that, when executed by the at least one processor, cause the second device to receive the abstract request in the following manner: receiving a third abstract request from the third device, the third abstract request comprising an indication of the machine learning entity and at least one metric for the at least one performance; sending a request for at least one metric value of the at least one performance to the first device; receiving, from the first device, a response including the at least one metric value for the at least one performance; as well as The at least one indicator corresponding to the at least one metric value is determined.

16. The second device of claim 15, wherein the at least one memory stores instructions that, when executed by the at least one processor, cause the second device to send the abstract report in the following manner: A third abstract report is sent to the third device, the third abstract report including the at least one indicator of the at least one performance.

17. The second device according to claim 12 or 15, wherein the indication of the machine learning entity comprises at least one of the following: The name of the machine learning entity, an identity of the machine learning entity, The domain name of the machine learning entity.

18. The second device of claim 11, wherein the at least one performance comprises at least one of the following: The precision, recall, F1 score, accuracy, mean absolute error, root mean square error, confidence, resource consumption, and speed of the machine learning entity.

19. The second device according to claim 11, wherein the at least one indicator is determined according to an indicator range for the at least one performance predetermined at the first device, the second device, and the third device.

20. The second device of claim 11, wherein the at least one memory stores instructions that, when executed by the at least one processor, further cause the second device to: The at least one metric of the at least one performance is published to a portal shared by the first device and the third device.

21. The second device of claim 11, wherein the first device comprises a machine learning management service producer or machine learning-enabled functionality, the second device comprises a performance abstraction management service producer providing performance abstraction, and the third device comprises a performance abstraction management service consumer.

22. A third device, comprising: at least one processor; as well as at least one memory storing instructions that, when executed by the at least one processor, cause the third device to at least: sending an abstract request to one of the first device or the second device, the abstract request being for at least one performance of a machine learning entity provided by the first device, the machine learning entity being used by the third device; as well as An abstract report is received from a corresponding one of the first device or the second device, the abstract report comprising the at least one indicator of the at least one performance, the at least one indicator being understandable by the third device.

23. The third device according to claim 22, wherein the at least one memory stores instructions, which, when executed by the at least one processor, cause the third device to send the abstract request in the following manner: sending a second abstract request to the first device, the second abstract request comprising an indication of the machine learning entity and at least one metric for the at least one performance, the sending of the second abstract request causing the first abstract request to be sent from the first device to the second device.

24. The third device of claim 23, wherein the at least one memory stores instructions that, when executed by the at least one processor, cause the third device to receive the abstract report by: A second abstract report is received from the first device, the second abstract report comprising the at least one indicator of the at least one performance.

25. The third device according to claim 22, wherein the at least one memory stores instructions, which, when executed by the at least one processor, cause the third device to send the abstract request in the following manner: sending a third abstract request to the second device, the third abstract request comprising an indication of the machine learning entity and at least one metric for the at least one performance, the sending of the third abstract request causing a request to be sent from the second device to the first device for at least one metric value for the at least one performance.

26. The third device of claim 25, wherein the at least one memory stores instructions that, when executed by the at least one processor, cause the third device to receive the abstract report by: A second abstract report is received from the second device, the second abstract report comprising the at least one indicator of the at least one performance.

27. The third device according to claim 23 or claim 25, wherein the indication of the machine learning entity comprises at least one of the following: The name of the machine learning entity, an identity of the machine learning entity, The domain name of the machine learning entity.

28. The third device according to claim 22, wherein the at least one performance comprises at least one of the following: The precision, recall, F1 score, accuracy, mean absolute error, root mean square error, confidence, resource consumption, and speed of the machine learning entity.

29. The third device according to claim 22, wherein an indicator range for the at least one performance is predetermined at the first device, the second device, and the third device.

30. A third device according to claim 22, wherein the first device includes a machine learning management service producer or a machine learning-enabled function, the second device includes a performance abstraction management service producer that provides performance abstraction, and the third device includes a performance abstraction management service consumer.

31. A method comprising: Sending, at a first device, a first abstract request to a second device, the first abstract request being for at least one performance of a machine learning entity provided by the first device, the machine learning entity being for a third device; as well as A first abstract report is received from the second device, wherein the first abstract report includes at least one indicator of the at least one performance, and the at least one indicator can be understood by the third device.

32. A method comprising: receiving, at a second device, from one of a first device or a third device, an abstract request for at least one capability of a machine learning entity provided by the first device, the machine learning entity being for the third device; determining at least one indicator of the at least one performance, the at least one indicator being understandable by the third device; as well as An abstract report is sent to a corresponding one of the first device or the third device, the abstract report comprising the at least one indicator of the at least one performance.

33. A method comprising: Sending, at a third device, an abstract request to one of the first device or the second device, the abstract request being for at least one performance of a machine learning entity provided by the first device, the machine learning entity being for the third device; as well as An abstract report is received from a corresponding one of the first device or the second device, the abstract report comprising the at least one indicator of the at least one performance, the at least one indicator being understandable by the third device.

34. A first device, comprising: means for sending a first abstract request to a second device, the first abstract request being for at least one capability of a machine learning entity provided by the first device, the machine learning entity being for a third device; as well as Means for receiving a first abstract report from the second device, the first abstract report comprising at least one indicator of the at least one capability, the at least one indicator being understandable by the third device.

35. A second device, comprising: means for receiving, from one of a first device or a third device, an abstract request for at least one capability of a machine learning entity provided by the first device, the machine learning entity being for the third device; means for determining at least one indicator of said at least one property, said at least one indicator being understandable by said third device; as well as Means for sending an abstract report to a corresponding one of the first device or the third device, the abstract report comprising the at least one indicator of the at least one performance.

36. A third device, comprising: means for sending an abstract request to one of the first device or the second device, the abstract request being for at least one capability of a machine learning entity provided by the first device, the machine learning entity being for the third device; as well as Means for receiving an abstract report from a corresponding one of the first device or the second device, the abstract report comprising the at least one indicator of the at least one performance, the at least one indicator being understandable by the third device.

37. A computer-readable medium comprising instructions stored thereon, the instructions being configured to cause an apparatus to perform at least the method of claim 31, the method of claim 32, or the method of claim 33.

Images