Device, method, and system for network analytics accuracy monitoring

Abstract

This disclosure relates to mobile communication. A first network function (NF) is adapted to manage machine learning (ML) model and provide suitable ML model(s) to a second NF. According to this disclosure, the first NF is configured to obtain model accuracy binding information from the second NF. The model accuracy binding information indicates an association between analytics information and ML model information. The analytics information indicates an association between an analytics ID, and analytics filter information and / or analytics target. The first NF obtains model accuracy information associated with model accuracy meta information from the second NF. The model accuracy meta information indicates an association of the ML model accuracy information and the analytics information. The first network function is configured to determine whether to update the ML model information based on the model accuracy binding information and / or the model accuracy meta information.

Description

[0001] DEVICE, METHOD, AND SYSTEM FOR NETWORK ANALYTICS ACCURACY MONITORING

[0002] TECHNICAL FIELD

[0003] The present disclosure relates to the field of communication. For instance, the disclosure relates to devices, methods and a system for network analytics accuracy monitoring.

[0004] BACKGROUND

[0005] In the context of 5G, 6G and beyond communications networks, the Network Data Analytics Function (NWDAF) plays a crucial role in collecting, processing, analysing, and providing insights from various network data sources. The NWDAF may be adapted to perform or may contain various logical functions for different purposes, such as Analytics Logical Function (AnLF) and Model Training Logical Function (MTLF).

[0006] An NWDAF comprising AnLF (may also be referred to as NWDAF-AnLF, or simply AnLF) is responsible for collecting the analytical request and sending the response to the consumer. For this purpose, the AnLF may be configured to collect meaningful information from network data and make a decision (e.g., perform prediction / estimation) using advanced analytics techniques, e.g., machine learning (ML) models (or algorithms). An NWDAF comprising MTLF (may also be referred to as NWDAF -MTLF, or simply MTLF), on the other hand, is responsible for training, managing, and updating the ML models used by the AnLF.

[0007] SUMMARY

[0008] The 3GPP specification TS 23.288 vl8.3.0 defines that the AnLF can monitor both the analytics accuracy and the ML model accuracy information. The accuracy information is provided to the MTLF if the MTLF is subscribed to receive such information, which can then trigger any of the following actions:

[0009] Re-training of ML models: If the accuracy information indicates that a model has degraded, the MTLF can trigger the re-training of the model. This process involves updating the model's parameters based on new data and improving its performance.

[0010] Re-selection of ML models: The MTLF can also re-select the ML model that is associated with a particular analytics ID. This may be necessary if the current model is no longer performing as well as it could, or if a new model has been developed that is more accurate or efficient.

[0011] The ability to monitor and manage accuracy information is essential for ensuring the reliability and effectiveness of network analytics. By actively updating and improving ML models, the AnLF can provide more accurate and timely insights, which can be used to optimize network performance, enhance user experience, and identify and resolve potential issues.

[0012] Current 3GPP specification allows for the AnLF to have multiple subscriptions to the same analytics ID (e.g., AMF subscribes to the same analytics ID with different analytics targets, i.e., UEs or group of UEs, and / or different analytics filter information) that are using the same unique ML model identifier. This means that one trained ML Model may be associated with a unique ML model identifier and a set of ML Model Filter Information and ML Model target (e.g., a general trained model for an analytics ID), may fit different specific subscriptions for the actual generation of the analytics ID (e.g., a specific scenario that could be detectab le / predictable by a generic trained ML Model).

[0013] As a consequence, the AnLF may be faced with the situation in which for a given specific usage of the trained ML model (e.g., a subscription for analytics ID “A” with filters “B”) the analytics accuracy is high, and for another specific usage of the same trained ML model (e.g., subscription for analytics ID “A” with filters “C”) the analytics accuracy is low. When AnLF generates the ML Model accuracy (per analytics ID and ML model identifier), these differences on the accuracy for specific usage of the model are diluted if not completely lost.

[0014] Overall, the current ML model provisioning, update and retraining scheme may have the following problems.

[0015] Firstly, the specific usage of ML models for different application scenarios is not considered. This can lead to inaccurate (or incomplete) ML model accuracy information being sent to the MTLF, which can in turn trigger unnecessary actions such as retraining or re-selection of ML models.

[0016] Secondly, it does not allow for the MTLF to differentiate between ML models that are suitable for different application scenarios. This can lead to the MTLF retraining or re-selecting ML models that are not actually suitable for specific usage scenario, resulting in wasted resources and sub-optimal decisions.

[0017] In view of the above-mentioned problems and disadvantages, this disclosure aims to improve network accuracy monitoring and reporting in communication networks. For instance, an objective of this disclosure may be to improve ML accuracy information reporting scheme between the AnLF and the MTLF. A further objective of this disclosure may be to prevent the MTLF from performing unnecessary signaling and / or determining imprecise ML model degradation information, and to prevent the MTLF from triggering of: imprecise or unnecessary reselection of ML models, or imprecise decision to not retrain or reselect a ML model, in response to ML model request from NWDAF with AnLF.

[0018] These and other objectives are achieved by the subject matter of the independent claims. Further implementation forms are apparent from the dependent claims, the description, and the drawings.

[0019] A first aspect of the present disclosure provides a first network function (NF) entity (e.g., MTLF) for ML model training (or provisioning) in a mobile communication network. The first NF entity is configured to obtain model accuracy binding information from a second network function entity (e.g., an AnLF). The model accuracy binding information is indicative of at least one association between an analytics information and ML model information. The analytics information is indicative of at least one association between an analytics identifier (ID), and one or more of analytics filter information and analytics target. The first NF entity is further configured to obtain model accuracy information associated with model accuracy meta information from the second NF entity. The model accuracy meta information is indicative of ML model accuracy information, and associated analytics information. The ML model accuracy information is of an association of the ML model accuracy information and the analytics information. The first NF entity is further configured to determine whether to update the ML model information (and / or the ML model identified by the ML model information) based on the model accuracy binding information and / or the model accuracy meta information.

[0020] Optionally, the ML model accuracy information may be indicative of performance and / or quality information about the ML model. Optionally, the model accuracy meta information may be comprised (or embedded) in the model accuracy information, or may be annexed to the model accuracy information, or may be linked to the model accuracy information, or the like.

[0021] The model accuracy binding information may be used to indicate not only which analytics (identified by an analytics ID) corresponds to which ML model, but also a specific usage scenario (identified by analytics filter information and / or analytics target information) of the analytics that is associated with the ML model. Based on the received model accuracy meta information and the binding information, the first NF entity can determine, in a specific usage scenario, whether an ML model is imprecise or underperformances. Therefore, the risk of the first NF entity of determining imprecise quality / correctness about the ML Model for an analytics ID used in different situations may be reduced.

[0022] It is noted that the first NF entity is further configured to determine whether to update the ML model based further on the model accuracy information.

[0023] In an implementation form of the first aspect, the model accuracy meta information may further indicate an association of analytics accuracy information and one or more of: an analytics ID, the analytics filter information, the analytics target, and the ML model information.

[0024] Optionally, the analytics accuracy information may be indicative of performance and / or quality information about an analytics identified by the analytics ID.

[0025] The use of analytics accuracy information and / or model accuracy information for the determining the retraining and / or reselection of ML Models is aligned with the definitions of TS 23.288 V18.3.0 for instance as described in Clause 6.2E.2 and / or Clause 6.2E.3.3. Therefore, for simplicity they are not described in detailed in this disclosure.

[0026] In a further implementation form of the first aspect, the first network function entity may be configured to obtain the model accuracy binding information from a request from the first network function entity. The request is indicative that the second network function entity is able to provide the ML model accuracy information with respect to the ML model information comprised in the model accuracy binding information.

[0027] In this way, the first network function may be aware of that the second network function supports the accuracy reporting with specific usage scenario.

[0028] In a further implementation form of the first aspect, the first network function entity may be configured to send indication information to the second network function entity for requesting the model accuracy meta information.

[0029] In this way, the first network function may explicitly request the second network function to provide the accuracy information with specific usage scenario(s).

[0030] In a further implementation form of the first aspect, the indication information for requesting the model accuracy meta information may be comprised in a subscription request sent by the first network function entity to the second network function entity.

[0031] In a further implementation form of the first aspect, the indication for requesting the model accuracy meta information may comprise one or more of:

[0032] - a flag indicating whether the model accuracy meta information is required - a ML model ID associated with an analytics ID;

[0033] - a ML model ID associated with one or more of an analytics ID analytics filter information, and analytics target;

[0034] - a ML model ID associated with one or more ML model accuracy IDs, wherein each ML model accuracy identifier identifies at least one ML model ID association with at least one analytics ID, and analytics filter information and / or analytics target; and

[0035] - one or more ML model accuracy IDs, wherein each ML model accuracy ID identifies a tuple of a ML model ID and corresponding analytics information.

[0036] In a further implementation form of the first aspect, the first NF entity may be further configured to update the ML model information based on the ML model accuracy meta information. The updated ML Model information is associated with a new ML Model ID or with a previous ML Model ID.

[0037] In a further implementation form of the first aspect, for updating the ML model information, the first NF entity may be configured to perform any of the following: determine new ML model information as the updated ML model information; or retrain the ML model to obtain a retrained ML model in order to update the ML model information.

[0038] In a further implementation form of the first aspect, the first NF entity may be further configured to provide the updated ML model information and / or a ML model specialization indication to the second network function entity. The ML model specialization indication is indicative of an association of the ML model information and / orthe updated ML model information, and corresponding analytics information.

[0039] That is, the first NF entity may be configured to provide the updated ML model information. Alternatively, the first NF entity may be configured to provide a ML model specialization indication. Alternatively, the first NF entity may be configured to the updated ML model information and the ML model specialization indication.

[0040] A second aspect of this disclosure provides a second NF entity (e.g., AnLF) for supporting ML model training in a mobile communication network. The second NF entity is configured to provide model accuracy binding information to a first NF entity (e.g., MTLF) for ML model training. The model accuracy binding information is indicative of at least one association between an analytics information and ML model information. The analytics information is indicative of at least one association between an analytics ID, and one or more of analytics filter information and analytics target. The second NF entity is further configured to obtain model accuracy information of the ML model information (or, of a ML model identified by the ML model information), and generate model accuracy meta information indicating an association of the model accuracy information and the analytics information. The second NF entity is configured to provide the model accuracy information associated with the model accuracy meta information to the first NF entity.

[0041] Optionally, for obtaining the model accuracy information, the second NF entity may be configured to generate an analytics output using an ML model identified by the ML model information, and verify the analytics output (e.g., at a later time point), to obtain the model accuracy information.

[0042] In an implementation form of the second aspect, the model accuracy meta information may further comprise analytics accuracy information associated with one or more of an analytics ID, the analytics filter information, the analytics target, and the ML model information. In a further implementation form of the second aspect, the second NF entity may be configured to provide the model accuracy binding information to the first network function entity through a request. The request is indicative that the second network function entity is able to provide the ML model accuracy information with respect to the ML model information comprised in the model accuracy binding information.

[0043] In a further implementation form of the second aspect, for providing the model accuracy meta information, the second network function entity may be configured to receive indication information from the first network function entity for requesting the model accuracy meta information.

[0044] In a further implementation form of the second aspect, the second NF entity may be further configured to: receive updated ML model information and / or a machine learning model specialization indication from the first network function entity, in which the machine learning model specialization indication is indicative of an association of the ML model information and / or the updated ML model information, and corresponding analytics information; and update the ML model information based on the updated ML model information and the machine learning model specialization indication.

[0045] The second NF entity of the second aspect may share the corresponding optional features and the same effect as the first NF entity of the first aspect.

[0046] A third aspect of this disclosure provides a system comprising at least one first NF entity according to the first aspect or any implementation form thereof, and at least one second NF entity according to the second aspect or nay implementation form thereof.

[0047] A fourth aspect of this disclosure provides a method applied to a first NF entity for ML model training in a mobile communication network. The method comprises the following steps: obtaining model accuracy binding information from a second NF entity, in which the model accuracy binding information is indicative of at least one association between an analytics information and ML model information, wherein the analytics information is indicative of at least one association between an analytics ID and one or more of analytics filter information and analytics target; and obtaining model accuracy information associated with model accuracy meta information from the second NF entity, wherein the model accuracy meta information is indicative of an association of the ML model accuracy information and the analytics information; and determining whether to update the ML model based on the model accuracy binding information and / or the model accuracy meta information.

[0048] The method of the fourth aspect may share the same optional features and the same effect as the first NF entity of the first aspect or any implementation form thereof.

[0049] A fifth aspect of this disclosure provides a method applied to a second NF entity for supporting ML model training in a mobile communication network. The method comprises the following steps: providing model accuracy binding information to a first NF entity, in which the model accuracy binding information is indicative of at least one association between an analytics information and ML model information, in which the analytics information is indicative of at least one association between an analytics ID, and one or more of analytics filter information and analytics target; obtaining model accuracy information of the ML model information; generating model accuracy meta information indicating an association of the model accuracy information and the analytics information; and providing the model accuracy information associated with the model accuracy meta information to the first NF entity .

[0050] The method of the fifth aspect may share the same optional features and the same effect as the second NF entity of the second aspect.

[0051] A sixth aspect of this disclosure provides a computer program comprising instructions which, when the program is executed by a computer, cause the computer to cany out the method according to the fourth aspect or the fifth aspect.

[0052] A fifth aspect of the present disclosure provides a computer-readable medium comprising instructions which, when executed by a computer, cause the computer to cany out the method according to the fourth aspect or the fifth aspect.

[0053] A fifth aspect of the present disclosure provides a chipset comprising instructions which, when executed by the chipset, cause the chipset to carry out the method according to the fourth aspect or the fifth aspect.

[0054] It has to be noted that all devices, terminals, elements, units, and means described in the present disclosure could be implemented in software or hardware elements or any kind of combination thereof. All steps which are performed by the various entities described in the present application as well as the functionalities described to be performed by the various entities are intended to mean that the respective entity is adapted to or configured to perform the respective steps and functionalities. Even if, in the following description, a specific functionality or step to be performed by external entities is not reflected in the description of a specific detailed element of that entity, which performs that specific step or functionality, it should be clear for a skilled person that these methods and functionalities can be implemented in respective software or hardware elements, or any kind of combination thereof.

[0055] BRIEF DESCRIPTION OF DRAWINGS

[0056] The above-described aspects and implementation forms will be explained in the following description in relation to the enclosed drawings, in which

[0057] FIG. 1 shows a first NF entity 110 and a second NF entity 120 this disclosure;

[0058] FIG. 2 shows an example of a method of this disclosure;

[0059] FIG. 3 shows a further example of a method of this disclosure;

[0060] FIG. 4 shows a diagram of a method applied to a first network function entity; and

[0061] FIG. 5 shows a diagram of a method applied to a second network function entity.

[0062] DETAILED DESCRIPTION OF EMBODIMENTS

[0063] Without loss of generality, exemplary explanations of terms used in this disclosure are given below. Accuracy Monitoring: A general term used to describe the process of identifying the performance (and / or an associated measurement) of an analytics (e.g., machine learning process such as inference or training).

[0064] Analytics Accuracy Monitoring: An activity relates to the accuracy monitoring of an Analytics ID during ML inference processes performed by an NWDAF with AnLF logical functionality.

[0065] ML Model Accuracy Monitoring assisted by AnLF: related to the accuracy monitoring of an ML model (e.g., identified by a unique ML Model identifier) and an Analytics ID performed by an NWDAF with AnLF logical functionality and provided to the NWDAF with MTLF logical functionality.

[0066] Feedback Type: defines the type of ML Model accuracy information to be generated (and / or provided) to an entity requesting such information. Another possible understanding for the feedback type is that it defines the tuple that is associated with the ML Model accuracy information generated (e.g., monitored and / or provisioned) or to be generated (e.g., monitored and / or provisioned) by the NWDAF with AnLF.

[0067] General Feedback Type: Information that indicates to NWDAF with AnLF that the ML model accuracy information should be monitored and / or generated and / or calculated and / or provisioned and / or calculated based on the tuple of [analytics ID, and unique ML Model identifier].

[0068] Specific Feedback Type: Information that indicates to NWDAF with AnLF that the ML model accuracy information should be monitored and / or generated and / or calculated and / or provisioned and / or calculated based on the ML model accuracy meta information, e.g., the tuple [unique ML Model identifier, analytics ID, analytics Filter Information and / or Analytics Target],

[0069] Indication for ML Model accuracy meta information request (or indication for requesting ML Model accuracy meta information or indication for ML Model accuracy meta information or indication information for requesting model accuracy meta information): defines a request and / or subscription to obtain the ML Model accuracy meta information.

[0070] ML model accuracy meta information (also referred to as model accuracy meta information): defines the association of a unique ML Model identifier and / or ML Model Information with an ML Model accuracy information for an analytics ID, and Analytics Filter Information and / or Analytics target. This association may be understood as a mapping.

[0071] Indication of ML Model specialization (or machine learning, ML, model specialization indication): In broader terms defines the information (and / or one or more parameters) used to map a unique ML Model ID and / or ML Model Information to an analytics ID, and Analytics Filter Information and / or Analytics target. It may also be understood as defining a relationship between a previous unique ML model identifier and / or ML Model Information associated with an analytics IDs to a new unique ML model identifier and / or new ML Model Information associated with the same analytics ID and further indicating the specific Analytics Filter Information and / or Analytics target related to the analytics ID.

[0072] Accuracy binding information (or model accuracy binding information): defines the information that associates and / or defines the mapping and / or defines the relationship among unique ML Model identifier (optionally a ML Model Filter Information and / or ML Model target) with analytics ID, analytics filter information and / or analytics target

[0073] ML model accuracy information (or model accuracy information): defines a performance and / or quality information about an ML Model. In FIGs. 1-5 below, corresponding elements may share the same features and function likewise.

[0074] FIG. 1 shows a first NF entity 110 and a second NF entity 120 this disclosure. The first NF entity 110 is for ML model training in a mobile communication network. For instance, the first NF entity 110 may be an MTLF. In this disclosure, MTLF is used to refer to the first NF entity 110. The second NF entity 120 is for supporting ML model training in a mobile communication network. The second NF entity 120 is adapted to provide analytic results based on trained model received from the first entity 110. For instance, the second NF entity 120 may be an AnLF. In this disclosure, AnLF is used to refer to the second NF entity 120.

[0075] In this disclosure, the NWDAF architecture and Nnwdaf service interface defined in 3GPP TS 23.288 vl8.3.0 may be considered as a baseline. Terms used in this disclose may the same definition as in the TS 23.288 (unless specified otherwise), which includes but not limited to: Analytics ID, Analytics Filter Information (or Analytics Filter), Analytics Target (or Analytics Target of reporting or Target of Analytics Reporting), ML Model, ML Model Filter Information, ML Model Information, unique ML Model identifier, ML Model Target (or Target of ML Model, or ML Model Target of reporting).

[0076] As depicted in FIG. 1, MTLF 110 and AnLF 120 are illustrated. For supporting ML training at MTLF 110, AnLF 120 is adapted to provide ML Model accuracy information to MTLF 110. Based on the ML model accuracy information, MTLF 110 may take follow-up steps when it detects that a ML Model is suffering some performance degradation. Optionally, AnLF 120 may be further adapted to provide analytics accuracy information to MTLF 110. Based on the model accuracy information, and / or model accuracy binding information, and / or the model accuracy meta information, and / or analytics accuracy information, the MTLF 110 may be adapted to determine whether to update the ML model if the ML model is not fit for the analytics, or underperformances for the analytics. The interactions may comprise the following steps.

[0077] It is noted that according to the definitions of analytics and ML accuracy monitoring defined in 3GPP TS 23.288 V18.3.0, different entities may be adapted to control the accuracy of the ML Model associated with an analytics ID. For instance, the NWDAF containing AnLF 120 may be adapted to track (or check) with its own parametrization the analytics accuracy information for a given ML Model #A associated with an analytics ID #A. It is also possible that the NWDAF containing the MTLF 110 may be adapted to track (or check) with its own parametrization the ML Model accuracy information for the same given ML Model #A associated with the analytics ID #A. Since these can be completely independent processes, it is possible that the two entities AnLF and MTLF 110 may have the same view of the accuracy for the association ML Model #A and Analytics ID #A. It is also possible that the two entities AnLF and MTLF 110 may have, at a given point in time, different views of the accuracy for the association ML Model #A and Analytics ID #A. For instance, the ML Model accuracy for ML Model #A with analytics ID #A at time T in the perspective of MTLF 110 may not be the same analytics accuracy for analytics ID #A with ML Model #A at time T from the perspective of AnLF. At time T, MTLF 110 may have an ML accuracy value still reflecting the accuracy for the ML Model #a trained at a different time T-x. When MTLF 110 obtains the analytics accuracy information from AnLF, the MTLF 110 can identify whether the „stored“ accuracy for ML model associated with analytics ID A should actually be rechecked.

[0078] Step 101 : AnLF 120 uses a service called “Nnwdaf_MLModelMonitor_Registef ’ from MTLF 110 in order to indicate to MTLF 110 that it is capable to generate ML Model accuracy information enhanced with accuracy meta information (e.g., with respect to a given Analytics ID and ML model ID). This step is optional and not essential for this disclosure.

[0079] Generally, MTLF 110 may be configured to obtain the model accuracy binding information from a request from AnLF 120. The request is indicative that AnLF 120 is able to provide the ML model accuracy information enhanced with meta information. Step 102: MTLF 110 receives from AnLF 120 the model accuracy binding information indicative of at least one association between analytics information and ML model information. The analytics information is indicative of at least one association between an analytics ID, and one or more of analytics filter information and analytics target. In another word, the accuracy binding information may be indicative of at least one mapping of an analytics ID, and analytics filter information and / or analytics target (e.g., [ analytics ID -> analytics filter and / or analytics target ]). The analytics information is further associated with the ML model information comprising an ML model ID identifying an ML model. Optionally, the ML model information may further comprise ML Model filter information and / or ML model target information associated with the ML model. For instance, the accuracy binding formation may comprise information of the following format: [ analytics ID -> analytics filter and / or analytics target ] -> [ML model ID, (optional) ML model filter information, (optional) ML model target information]).

[0080] It is noted that the analytics / ML model filter information may be used to indicate a service type that the analytics / ML model is applied to, and the target may be used to indicate to which UE(s) / terminal(s) the analytics / ML model is applied.

[0081] Optionally, step 102 may be performed together with step 101. Alternatively, step 102 may be performed separately, for instance, using a different service invoked between ANLF 120 and MTLF 110.

[0082] Step 103 : AnLF 120 performs its tasks of analytics collection (or generation), for each subscription and for an Analytics ID, using a ML model (identified by a corresponding ML model ID) indicated by MLTF 110.

[0083] AnLF 120 is configured to use a ML model for providing analytics information to an analytics consumer. During providing the analytics information, AnLF 120 may be configured to collect / determine analytics accuracy associated with the ML model supporting the generation of the analytics information. For instance, AnLF 120 may use a ML model for the generation of an analytics for traffic prediction. Later on, when AnLF 120 obtains a real traffic statistic, AnLF 120 may use the real traffic statics (as a ground-truth) to evaluate the performance of the ML model used for the generation of the analytics information, in order to obtain the analytics accuracy information of the ML model. That is, from the perspective of AnLF 120 in a given time, AnLF 120 determines that the ML model for the analytics for traffic predictions. For instance, the analytics model accuracy information may comprise confidence level (0-100%) of a prediction result for the ML model used at that moment y AnLF.

[0084] Step 104: MTLF 110 is configured to obtain model accuracy information associated with model accuracy meta information from AnLF 120. The model accuracy meta information is indicative of an association of the ML model accuracy information and the analytics information.

[0085] Optionally, MTLF 110 may be configured to send indication information to AnLF 120 for requesting the model accuracy meta information. Optionally, MTLF 110 may be configured to subscribe to ML model accuracy monitoring information from a registered AnLF 120 for a given analytics ID and unique ML model ID (e.g., through a subscription request). MTLF 110 may be configured to indicate to AnLF 120 whether the ML model accuracy monitoring should be generated with further refinements considering the mapping of a unique ML model ID to the association of analytics ID, and analytics filter information and / or analytics target information. When AnLF 120 accepts such subscription, AnLF 120 may be configured to monitor the ML model accuracy information as per indication received from MTLF 110, and provide to MTLF 110 with the ML model accuracy information comprising the ML Model accuracy meta information.

[0086] Optionally, the indication information for requesting the model accuracy meta information may comprise one or more of:

[0087] - a flag indicating whether the model accuracy meta information is required;

[0088] - a ML model ID associated with an analytics ID;

[0089] - a ML model ID associated with one or more of an analytics ID analytics filter information, and analytics target; - a ML model ID associated with one or more ML model accuracy IDs, in which each ML model accuracy identifier identifies at least one ML model ID association with at least one analytics ID, and analytics filter information and / or analytics target; and

[0090] - one or more ML model accuracy IDs, in which each ML model accuracy ID identifies a tuple of a ML model ID and corresponding analytics information.

[0091] NOTE: It is considered that analytics ID and / or ML model ID are included as mandatory information in the indication information for requesting the model accuracy meta information (if sent).

[0092] Optionally, the ML model accuracy meta information may comprise a tuple (or an association) of: analytics ID, and analytics filter and / or analytics target.

[0093] Optionally, the ML model accuracy meta information may further comprise reached analytics accuracy information per distinct analytics filter and / or analytics target and / or analytics subscription associated with the unique ML model ID.

[0094] Step 105: Based on the model accuracy binding information and / or the model accuracy meta information, MTLF 110 is configured to determine whether to update the ML model information.

[0095] Either one of the model accuracy binding information and the model accuracy meta information may have sufficient information for MTLF 110 to determine specific analytics usage scenario. Thus, based on either one or both of the model accuracy binding information and the model accuracy meta information, MTLF 110 is able to determine whether to update the ML model information (and / or the ML model identified by the ML model information).

[0096] According to this disclosure, based on the received ML model accuracy information enhanced with ML model accuracy meta information from AnLF, MTLF 110 may be configured to determine whether to update the ML model based on the model accuracy binding information and / or the ML model accuracy meta information. For instance, MTLF 110 may be configured to trigger ML model re-training for a given unique ML model identifier associated with model accuracy meta information, or select a new unique ML Model identifier (and associated ML Model information) to be re-associated with an analytics ID.

[0097] Optionally, MTLF 110 may be configured to update the ML model information based on the model accuracy meta information. The updated ML Model information may be associated with a new ML Model ID or with a previous ML Model ID.

[0098] Optionally, for updating the ML model information, MTLF 110 may be configured to perform any of the following: determine new ML model information as the updated ML model information; or retrain the ML model to obtain a retrained ML model in order to update the ML model information.

[0099] Step 106: MTLF 110 may be configured to generate an ML model specialization indication, which comprises an indication of updates (or changes) in: the ML model information; and / or ML model filter information; and / or ML model target; and / or ML Model ID associated to the tuple of Analytics IDs, and Analytics Filter(s) and / or Target / s), and / or to the ML Model accuracy meta information.

[0100] Step 107 : MTLF 110 may be configured to provide to AnLF 120 the ML Model specialization indication for an analytics ID when it sends a notification for providing new ML Model related information. Based on the received information, AnLF 120 updates the unique ML model identifier and / or the ML model information used for the generation of the analytics IDs matching the tuple of Analytics IDs, and Analytics Filters) and / or Target(s), which is comprised in the ML Model specialization indication for an analytics ID.

[0101] In general, MTLF 110 may be configured to provide the updated ML model information and / or a machine learning model specialization indication to AnLF 120. The machine learning model specialization indication is indicative of an association of the ML model information and / or the updated ML model information, and corresponding analytics information

[0102] In general, an NWDAF with MTLF capable of differentiating ML model accuracy per analytics ID, ML model identifier, and ML Model Filter and / or Analytics Filter Information is proposed. Accordingly, the NWDAF with MTLF may trigger a ML- retraining to refine a generic ML Model (e.g., with broad ML Model Filter Information) into a more specific trained ML model (e.g., with specialized ML Filter Information). For binding a unique ML Model ID to multiple use in analytics inference, MTLF is configured to obtain the accuracy binding information, which is the mapping of an analytics ID, and analytics filter information and / or analytics target. The mapping is associated with a given unique ML Model identifier, optionally with ML Model Filter Information and / or ML Model target.

[0103] In this way, the risk of MTLF to determine imprecise quality about the ML Model for an analytics ID used in different situations may be reduced. Further, the changes of wasting network resources or computation capabilities in re-training of ML models may be reduced. Thus, the MTLF capability to effectively identify which an inferior ML model can be improved.

[0104] In general, the model accuracy binding information of this disclosure may comprise the following information: 1. analytics information, 2. ML model information associated with the analytics information.

[0105] The analytics information is used to identify an analytics service and its specific usage scenario. The analytics information is indicative of (or comprises) the following information: 1. an analytics ID, 2. analytics target and / or analytics filter information associated with the analytics ID.

[0106] Forinstance, the analytics information may comprise: 1. an analytics ID, 2. analytics filter information associated the analytics ID. Alternatively, the analytics information may comprise: 1. an analytics ID, 2. analytics target information associated with the analytics ID. Alternatively, the analytics information may comprise: 1. an analytics ID, 2. analytics target and analytics filter information associated with the analytics ID.

[0107] The ML model information is used to identify an ML model. Optionally, the ML model information may indicate specific usage scenario of the ML model. The ML model information comprises an ML model ID, and optionally may comprise ML model filter and / or ML target of a ML model identified by the ML model ID.

[0108] It is noted that the model accuracy binding information may comprise association information between multiple pieces of analytics information and multiple pieces of ML model information. An example of the model accuracy binding information may be illustrated in the Table 1 below.

[0109] Table 1

[0110] Table 1 exemplarily shows four associations. Al and A2 referto two different analytics services. Fl, F3 and F5 refer to different analytics filter information. Al-Fl refers to analytics filter Fl is applied to (or is associated with) analytics service Al. The similar notion applies to analytics analytics target information T1 and T3. Ml, M2, M3 andM4 referto three different models. F2 and F4 refer to ML model filter information. M1-F2 refers to filter information F2 is applied to (or is associated with) ML model Ml . The similar notion applies to ML model target information T2 and T4.

[0111] It is noted that the information in Table 1 is only for illustrations purpose only and does not mean the actual content comprises in the model accuracy binding information. The model accuracy binding information may comprise one or more of such association illustrated in Table 1. Optionally, one analytics ID may be associated with more than one analytics filter information, and / or with more than one analytics target information. For instances, as shown above in Table 1 , analytics filters F5 and F6 are associated with analytics ID A2.

[0112] Optionally, the ML Model accuracy meta information may comprise (or indicate information on) one or more of the following information: an association of [unique ML Model ID, Analytics ID, and Analytics Filter and / or Analytics Target]; an association of [unique ML Model ID, ML Model accuracy information, Analytics ID, and Analytics Filter and / or Analytics Target]; an association of [ML Model accuracy information, Analytics ID, and Analytics Filter and / or Analytics Target]; an association of [Analytics ID, and Analytics Filter and / or Analytics Target] ; consumer NF ID; unique ML model ID;

[0113] ML Model Information;

[0114] Analytics ID;

[0115] Analytics Filter;

[0116] Analytics Target; and

[0117] ML Model accuracy information.

[0118] NOTE: It is considered that either the association of [unique ML Model ID, Analytics ID, and Analytics Filter and / or Analytics Target] or the association of [Analytics ID, and Analytics Filter and / or Analytics Target] are included as mandatory information in the ML Model accuracy meta information.

[0119] Optionally, the ML model accuracy meta information request may comprise one or more of the following: a flag indicating the request for ML Accuracy meta information; feedback type defining the type of ML Model accuracy information to be generated (and / or provided) to an entity requesting such information. Another possible understanding for the feedback type is that it defines the tuple that identifies the generated (and / or monitored and / or provisioned) ML Model accuracy information. Examples of possible values for the feedback type are: general, or specific;

[0120] Analytics Accuracy Request information; an association of [analytics ID, and Analytics Filter and / or Analytics Target]; an association of [unique ML Model Identifier, analytics ID, and Analytics Filter and / or Analytics Target]; and an association of [unique ML Model Identifier, ML Model Filter Identifier, analytics ID, and Analytics Filter and / or Analytics Target],

[0121] FIG. 2 shows an example of a method of this disclosure. In this example, the following principles are used:

[0122] An ML Model ID is directly associated to an association of [analytics ID, and Analytics Filter and / or Analytics Target]; and

[0123] ML Model accuracy meta information is indicative of the association of [analytics ID, and Analytics Filter and / or Analytics Target],

[0124] The method in FIG. 2 comprises the following steps 201-208, which are built based on the steps shown in FIG. 1. Thus, features mentioned in FIG. 2 may be applied to the steps in FIG. 1.

[0125] Step 201 : The AnLF subscribes for one or more trained ML Model(s) associated with one or more Analytics ID(s) by invoking the Nnwdaf_MLModelProvision_Subscribe service operation from MTLF. The AnLF receives the ML Model and its associated information. For each analytics ID, AnLF is aware of the following association: analytics ID, unique ML Model identifier, ML Model Information. This association information enables AnLF to uniquely related an analytics ID to a ML Model.

[0126] Step 202: The AnLF is configured to register with the MTLF that provided an ML Model to an analytics ID. The AnLF may invoke the Nnwdaf JMLModelMonitor_Register service operation from MTLF including the accuracy binding information comprising one or more associations of [unique ML Model identifier and / or ML Model Filter Information, Analytics ID, Analytics Filter and / or Analytics Target],

[0127] Step 203 (as a general step of steps 203 A-C): The MTLF determines that it needs the ML Model accuracy information from AnLF registered with the ML Model (e.g., denoting it can provide the ML Model accuracy information for such ML Model). MTLF may invoke the Nnwdaf JMLModelMonitor_Subscribe service operation from AnLF including an indication for requesting the ML Model accuracy meta information. The MTLF for instance can decide to include such indication based on the received accuracy binding information. When MTLF receives the accuracy binding information, the MTLF identifies that the AnLF that provided such accuracy binding information is capable of providing the ML model accuracy meta information.

[0128] There are at least three possible examples for indication for requesting the ML Model accuracy meta information, which are shown below in Step 203 A-C.

[0129] Step 203A: MTLF invokes the Nnwdaf JMLModelMonitor_Subscribe service operation from AnLF. In the subscription (or request) the MTLF includes the Unique ML Model identifier, and the indication for requesting the ML Model accuracy meta information. In this example, the indication for requesting the ML Model accuracy meta information may be equivalent to the following parameters in the service request: Analytics ID, ML Model Accuracy Feedback Type: [General, Specific], optional Analytics Accuracy Request information.

[0130] Step 203B: MTLF invokes the Nnwdaf JMLModelMonitor_Subscribe service operation from AnLF. In the subscription (or request) the MTLF includes the Unique ML Model identifier, and the indication for requesting the ML Model accuracy meta information. In this example, the indication for requesting the ML Model accuracy meta information is equivalent to the following parameters in the service request: Analytics ID, Analytics Filter and / or Analytics Target, optional Analytics Accuracy Request information. Step 203C: MTLF invokes the Nnwdaf_MLModelMonitor_Subscribe service operation from AnLF. In the subscription (or request) the MTLF includes the Unique ML Model identifier, and the indication for requesting the ML Model accuracy meta information. In this example, the indication for requesting the ML Model accuracy meta information is equivalent to the following parameters in the service request: Analytics ID, ML Model Accuracy meta information request.

[0131] It is also possible that the indication for requesting the ML Model accuracy meta information may comprise different combinations of the aforementioned information.

[0132] Step 204: Based on the information received from MTLF (e.g., indication for requesting the ML model accuracy meta information), AnLF generates the requested ML Model Accuracy Information.

[0133] If the MTLF did not include any indication for requesting ML Model accuracy meta information and / or if the indication for requesting ML Model accuracy meta information comprised the ML Model Accuracy Feedback Type set to General, the AnLF calculates the ML Model accuracy by taking into account the information (e.g., the ground truth and / or the analytics accuracy information) for a given analytics ID and unique ML Model identifier independent of the analytics Filter and / or Analytics target associated with a subscription to an analytics ID.

[0134] For instance, if ANLF has a subscription A for analytics ID XI and analytics filter for areas of interest set to area 1, area 2 with the unique ML Model identifier Y, and another subscription B for the same analytics ID XI with the same unique ML Model identifier Y but with analytics filter for areas of interest set to area 3, area 4, all the data related to these 2 distinct subscriptions for analytics ID will be used for the single value calculation of the ML Model accuracy information. If for subscription A the analytics accuracy information indicates a performance of 60% and for the subscription B the performance is 99%; the AnLF may calculate that the ML Model accuracy for the unique ML Model identifier Y is near 80%. This kind of calculation including together all data related to a unique ML Model identifier may imprecisely lead MTLF to believe that unique ML Model identifier Y is well trained.

[0135] If the MTLF included an indication for requesting ML Model accuracy meta information and / or if the indication for requesting ML Model accuracy meta information comprised the ML Model Accuracy Feedback Type set to Specific, the AnLF calculates the ML Model accuracy by taking into account the information (e.g., the ground truth and / or the analytics accuracy information) for a given analytics ID and unique ML Model identifier and each different combination of accuracy binding information (e.g., analytics Filter and / or Analytics target associated and analytics ID and unique ML model identifier).

[0136] For instance, if ANLF has a subscription A for analytics ID XI and analytics filter for areas of interest set to area 1, area 2 with the unique ML Model identifier Y, and another subscription B for the same analytics ID XI with the same unique ML Model identifier Y but with analytics filter for areas of interest set to area 3, area 4, AnLF will use the specific data of each different subscription for analytics ID (e.g., information related to the accuracy binding information) to generate the ML Model accuracy information and include the ML Model accuracy meta information. For instance, the algorithm inside AnLF operating with ML Model accuracy Meta Information may be designed to consider put different weight in the ground truth of the different areas that can be included in an Analytics filter. In this way, the AnLF could then detect that for areas 1 and 2 (e.g., with weights that highlight good calculated performance) may have a ML Model accuracy information of 80%, while for the same unique ML model identifier used in different areas of the mobile network (e.g., with weights that highlight a bad performance) the ML Model Accuracy information could be 30%. With such distinguish information, AnLF is able to provide refined ML Model accuracy information to MTLF with the meta information supporting MTLF decisions. Step 205A: If the request from MTLF did not include any indication for requesting ML Model accuracy meta information and / or if the indication for requesting ML Model accuracy meta information comprised the ML Model Accuracy Feedback Type set to General, the AnLF provides to MTLF the ML Model accuracy information by invoking the Nnwdaf_MLModelMonitor_Notify service operation from MTLF. In this case, no ML Model accuracy meta information is included in the notification.

[0137] Step 205B: In alternative to step 205A, if the request from MTLF included an indication for requesting ML Model accuracy meta information and / or if the indication for requesting ML Model accuracy meta information comprised the ML Model Accuracy Feedback Type set to Specific, the AnLF provides to MTLF the one or more tuple of [ML Model accuracy information with the associated ML Model accuracy meta information] by invoking the Nnwdaf_MLModelMonitor_Notify service operation from MTLF.

[0138] For instance, a possible example of one instance of the tuple can be defined as [unique ML Model identifier, ML Model Accuracy information, Analytics ID, Analytics Filter and / or Analytics Target, optional ML Model Filter Information and / or ML Model Target Information], In this example, the ML Model accuracy meta information is equivalent to the following terms of the tuple [Analytics ID, Analytics Filter and / or Analytics Target, optionally ML Model Filter Information and / or ML Model Target Information], and such ML model accuracy information defines that the ML Model accuracy information associated with the unique ML Model identifier is further associated with the further term of such tuple (i.e., the ML Model accuracy meta information). In another possible embodiment of the ML Model Accuracy meta information, such information is comprised of the following terms: (ML Model accuracy information, Analytics ID, Analytics Filter and / or Analytics Target, optionally ML Model Filter Information and / or ML Model Target Information). In this case, the association of the ML Model accuracy information is part of the ML Model accuracy meta information, and it would be equivalent to define the unique ML model identifier is associated with the ML Model Accuracy meta information.

[0139] Step 206: Based on the information received from AnLF (e.g., for one unique ML model identifier one or more tuple of (ML model accuracy information, ML Model accuracy meta information), the MTLF can detect whether for a given unique ML model identifier, changes are necessary.

[0140] Examples of possible changes enforced by the MTLF based on the ML Model accuracy meta information received from AnLF are described as follows. a) Creating and / or train a new ML Model with a new unique ML Model identifier based on the previous ML Model associated with the unique ML model identifier further related to a received ML Model Accuracy meta information. In this case, the new trained and / or created ML Model can be a refinement of the previous ML model, for instance, can be trained with only part of the entire dataset (and / or type of data set information and / or type of data set characteristics). For instance, if a previous ML Model was trained with the data set characteristic, for instance, representing all tracking areas of the mobile network, the new ML model is a specialization of the previous ML model for instance because it is trained for a reduce list (or set) of tracking areas of the mobile network. This is only to possible type o data set characteristic, other example may include: type of application, type of slice, slice identification, type of radio access technology, type of users (e.g., stationary users, mobile users), type sessions, data network identifications, and any other field defined in the Analytics Filter Information as per 3GPP TS 23.288). b) Selecting a new ML model and associated new unique ML Model identifier, where the new ML Model is associated with ML Model Filters that are better suitable analytics ID associated with the ML Model accuracy meta information. For instance, the MTLF could select a new ML Model that was already trained to support the specific analytics filter information indicated in the ML Model accuracy meta information. Step 207 : The MTLF provides to AnLF a notification indicating changes in the ML Model association to an analytics ID the Nnwdaf_MLModelProvision_Notify service operation. In one possible embodiment, the input parameters of the message include the indication of ML Model specialization which can comprise any of the possible information:

[0141] Tuple with [new unique ML Model identifier, analytics ID, Analytics Filter Information and / or Analytics Target, new ML Model Information, new ML Model Filter Information]

[0142] Tuple with [previous unique ML model identifier, new unique ML Model identifier, analytics ID, Analytics Filter Information and / or Analytics Target, new ML Model Information, new ML Model Filter Information],

[0143] Step 208: Based on the ML Model specialization (or indication of ML model specialization), the AnLF determines which analytics IDs associated with a previous unique ML model identifier should be re-associated with the new unique ML model identifier and / or new ML Model Information. The AnLF then re-associates the analytics IDs with Analytics Filter and / or Analytics target to the new unique ML Model identifier and / or ML Model Information .

[0144] The AnLF may trigger the its registration as provider of ML Model accuracy information as per step 202 with the new unique ML model identifier and respective accuracy binding information.

[0145] It is also possible that the AnLF changes the existing accuracy binding information from the existing registration information for the previous unique ML model with the analytics ID to reflect the dissociation of the analytics ID with analytics filters and / or analytics target with the previous unique ML model identifier. This allows MTLF to keep the consistent map of which ANLF is providing the ML Model accuracy for the unique ML Model identifier with its proper accuracy binding information.

[0146] FIG. 3 shows a further example of a method of this disclosure. In this example, the following principles are used:

[0147] When AnLF register to MTLF to indicate its capability of ML Model monitoring for a given unique ML model identifier, these two entities exchange information in order to pair the association of unique ML Model identifier to new proposed Unique ML Model Accuracy Identifier; and

[0148] ML Model accuracy meta information is indicative of unique ML Model Accuracy Identifier, which is associated with the tuple [analytics ID, and Analytics Filter and / or Analytics Target],

[0149] The method in FIG. 3 comprises the following steps 301-308, which are built based on the steps in FIG. 1. Features introduced with respect to FIG. 3 may be similarity applied to FIG. 1.

[0150] Step 301: The same as Step 201 in FIG. 2.

[0151] Step 302: The AnLF decides to register with the MTLF that provided an ML Model to an analytics ID.

[0152] Step 302A: The AnLF may invoke the Nnwdaf_MLModelMonitor_Register request service operation from MTLF including the one or more accuracy binding information, comprising one or more tuples of [unique ML Model identifier and / or ML Model Filter Information, Analytics ID, Analytics Filter and / or Analytics Target],

[0153] Step 302B: The MTLF is configured to map, for each tuple [unique ML model identifier, accuracy binding information], a unique ML Model Accuracy identifier, and then provide to the AnLF the in the Nnwdaf JMLModelMonitor_Register response service operation such mapping information, e.g., for each unique ML model identifier the unique ML Model accuracy identifier for an accuracy binding information. This received mapping information is stored by AnLF so that it can use such mapping information later on to generate the ML model accuracy meta information for the one or more ML Model accuracy information associated with the same unique ML Model identifier. Step 303 : The MTLF determines that it needs the ML Model accuracy information from AnLF registered with the ML Model (e.g., denoting it can provide the ML Model accuracy information for such ML Model). MTLF may invokes the Nnwdaf_MLModelMonitor_Subscribe service operation from AnLF including an indication for requesting the ML Model accuracy meta information. The MTLF for instance may decide to include such indication based on the received accuracy binding information. When MTLF receives the accuracy binding information, the MTLF identifies that the AnLF that provided such accuracy binding information is capable of providing the ML model accuracy meta information.

[0154] In this example, the indication for requesting the ML model accuracy meta information may be equivalent to the MTLF providing (e.g., including in the request) the one or more unique ML Model accuracy identifiers associated with the unique ML Model identifier. It is also possible that the indication for requesting the ML Model accuracy meta information may comprise ML Model Accuracy Feedback Type: [General, Specific], and / or Analytics Accuracy Request information.

[0155] Step 304: Based on the information received from MTLF (e.g., indication for requesting the ML model accuracy meta information), AnLF generates the requested ML Model Accuracy Information.

[0156] If the MTLF does not include any indication for requesting ML Model accuracy meta information and / or if the indication for requesting ML Model accuracy meta information comprised the ML Model Accuracy Feedback Type set to General, AnLF calculates the ML Model accuracy in the same way as described in Step 204 in Figure 2.

[0157] If the MTLF includes an indication for requesting ML Model accuracy meta information and / or if the indication for requesting ML Model accuracy meta information comprised the ML Model Accuracy Feedback Type set to Specific, the AnLF calculates the ML Model accuracy by taking into account the information (e.g., the ground truth and / or the analytics accuracy information) for a given analytics ID and unique ML Model identifier and each different combination of unique ML Model accuracy identifier and / or accuracy binding information (e.g., analytics Filter and / or Analytics target associated and analytics ID and unique ML model identifier). The same example of calculation listed in step 204 from FIG. 2 may apply to this embodiment.

[0158] Step 305 A: If the request from MTLF does not include any indication for requesting ML Model accuracy meta information and / or if the indication for requesting ML Model accuracy meta information comprised the ML Model Accuracy Feedback Type set to General, the AnLF provides to MTLF the ML Model accuracy information by invoking the Nnwdaf_MLModelMonitor_Notify service operation from MTLF. In this case, no ML Model accuracy meta information is included in the notification.

[0159] Step 305B: If the request from MTLF included an indication for requesting ML Model accuracy meta information and / or if the indication for requesting ML Model accuracy meta information comprised the ML Model Accuracy Feedback Type set to Specific, the AnLF provides to MTLF the one or more tuple of [ML Model accuracy information with the associated ML Model accuracy meta information] by invoking the Nnwdaf_MLModelMonitor_Notify service operation from MTLF.

[0160] For instance, an example of one instance of the tuple can be defined as ML Model Accuracy information for a unique ML Model identifier associated with a unique ML Model accuracy identifier. In this possible example, the ML Model accuracy meta information is equivalent to the following terms of such tuple [a unique ML Model identifier associated with a unique ML Model accuracy identifier].

[0161] Step 306: The same as Step 206 in FIG. 2.

[0162] Step 307 : Based on step 207 in FIG.2 with the difference that: The MTLF provides to AnLF a notification indicating changes in the ML Model association to an analytics ID the Nnwdaf_MLModelProvision_Notify service operation. In one possible embodiment, the input parameters of the message include the indication of ML Model specialization which can comprise any of the possible information:

[0163] Tuple with [new unique ML Model identifier, analytics ID, unique ML Model Accuracy identifier, new ML Model Information, new ML Model Filter Information]

[0164] Tuple with [previous unique ML model identifier, new unique ML Model identifier, analytics ID, unique ML Model Accuracy identifier, new ML Model Information, new ML Model Filter Information].

[0165] Step 308: The same as step 208 in FIG. 2.

[0166] FIG. 4 shows a diagram of a method applied to a first network function entity of this disclosure. The method comprises the following steps:

[0167] Step 401 : obtaining, by the first network function entity, model accuracy binding information from a second network function entity, in which the model accuracy binding information is indicative of at least one association between analytics information and ML model information, in which the analytics information is indicative of at least one association between an analytics ID, and one or more of analytics filter information and analytics target;

[0168] Step 402: obtaining, by the first network function entity, model accuracy information associated with model accuracy meta information from the second network function entity (120), wherein the model accuracy meta information is indicative of an association of the ML model accuracy information and the analytics information; and

[0169] Step 403 : determining, by the first network function entity, whether to update the ML model information based on the model accuracy binding information and / or the model accuracy meta information.

[0170] The method of FIG. 4 may share the corresponding features mentioned above with respect to FIG. 1-3.

[0171] FIG. 5 shows a diagram of a method applied to a second NF entity of this disclosure. The method comprises the following steps:

[0172] Step 501: providing, by the second network function entity, model accuracy binding information to a first network function entity, wherein the model accuracy binding information is indicative of at least one association between an analytics information and ML model information, wherein the analytics information is indicative of at least one association between an analytics identifier, ID, and one or more of analytics filter information and analytics target;

[0173] Step 502: obtaining, by the second network function entity, model accuracy information of the ML model information;

[0174] Step 503 : generating, by the second network function entity, model accuracy meta information indicating an association of the ML model accuracy information and the analytics information; and

[0175] Step 504: providing, by the second network function entity, the model accuracy information associated with the model accuracy meta information to the first network function entity.

[0176] The method of FIG. 5 may share the corresponding features mentioned above with respect to FIG. 1-4.

[0177] In summary, the present disclosure provides a solution for optimizing ML model accuracy monitoring, provisioning, re-training, and update in a mobile communication network. This disclosure may be applied to any type of mobile communication network including but not limited to 5G, 6G cellular system, or any network built based on such mobile communication network (e.g., loT network, V2X network, etc.).

[0178] It is noted that the entities in the present disclosure may comprise processing circuitry configured to perform, conduct or initiate the various operations of the device described herein, respectively. The processing circuitry may comprise hardware and software. The hardware may comprise analog circuitry or digital circuitry, or both analog and digital circuitry. The digital circuitry may comprise components such as application-specific integrated circuits (ASICs), field-programmable arrays (FPGAs), digital signal processors (DSPs), or multi-purpose processors. Optionally, the processing circuitry comprises one or more processors and a non-transitory memory connected to the one or more processors. The non-transitory memory may carry executable program code which, when executed by the one or more processors, causes the device to perform, conduct or initiate the operations or methods described herein, respectively.

[0179] The present disclosure has been described in conjunction with various aspects as examples as well as implementations. However, other variations can be understood and effected by those persons skilled in the art and practicing the claimed subject matter, from the studies of the drawings, this disclosure and the independent claims. In the claims as well as in the description the word “comprising” does not exclude other elements or steps and the indefinite article “a” or “an” does not exclude a plurality. A single element or another unit may fulfill the functions of several entities or items recited in the claims. The mere fact that certain measures are recited in the mutual different dependent claims does not indicate that a combination of these measures cannot be used in an advantageous implementation.

Claims

CLAIMS1. A first network function entity (110) for machine learning, ML, model training in a mobile communication network, the first network function entity (110) being configured to: obtain model accuracy binding information from a second network function entity (120), wherein the model accuracy binding information is indicative of at least one association between analytics information and ML model information, wherein the analytics information is indicative of at least one association between an analytics identifier, ID, and one or more of analytics filter information and analytics target; obtain model accuracy information associated with model accuracy meta information from the second network function entity (120), wherein the model accuracy meta information is indicative of an association of the ML model accuracy information and the analytics information; and determine whether to update the ML model information based on the model accuracy binding information and / or the model accuracy meta information.

2. The first network function entity (110) according to claim 1, wherein the model accuracy meta information is further indicative of an association of analytics accuracy information and one or more of: an analytics ID, the analytics filter information, the analytics target, and the ML model information.

3. The first network function entity (110) according to claim 1 or 2, wherein the first network function entity ( 110) is configured to obtain the model accuracy binding information from a request from the second network function entity (120), wherein the request is indicative that the second network function entity (120) is able to provide the ML model accuracy information with respect to the ML model information comprised in the model accuracy binding information.

4. The first network function entity (110) according to any one of claims 1 to 3, wherein the first network function entity (110) is configured to send indication information to the second network function entity (120) for requesting the model accuracy meta information.

5. The first network function entity (110) according to claim 4, wherein the indication information for requesting the model accuracy meta information is comprised in a subscription request sent by the first network function entity (110) to the second network function entity (120).

6. The first network function entity (110) according to claim 4 or 5, wherein the indication information for requesting the model accuracy meta information comprises one or more of:- a flag indicating whether the model accuracy meta information is required;- a ML model ID associated with an analytics ID;- a ML model ID associated with one or more of an analytics ID analytics filter information, and analytics target;- a ML model ID associated with one or more ML model accuracy IDs, wherein each ML model accuracy identifier identifies at least one ML model ID association with at least one analytics ID, and analytics filter information and / or analytics target; and- one or more ML model accuracy IDs, wherein each ML model accuracy ID identifies a tuple of a ML model ID and corresponding analytics information.

7. The first network function entity (110) according to any one of claims 1 to 6, further configured to: update the ML model information based on the model accuracy meta information, wherein the updated ML Model information is associated with a new ML Model ID or with a previous ML Model ID.

8. The first network function entity (110) according to claim 7, wherein for updating the ML model information, the first network function entity (110) is configured to perform any of the following: determine new ML model information as the updated ML model information; or retrain the ML model to obtain a retrained ML model in order to update the ML model information.

9. The first network function entity (110) according to claim 7 or 8, further configured to: provide the updated ML model information and / or a machine learning model specialization indication to the second network function entity (120), wherein the machine learning model specialization indication is indicative of an association of the ML model information and / or the updated ML model information, and corresponding analytics information.

10. A second network function entity (120) for supporting machine learning, ML, model training in a mobile communication network, the second network function entity (120) being configured to: provide model accuracy binding information to a first network function entity (110), wherein the model accuracy binding information is indicative of at least one association between analytics information and ML model information, wherein the analytics information is indicative of at least one association between an analytics identifier, ID, and one or more of analytics filter information and analytics target; obtain model accuracy information of the ML model information; generate model accuracy meta information indicating an association of the model accuracy information and the analytics information; and provide the model accuracy information associated with the model accuracy meta information to the first network function entity (110).

11. The second network function entity (120) according to claim 10, wherein the model accuracy meta information further comprises analytics accuracy information associated with one or more of an analytics ID, the analytics filter information, the analytics target, and the ML model information.

12. The second network function entity (120) according to claim 10 or 11, wherein the second network function entity (120) is configured to provide the model accuracy binding information to the first network function entity (110) through a request, wherein the request is indicative that the second network function entity (120) is able to provide the ML model accuracy information with respect to the ML model information comprised in the model accuracy binding information.

13. The second network function entity (120) according to any one of claims 10 to 13, wherein for providing the model accuracy meta information, the second network function entity (120) is further configured to receive indication information from the first network function entity (110) for requesting the model accuracy meta information.

14. The second network function entity (120) according to any one of claims 10 to 13, further configured to: receive updated ML model information and / or a machine learning model specialization indication from the first network function entity (110), wherein the machine learning model specialization indication is indicative of an association of the ML model information and / or the updated ML model information, and corresponding analytics information; and update the ML model information based on the updated ML model information and / or the machine learning model specialization indication.

15. A system comprising at least one first network function entity (110) according to any one of claims 1 to 9, and at least one second network function entity (120) according to any one of claims 10 to 14.

16. A method (400) applied to a first network function entity for machine learning, ML, model training in a mobile communication network, the method comprising: obtaining (401) model accuracy binding information from a second network function entity, wherein the model accuracy binding information is indicative of at least one associationbetween analytics information and ML model information, wherein the analytics information is indicative of at least one association between an analytics identifier, ID, and one or more of analytics filter information and analytics target; and obtaining (402) model accuracy information associated with model accuracy meta information from the second network function entity (120), wherein the model accuracy meta information is indicative of an association of the ML model accuracy information and the analytics information; and determining (403) whether to update the ML model information based on the model accuracy binding information and / or the model accuracy meta information.

17. A method (500) applied to a second network function entity for supporting machine learning, ML, model training in a mobile communication network, the method comprising: providing (501) model accuracy binding information to a first network function entity, wherein the model accuracy binding information is indicative of at least one association between an analytics information and ML model information, wherein the analytics information is indicative of at least one association between an analytics identifier, ID, and one or more of analytics filter information and analytics target; obtaining (502) model accuracy information of the ML model information; generating (503) model accuracy meta information indicating an association of the ML model accuracy information and the analytics information; and providing (504) the model accuracy information associated with the model accuracy meta information to the first network function entity.

18. A computer program comprising instructions which, when the program is executed by a computer, cause the computer to carry out the method of claim 16 or 17.

Images