Apparatus, method and system for network analysis accuracy monitoring

Abstract

This invention relates to a mobile communication. A first network function (NF) manages ML models and provides one or more suitable ML models to a second network function (NF). According to the invention, the first NF is used to obtain model accuracy binding information from the second NF. The model accuracy binding information indicates the association between analysis information and ML model information. The analysis information indicates the association between an analysis ID and analysis filter information and / or an analysis target. The first NF obtains model accuracy information associated with model accuracy metadata from the second NF. The model accuracy metadata indicates the association between the ML model accuracy information and the analysis information. The first network function is used to determine whether to update the ML model information based on the model accuracy binding information and / or the model accuracy metadata.

Description

Technical Field

[0001] This invention relates to the field of communications. For example, this invention relates to devices, methods, and systems for monitoring the accuracy of network analysis. Background Technology

[0002] In the context of 5G, 6G and higher versions of communication networks, network data analytics functions (NWDAF) play a crucial role in collecting, processing, analyzing, and providing insights from various network data sources. NWDAF can be used to perform or can include various logical functions for different purposes, such as analytics logical functions (AnLF) and model training logical functions (MTLF).

[0003] The NWDAF including AnLF (also known as NWDAF-AnLF, or simply AnLF) is responsible for collecting analytics requests and sending responses to consumers. To this end, AnLF can be used to gather meaningful information from web data and make decisions (e.g., perform predictions / estimations) using advanced analytics techniques such as machine learning (ML) models (or algorithms). On the other hand, the NWDAF including MTLF (also known as NWDAF-MTLF, or simply MTLF) is responsible for training, managing, and updating the ML models used by AnLF. Summary of the Invention

[0004] 3GPP specification TS 23.288 v18.3.0 defines that AnLF can monitor analysis accuracy and ML model accuracy information. If MTLF is subscribed to receive accuracy information, this information is provided to MTLF, and MTLF can then trigger any of the following actions: - Retraining of ML Models: If accuracy information indicates model degradation, MTLF can trigger model retraining. This process involves updating the model's parameters based on new data and improving its performance.

[0005] - Reselection of ML Models: MTLF also allows reselection of the ML model associated with a specific analysis identifier (ID). This may be necessary if the current model's performance is no longer as expected, or if a new, more accurate or efficient model has been developed.

[0006] The ability to monitor and manage accurate information is crucial for ensuring the reliability and effectiveness of network analytics. By proactively updating and improving ML models, AnLF can provide more accurate and timely insights that can be used to optimize network performance, enhance user experience, and identify and resolve potential problems.

[0007] Current 3GPP specifications allow AnLF to subscribe to the same analytics ID multiple times (e.g., AnLF subscribes to the same analytics ID using different analytics targets (i.e., UEs or UE groups) and / or different analytics filter information), with these subscriptions using the same unique ML model identifier. This means that a trained ML model can be associated with a unique ML model identifier, and the set of ML model filter information and ML model targets (e.g., a general training model for analytics IDs) can be adapted to different specific subscriptions for the actual generation of analytics IDs (e.g., specific scenarios that can be detected / predicted by a general trained ML model).

[0008] Therefore, AnLF may encounter situations where, for a given specific use of a trained ML model (e.g., subscribing to analysis ID "A" using filter "B"), the analysis accuracy is high, but for another specific use of the same trained ML model (e.g., subscribing to analysis ID "A" using filter "C"), the analysis accuracy is low. When AnLF generates ML model accuracy (based on analysis ID and ML model identifier), the accuracy differences for model-specific uses are diluted or even completely eliminated.

[0009] Overall, current ML model provisioning, updating, and retraining schemes may have the following problems.

[0010] First, it fails to consider the specific uses of ML models in different application scenarios. This could lead to sending inaccurate (or incomplete) ML model accuracy information to MTLF, potentially triggering unnecessary actions such as retraining or reselecting the ML model.

[0011] Second, it does not support MTLF in distinguishing between ML models suitable for different application scenarios. This may lead to MTLF retraining or reselecting ML models that are not actually suitable for a specific use case, resulting in wasted resources and suboptimal decisions.

[0012] In view of the above-mentioned problems and drawbacks, the present invention aims to improve network accuracy monitoring and reporting in communication networks. For example, an objective of the present invention could be to improve the ML accuracy information reporting scheme between AnLF and MTLF. Another objective of the present invention could be to prevent MTLF from performing unnecessary signaling and / or determining inaccurate ML model degradation information, and to prevent MTLF from triggering: inaccurate or unnecessary reselection of ML models, or inaccurate decisions not to retrain or reselect ML models, in response to ML model requests from NWDAF and AnLF.

[0013] These and other objectives are achieved through the subject matter of the independent claims. Other implementations are apparent from the dependent claims, the specification, and the drawings.

[0014] A first aspect of the invention provides a first network function (NF) entity (e.g., MTLF) for training (or pre-configuring) an ML model in a mobile communication network. The first NF entity is used to obtain model accuracy binding information from a second network function entity (e.g., AnLF). The model accuracy binding information indicates at least one association between analytics information and ML model information. The analytics information indicates at least one association between an analytics (ID) and one or more of analytics filter information and analytics targets. The first NF entity is also used to obtain model accuracy information associated with model accuracy metadata from the second NF entity. The model accuracy metadata indicates ML model accuracy information and associated analytics information. The ML model accuracy information is the association between the ML model accuracy information and the analytics information. The first NF entity is also used to determine whether to update the ML model information (and / or the ML model identified by the ML model information) based on the model accuracy binding information and / or the model accuracy metadata.

[0015] Optionally, ML model accuracy information can indicate information about the performance and / or quality of the ML model.

[0016] Optionally, model accuracy metadata may be included (or embedded) in model accuracy information, or may be attached to model accuracy information, or may be linked to model accuracy information, etc.

[0017] Model accuracy binding information can be used not only to indicate which analysis (identified by the analysis ID) corresponds to which ML model, but also to indicate the specific use case of the analysis associated with the ML model (identified by analysis filter information and / or analysis target information). Based on the received model accuracy metadata and binding information, the first NF entity can determine whether the ML model is inaccurate or underperforming in a specific use case. Therefore, the risk of the first NF entity making inaccurate judgments about the quality / correctness of the ML model can be reduced for analysis IDs used in different situations.

[0018] It should be noted that the first NF entity is also used to further determine whether to update the ML model based on the model accuracy information.

[0019] In one implementation of the first aspect, the model accuracy metadata may also indicate the association between the analysis accuracy information and one or more of the following: analysis ID, analysis filter information, analysis target, and ML model information.

[0020] Optionally, the analysis accuracy information may indicate performance and / or quality information about the analysis identified by the analysis ID.

[0021] The analysis of accuracy information and / or model accuracy information is used to determine whether the retraining and / or reselection of the ML model conforms to the definition in TS 23.288 V18.3.0, for example, as described in Clauses 6.2E.2 and / or 6.2E.3.3. Therefore, for simplicity, it is not described in detail in this invention.

[0022] In another implementation of the first aspect, a first network functional entity may be used to obtain model accuracy binding information from a request received from the first network functional entity. This request instructs a second network functional entity to provide ML model accuracy information regarding the ML model information included in the model accuracy binding information.

[0023] In this way, the first network function can know that the second network function supports accuracy reporting in specific use cases.

[0024] In another implementation of the first aspect, the first network functional entity can be used to send indication information to the second network functional entity for requesting model accuracy metadata.

[0025] In this way, the first network function can explicitly request the second network function to provide one or more accuracy information for a specific use case.

[0026] In another implementation of the first aspect, the indication information for requesting model accuracy metadata may be included in the subscription request sent by the first network functional entity to the second network functional entity.

[0027] In another implementation of the first aspect, the indication for requesting model precision metadata may include one or more of the following: - A flag indicating whether the model accuracy metadata is required; - The ML model ID associated with the analytics ID; - An ML model ID associated with one or more of the analysis ID, analysis filter information, and analysis target; - An ML model ID associated with one or more ML model precision IDs, wherein each ML model precision identifier identifies the association of at least one ML model ID with at least one analysis ID, analysis filter information and / or analysis target; - One or more ML model precision IDs, where each ML model precision ID identifies a tuple of ML model ID and corresponding analysis information.

[0028] In another implementation of the first aspect, the first NF entity can also be used to update ML model information based on ML model precision metadata. The updated ML model information is associated with either a new ML model ID or a previous ML model ID.

[0029] In another implementation of the first aspect, in order to update the ML model information, the first NF entity can be used to perform any of the following: The new ML model information is determined as the updated ML model information; or The ML model is retrained to obtain a retrained ML model, thereby updating the ML model information.

[0030] In another implementation of the first aspect, the first NF entity can also be used to provide the second network functional entity with updated ML model information and / or ML model specialization instructions. The ML model specialization instructions indicate the association between the ML model information and / or the updated ML model information and the corresponding analysis information.

[0031] In other words, the first NF entity can be used to provide updated ML model information. Alternatively, the first NF entity can be used to provide ML model specialization instructions. Or, the first NF entity can be used to provide both updated ML model information and ML model specialization instructions.

[0032] A second aspect of the invention provides a second NF entity (e.g., AnLF) for supporting ML model training in a mobile communication network. The second NF entity provides model accuracy binding information to a first NF entity (e.g., MTLF) for ML model training. The model accuracy binding information indicates at least one association between analytics information and ML model information. The analytics information indicates at least one association between an analytics ID and one or more of analytics filter information and analytics targets. The second NF entity is also used to acquire model accuracy information (or, of the ML model identified by the ML model information) from the ML model information and generate model accuracy metadata indicating the association between the model accuracy information and the analytics information. The second NF entity provides the first NF entity with the model accuracy information associated with the model accuracy metadata.

[0033] Optionally, in order to obtain model accuracy information, the second NF entity can be used to generate analysis output using the ML model identified by the ML model information, and to verify the analysis output (e.g., at a later point in time) to obtain model accuracy information.

[0034] In one implementation of the second aspect, the model accuracy metadata may further include analysis accuracy information, which is associated with one or more of the analysis ID, analysis filter information, analysis target, and ML model information.

[0035] In another implementation of the second aspect, the second NF entity can be used to request model accuracy binding information from the first network function entity. This request instructs the second network function entity to provide ML model accuracy information regarding the ML model information included in the model accuracy binding information.

[0036] In another implementation of the second aspect, in order to provide model accuracy metadata, the second network functional entity can be used to receive indication information from the first network functional entity for requesting model accuracy metadata.

[0037] In another implementation of the second aspect, the second NF entity can also be used for: Receive updated ML model information and / or machine learning model specialization instructions from the first network functional entity, wherein the machine learning model specialization instructions indicate the association between the ML model information and / or the updated ML model information and the corresponding analysis information; Update the ML model information based on the updated ML model information and machine learning model specialization instructions.

[0038] The second NF entity in the second aspect can have corresponding optional features and the same effects as the first NF entity in the first aspect.

[0039] A third aspect of the present invention provides a system comprising at least one first NF entity according to the first aspect or any implementation thereof and at least one second NF entity according to the second aspect or any implementation thereof.

[0040] A fourth aspect of the present invention provides a method for training an ML model in a mobile communication network for a first NF entity. The method includes the following steps: Obtain model precision binding information from the second NF entity, wherein the model precision binding information indicates at least one association between analysis information and ML model information, wherein the analysis information indicates at least one association between analysis ID and one or more of analysis filter information and analysis target; Obtain model accuracy information associated with model accuracy metadata from the second NF entity, where model accuracy metadata indicates the association between ML model accuracy information and analysis information; Based on model accuracy binding information and / or model accuracy metadata, determine whether to update the ML model.

[0041] The method of the fourth aspect can have the same optional features and the same effects as the first NF entity of the first aspect or any implementation thereof.

[0042] A fifth aspect of the present invention provides a method for supporting ML model training in a mobile communication network for second NF entities. The method includes the following steps: Provide model precision binding information to the first NF entity, wherein the model precision binding information indicates at least one association between analysis information and ML model information, wherein the analysis information indicates at least one association between analysis ID and one or more of analysis filter information and analysis target; Obtain model accuracy information from ML model information; Generate model accuracy metadata, which indicates the correlation between model accuracy information and analysis information; Provide the first NF entity with model accuracy information associated with the model accuracy metadata.

[0043] The fifth aspect of the method can have the same optional features and the same effects as the second NF entity in the second aspect.

[0044] A sixth aspect of the invention provides a computer program including instructions that, when executed by a computer, cause the computer to perform the method according to a fourth or fifth aspect.

[0045] A fifth aspect of the invention provides a computer-readable medium including instructions that, when executed by a computer, cause the computer to perform the method according to the fourth or fifth aspect.

[0046] A fifth aspect of the invention provides a chipset including instructions that, when executed by the chipset, cause the chipset to perform the method according to the fourth or fifth aspect.

[0047] It should be noted that all devices, terminals, elements, units, and methods described in this invention can be implemented using software or hardware elements or any combination thereof. All steps performed by the various entities described in this application, and the functions described as being performed by the various entities, are intended to indicate that the respective entities are used to perform the corresponding steps and functions. Although specific functions or steps performed by external entities are not reflected in the detailed descriptions of the specific elements of the entities performing those steps or functions in the following description, those skilled in the art will understand that these methods and functions can be implemented using corresponding hardware or software elements or any combination thereof. Attached Figure Description

[0048] The following description, in conjunction with the accompanying drawings, will illustrate the various aspects and implementations described above, as shown in the drawings: Figure 1 The first NF entity 110 and the second NF entity 120 of the present invention are shown; Figure 2 An example of the method of the present invention is shown; Figure 3 Another example of the method of the present invention is shown; Figure 4 A diagram illustrating the method applied to the first network functional entity is shown; Figure 5 A diagram illustrating the method applied to the second network functional entity is shown. Detailed Implementation

[0049] Without loss of generality, exemplary explanations of the terminology used in this invention are given below.

[0050] -Accuracy monitoring: A general term used to describe the process of measuring the performance (and / or associated measurements) of identification analytics (e.g., machine learning processes such as inference or training).

[0051] -Analysis Accuracy Monitoring: This activity involves monitoring the accuracy of the analysis ID during the ML inference process performed by an NWDAF with AnLF logic functionality.

[0052] - AnLF-assisted ML model accuracy monitoring: Related to the accuracy monitoring of the ML model (e.g., identified by a unique ML model identifier) ​​and the analysis ID, which is performed by an NWDAF with AnLF logic and provided to an NWDAF with MTLF logic.

[0053] - Feedback Type: Defines the type of ML model accuracy information to be generated (and / or provided) to entities requesting such information. Another possible understanding of the feedback type is that it defines a tuple associated with ML model accuracy information generated (e.g., monitored and / or pre-configured) by or to be generated (e.g., monitored and / or pre-configured) by an NWDAF with AnLF.

[0054] - Typical feedback type: Information to NWDAF with AnLF, i.e., should be monitored and / or generated and / or calculated and / or pre-configured and / or calculated ML model accuracy information based on tuples of [analysis ID, and unique ML model identifier].

[0055] - Specific feedback type: Information indicated by NWDAF with AnLF, i.e., should be monitored and / or generated and / or calculated and / or pre-configured and / or calculated based on ML model accuracy meta-information (e.g., tuples of [unique ML model identifier, analysis ID, analysis filter information and / or analysis target]).

[0056] - Instructions for requesting ML model precision metadata (or instructions for requesting ML model precision metadata or instructions for requesting ML model precision metadata): Defines the request and / or subscription to obtain ML model precision metadata.

[0057] - ML Model Accuracy Meta-information (also known as Model Accuracy Meta-information): Defines the association between a unique ML model identifier and / or ML model information and the ML model accuracy information for the analysis ID, as well as analysis filter information and / or analysis objectives. This association can be understood as a mapping.

[0058] - ML model specialization indication (or ML, model specialization indication): In broader terms, it defines the information (and / or one or more parameters) used to map a unique ML model ID and / or ML model information to an analysis ID, as well as analysis filter information and / or analysis targets. It can also be understood as defining the relationship between a previously unique ML model identifier and / or ML model information associated with an analysis ID and a new unique ML model identifier and / or new ML model information associated with the same analysis ID, and further indicating the specific analysis filter information and / or analysis targets associated with the analysis ID.

[0059] - Precision binding information (or model precision binding information): Defines the information that associates a unique ML model identifier (optionally, ML model filter information and / or ML model target) with the analysis ID, analysis filter information and / or analysis target and / or defines the mapping between them and / or defines the relationship between them.

[0060] -ML model accuracy information (or model accuracy information): Defines performance and / or quality information about the ML model.

[0061] exist Figures 1 to 5 In this context, corresponding components can have the same characteristics and similar functions.

[0062] Figure 1 The first NF entity 110 and the second NF entity 120 of the present invention are illustrated. The first NF entity 110 is used to train an ML model in a mobile communication network. For example, the first NF entity 110 may be an MTLF. In this invention, MTLF is used to refer to the first NF entity 110. The second NF entity 120 is used to support ML model training in a mobile communication network. The second NF entity 120 is used to provide analysis results based on the trained model received from the first entity 110. For example, the second NF entity 120 may be an AnLF. In this invention, AnLF is used to refer to the second NF entity 120.

[0063] In this invention, the NWDAF architecture and Nnwdaf service interface defined in 3GPP TS 23.288 v18.3.0 can be considered as the baseline. The terminology used in this invention may have the same definitions as in TS 23.288 (unless otherwise stated), including but not limited to: analysis ID, analysis filter information (or analysis filter), analysis target (or reported analysis target or target of analysis report), ML model, ML model filter information, ML model information, unique ML model identifier, ML model target (or target of ML model, or reported ML model target).

[0064] like Figure 1 As shown in the figure, MTLF 110 and AnLF 120 are illustrated. To support ML training at MTLF 110, AnLF 120 is used to provide ML model accuracy information to MTLF 110. Based on the ML model accuracy information, subsequent steps can be taken when MTLF 110 detects that the ML model is suffering some performance degradation. Optionally, AnLF 120 can also be used to provide analysis accuracy information to MTLF 110. Based on model accuracy information, and / or model accuracy binding information, and / or model accuracy metadata, and / or analysis accuracy information, MTLF 110 can be used to determine whether to update the ML model if it is unsuitable for analysis or performs poorly. The interaction may include the following steps.

[0065] It should be noted that, according to the definition of analytics and ML accuracy monitoring in 3GPP TS 23.288 V18.3.0, different entities can be used to control the accuracy of the ML model associated with the analytics ID. For example, the NWDAF including AnLF 120 can be used to track (or inspect) the analytics accuracy information of a given ML model #A associated with analytics ID #A through its own parametric tracking (or inspection). The NWDAF including MTLF 110 can also be used to track (or inspect) the ML model accuracy information of the same given ML model #A associated with analytics ID #A through its own parametric tracking (or inspection). Since these can be completely independent processes, it is possible that two entities, AnLF and MTLF 110, can have the same view of the accuracy of the associated ML model #A and analytics ID #A. At a given point in time, the two entities, AnLF and MTLF 110, may also have different views of the accuracy of the associated ML model #A and analytics ID #A. For example, from the perspective of MTLF 110, the ML model accuracy of ML model #A with analysis ID #A at time T may differ from the analysis accuracy of ML model #A with analysis ID #A at time T from the perspective of AnLF. At time T, MTLF 110 may have an ML accuracy value that still reflects the accuracy of ML model #a trained at different times T–x. When MTLF 110 obtains analysis accuracy information from AnLF, MTLF 110 can identify whether the "stored" accuracy of the ML model associated with analysis ID IDA should actually be re-examined.

[0066] Step 101: AnLF 120 uses a service called "Nnwdaf_MLModelMonitor_Register" from MTLF 110 to indicate to MTLF 110 that it is capable of generating ML model accuracy information enhanced with accuracy metadata (e.g., regarding a given analysis ID and ML model ID). This step is optional and not required for the present invention.

[0067] Typically, MTLF 110 can be used to retrieve model accuracy binding information from a request from AnLF 120. This request instructs AnLF 120 to provide ML model accuracy information enhanced with meta-information.

[0068] Step 102: MTLF 110 receives model precision binding information from AnLF 120, which indicates at least one association between analysis information and ML model information. The analysis information indicates at least one association between an analysis ID and one or more of analysis filter information and analysis targets. In other words, the precision binding information may indicate at least one mapping between the analysis ID and analysis filter information and / or analysis targets (e.g., [analysis ID -> analysis filter and / or analysis target]). The analysis information is also associated with ML model information, which includes an ML model ID that identifies the ML model. Optionally, the ML model information may also include ML model filter information and / or ML model target information associated with the ML model. For example, precision binding formation may include information in the following format: [analysis ID -> analysis filter and / or analysis target] -> [ML model ID, (optional) ML model filter information, (optional) ML model target information].

[0069] It should be noted that the analysis / ML model filter information can be used to indicate the type of service to which the analysis / ML model is applied, and the target can be used to indicate one or more UEs / one or more terminals to which the analysis / ML model is applied.

[0070] Optionally, step 102 can be performed together with step 101. Alternatively, step 102 can be performed separately, for example, using different services invoked between ANLF 120 and MTLF 110.

[0071] Step 103: AnLF 120 uses the ML model (identified by the corresponding ML model ID) indicated by MLTF 110 to perform its analysis collection (or generation) task for each subscription and analysis ID.

[0072] The AnLF 120 is used to provide analytical information to analytics consumers using ML models. During the provision of analytical information, the AnLF 120 can be used to collect / determine the analytical accuracy associated with the ML model supporting the generation of the analytical information. For example, the AnLF 120 can use an ML model to generate analytics for traffic forecasting. Subsequently, when the AnLF 120 acquires actual traffic statistics, it can use these actual traffic statistics (as ground truth) to evaluate the performance of the ML model used to generate the analytical information, thereby obtaining analytical accuracy information for the ML model. In other words, from the AnLF 120's perspective, at a given time, the AnLF 120 determines the ML model used for traffic forecasting analytics. For example, analytical model accuracy information may include the confidence level (0 to 100%) of the prediction results of the ML model used by the AnLF at that moment.

[0073] Step 104: MTLF 110 is used to obtain model accuracy information associated with the model accuracy metadata from AnLF 120. The model accuracy metadata indicates the association between ML model accuracy information and analysis information.

[0074] Optionally, MTLF 110 can be used to send an instruction to AnLF 120 requesting model accuracy metadata. Optionally, MTLF 110 can be used to subscribe to ML model accuracy monitoring information from a registered AnLF 120 for a given analysis ID and a unique ML model ID (e.g., via a subscription request). MTLF 110 can be used to indicate to AnLF 120 whether further refinement should be used to generate ML model accuracy monitoring, taking into account the association mapping from the unique ML model ID to the analysis ID and analysis filter information and / or analysis target information. When AnLF 120 accepts the subscription, AnLF 120 can be used to monitor ML model accuracy information according to the instruction received from MTLF 110 and provide MTLF 110 with ML model accuracy information including ML model accuracy metadata.

[0075] Optionally, the indication information requesting model accuracy metadata may include one or more of the following: - A flag indicating whether the model accuracy metadata is required; - The ML model ID associated with the analytics ID; - An ML model ID associated with one or more of the analysis ID, analysis filter information, and analysis target; - An ML model ID associated with one or more ML model precision IDs, wherein each ML model precision identifier identifies the association of at least one ML model ID with at least one analysis ID, analysis filter information and / or analysis target; - One or more ML model precision IDs, where each ML model precision ID identifies a tuple of ML model ID and corresponding analysis information.

[0076] Note: It can be assumed that the analysis ID and / or ML model ID are mandatory information included in the indication information for requesting model accuracy metadata (if sent).

[0077] Optionally, the ML model accuracy metadata may include tuples (or associations) of the following items: -Analyze ID; -Analyze filters and / or analyze targets.

[0078] Optionally, the ML model accuracy metadata may also include the analytical accuracy information achieved by each different analytical filter and / or analytical target and / or analytical subscription associated with the unique ML model ID.

[0079] Step 105: Based on model accuracy binding information and / or model accuracy metadata, MTLF 110 is used to determine whether to update the ML model information.

[0080] Either the model accuracy binding information or the model accuracy metadata can provide sufficient information for MTLF 110 to determine a specific analysis use case. Therefore, based on one or both of the model accuracy binding information and the model accuracy metadata, MTLF 110 can determine whether to update the ML model information (and / or the ML model identified by the ML model information).

[0081] According to the present invention, based on ML model accuracy information enhanced with ML model accuracy metadata received from AnLF, MTLF 110 can be used to determine whether to update the ML model based on model accuracy binding information and / or ML model accuracy metadata. For example, MTLF 110 can be used to trigger ML model retraining for a given unique ML model identifier associated with the model accuracy metadata, or to select a new unique ML model identifier (and associated ML model information) to re-associate with the analytics ID.

[0082] Optionally, MTLF 110 can be used to update ML model information based on model accuracy metadata. The updated ML model information can be associated with a new ML model ID or a previous ML model ID.

[0083] Alternatively, to update ML model information, MTLF 110 can be used to perform any of the following: The new ML model information is determined as the updated ML model information; or The ML model is retrained to obtain a retrained ML model, thereby updating the ML model information.

[0084] Step 106: MTLF 110 can be used to generate ML model specialization indications, which include indications of updates (or changes) to the following: ML model information; and / or ML model filter information; and / or ML model targets; and / or ML model ID, associated with an analysis ID and a tuple of one or more analysis filters and / or one or more targets and / or associated with ML model accuracy metadata.

[0085] Step 107: MTLF 110 may provide AnLF 120 with an ML model specialization instruction for the analysis ID when it sends a notification providing information related to a new ML model. Based on the received information, AnLF 120 updates the unique ML model identifier and / or ML model information used to generate the analysis ID, which matches the analysis ID, one or more analysis filters, and / or one or more targets in the ML model specialization instruction for the analysis ID.

[0086] Typically, MTLF 110 can be used to provide AnLF 120 with updated ML model information and / or machine learning model specialization instructions. Machine learning model specialization instructions indicate the association between ML model information and / or updated ML model information and corresponding analysis information.

[0087] Typically, a non-dimensional data retrieval and analysis (NWDAF) method with MTLF is proposed, capable of distinguishing ML model precision based on the analysis ID, ML model identifier, and ML model filter and / or analysis filter information. Accordingly, an NWDAF with MTLF can trigger ML retraining to refine general ML models (e.g., with generalized ML model filter information) into more specific trained ML models (e.g., with dedicated ML filter information). To bind a unique ML model ID to multiple uses in analysis inference, MTLF is used to obtain precision binding information, which is a mapping between the analysis ID and analysis filter information and / or analysis objective. This mapping is associated with a given unique ML model identifier, and optionally with ML model filter information and / or ML model objective.

[0088] In this way, the risk of MTLF's inaccurate judgment of ML model quality can be reduced for analysis IDs used in different situations. Furthermore, it can reduce the waste of network resources or computational power during ML model retraining. Therefore, it can improve MTLF's ability to effectively identify which ML models are of poor quality.

[0089] Typically, the model accuracy binding information of this invention may include the following information: 1. Analysis information, 2. ML model information associated with the analysis information.

[0090] Analysis information is used to identify the analysis service and its specific use case. Analysis information indicates (or includes) the following: 1. Analysis ID, 2. Analysis target and / or analysis filter information associated with the analysis ID.

[0091] For example, the analysis information may include: 1. Analysis ID, 2. Analysis filter information associated with the analysis ID. Alternatively, the analysis information may include: 1. Analysis ID, 2. Analysis target information associated with the analysis ID. Or, the analysis information may include: 1. Analysis ID, 2. Analysis target and analysis filter information associated with the analysis ID.

[0092] ML model information is used to identify ML models. Optionally, ML model information may indicate a specific use case for the ML model. ML model information includes the ML model ID, and optionally may include ML model filters and / or the ML target of the ML model identified by the ML model ID.

[0093] It should be noted that model accuracy binding information can include the association information between multiple analysis information and multiple ML model information. Examples of model accuracy binding information are shown in Table 1 below.

[0094] Table 1

[0095] Table 1 illustrates four types of associations. A1 and A2 refer to two different analysis services. F1, F3, and F5 refer to different analysis filter information. A1-F1 means that analysis filter F1 is applied to (or associated with) analysis service A1. A similar concept applies to analysis target information T1 and T3. M1, M2, M3, and M4 refer to three different models. F2 and F4 refer to ML model filter information. M1-F2 means that filter information F2 is applied to (or associated with) ML model M1. A similar concept applies to ML model target information T2 and T4.

[0096] It should be noted that the information in Table 1 is for illustrative purposes only and does not imply the actual content included in the model accuracy binding information. Model accuracy binding information may include one or more of the associations shown in Table 1. Optionally, an analysis ID may be associated with more than one analysis filter and / or more than one analysis target. For example, as shown in Table 1 above, analysis filters F5 and F6 are associated with analysis ID A2.

[0097] Optionally, the ML model accuracy metadata may include one or more of the following information (or information indicating one or more of the following information): - The association of [unique ML model ID, analysis ID, and analysis filter and / or analysis target]; - The association of [unique ML model ID, ML model accuracy information, analysis ID, and analysis filters and / or analysis targets]; - The association of [ML model accuracy information, analysis ID, and analysis filters and / or analysis targets]; - Association of [analysis ID, analysis filter, and / or analysis target]; -Consumer NF ID; - Unique ML model ID; -ML model information; -Analyze ID; -Analyze filters; -Analysis objectives; -ML model accuracy information.

[0098] Note: It can be assumed that the association of [unique ML model ID, analysis ID, and analysis filter and / or analysis target] or the association of [analysis ID and analysis filter and / or analysis target] includes mandatory information in the ML model accuracy metadata.

[0099] Optionally, the ML model accuracy metadata request may include one or more of the following: - A flag indicating a request for ML precision metadata; - Feedback type defines the type of ML model accuracy information to be generated (and / or provided) to entities requesting such information. Another possible understanding of feedback type is that it defines a tuple that identifies the ML model accuracy information generated (and / or monitored and / or pre-configured). Examples of possible values ​​for feedback type include: general or specific; -Analyze accuracy request information; - Association of [analysis ID, analysis filter, and / or analysis target]; - The association of [unique ML model identifier, analysis ID, and analysis filter and / or analysis target]; - Association of [unique ML model identifier, ML model filter identifier, analysis ID and analysis filter and / or analysis target].

[0100] Figure 2 An example of the method of the present invention is shown. In this example, the following principles are used: - The ML model ID is directly associated with the [analysis ID, analysis filter, and / or analysis target]; -ML model accuracy metadata indicates the association of [analysis ID, analysis filter, and / or analysis target].

[0101] Figure 2 The method includes steps 201 to 208, which are based on Figure 1 The steps shown are used to construct it. Therefore, Figure 2 The features mentioned can be applied to Figure 1 The steps.

[0102] Step 201: AnLF subscribes to one or more trained ML models associated with one or more analytics IDs by calling the Nnwdaf_MLModelProvision_Subscribe service operation from MTLF. AnLF receives the ML models and their association information. For each analytics ID, AnLF knows the following associations: analytics ID, unique ML model identifier, and ML model information. This association information allows AnLF to uniquely associate an analytics ID with an ML model.

[0103] Step 202: AnLF is used to register with the MTLF that provides the ML model for the analysis ID. AnLF can invoke the Nnwdaf_MLModelMonitor_Register service operation from the MTLF, which includes precision binding information, including one or more associations of [unique ML model identifier and / or ML model filter information, analysis ID, analysis filter and / or analysis target].

[0104] Step 203 (as a general step in steps 203A through 203C): The MTLF determines that it needs ML model precision information from an AnLF that has registered with the ML model (e.g., indicating that it can provide ML model precision information for such an ML model). The MTLF may invoke the Nnwdaf_MLModelMonitor_Subscribe service operation to the AnLF, which includes an indication for requesting ML model precision metadata. For example, the MTLF may decide to include this indication based on received precision binding information. When the MTLF receives precision binding information, the MTLF recognizes that the AnLF providing this precision binding information is capable of providing ML model precision metadata.

[0105] There are at least three possible examples of instructions for requesting ML model precision metadata, as shown in steps 203A to 203C below.

[0106] Step 203A: MTLF invokes the Nnwdaf_MLModelMonitor_Subscribe service operation from AnLF. In the subscription (or request), MTLF includes a unique ML model identifier and an indication for requesting ML model accuracy metadata. In this example, the indication for requesting ML model accuracy metadata can be equivalent to the following parameters in the service request: Analysis ID, ML model accuracy feedback type: [General, Specific], and optional analysis accuracy request information.

[0107] Step 203B: MTLF invokes the Nnwdaf_MLModelMonitor_Subscribe service operation from AnLF. In the subscription (or request), MTLF includes a unique ML model identifier and an indication for requesting ML model precision metadata. In this example, the indication for requesting ML model precision metadata is equivalent to the following parameters in the service request: analysis ID, analysis filter and / or analysis target, and optional analysis precision request information.

[0108] Step 203C: MTLF invokes the Nnwdaf_MLModelMonitor_Subscribe service operation from AnLF. In the subscription (or request), MTLF includes a unique ML model identifier and an indication for requesting ML model precision metadata. In this example, the indication for requesting ML model precision metadata is equivalent to the following parameters in the service request: Analysis ID, ML model precision metadata request.

[0109] Indicators used to request ML model accuracy metadata may also include different combinations of the above information.

[0110] Step 204: Based on the information received from MTLF (e.g., an indication for requesting ML model accuracy metadata), AnLF generates the requested ML model accuracy information.

[0111] If MTLF does not include any indication for requesting ML model accuracy metadata and / or if the indication for requesting ML model accuracy metadata includes an ML model accuracy feedback type set to "general", then AnLF calculates ML model accuracy by considering information (e.g., ground truth and / or analysis accuracy information) of a given analysis ID and a unique ML model identifier, regardless of the analysis filters and / or analysis targets associated with the subscription to the analysis ID.

[0112] For example, if ANLF has subscription A with analysis ID X1, analysis filters for regions of interest set to regions 1 and 2, and a unique ML model identifier Y, and another subscription B with the same analysis ID X1, but analysis filters for regions of interest set to regions 3 and 4, and the same unique ML model identifier Y, then all data associated with these two different analysis ID subscriptions will be used to calculate a single value for the ML model accuracy information. If the analysis accuracy information indicates a performance of 60% for subscription A and 99% for subscription B, then ANLF can calculate the ML model accuracy for the unique ML model identifier Y to be close to 80%. This calculation, which includes all data associated with the unique ML model identifier, may mislead ANLF into believing that the unique ML model identifier Y is well-trained.

[0113] If the MTLF includes an indication for requesting ML model accuracy metadata and / or if the indication for requesting ML model accuracy metadata includes an ML model accuracy feedback type set to "specific", then AnLF calculates ML model accuracy by considering each different combination of information (e.g., ground truth and / or analysis accuracy information) and accuracy binding information (e.g., analysis filters and / or associated analysis objectives and analysis IDs and unique ML model identifiers) given an analysis ID and unique ML model identifier.

[0114] For example, if AnLF has subscription A with analysis ID X1, analysis filters for regions of interest set to regions 1 and 2, and a unique ML model identifier Y, and another subscription B with the same analysis ID X1, but analysis filters for regions of interest set to regions 3 and 4, and the same unique ML model identifier Y, then AnLF will use specific data (e.g., information related to accuracy binding information) for each different analysis ID subscription to generate ML model accuracy information, including ML model accuracy metadata. For example, the algorithm within AnLF using ML model accuracy metadata can be designed to consider placing different weights in the ground truth values ​​of different regions that can be included in the analysis filters. Thus, AnLF can then detect that 80% ML model accuracy information is possible for regions 1 and 2 (e.g., weights highlighting good computational performance), while 30% might be possible for the same unique ML model identifier used in different regions of the mobile network (e.g., weights highlighting poor performance). Through this differentiation, AnLF can provide MTLF with fine-grained ML model accuracy information and metadata supporting MTLF decision-making.

[0115] Step 205A: If the request from MTLF does not include any indication for requesting ML model accuracy metadata and / or if the indication for requesting ML model accuracy metadata includes an ML model accuracy feedback type set to "General", then AnLF provides the ML model accuracy information to MTLF by invoking the Nnwdaf_MLModelMonitor_Notify service operation from MTLF. In this case, the notification does not include ML model accuracy metadata.

[0116] Step 205B: As an alternative to step 205A, if the request from MTLF includes an indication for requesting ML model accuracy metadata and / or if the indication for requesting ML model accuracy metadata includes an ML model accuracy feedback type set to "specific", then AnLF provides MTLF with one or more tuples of [ML model accuracy information with associated ML model accuracy metadata] by invoking the Nnwdaf_MLModelMonitor_Notify service operation from MTLF.

[0117] For example, a possible instance of a tuple could be defined as [unique ML model identifier, ML model precision information, analysis ID, analysis filter and / or analysis target, optional ML model filter information and / or ML model target information]. In this example, the ML model precision metadata is equivalent to the following items in the tuple [analysis ID, analysis filter and / or analysis target, optional ML model filter information and / or ML model target information], and this ML model precision information defines that the ML model precision information associated with the unique ML model identifier is further associated with another item in this tuple (i.e., the ML model precision metadata). In another possible embodiment of the ML model precision metadata, this information consists of the following items: (ML model precision information, analysis ID, analysis filter and / or analysis target, optional ML model filter information and / or ML model target information). In this case, the association of the ML model precision information is part of the ML model precision metadata, which is equivalent to defining the unique ML model identifier associated with the ML model precision metadata.

[0118] Step 206: Based on the information received from AnLF (e.g., one or more tuples of (ML model precision information, ML model precision meta information) for a unique ML model identifier), MTLF can detect whether a change is needed for a given unique ML model identifier.

[0119] Examples of possible changes that MTLF may implement based on the ML model accuracy metadata received from AnLF are described below.

[0120] - (a) Create and / or train a new ML model with a new unique ML model identifier based on a previous ML model associated with a unique ML model identifier (further related to the received ML model precision metadata). In this case, the new trained and / or created ML model can be an improvement on the previous ML model, for example, it can be trained using only a portion of the entire dataset (and / or the type of dataset information and / or the type of dataset features). For example, if the previous ML model was trained with dataset features, for example, representing all tracking areas of a mobile network, the new ML model is, for example, a specialization of the previous ML model because it was trained on a reduced list (or set) of tracking areas of the mobile network. This applies only to possible dataset feature types; other examples may include: application type, slice type, slice identifier, radio access technology type, user type (e.g., fixed user, mobile user), type session, data network identifier, and any other fields defined in the analysis filter information according to 3GPP TS23.288.

[0121] - (b) Select a new ML model and an associated new unique ML model identifier, wherein the new ML model is associated with an ML model filter that is more suitable for the analysis ID associated with the ML model accuracy metadata. For example, MTLF can select a new ML model that has been trained to support the specific analysis filter information indicated in the ML model accuracy metadata.

[0122] Step 207: MTLF provides a notification to AnLF via the Nnwdaf_MLModelProvision_Notify service operation, indicating changes in the ML model associated with the analytics ID. In one possible embodiment, the message's input parameters include an indication of ML model specialization, which may include any possible information: - A tuple containing: [new unique ML model identifier, analysis ID, analysis filter information and / or analysis target, new ML model information, new ML model filter information]; - A tuple of [previous unique ML model identifier, new unique ML model identifier, analysis ID, analysis filter information and / or analysis target, new ML model information, new ML model filter information].

[0123] Step 208: Based on ML model specialization (or an indication of ML model specialization), AnLF determines which analysis IDs associated with the previous unique ML model identifier should be reassociated with the new unique ML model identifier and / or the new ML model information. AnLF then reassociates the analysis IDs with the analysis filters and / or analysis targets to the new unique ML model identifier and / or ML model information.

[0124] AnLF can trigger its registration as an ML model precision information provider based on step 202 with a new unique ML model identifier and corresponding precision binding information.

[0125] AnLF may also modify existing precision binding information from existing registration information of previous unique ML models with analysis IDs to reflect the deassociation of analysis IDs with analysis filters and / or analysis targets from previous unique ML model identifiers. This allows MTLFs to maintain a consistent mapping of which ANLF provides ML model precision for a unique ML model identifier and its correct precision binding information.

[0126] Figure 3 Another example of the method of the present invention is shown. In this example, the following principles are used: - When AnLF registers with MTLF to indicate its ability to perform ML model monitoring on a given unique ML model identifier, the two entities exchange information to pair the association of the unique ML model identifier to a newly proposed unique ML model precision identifier. - ML model accuracy metadata indicates a unique ML model accuracy identifier, which is associated with the tuple [analysis ID, analysis filter, and / or analysis target].

[0127] Figure 3 The method includes the following steps 301 to 308, which are based on Figure 1 The steps are as follows: Figure 3 The features introduced in [the document] can be similarly applied to [other applications]. Figure 1 .

[0128] Step 301: with Figure 2 The same as step 201 in the previous section.

[0129] Step 302: AnLF decides to register with MTLF, which provides the ML model for the analysis ID.

[0130] Step 302A: AnLF can invoke Nnwdaf_MLModelMonitor_Register to request a service operation from an MTLF that includes one or more precision bindings, which include one or more tuples of [unique ML model identifier and / or ML model filter information, analysis ID, analysis filter and / or analysis target].

[0131] Step 302B: MTLF maps a unique ML model precision identifier to each tuple [unique ML model identifier, precision binding information], and then provides this mapping information to AnLF in the Nnwdaf_MLModelMonitor_Register response service operation. For example, for each unique ML model identifier, there is a unique ML model precision identifier with precision binding information. This received mapping information is stored by AnLF for subsequent use in generating ML model precision metadata for one or more ML model precision information associated with the same unique ML model identifier.

[0132] Step 303: The MTLF determines that it needs ML model precision information from an AnLF that has registered with the ML model (e.g., indicating that it can provide ML model precision information for this ML model). The MTLF can invoke the Nnwdaf_MLModelMonitor_Subscribe service operation to the AnLF, which includes an indication for requesting ML model precision metadata. For example, the MTLF can decide to include this indication based on received precision binding information. When the MTLF receives precision binding information, it recognizes that the AnLF providing this precision binding information is capable of providing ML model precision metadata.

[0133] In this example, the indication for requesting ML model accuracy metadata can be equivalent to MTLF providing (e.g., included in the request) one or more unique ML model accuracy identifiers associated with a unique ML model identifier. The indication for requesting ML model accuracy metadata may also include ML model accuracy feedback type: [general, specific] and / or analysis accuracy request information.

[0134] Step 304: Based on the information received from MTLF (e.g., an indication for requesting ML model accuracy metadata), AnLF generates the requested ML model accuracy information.

[0135] If MTLF does not include any indication for requesting ML model accuracy metadata and / or if the indication for requesting ML model accuracy metadata includes an ML model accuracy feedback type set to "general", then AnLF is consistent with... Figure 2 The accuracy of the ML model is calculated in the same manner as described in step 204.

[0136] If the MTLF includes an indication for requesting ML model accuracy metadata and / or if the indication for requesting ML model accuracy metadata includes an ML model accuracy feedback type set to "specific", then AnLF calculates ML model accuracy by considering each different combination of information (e.g., ground truth and / or analysis accuracy information) and unique ML model accuracy identifier and / or accuracy binding information (e.g., analysis filters and / or associated analysis targets and analysis IDs and unique ML model identifiers). Figure 2 The same calculation example listed in step 204 is applicable to this embodiment.

[0137] Step 305A: If the request from MTLF does not include any indication for requesting ML model accuracy metadata and / or if the indication for requesting ML model accuracy metadata includes an ML model accuracy feedback type set to "General", then AnLF provides the ML model accuracy information to MTLF by invoking the Nnwdaf_MLModelMonitor_Notify service operation from MTLF. In this case, the notification does not include ML model accuracy metadata.

[0138] Step 305B: If the request from MTLF includes an indication for requesting ML model accuracy metadata and / or if the indication for requesting ML model accuracy metadata includes an ML model accuracy feedback type set to "specific", then AnLF provides MTLF with one or more tuples of [ML model accuracy information with associated ML model accuracy metadata] by invoking the Nnwdaf_MLModelMonitor_Notify service operation from MTLF.

[0139] For example, an instance of a tuple can be defined as the ML model precision information of a unique ML model identifier associated with a unique ML model precision identifier. In this possible example, the ML model precision meta-information is equivalent to the following item of such a tuple: [unique ML model identifier associated with a unique ML model precision identifier].

[0140] Step 306: with Figure 2 The same as step 206 in the previous section.

[0141] Step 307: Based on Figure 2 Step 207 in the text differs in that: MTLF provides a notification to AnLF via the Nnwdaf_MLModelProvision_Notify service operation, indicating changes in the ML model associated with the analytics ID. In one possible embodiment, the message's input parameters include an indication of ML model specialization, which can include any possible information: - A tuple containing: [new unique ML model identifier, analysis ID, unique ML model accuracy identifier, new ML model information, new ML model filter information]; -A tuple of [previous unique ML model identifier, new unique ML model identifier, analysis ID, unique ML model precision identifier, new ML model information, new ML model filter information].

[0142] Step 308: with Figure 2 The same as step 208 in the previous section.

[0143] Figure 4 A diagram illustrating a method for a first network functional entity of the present invention is shown. The method includes the following steps: Step 401: The first network functional entity obtains model accuracy binding information from the second network functional entity, wherein the model accuracy binding information indicates at least one association between analysis information and ML model information, and wherein the analysis information indicates at least one association between analysis ID and one or more of analysis filter information and analysis target.

[0144] Step 402: The first network functional entity obtains model accuracy information associated with the model accuracy metadata from the second network functional entity (120), wherein the model accuracy metadata indicates the association between the ML model accuracy information and the analysis information.

[0145] Step 403: The first network functional entity determines whether to update the ML model information based on the model accuracy binding information and / or model accuracy metadata.

[0146] Figure 4 The method can have the above-mentioned relationship with Figures 1 to 3 The corresponding features mentioned.

[0147] Figure 5 A diagram illustrating a method for a second NF entity according to the present invention is shown. The method includes the following steps: Step 501: The second network functional entity provides model accuracy binding information to the first network functional entity, wherein the model accuracy binding information indicates at least one association between analysis information and ML model information, and wherein the analysis information indicates at least one association between analysis ID and one or more of analysis filter information and analysis target.

[0148] Step 502: The second network functional entity obtains the model accuracy information of the ML model information.

[0149] Step 503: The second network functional entity generates model accuracy metadata, which indicates the correlation between ML model accuracy information and analysis information.

[0150] Step 504: The second network functional entity provides the first network functional entity with model accuracy information associated with the model accuracy metadata.

[0151] Figure 5 The method can have the above-mentioned relationship with Figures 1 to 4 The corresponding features mentioned.

[0152] In summary, this invention provides a scheme for optimizing the accuracy monitoring, pre-configuration, retraining, and updating of ML models in mobile communication networks. This invention can be applied to any type of mobile communication network, including but not limited to 5G, 6G cellular systems, or any network built upon such mobile communication networks (e.g., IoT networks, V2X networks, etc.).

[0153] It should be noted that the entities in this invention may include processing circuitry for executing, performing, or initiating various operations of the device described herein. The processing circuitry may include hardware and software. Hardware may include analog or digital circuitry, or both. Digital circuitry may include components such as application-specific integrated circuits (ASICs), field-programmable arrays (FPGAs), digital signal processors (DSPs), or multi-purpose processors. Optionally, the processing circuitry includes one or more processors and non-transitory memory connected to the one or more processors. The non-transitory memory may carry executable program code, which, when executed by the one or more processors, causes the device to execute, perform, or initiate the operations or methods described herein.

[0154] The invention has been described in conjunction with various aspects as examples and implementations. However, those skilled in the art will understand and derive other variations by implementing the claimed subject matter, studying the drawings, the invention, and the independent claims. In the claims and description, the word "comprising" does not exclude other elements or steps, and "a" does not exclude a plurality. A single element or other unit may fulfill the function of several entities or items described in the claims. The enumeration of certain measures in dissimilar dependent claims does not imply that combinations of these measures cannot be used in advantageous implementations.

Claims

1. A first network functional entity (110) for training machine learning (ML) models in a mobile communication network, characterized in that, The first network functional entity (110) is used for: Model accuracy binding information is obtained from the second network functional entity (120), wherein the model accuracy binding information indicates at least one association between analysis information and ML model information, wherein the analysis information indicates at least one association between analysis identifier ID and one or more of analysis filter information and analysis target; Obtain model accuracy information associated with model accuracy metadata from the second network functional entity (120), wherein the model accuracy metadata indicates the association between the ML model accuracy information and the analysis information; Based on the model accuracy binding information and / or the model accuracy metadata, determine whether to update the ML model information.

2. The first network functional entity (110) according to claim 1, characterized in that, The model accuracy metadata also indicates the association between the analysis accuracy information and one or more of the following: Analysis ID, analysis filter information, analysis target, and ML model information.

3. The first network functional entity (110) according to claim 1 or 2, characterized in that, The first network functional entity (110) is used to obtain the model accuracy binding information from a request from the second network functional entity (120), wherein the request indicates that the second network functional entity (120) can provide the ML model accuracy information regarding the ML model information included in the model accuracy binding information.

4. The first network functional entity (110) according to any one of claims 1 to 3, characterized in that, The first network functional entity (110) is used to send an indication message to the second network functional entity (120) to request the model accuracy metadata.

5. The first network functional entity (110) according to claim 4, characterized in that, The indication information used to request the model accuracy metadata is included in the subscription request sent by the first network function entity (110) to the second network function entity (120).

6. The first network functional entity (110) according to claim 4 or 5, characterized in that, The indication information used to request the model accuracy metadata includes one or more of the following: A flag indicating whether the model accuracy metadata is required; The ML model ID associated with the analysis ID; The ML model ID associated with one or more of the analysis ID, analysis filter information, and analysis target; ML model IDs associated with one or more ML model precision IDs, wherein each ML model precision identifier identifies the association of at least one ML model ID with at least one analysis ID, analysis filter information and / or analysis target; One or more ML model precision IDs, where each ML model precision ID identifies a tuple of ML model ID and corresponding analysis information.

7. The first network functional entity (110) according to any one of claims 1 to 6, characterized in that, Also used for: The ML model information is updated based on the model accuracy metadata, wherein the updated ML model information is associated with a new ML model ID or a previous ML model ID.

8. The first network functional entity (110) according to claim 7, characterized in that, In order to update the ML model information, the first network functional entity (110) performs any of the following: The new ML model information is determined as the updated ML model information; or The ML model is retrained to obtain a retrained ML model, thereby updating the ML model information.

9. The first network functional entity (110) according to claim 7 or 8, characterized in that, Also used for: The updated ML model information and / or machine learning model specialization indication are provided to the second network functional entity (120), wherein the machine learning model specialization indication indicates the association between the ML model information and / or the updated ML model information and the corresponding analysis information.

10. A second network functional entity (120) for supporting machine learning (ML) model training in a mobile communication network, characterized in that, The second network functional entity (120) is used for: Provide model accuracy binding information to the first network functional entity (110), wherein the model accuracy binding information indicates at least one association between analysis information and ML model information, wherein the analysis information indicates at least one association between analysis identifier ID and one or more of analysis filter information and analysis target; Obtain the model accuracy information of the ML model information; Generate model accuracy metadata, which indicates the correlation between the model accuracy information and the analysis information; Provide the first network functional entity (110) with the model accuracy information associated with the model accuracy metadata.

11. The second network functional entity (120) according to claim 10, characterized in that, The model accuracy metadata also includes analysis accuracy information, which is associated with one or more of the analysis ID, analysis filter information, analysis target, and ML model information.

12. The second network functional entity (120) according to claim 10 or 11, characterized in that, The second network functional entity (120) is used to provide the model accuracy binding information to the first network functional entity (110) by requesting it, wherein the request indicates that the second network functional entity (120) is able to provide the ML model accuracy information regarding the ML model information included in the model accuracy binding information.

13. The second network functional entity (120) according to any one of claims 10 to 13, characterized in that, In order to provide the model accuracy metadata, the second network functional entity (120) is also configured to receive indication information from the first network functional entity (110) for requesting the model accuracy metadata.

14. The second network functional entity (120) according to any one of claims 10 to 13, characterized in that, Also used for: Receive updated ML model information and / or machine learning model specialization indication from the first network functional entity (110), wherein the machine learning model specialization indication indicates the association between the ML model information and / or the updated ML model information and the corresponding analysis information; The ML model information is updated based on the updated ML model information and / or the machine learning model specialization indication.

15. A system, characterized in that, It includes at least one first network function entity (110) according to any one of claims 1 to 9 and at least one second network function entity (120) according to any one of claims 10 to 14.

16. A method (400) for training a machine learning (ML) model in a mobile communication network for a first network functional entity, characterized in that, The method includes: (401) Model precision binding information is obtained from the second network functional entity, wherein the model precision binding information indicates at least one association between analysis information and ML model information, wherein the analysis information indicates at least one association between analysis identifier ID and one or more of analysis filter information and analysis target; Obtain (402) model accuracy information associated with model accuracy metadata from the second network functional entity (120), wherein the model accuracy metadata indicates the association between the ML model accuracy information and the analysis information; Based on the model accuracy binding information and / or the model accuracy metadata, determine (403) whether to update the ML model information.

17. A method (500) for supporting machine learning (ML) model training in a mobile communication network for a second network functional entity, characterized in that, The method includes: Provide (501) model precision binding information to the first network functional entity, wherein the model precision binding information indicates at least one association between analysis information and ML model information, wherein the analysis information indicates at least one association between analysis identifier ID and one or more of analysis filter information and analysis target; Obtain the model accuracy information of the ML model information mentioned in (502); Generate (503) model accuracy metadata, wherein the model accuracy metadata indicates the association between the ML model accuracy information and the analysis information; Provide the first network functional entity with (504) the model accuracy information associated with the model accuracy metadata.

18. A computer program comprising instructions, characterized in that, When the program is executed by a computer, the instructions cause the computer to perform the method according to claim 16 or 17.

Images