Analyzing continuous use of model accuracy information during migration

Abstract

The present disclosure relates to providing and using model accuracy information for a model associated with a model identifier (ID) and / or an analysis ID, for example, a model for generating and / or providing analysis information for the analysis ID. The present disclosure relates, inter alia, to enable continuous use of model accuracy information, even in the case of analytic migration, e.g., analytic ID migration from one network entity to another. To this end, the present disclosure proposes various network entities and corresponding methods in a mobile communication network. These network entities are used to provide and receive indications for altering the provision of model accuracy information, and indications relating to alteration of the provision of model accuracy information, respectively.

Description

TECHNICAL FIELD

[0001] The present disclosure relates to a new generation of communication networks, e.g., a 5th generation (5G) or a 6th generation (6G) mobile network. The present disclosure relates to providing and using model accuracy information of a model associated with a model identifier (ID) and / or an analytics identifier (ID), e.g., a model used for generating and / or providing analytics information of an analytics ID. The present disclosure particularly relates to enabling a continuous usage of model accuracy information even in case of analytics migration, e.g., analytics ID migration from one network entity to another network entity. To this end, the present disclosure proposes various network entities and corresponding methods. BACKGROUND

[0002] According to the definition in TS 23.288 in 3rd generation partnership project (3GPP) Release 17 (R17), an analytics ID can be migrated from a first network entity to a second network entity in different cases. Specifically, an analytics ID can be migrated from a source network data analytics function (NWDAF) to a target NWDAF. For example, if the source NWDAF has to be shut down and needs to redirect one or more subscriptions to a target NWDAF, the subscriptions for which analytics are continued to be computed and provided in the target NWDAF. An advantage of such an analytics migration procedure is that continuity of one or more services provided by the NWDAF to a service consumer can be achieved. In R17, according to the definition in TS 23.288, the NWDAF provides an analytics output generation service.

[0003] Progress in the Automation Enablement Technologies Research Project TR 23.700-81 of 3GPP Release 18 (R18) and the R18 specification in TS 23.288 has enhanced NWDAF services—particularly NWDAFs with analytics logical functions (AnLF)—providing them with two important new features and services: (a) NWDAFs with AnLF can check and provide the accuracy of analytics IDs used by network functions (NFs); and (b) NWDAFs with AnLF can check and provide the accuracy of analytics IDs associated with machine learning (ML) models to NWDAFs with model training logical functions (MTLFs). NWDAFs with MTLFs are used to provide ML models to NWDAFs with AnLFs.

[0004] It's important to note the difference between the analytics accuracy information used to generate analytics IDs and the accuracy information used to generate ML models. In the first case, an NWDAF can use collected information associated with a subscribed analytics ID to generate the analytics ID, considering the possibility of using different ML models for the same analytics ID. Alternatively, it can use collected information associated with a subscribed analytics ID and a given ML model associated with that analytics ID. In the second case, when generating ML model accuracy information, an NWDAF with AnLF can use information from different subscriptions for analytics IDs using the same ML model to compute the ML model accuracy information. These are two distinct processes involving two different types of users, different datasets (which may or may not overlap) to be used for computation, and ultimately, two different types of information provided to different users.

[0005] The NWDAF with MTLF neither participates in nor is aware of any analysis migration process. In current standard specifications, the target NWDAF receives information from the NWDAF with MTLF that provides the model. If the ML model ID of the migrated analysis ID is already stored in the target NWDAF, there is no reason for the target NWDAF to interact with the NWDAF with MTLF that provides the ML model. Summary of the Invention

[0006] This disclosure and its proposed solutions are also based on the following considerations.

[0007] Current standard specifications do not address the following issue: how should model accuracy information of the ML model be handled when the analysis ID associated with the ML model is migrated to a target NWDAF with AnLF, where the ML model is provided by a source NWDAF with AnLF to an NWDAF with MTLF.

[0008] This disclosure assumes a scenario where an NWDAF with MTLF subscribes to an NWDAF with AnLF to obtain ML model accuracy information. According to TS 23.288 V18.1.0, upon receiving a registration from an NWDAF with AnLF, the NWDAF with MTLF can determine that it needs to subscribe to the accuracy information of the ML model, wherein the registration indicates that such an NWDAF intends to monitor the accuracy information of an analytics ID using a given ML model and a local policy. The NWDAF with MTLF then subscribes to the NWDAF with AnLF to retrieve the model accuracy information of the ML model. In the subscription, the NWDAF with MTLF includes one or more unique identifiers of one or more ML models to be monitored, the accuracy metric to be monitored, and an optional reporting period or one or more reporting thresholds.

[0009] NWDAF with MTLF can also determine the need to retrain the ML model and / or reselect different ML models to be used for the analysis ID based on the ML model accuracy information received from NWDAF with AnLF.

[0010] As defined in Section 6.2E.3.2 of TS 23.288, a local policy or a request from a service user (i.e., an NF user of the analysis output that also requires the generation of accuracy information for the analysis ID as defined in Section 6.2D) can trigger an NWDAF with AnLF to register with an NWDAF with MTLF (therefore triggering such an NWDAF with MTLF to subscribe to ML model accuracy information).

[0011] In the considered scenario, migrating the analytics ID will not cause the target NWDAF with AnLF to start providing analytics accuracy information, because subscribing to the analytics ID does not require generating analytics accuracy information. This will also prevent the target NWDAF from registering with the NWDAF with MTLF to become a provider of model accuracy information for the ML model migrated along with the analytics ID. Therefore, the following issues may arise: To determine the appropriate action for retraining or reselecting one or more ML models, the use of model accuracy information can be affected by misconfigurations in different NWDAFs, as the (new) target NWDAF may have a different configuration from the source NWDAF during / after the analysis transfer. Taking into account any gaps in model accuracy information from different NWDAFs, or not, can lead to instability in determining the overall action to take for an NWDAF with MTLF.

[0012] Unregistering from an NWDAF with MTLF can result in high overhead and unnecessary indications to the source NWDAF with AnLF. This causes the NWDAF with MTLF to unnecessarily remove its subscription to model accuracy information (according to Section 6.2E.3). Furthermore, in parallel, a new target NWDAF with AnLF triggers registration to the NWDAF with MTLF, and subsequently subscriptions to ML model accuracy information.

[0013] In view of these issues, this disclosure aims to provide an improvement for analytics transfer, particularly when considering the provision and use of model accuracy information. One objective is to prevent network entities using model accuracy information (e.g., NWDAF with MTLF in the above scenario) from making unnecessary instructions and / or determining inaccurate ML model (degradation) information. Another objective is to avoid inaccurately triggering (unnecessary) ML model reselection, or making inaccurate decisions not to retrain and / or reselect the ML model associated with an analytics ID that migrates between the source and target network entities (e.g., source and target NWDAFs with AnLF in the above scenario).

[0014] These and other objectives are achieved through this disclosure based on the technical solutions described in the independent claims. Advantageous implementation methods are further described in the dependent claims.

[0015] A first aspect of this disclosure provides a first network entity for generating model accuracy information for a model associated with a model identifier and / or analysis identifier (ID). The first network entity is configured to: obtain a first instruction for changing the provision of model accuracy information, wherein the provision of the model accuracy information is used by a third network entity for using the model accuracy information and is associated with the model and one or more parameter information; the model accuracy information represents quality information of the model; and, based on the first instruction, provide the third network entity with a second instruction related to the change in the provision of the model accuracy information, wherein the second instruction includes at least one of the following: first information indicating termination of the provision of the model accuracy information for the model ID and / or analysis ID at the first network entity; second information indicating redirection of the provision of the model accuracy information for the model ID and / or analysis ID from the first network entity to a second network entity; third information indicating one or more changes related to the provision of the model accuracy information for the model ID and / or analysis ID; and fourth information indicating registration of the second network entity as a provider of model accuracy information for the model ID and / or analysis ID according to the first instruction.

[0016] In this disclosure, the first information is also referred to as "ML model accuracy provision termination". In this disclosure, the second information is also referred to as "ML model accuracy provision redirection".

[0017] The first network entity can be a source NWDAF with AnLF, and the second network entity can be a target NWDAF with AnLF. The third network entity can be an NWDAF with MTLF.

[0018] The second instruction provided by the first network entity to the third network entity can provide the third network entity with the information required to achieve the above-mentioned purpose, wherein the third network entity is a network entity that uses model accuracy information.

[0019] In one implementation of the first aspect, the first network entity is further configured to: obtain the first instruction for changing the provision of the model accuracy information based on at least one of the following: by identifying that the model ID and / or analysis ID associated with the provision of the model accuracy information is related to an analysis migration from the first network entity to the second network entity; by receiving from the second network entity any one of the following: a model accuracy information context type, indicating that migration information related to the provision of the model accuracy information should be included in the analysis context information; a model accuracy information context flag, indicating that the model accuracy information context type should be requested; one or more parameters related to the provision of the model accuracy information, wherein the provision of the model accuracy information is associated with the model.

[0020] In this disclosure, one or more parameters are also referred to as "transition information related to the accuracy generation of ML models". Different methods of obtaining the first indication make the scheme of this disclosure compatible with existing standard procedures.

[0021] In one implementation of the first aspect, the first network entity is further configured to: if the first indication is obtained by receiving the model accuracy information context type or the model accuracy information context flag from the second network entity, request the one or more parameters related to the provision of the model accuracy information from the second network entity, wherein the provision of the model accuracy information is associated with the model.

[0022] This is an efficient way to request parameters and is compatible with existing standard procedures.

[0023] In one implementation of the first aspect, the first network entity is configured to: provide the second instruction to the third network entity, wherein the second instruction is included in a cancellation message or in a notification message having or not having model accuracy information.

[0024] This is an efficient way to provide a second instruction, and it is compatible with existing standard procedures.

[0025] In one implementation of the first aspect, the second instruction further includes at least one of the following: a registration ID of the first network entity, the registration ID being associated with the provision of model accuracy information performed by the first network entity, wherein the provision of the model accuracy information is associated with the model; and a subscription ID of the third network entity at the first network entity, the subscription ID being used for the provision of the model accuracy information performed by the first network entity, wherein the provision of the model accuracy information is associated with the model.

[0026] In one implementation of the first aspect, when the second network entity registers as a provider of model accuracy information for the model ID and / or analysis ID, the fourth information includes the ID of the second network entity.

[0027] A second aspect of this disclosure provides a third network entity for using model accuracy information of a model, wherein the model is associated with a model identifier and / or analysis identifier (ID), and the model accuracy information represents quality information of the model; the third network entity is configured to: use model accuracy information from a first network entity used to generate model accuracy information, wherein the model accuracy information is associated with the model ID and / or analysis ID; obtain a second instruction from the first network entity or a second network entity, wherein the second instruction includes at least one of the following: first information indicating termination of the provision of model accuracy information for the model ID and / or analysis ID at the first network entity; second information indicating redirection of the provision of the model accuracy information for the model ID and / or analysis ID from the first network entity to the second network entity; third information indicating one or more changes related to the provision of the model accuracy information for the model ID and / or analysis ID from the second network entity; and fourth information indicating registration of the second network entity as a provider of model accuracy information for the model ID and / or analysis ID.

[0028] The third network entity is a network entity that uses model accuracy information. For example, the third network entity could be an NWDAF with MTLF, while the first and second network entities are the source NWDAF and target NWDAF with AnLF, respectively. The third network entity can achieve the aforementioned objective by receiving a second instruction from the first network entity.

[0029] In one implementation of the second aspect, the third network entity is further configured to: determine a relationship and / or mapping between the acquired second indication and the third network entity's use of the model accuracy information for the model ID and / or analysis ID, wherein the use of the model accuracy information is associated with the model and one or more parameter information.

[0030] A reference to the use of model accuracy information for a model ID and / or analysis ID is equivalent to a reference to the provision of model accuracy information for a model ID and / or analysis ID. The information associated with these equivalent terms is the same. The difference lies in that the first term refers to this information from the perspective of the information user, while the latter describes this information from the perspective of the information provider. Furthermore, the use or provision of model accuracy information for a model ID and / or analysis ID can also be referred to as a subscription to model accuracy information for a model ID and / or analysis ID. When described from the user's perspective, this term is called a subscription to model accuracy information for a model ID and / or analysis ID requested by a third network entity, while when described from the provider's perspective, this term is called a subscription to model IDs and / or analysis IDs provided by a first network entity (or second network entity).

[0031] In one implementation of the second aspect, the third network entity is configured to: if the second indication includes the first information or the second information, then, based on local configuration and / or based on the second indication, suspend the use of data and / or one or more processes related to the use of the model accuracy information from the first network entity.

[0032] This ensures that third network entities do not attempt to make changes to data or processes related to the model accuracy information used by the first network entity, even if an analytics migration occurs, which includes the migration of model accuracy information.

[0033] In one implementation of the second aspect, in order to suspend the one or more processes associated with the use of the model accuracy information, the third network entity is configured to: (a) stop modifying or deleting any of the following for a defined period of time: data associated with the use of the model accuracy information from the first network entity; training or retraining processes associated with the model, the model being associated with the use of the model accuracy information from the first network entity; or (b) suspend the use of the model accuracy information for a defined period of time without modifying or deleting any of the following: subscriptions to the use of the model accuracy information from the first network entity; data associated with the use of the model accuracy information from the first network entity; processes associated with the use of the model accuracy information from the first network entity.

[0034] In one implementation of the second aspect, the third network entity is configured to: provide a request to the second network entity for a new provision of model accuracy information to generate model accuracy information, wherein the request is based on the second instruction if the second instruction includes the fourth information.

[0035] In one implementation of the second aspect, the third network entity is further configured to: reuse or redirect or re-associate data and / or information from the one or more processes to a new provision of the model accuracy information, wherein the one or more processes are associated with the use of the model accuracy information from the first network entity (100), and the new provision of the model accuracy information is associated with the model ID and / or analysis ID from the second network entity.

[0036] This allows the third network entity to continue using model accuracy information from the second network entity, but not the first, even in the case of analysis transfer. The use of model accuracy information can be continuous.

[0037] In one implementation of the second aspect, the third network entity is further configured to: if the second indication includes the third information, update the information at the third network entity based on the second indication, wherein the information is related to the use of the model accuracy information, the use of the model accuracy information being associated with the model ID and / or analysis ID from the second network entity.

[0038] In one implementation of the second aspect, the third network entity is further configured to: reuse or redirect or re-associate data and / or information from the one or more processes to the updated information related to the use of the model accuracy information, wherein the one or more processes are associated with the use of the model accuracy information from the first network entity, and the use of the model accuracy information is associated with the model ID and / or analysis ID from the second network entity.

[0039] In one implementation of the second aspect, the third network entity is further configured to: based on the second instruction, restore the data and / or one or more processes related to the use of the model accuracy information.

[0040] In one implementation of the second aspect, the third network entity is configured to: determine that the second network entity is associated with the suspended data and / or one or more processes related to the use of the model accuracy information, wherein the use of the model accuracy information is associated with the model ID and / or analysis ID provided by the first network entity; and resume the use of the data and / or one or more processes related to the use of the model accuracy information from the first network entity.

[0041] In one implementation of the second aspect, in order to resume the one or more processes associated with the use of the model accuracy information, the third network entity is configured to: reuse or redirect or re-associate data and / or information from the suspended one or more processes to a new provision of the model accuracy information, wherein the suspended one or more processes are associated with the use of the model accuracy information from the first network entity, and the new provision of the model accuracy information is associated with the model ID and / or analysis ID from the second network entity.

[0042] In one implementation of the second aspect, the third network entity is further configured to: if the time period associated with the suspended one or more processes times out before the third network entity receives the third or fourth information from the second network entity, wherein the suspended one or more processes are associated with the use of model accuracy information, then the third network entity is configured to: deactivate the suspended one or more processes associated with the use of the model accuracy information, and / or delete data associated with the suspended one or more processes, wherein the suspended one or more processes are associated with the use of the model accuracy information.

[0043] A third aspect of this disclosure provides a method for generating model accuracy information for a model, wherein the model is associated with an analysis identifier (ID), the method being performed by a first network entity and comprising: obtaining a first instruction for changing the provision of model accuracy information, wherein the provision of the model accuracy information is used by a third network entity for training the model and is associated with the model and one or more parameter information; the model accuracy information represents quality information of the model; based on the first instruction, providing the third network entity with a second instruction related to the change in the provision of the model accuracy information, wherein the second instruction includes at least one of the following: first information indicating termination of the provision of the model accuracy information for the model ID and / or analysis ID at the first network entity; second information indicating redirection of the provision of the model accuracy information for the model ID and / or analysis ID from the first network entity to a second network entity; third information indicating one or more changes related to the provision of the model accuracy information for the model ID and / or analysis ID; and fourth information indicating registration of the second network entity as a provider of model accuracy information for the model ID and / or analysis ID according to the first instruction.

[0044] The method described in the third aspect can be implemented in a manner corresponding to the implementation of the first network entity described in the first aspect. The method described in the third aspect and its implementation can achieve the same purpose and advantages as the first network entity described in the first aspect and its corresponding implementation.

[0045] A fourth aspect of this disclosure provides a method for using model accuracy information of a model, wherein the model accuracy information represents quality information of the model; the method is performed by a third network entity and includes: using model accuracy information from a first network entity used to generate model accuracy information associated with the analysis ID, wherein the model accuracy information is associated with the model ID and / or analysis ID; obtaining a second instruction from the first network entity or a second network entity, wherein the second instruction includes at least one of the following: first information indicating termination of the provision of model accuracy information for the model ID and / or analysis ID at the first network entity; second information indicating redirection of the provision of the model accuracy information for the model ID and / or analysis ID from the first network entity to the second network entity; third information indicating one or more changes related to the provision of the model accuracy information for the model ID and / or analysis ID from the second network entity; and fourth information indicating registration of the second network entity as a provider of model accuracy information for the model ID and / or analysis ID.

[0046] The implementation of the method described in the fourth aspect can correspond to the implementation of the third network entity described in the second aspect. The method described in the fourth aspect and its implementation can achieve the same purpose and advantages as the third network entity described in the second aspect and its corresponding implementation.

[0047] The fifth aspect of this disclosure provides a computer program including instructions that, when executed by a computer, cause the computer to perform the method described according to the third or fourth aspect or any implementation thereof.

[0048] A sixth aspect of this disclosure provides a non-transitory storage medium for storing executable program code, which, when executed by a processor, causes the method described according to the third or fourth aspect or any implementation thereof to be performed.

[0049] In summary, this disclosure provides the following advantages. First, this disclosure can redirect or terminate the provision of model accuracy information to a third network entity (e.g., an NWDAF with MTLF). Second, this disclosure enables the third network entity to monitor and update ML model accuracy information.

[0050] It should be noted that all devices, elements, units, and modules described in this application can be implemented in software or hardware elements or any combination thereof. All steps performed by the various entities described in this application, and functions described as being performed by the various entities, are intended to indicate that the respective entities are suitable for or used to perform the respective steps and functions. Although in the following description of specific embodiments, the specific functions or steps to be performed by an external entity are not reflected in the description of the specific detailed elements of the entity performing that specific step or function, those skilled in the art should understand that these methods and functions can be implemented in the corresponding software or hardware elements or any combination thereof. Attached Figure Description

[0051] The above aspects and various implementations will be explained below in conjunction with the accompanying drawings and specific embodiments.

[0052] Figure 1 The first network entity according to this disclosure, as well as the second and third network entities according to this disclosure, are shown.

[0053] Figure 2 The system architecture is illustrated with a first network entity (source NWDAF with AnLF), a second network entity (target NWDAF with AnLF), and a third network entity (NWDAF with MTLF). This system architecture is related to the generation and provision of model accuracy information and supports the redirection of model accuracy information.

[0054] Figure 3 The source-triggered analysis migration process is illustrated, in which the source NWDAF-triggered MTLF prepares to redirect the provision of model accuracy information (Alternative A).

[0055] Figure 4 The source-triggered augmentation analysis migration process is illustrated, in which the source NWDAF-triggered MTLF prepares to redirect the provision of model accuracy information (Alternative B).

[0056] Figure 5 The source-triggered augmentation analysis migration process is illustrated, in which the target NWDAF-triggered MTLF prepares to redirect the provision of model accuracy information (Alternative C).

[0057] Figure 6 The target-triggered analysis migration process is illustrated, in which the source NWDAF-triggered MTLF prepares to redirect the provision of model accuracy information (Alternative D).

[0058] Figure 7 The target-triggered analysis migration process is illustrated, in which the target NWDAF-triggered MTLF prepares to redirect the provision of model accuracy information (alternative E).

[0059] Figure 8 A method for generating model accuracy information according to this disclosure is shown, which can be performed by a first network entity.

[0060] Figure 9 A method for using model accuracy information according to this disclosure is shown, which can be performed by a third network entity.

[0061] Explanation of terms used in the present disclosure ML Model Accuracy Context Flag: Indicates to NWDAFs with AnLF that the ML model accuracy monitoring context type should be requested when performing the analysis context migration process.

[0062] ML Model Accuracy Monitoring (or Generation or Provision) Context Type: This indicates a request (or indication) to include migration information related to ML model accuracy generation in the analysis context information. This indication allows a target NWDAF with AnLF to request this information from a source NWDAF with AnLF to trigger ML model accuracy information monitoring based on previously used parameterizations (or information) by the source NWDAF with AnLF, in order to generate ML model accuracy information.

[0063] Transfer information related to ML model accuracy generation (source NWDAF with AnLF) A target NWDAF with AnLF: represents information (or one or more parameters) related to an existing subscription to provide ML model accuracy information associated with an analytics ID, where the analytics ID is also related to the ML model and / or analytics migration process (or the need for the analytics migration process, or the migration of analytics IDs). Synonyms for this term are ML model accuracy monitoring related information. This information may include any of the following: ML Model Accuracy Activation Flag: Indicates that a target NWDAF with AnLF that has accuracy checking capabilities (e.g., the ability to generate analysis accuracy information and / or ML model accuracy information) needs to activate the ML model accuracy check (or ML model accuracy monitoring) associated with the ML model, which is related to the analysis ID that is migrating from a source NWDAF with AnLF to a target NWDAF with AnLF.

[0064] ML Model Accuracy Context Flag: This parameter can also be used alone, rather than as part of the migration information related to ML model accuracy generation.

[0065] The ML model accuracy subscription identifier at the source NWDAF (e.g., the original subscription ID of the ML model accuracy).

[0066] ML Model Accuracy Information Redirection (or Termination) Instruction: Instructs the target NWDAF that the source NWDAF with AnLF to terminate (or will terminate) the generation of ML model accuracy information associated with the analysis ID, which is related to the analysis migration. This instruction can also be understood as the source NWDAF with AnLF instructing the target NWDAF with AnLF to redirect the subscription to ML model accuracy information associated with the analysis ID.

[0067] ML Model Accuracy Information Splitting Instruction: Instructs the target NWDAF that the source NWDAF with AnLF will retain (or maintain) a subscription to ML model accuracy information for the analysis IDs related to the analysis migration. In other words, it indicates that both the source NWDAF with AnLF and the target NWDAF are capable of (or responsible for or have subscribed to) generating ML model accuracy information for the analysis IDs related to the analysis migration.

[0068] Accuracy metrics monitored at the source NWDAF with AnLF, where the accuracy metric defines one or more metrics to compute accuracy information for the ML model.

[0069] One or more accuracy reporting thresholds: These indicate the reporting conditions under which ML model accuracy information needs to be reported.

[0070] Accuracy reporting cycle: This indicates the reporting cycle during which ML model accuracy information can be reported.

[0071] Notification endpoints that use ML model accuracy information to perform service operations (e.g., Nnwdaf_MLModelMonitor_Notify service operations) at the NWDAF containing MTLF.

[0072] Identifiers for existing ML model monitoring subscriptions (e.g., subscription association IDs for ML model accuracy information subscription requests).

[0073] ML Model Accuracy Transfer Indicator (NWDAF with AnLF) (NWDAF with MTLF): Information related to an existing subscription for ML model accuracy information used for the ML model ID and / or analysis ID. This indication includes any of the following: The original (or existing) subscription identifier associated with the ML model accuracy information at the source NWDAF with AnLF.

[0074] A new subscription ID associated with a subscription that has ML model accuracy information at the target NWDAF with AnLF.

[0075] ML model identifier (e.g., a unique ML model identifier).

[0076] One or more analytics IDs.

[0077] The NF ID of the NWDAF containing AnLF associated with the target NWDAF (e.g., the NF identifier of the NWDAF with AnLF that received the analysis migration).

[0078] Subscription endpoints that contain service operations (e.g., the Nnwdaf_MLModelMonitor_Subscribe service operation) at the target NWDAF of AnLF.

[0079] A measure of accuracy monitored at the source NWDAF with AnLF.

[0080] Types of ML model accuracy subscription migrations. Examples of possible types of ML model accuracy subscription migrations: Split: Indicates that both the source NWDAF and the target NWDAF with AnLF are capable (in this disclosure, the term is synonymous with: responsible for or subscribed to) generating ML model accuracy information for analysis IDs related to analysis migration.

[0081] Redirection: Indicates that only the target NWDAF with AnLF is able to (or responsible for or subscribed to) generate ML model accuracy information for analysis IDs related to analysis migration.

[0082] The set of UEs associated with migration subscriptions to ML model accuracy.

[0083] ML Model Accuracy Provision Termination: Indicates to the NWDAF with MTLF that the registration and / or subscription associated with ML model accuracy monitoring used for analysis ID has been terminated. ML model accuracy provision termination may include any of the following: ML model accuracy monitoring terminated (e.g., implemented as a flag): indicating any of the following: (a) the NWDAF with AnLF cancels its ability to provide ML model accuracy information for analysis ID; (b) unsubscribes (e.g., cancels subscription) to the ML model accuracy information of the NWDAF with AnLF.

[0084] Reasons for termination of ML model accuracy monitoring: Indicates the reason for cancellation and / or unsubscription of ML model accuracy monitoring. Examples of possible reasons are as follows: "Cancellation due to analysis migration", "Termination due to analysis ID redirection".

[0085] The target NWDAF contains an AnLF NF ID (e.g., the NF identifier of the NWDAF with AnLF that receives the analysis ID, associated with the model accuracy monitoring subscription for the analysis ID at the source NWDAF with AnLF).

[0086] The original (or existing or source NWDAF with AnLF) subscription identifier (also known as subscription association ID) of the ML model accuracy information associated with the subscription at the source NWDAF with AnLF.

[0087] The subscription endpoint for the Nnwdaf_MLModelMonitor_Subscribe service operation at the target NWDAF of AnLF.

[0088] ML model identifier (e.g., a unique ML model identifier).

[0089] One or more analytics IDs.

[0090] Analyze the user's ID (e.g., each analytics ID).

[0091] The original (or existing) subscription identifier (also known as the subscription association ID) of the analysis output at the source NWDAF of AnLF.

[0092] A rollback timer instructs an NWDAF with an MTLF to wait for a specified period before triggering a change in a process associated with an ML model (e.g., a unique ML model identifier) ​​that is linked to an NWDAF with an AnLF that indicates deregistration. Examples of these processes include any of the following: (a) Clear (or clean up or stop) training or retraining associated with an ML model that is associated with an NWDAF with an AnLF indicating deregistration; (b) Clear (or clean up or stop) the model reselection used for the analysis ID associated with the NWDAF with AnLF that indicates deregistration; (c) Clear (or clean up or stop) the markers (or selections of collected data) from the NWDAF with AnLF that indicates deregistration. (d) Clear (or clean up or stop) data collection associated with NWDAFs with AnLF that are indicated for deregistration.

[0093] ML Model Accuracy Provision Redirection: Information indicating to an NWDAF with MTLF that a new NWDAF with AnLF will begin or is providing ML model accuracy information for the same ML model identifier and / or analysis ID, where both the ML model identifier and analysis ID are associated with existing subscriptions to ML model accuracy information from different NWDAFs with AnLF. ML model accuracy provision redirection may include any of the following: ML model accuracy monitoring redirection (e.g., implemented as a flag): indicates that a new NWDAF with AnLF will begin or is providing an analysis ID associated with an existing subscription for ML model accuracy information for such an analysis ID associated with an NWDAF with MTLF. Such a flag allows or enables an NWDAF with MTLF to consider completely separating the ML model accuracy information from the subscription of the initial NWDAF or source NWDAF with AnLF, or to bind the outputs (or notifications) of two subscriptions for ML model accuracy information associated with the same analysis ID to a single (or unique, centralized, combined, aggregated, associated, or bound) accuracy information of the ML model.

[0094] The target NWDAF contains the AnLF NF ID (e.g., the NF identifier of the NWDAF with AnLF that receives the analysis migration and redirects the ML model accuracy monitoring function).

[0095] The subscription endpoint for the Nnwdaf_MLModelMonitor_Subscribe service operation at the target NWDAF of AnLF.

[0096] The original (or existing or source NWDAF with AnLF) subscription identifier (also known as subscription association ID) of the ML model accuracy information associated with the subscription at the source NWDAF with AnLF.

[0097] ML model identifier (e.g., a unique ML model identifier).

[0098] One or more analytics IDs.

[0099] Analyze the user's ID (e.g., each analytics ID).

[0100] The original (or existing) subscription identifier (also known as the subscription association ID) of the analysis output at the source NWDAF of AnLF.

[0101] A rollback timer instructs an NWDAF with an MTLF to wait for a specified period before triggering a change in a process associated with an ML model (e.g., a unique ML model identifier) ​​that is linked to an NWDAF with an AnLF that indicates a redirection. Examples of these processes include any of the following: (a) Clear (or clean up or stop) training or retraining associated with an ML model that is associated with an NWDAF with an AnLF indicating deregistration; (b) Clear (or clean up or stop) the model reselection used for the analysis ID, which is associated with the NWDAF with AnLF indicating redirection; (c) Clear (or clean up or stop) the flags (or selections of collected data) from the NWDAF with AnLF that indicates redirection. (d) Clear (or clean up or stop) data collection associated with NWDAFs with AnLF that are indicated for redirection.

[0102] Change indicators related to ML model accuracy generation: Information indicating that one or more analysis IDs or data sources used for ML model accuracy generation have changed (e.g., since the last call to the Nnwdaf_MLModelMonitor_Subscribe service to update the ML model accuracy subscription) or have been created (e.g., the first call to the Nnwdaf_MLModelMonitor_Subscribe service to create a subscription for ML model accuracy information) since the last update of the ML model subscription. Change indicators related to ML model accuracy generation may include any of the following: A flag indicating a change in subscription.

[0103] Change type, possible values ​​are any of the following: include new analysis ID, delete analysis ID, include new data source, delete data source.

[0104] Reasons for change: new subscription to analytics ID, unsubscription of analytics ID, receiving analytics ID migration from a source NWDAF with AnLF, migrating analytics ID to a new NWDAF with AnLF.

[0105] One or more analytics IDs.

[0106] Analyze the user's ID.

[0107] The subscription association ID associated with the subscription that requests the analysis ID output.

[0108] An existing subscription identifier (also known as a subscription association ID) that is associated with the ML model accuracy information at the target NWDAF with AnLF.

[0109] ML Model Accuracy Migration Indicator: Information related to existing subscriptions that use ML model accuracy information for ML model IDs and / or analysis IDs. Detailed Implementation

[0110] Figure 1 A first network entity 100 for generating model accuracy information according to the present disclosure, and a second network entity 102 and a third network entity 103 for using model accuracy information according to the present disclosure are shown.

[0111] The first network entity 100 is an entity applicable to the model accuracy information 105 used to generate the model, which is associated with a model ID and / or analysis ID. For example, the first network entity 100 could be an NWDAF with AnLF. Specifically, the first network entity 100 could be... Figure 1 In the example, the source NWDAF has an AnLF. The second network entity 102 can also be an NWDAF with an AnLF, and in this example, it can be a target NWDAF with an AnLF. The third network entity 103 can be... Figure 1 The example has an NWDAF with MTLF.

[0112] The first network entity 100 is used to obtain a first instruction 101 for providing model accuracy information 105. There are several ways in which the first network entity 100 obtains the first instruction 101. For example, the first network entity 100 may recognize that the model ID and / or analysis ID associated with the provision of model accuracy information 105 is related to an analysis migration from the first network entity 100 to the second network entity 102. For example, the analysis ID may be redirected from the first network entity 100 to the second network entity 102. Furthermore, the first network entity 100 may receive such an instruction 101 from the second network entity 102. For example, the first network entity 100 may receive any of the following from the second network entity: model accuracy information context type, model accuracy information context flag, and one or more parameters associated with the provision of model accuracy information 105.

[0113] Model accuracy information 105 is used by a third network entity 103. The third network entity 103 can use the used model accuracy information 105 to train a model, especially the model associated with the model accuracy information 105. The third network entity 103 can also use the model accuracy information 105 to select one or more new models for an analysis ID. The third network entity 103 can provide models associated with the model accuracy information 105 and / or any other models associated with or selected for an analysis ID. Model accuracy information 105 is associated with the model and one or more parameter information, and represents the quality information of the model.

[0114] First network entity 100 may provide a second instruction 104 to third network entity 103. The second instruction 104 is related to changes provided based on model accuracy information 105 according to the first instruction 101. The second instruction 104 includes at least one of first information, second information, third information, and fourth information. Using the model accuracy information 105 associated with the model ID and / or analysis ID from first network entity 100, third network entity 103 may accordingly obtain the second instruction 104 from first network entity 100 (as shown). Alternatively, third network entity 103 may obtain the second instruction 104 from second network entity 102.

[0115] The first message indicates that the provision of model accuracy information 105 for model ID and / or analysis ID be terminated at the first network entity 100. The second message indicates that the provision of model accuracy information 105 for model ID and / or analysis ID be redirected from the first network entity to the second network entity 102. The third message indicates one or more changes related to the provision of model accuracy information 105 for model ID and / or analysis ID. The fourth message indicates that, according to the first instruction 101, the second network entity 102 be registered as a provider of model accuracy information 105 for model ID and / or analysis ID.

[0116] The first network entity 100 and the third network entity 103 may each include a processor or processing circuitry (not shown) for executing, performing, or initiating various operations of the respective network entities 100, 103 described herein. The processing circuitry may include hardware, and / or may be controlled by software. The hardware may include analog or digital circuitry, or both. The digital circuitry may include components such as application-specific integrated circuits (ASICs), field-programmable arrays (FPGAs), digital signal processors (DSPs), or multi-purpose processors. The first network entity 100 and the third network entity 103 may also each include memory circuitry storing one or more instructions that can be executed by the processor or processing circuitry (specifically, under software control). For example, the memory circuitry may include a non-transitory storage medium storing executable software code that, when executed by the processor or processing circuitry, causes the various operations of the respective network entities 100, 103 to be performed. In one embodiment, the processing circuitry includes one or more processors and a non-transitory memory connected to the one or more processors. Non-transient memory may carry executable program code that, when executed by one or more processors, causes the corresponding network entities 100, 103 to perform, conduct, or initiate the operations or methods described herein.

[0117] Further disclosure focuses on Figure 1 The interaction of network entities described herein, specifically, refers to the exemplary cases of a first network entity 100 and a second network entity 102 having a source NWDAF and a target NWDAF respectively having AnLF (hereinafter referred to as AnLF), and a third network entity 103 having an NWDAF with MTLF. The first network entity 100 and the second network entity 102 have enhanced capabilities to perform analytical migrations and have extensions that also support the migration of information related to the generated model accuracy information 105.

[0118] Figure 2 The system architecture, comprising the first, second, and third network entities, is illustrated. This architecture is used for the generation, provision, and redirection of model accuracy information 105. The system architecture is based on the 3GPP R18 specification for NWDAF. The user 200 of the analysis output initially interacts with the source NWDAF 100 with AnLF to request access to the analysis output, such as... Figure 2 As shown in step 1.

[0119] When the source NWDAF 100 with AnLF starts generating analysis accuracy information for the analysis ID (e.g., due to receiving a request for analysis output from user 200) and / or when the source NWDAF 100 with AnLF starts using the ML model for the analysis ID, the source NWDAF 100 with AnLF can trigger an interaction with the NWDAF 103 with MTLF (or simply MTLF), such as... Figure 2 As shown in step 2 of the document.

[0120] An NWDAF 100 with AnLF can register itself with an NWDAF 103 with MTLF. For example, 3GPP TS23.288 defines the service Nnwdaf_MLModelMonitor_Register for performing such a registration process. This registration of the NWDAF 100 with AnLF indicates that it can provide ML model accuracy information 105 to the NWDAF 103 with MTLF, which provides the ML model used by the NWDAF 100 with AnLF to generate the analysis output. Furthermore, as defined in TS 23.288, this registration and / or local policies may cause (or trigger) the NWDAF 103 with MTLF to request the generation of ML model accuracy information 105 from the registered NWDAF 100 with AnLF. TS 23.288 defines an example of how to implement this request through the service Nnwdaf_MLModelMonitor_Subscribe, where the input parameters are: one or more analysis IDs, one or more unique identifiers of one or more ML models to be monitored, the accuracy metric to be monitored, and optional reporting period or one or more reporting thresholds.

[0121] The NWDAF 100 with AnLF receives the request and initiates a mechanism (or process) for accuracy monitoring of the ML model (also known as ML model accuracy generation), and provides the generated ML model accuracy information 105 to the NWDAF 103 with MTLF that has subscribed to such information.

[0122] Simultaneously, various reasons may trigger the analytics migration process, such as those defined in Section 6.1B of TS 23.288. Examples of such reasons include the need for the NWDAF 100 with AnLF to shut down or become overloaded and decide to move some analytics ID subscriptions to other NWDAFs, or the UE being the target of the analytics ID being generated being moved to a different area of ​​a network not served by the NWDAF 100 with AnLF. Therefore, the new target NWDAF 102 with AnLF needs to take over the generation of analytics IDs for such UEs, as well as all other processes related to such analytics IDs, such as ML model accuracy information generation.

[0123] In addition to the reasons stated above, this disclosure specifically defines interactions and information exchanges that enable the redirection (or rebinding, remapping, relinking, or migration) of subscriptions monitored (or generated) by ML model accuracy information 105 to a new NWDAF 102 with AnLF, which is now capable of providing ML model accuracy information 105 for analyzing IDs and / or ML models. For example, it is possible to provide new subscriptions associated with existing ML model accuracy information subscriptions that have been migrated to or need to be started in the new NWDAF 102 with AnLF. These interactions and associated information are summarized below.

[0124] Interaction between source NWDAF 100 with AnLF and NWDAF 103 with MTLF ( Figure 1 Step 2), wherein at least one of the following information is exchanged between these entities: ML model accuracy provides termination, ML model accuracy provides redirection.

[0125] Interaction between source NWDAF 100 with AnLF and target NWDAF 102 with AnLF ( Figure 1 Step 3), wherein at least one of the following information is exchanged between these entities: migration information related to ML model accuracy generation, ML model accuracy context flags, and ML accuracy generation context type.

[0126] The interaction between the target NWDAF 102 with AnLF and the NWDAF 103 with MTLF (which is using ML model accuracy information from the source NWDAF 100 with AnLF) involves exchanging at least one of the following information between these entities: ML model accuracy migration instructions, subscriptions based on information from ML model accuracy migration instructions, subscriptions based on information from ML model accuracy provision termination, and instructions for changes related to ML model accuracy generation.

[0127] The embodiments of this disclosure describe in detail the following: Figure 2 The specific circumstances under which such information is exchanged in each step of the description.

[0128] More specific exemplary embodiments of this disclosure are described below. All exemplary embodiments of this disclosure contemplate the 5G network architecture defined by 3GPP and documented in TS 23.501. Specifically, the embodiments focus on extensions related to NWDAF network functionality, which are defined in the 3GPP TS 23.288 specification. Exemplary embodiments are based on Figures 3 to 7 The process (alternative) is shown and described in conjunction with these diagrams.

[0129] exist Figures 3 to 7 In other embodiments of the alternatives described, it is possible that no explicit mechanism for pausing and resuming subscriptions to its data and / or one or more related processes (e.g., data collection) is implemented at the NWDAF with MTLF. In these possible embodiments, Figures 3 to 7 The steps shown for pausing and resuming processes at the NWDAF with MTLF (third network entity 103) are unnecessary. Instead, the NWDAF with MTLF can still interact with the source NWDAF and / or target NWDAF with AnLF and can determine the mapping between the provision of model accuracy information 105 from the source NWDAF with AnLF (first network entity 100) and the provision of model accuracy information 105 from the target NWDAF with AnLF (second network entity 102), for example, by reusing or reassociating data and / or one or more processes related to the provision of model accuracy information 105 from the source NWDAF with AnLF to the provision of model accuracy information 105 from the target NWDAF with AnLF.

[0130] Figure 3 Alternative A is illustrated: a source-triggered analytics migration process where the source NWDAF 100 triggers a redirection of ML model accuracy preparation via MTLF. This alternative aims to preserve the analytics migration process with no or very minor changes and centralize these changes to redirect the ML model accuracy information monitoring process (also known as the redirection of ML model accuracy information subscription) in the interaction between the source NWDAF 100 with AnLF and the NWDAF 103 with MTLF. These changes are then centralized in extensions of the Nnwdaf_MLModelMonitor service operations, such as registration, deregistration, subscription, and notification.

[0131] Step 0: These steps are performed when the source NWDAF 100 with AnLF recognizes that it is able to generate ML model accuracy information 105 for the NWDAF 103 with MTLF. That is, the source NWDAF 100 recognizes that it is using a given ML model provided by a specific NWDAF 103 with MTLF, and the source NWDAF 100 with AnLF is able to monitor the ML model accuracy information 105. This recognition is performed when the source NWDAF 100 with AnLF begins monitoring the analysis accuracy information of the analysis ID based on its local policy, or when it receives feedback from the NF user of the analysis ID. Based on this recognition, the source NWDAF 100 triggers the following steps (these steps are performed before any analysis migration process occurs): Step 0a: The source NWDAF 100 with AnLF invokes a service related to the ML model from the NWDAF 103 with MTLF, thereby enabling the NWDAF 100 with AnLF to monitor ML model accuracy information 105. For example, the source NWDAF 100 may invoke the service "Nnwdaf_MLModelMonitor_Register" defined in TS 23.288 to indicate that the source NWDAF 100 is registering with the NWDAF 103 with MTLF as a provider of ML accuracy information 105 for a given ML model.

[0132] Step 0b: The NWDAF 103 with MTLF decides to request ML model accuracy information 105 from the (source) NWDAF 100 with AnLF, which is capable of providing such information for a specific model. This decision may be based on a local policy and / or when it receives a registration from an NWDAF with AnLF, which indicates that it is capable of providing ML model accuracy information 105 for the ML model.

[0133] Step 0c: NWDAF 103 with MTLF subscribes to a service provided by (source) NWDAF 100 with AnLF, wherein (source) NWDAF 100 with AnLF is able to provide ML model accuracy information 105 for the desired ML model. NWDAF 103 with MTLF can, for example, invoke the service “Nnwdaf_MLModelMonitor_Subscribe” to request a subscription to the ML model accuracy information 105.

[0134] Step 0d: The source NWDAF 100 with AnLF notifies the NWDAF 103 with MTLF of the computed ML model accuracy information 105. An example of a service that can be used to deliver a notification including the ML model accuracy information 105 is the service "Nnwdaf_MLModelMonitor_Notify".

[0135] Step 1: The source NWDAF 100 with AnLF recognizes the need to perform an analytics migration. When preparing information for migrating analytics subscriptions associated with analytics IDs to the target NWDAF 102 with AnLF, the source NWDAF 100 with AnLF recognizes that for such analytics IDs and / or the ML models associated with those analytics IDs, there are existing subscriptions from the NWDAF 103 with MTLF to receive ML model accuracy information 105.

[0136] Step 2: Based on the identification of the need to perform the analysis migration and the existence of an ML model accuracy subscription with MTLF associated with the analysis ID and / or ML model related to the analysis migration process, the source NWDAF 100 with AnLF provides termination of the ML model accuracy provision related to the existing ML model accuracy subscription to the NWDAF 103 with MTLF, or indicates a redirection of the ML model accuracy provision. Such indication can be achieved in any of the ways described below: Option 1: Achieve this via service deregistration: In this case, the source NWDAF 100 with AnLF calls the “Nnwdaf_MLModelMonitor_Deregister” service, which includes any of the following parameters: information identifying the registration of the NWDAF with AnLF that provides ML model accuracy information 105, such as a registration or subscription identifier (or subscription association ID), and ML model accuracy provision termination or ML model accuracy provision termination.

[0137] Option 2: Implemented via notification: In this case, the source NWDAF 100 with AnLF calls "Nnwdaf_MLModelMonitor_Notify" with any of the following parameters: subscription identifier (or subscription association ID) and ML model accuracy provision termination or ML model accuracy provision termination.

[0138] Step 3: Based on the termination of ML model accuracy provision received from the source NWDAF 100 with AnLF or the termination of ML model accuracy provision, the NWDAF 103 with MTLF determines that before deleting all data related to the subscription of ML model accuracy information 105 (e.g., previous records of ML model accuracy information, and / or the association of ML model accuracy with ML model identifier, and / or the association of ML model accuracy information 105 with other data collection processes) and / or clearing (or stopping, or deleting) any other processes related to ML model accuracy information (or triggered based on ML model accuracy information or used to trigger based on ML model accuracy information), the NWDAF 103 with MTLF suspends the subscription to ML model accuracy information for a period of time without forcing any changes to such subscription and / or associated data and / or associated processes. The NWDAF 103 with MTLF can use information received from the source NWDAF 100 with AnLF (e.g., including fallback triggers in received ML model accuracy provision termination or ML model accuracy provision termination), or information configured locally, to trigger the process of waiting for new registrations via the Nnwdaf_MLModelMonitor_Register service and / or new notifications from the new NWDAF 102 with AnLF, where the new NWDAF 102 with AnLF takes over the paused subscription to ML model accuracy provision.

[0139] Step 4: The source NWDAF 100 with AnLF triggers an analytics migration to the target NWDAF 102 with AnLF. At this point, the migration process when the source NWDAF calls the Nnwdaf_AnalyticsSubscription_Transfer() service does not need to be changed. The target NWDAF 102 with AnLF receives the analytics migration request from the source NWDAF 100 with AnLF and triggers processing to retrieve analytics context information (if needed) from the source NWDAF 100 with AnLF. The target NWDAF 102 finally completes the analytics migration process and takes over the generation of analytics output.

[0140] Note 1: Step 4 or steps 2 and 3 can occur in parallel or in any order.

[0141] Step 5: Based on its internal logic and / or on indications for providing analysis accuracy information associated with the analysis ID received by the source NWDAF 100 with AnLF (e.g., as described in TS 23.288), the target NWDAF 102 with AnLF determines that it should also trigger monitoring of ML model accuracy information associated with the analysis ID and / or ML model, wherein the analysis ID and / or ML model is associated with a newly activated analysis output subscription generated by the analysis migration process.

[0142] Step 6: (a – Option 1) If the target NWDAF 102 with AnLF has not yet registered itself as a provider of ML model accuracy information for the ML model used by the migrated analytics ID, then the NWDAF 102 with AnLF invokes the “Nnwdaf_MLModelMonitor_Register” service to register itself as a provider of ML model accuracy information for the ML model (e.g., identified by a unique ML model identifier) ​​and / or analytics ID, possibly including other parameters, such as the subscription endpoint for the Nnwdaf_MLModelMonitor_Subscribe service operation at the target NWDAF with AnLF. Additionally, the NWDAF with the AnLF NF ID parameter associated with the target NWDAF 102 with AnLF is added to the “Nnwdaf_MLModelMonitor_Register” service request parameters so that the NWDAF 103 with MTLF can determine whether this registration is related to a suspension of subscription to ML model accuracy information. In this case, the NWDAF with the AnLF NF ID parameter can be considered equivalent to a subscription change indication.

[0143] Step 6 (b – Option 2) If the target NWDAF 102 with AnLF has registered as an ML model accuracy information provider (also known as an AnLF-enabled NWDAF capable of monitoring ML model accuracy), then the target NWDAF 102 with AnLF provides the NWDAF 103 with MTLF with an indication (or subscription change indication) of changes related to ML model accuracy generation. For example, the target NWDAF 102 uses the Nnwdaf_MLModelMonitor_Notify service, which includes subscription change indications, to provide notifications. This notification may contain only subscription change indications, or it may contain both subscription change indications and generated ML model accuracy information.

[0144] Note 2: If the timer for the paused ML model accuracy subscription has not yet expired, proceed to steps 7 through 9.

[0145] Step 7: The NWDAF 102 with MTLF provides termination or redirection based on ML model accuracy and information received from the target NWDAF 102 with AnLF. The NWDAF 103 with MTLF determines that the target NWDAF 102 with AnLF is associated with a suspended ML model accuracy subscription and then resumes such subscription.

[0146] The following scenarios may apply to the decision to resume a paused ML model accuracy subscription.

[0147] (If step 6a was performed) If the target NWDAF 102 with AnLF registers itself using the Nnwdaf_MLModelMonitor_Register service operation, then the NWDAF 103 with MTLF is able to map the NWDAF containing the AnLF NF ID received during registration to the target NWDAF containing the AnLF NF ID, which is included in the ML model accuracy provision termination or ML model accuracy provision redirection received from the source NWDAF 100 with AnLF. This mapping allows the NWDAF 103 with MTLF to recognize that the target NWDAF 102 with AnLF is now a new provider of ML model accuracy information related to the suspension of subscription.

[0148] (If step 6b was performed) If the target NWDAF 102 with AnLF uses the Nnwdaf_MLModelMonitor_Notify service, the target NWDAF with MTLF compares any of the following information included in the ML model accuracy provision termination or ML model accuracy provision redirection received from the source NWDAF 100 with AnLF—one or more analytics IDs, the analytics user's ID, and the subscription association ID associated with the subscription for the analytics ID output request—with the same information received from the target NWDAF 102 with AnLF (i.e., the subscription change information indication included in the notification of that service) in the parameters included in the Nnwdaf_MLModelMonitor_Notify service. If one or more of these parameters are the same, the NWDAF 103 with MTLF can determine that the existing subscription to the ML model accuracy information of the target NWDAF 102 with AnLF is also associated with the suspended subscription to the ML model accuracy information. This mapping allows the NWDAF 103 with MTLF to recognize that the target NWDAF 102 with AnLF is now a new provider of ML model accuracy information associated with the suspended subscription.

[0149] The process of resuming a suspended subscription to ML model accuracy information involves reassociating the parameterization of the ML model generation, and / or the data and / or MTLF processes (e.g., ML model retraining, ML model reselection) with the target NWDAF102 having AnLF. In other words, resuming a suspended subscription to ML model accuracy information means reusing, moving, or reassociating the data and / or information from the suspended subscription to a subscription (new or existing) for ML model accuracy information related to the target NWDAF102.

[0150] Note 3: If the NWDAF 103 with MTLF does not perform the process of pausing the subscription to the ML model accuracy information 105 from the source NWDAF 100 with AnLF, then the NWDAF 103 with MTLF can still perform ML model generation and / or reassociation of data and / or MTLF processes (e.g., ML model retraining, ML model reselection) by associating the information received in step 2 with the information received in step 6. In this case, steps 3, 7, and 10 will not be executed; only steps 8 and 9 will be executed. Further references to pausing subscriptions in this application can also be understood as "reassigned subscriptions." This is a simplification of the text to express the complete sentence "reuse or move or reassign data and / or parameters and / or one or more processes associated with the subscription to the ML model accuracy from the source NWDAF with AnLF."

[0151] Step 8: When NWDAF 103 with MTLF determines that it should resume its previous suspended subscription to ML model accuracy information (associated with the source NWDAF with AnLF), or re-associate (or reuse, move, or copy) parameterizations and / or data and / or processes to (or reuse, move, or copy) a new target NWDAF 102 with AnLF capable of providing ML model accuracy information 105, NWDAF 103 with MTLF invokes an Nnwdaf_MLModelMonitor_Subscribe request from the target NWDAF 102 with AnLF to request a new subscription to receive ML model accuracy information associated with the ML model and / or analysis ID, where the ML model ID and / or analysis ID is associated with the suspended subscription; or updates an existing subscription using the target NWDAF 102 with AnLF. In both cases, the subscription parameters of the NWDAF with MTLF are based on information from the termination or redirection of ML model accuracy provision or parameters associated with the suspended subscription, thereby enabling the use of the same parameterizations being used by the suspended subscription in the new NWDAF. Step 9: Based on the subscription information received from the NWDAF 103 with MTLF, the target NWDAF 102 with AnLF begins to generate (or monitor) ML model accuracy information and provides such information to the target NWDAF 102 with AnLF using the Nnwdaf_MLModelMonitor_Notify service operation.

[0152] Step 10: [Conditional: If the timer for the paused subscription times out] If the timeout associated with the paused subscription times out, and the NWDAF 103 with MTLF does not receive any new registration requests or notifications from a different NWDAF with AnLF (as associated with the paused subscription), the NWDAF 103 with MTLF will consider that such a paused subscription should be deactivated, thereby deleting all data associated with such a paused subscription.

[0153] Figure 4 Alternative B is illustrated: a source-triggered enhanced analytics migration process, where the source NWDAF triggers MTLF preparation for ML model accuracy redirection. This alternative aims to achieve transparency in the process of redirecting ML model accuracy information monitoring (also known as redirecting ML model accuracy information subscription). In this case, the analytics migration process and the interaction between the source NWDAF 100 with AnLF and the NWDAF 103 with MTLF are altered.

[0154] Note: Figure 4 Steps 0 to 3 in the middle Figure 3 Steps 0 through 3 are the same as in [previous steps], and therefore are described in the text above. At this point, a possible embodiment of the NWDAF 103 with MTLF that does not pause and resume subscriptions is also [as described]. Figure 3 The related descriptions are consistent.

[0155] Step 4: The source NWDAF 100 with AnLF triggers an analysis migration to the target NWDAF 102 with AnLF. At this point, the source NWDAF 100 calls Nnwdaf_AnalyticsSubscription_Transfer() from the target NWDAF 102 with AnLF to request a service operation, which includes migration information or ML model accuracy context flags related to the generation of ML model accuracy information.

[0156] When migration information related to the generation of ML model accuracy information is included, steps 4b and 4c can be skipped because all relevant information for triggering ML model accuracy monitoring is included in the migration information, where the ML model is associated with the analysis ID being migrated.

[0157] When the Nnwdaf_AnalyticsSubscription_Transfer() service request only includes the ML model accuracy context flag, the ML model accuracy context flag indicates to the target NWDAF 102 with AnLF that analysis context information must be retrieved. Therefore, steps 4b and 4c must be performed, and when retrieving the analysis context, the target NWDAF 102 with AnLF needs to request migration information related to ML model accuracy generation. This is achieved by the target NWDAF 102 with AnLF requesting the Nnwdaf_AnalyticsInfo_ContextTransfer service operation from the source NWDAF 100 with AnLF and including the ML accuracy generation context type in that request. Based on the received ML accuracy generation context type, the source NWDAF 100 with AnLF includes migration information related to ML model accuracy generation (possibly included in the analysis context information) in its Nnwdaf_AnalyticsInfo_ContextTransfer response to the target NWDAF 102 with AnLF. The target NWDAF 102 with AnLF then completes the analysis migration process. Step 5: Based on the migration information related to ML model accuracy generation received from the source NWDAF 100 with AnLF, the target NWDAF 102 with AnLF triggers monitoring of ML model accuracy information 105 associated with the analysis ID and / or ML model received during the analysis migration process.

[0158] Step 6 (a – Option 1) If the target NWDAF 102 with AnLF has not yet registered itself as a provider of ML model accuracy information for the ML model used by the migrated analytics ID, then the NWDAF 102 with AnLF invokes the “Nnwdaf_MLModelMonitor_Register Request” operation from the NWDAF 103 with MTLF to register itself as a provider of ML model accuracy information 105 for the ML model (e.g., identified by a unique ML model identifier) ​​and / or analytics ID, possibly including other parameters, such as the subscription endpoint of the Nnwdaf_MLModelMonitor_Subscribe service operation at the target NWDAF 102 with AnLF. Additionally, the target NWDAF 102 with AnLF includes an ML model accuracy migration indication in the “Nnwdaf_MLModelMonitor_Register” service request parameters so that the NWDAF 103 with MTLF can determine whether this registration is related to a suspension of subscription to ML model accuracy information.

[0159] Step 6 (b – Option 2) If the target NWDAF 102 with AnLF has been registered as an ML model accuracy information provider (also known as an AnLF-enabled NWDAF capable of monitoring ML model accuracy), the target NWDAF 102 with AnLF provides an ML model accuracy migration indication to the NWDAF 103 with MTLF using the Nnwdaf_MLModelMonitor_Notify service. This notification may contain only the ML model accuracy migration indication, or it may contain both the ML model accuracy migration indication and the generated ML model accuracy information.

[0160] Step 7: The NWDAF 103 with MTLF provides termination or redirection based on ML model accuracy and receives ML model accuracy migration instructions from the target NWDAF 102 with AnLF. The NWDAF 103 with MTLF determines that the target NWDAF 102 with AnLF is associated with a suspended ML model accuracy subscription and then resumes such subscription.

[0161] Step 8: When the NWDAF with MTLF determines that it should resume its previously suspended subscription to ML model accuracy information (as associated with the source NWDAF with AnLF), the NWDAF 103 with MTLF invokes the Nnwdaf_MLModelMonitor_Subscribe request from the target NWDAF 102 with AnLF to request a new subscription to receive ML model accuracy information associated with the ML model and / or analysis ID, where the ML model ID and / or analysis ID are associated with the suspended subscription; or it updates an existing subscription using the target NWDAF 102 with AnLF. In both cases, the subscription parameters of the NWDAF with MTLF are based on information from an indication of termination, redirection, or migration of the ML model accuracy provision. Note: Figure 4 Steps 9 and 10 in the text are the same as those in the text. Figure 3 Steps 9 and 10 are the same, so they are described in the text above.

[0162] Figure 5Alternative C is illustrated: a source-triggered enhanced analysis migration process, where the target NWDAF triggers MTLF preparation for ML model accuracy redirection. This alternative aims to centralize the responsibility of the redirection of ML model accuracy information monitoring process (also known as the redirection of ML model accuracy information subscription) onto the target NWDAF 102 with AnLF. In this case, the analysis migration process and the interaction between the target NWDAF 102 with AnLF and the NWDAF 103 with MTLF are altered.

[0163] Note 1: Figure 5 Steps 0 and 1 in the middle Figure 3 Steps 0 and 1 are the same, so they are described in the text above.

[0164] Note 2: Figure 5 Steps 2 and 3 in the text are respectively related to Figure 4 Steps 4 and 5 are the same, so they are described in the text above.

[0165] Step 4: The source NWDAF 100 with AnLF is deregistered from the NWDAF, which calls the Nnwdaf_MLModelMonitor_DeRegister() service operation provided by NWDAF103 with MTLF.

[0166] Step 5: When the NWDAF 103 with MTLF receives a deregistration request from the NWDAF with AnLF based on its internal logic, the NWDAF 103 with MTLF determines that before deleting all data related to the subscription of ML model accuracy information (e.g., previous records of ML model accuracy information, and / or the association of ML model accuracy with ML model identifier, and / or the association of ML model accuracy information with other data collection processes) and / or clearing (or stopping, or deleting) any other processes related to ML model accuracy information (or triggered based on ML model accuracy information or used to trigger based on ML model accuracy information), the NWDAF 103 with MTLF suspends the subscription to ML model accuracy information for a period of time without forcing any changes to such subscription and / or associated data and / or associated processes. The NWDAF with MTLF can use locally configured information to trigger a process of waiting for a new registration via the Nnwdaf_MLModelMonitor_Register service and / or a new notification via the Nnwdaf_MLModelMonitor_Notify service from the new NWDAF 102 with AnLF, where the new NWDAF 102 with AnLF takes over the pause subscription provided by ML model accuracy.

[0167] Note 3:Figure 5 Step 6 in 5 and Figure 4 Step 6 is the same as in the previous step, so it is described in the text above.

[0168] Step 7: The NWDAF 103 with MTLF is able to determine that the target NWDAF with AnLF is associated with a suspended ML model accuracy subscription based on the ML model accuracy migration indication received from the target NWDAF 102 with AnLF, and then resume such subscription.

[0169] Note 4: Figure 5 Steps 8 to 10 in the middle Figure 4 Steps 8 through 10 are the same, and therefore are described in the text above.

[0170] Figure 6 Alternative D is illustrated: a target-triggered analytics migration process where the source NWDAF triggers MTLF preparation for ML model accuracy redirection. This alternative aims to preserve the analytics migration process with no or very minor changes and centralize these changes to redirect the ML model accuracy information monitoring process (also known as the ML model accuracy information subscription redirection) in the interaction between the source NWDAF 100 with AnLF and the NWDAF 103 with MTLF. These changes are then centralized in extensions of the Nnwdaf_MLModelMonitor service operations, such as registration, deregistration, subscription, and notification. The difference from Alternatives A and D is that the target NWDAF 102 is the entity that initiates the analytics migration.

[0171] Note 1: Figure 6 Step 0 and Figure 3 Step 0 is the same as in step 0, so it is described in the text above.

[0172] Step 1: The target NWDAF 102 with AnLF determines that an analysis migration should be performed from the source NWDAF 100 with AnLF. The target NWDAF 102 with AnLF triggers and performs the analysis migration, for example, by requesting analysis context information from the source NWDAF 100 with AnLF.

[0173] Note 2: Figure 6 All subsequent steps are related to Figure 3 The steps are the same as in the previous section, where, Figure 6 Steps 2, 3, and 4 in the text are respectively related to Figure 3 Steps 1, 2, and 3 are the same. Figure 6 Steps 5 to 10 in the middle and Figure 3 Steps 5 through 10 are the same.

[0174] Figure 7 Alternative E is shown: a target-triggered analysis migration process where the target NWDAF triggers the MTLF preparation for ML model accuracy redirection. This alternative aims to centralize the responsibility of the redirection of ML model accuracy information monitoring process (also known as the redirection of ML model accuracy information subscription) to the target NWDAF with AnLF. In this case, the analysis migration process and the interaction between the target NWDAF 102 with AnLF and the NWDAF 103 with MTLF are changed. The difference from Alternatives C and E is that the target NWDAF 102 is the entity that initiates the analysis migration.

[0175] Note 1: Figure 7 Steps 0, 2 to 10 in the middle and Figure 3 Steps 0, 2 through 10 are the same, and therefore are described in the text above.

[0176] Figure 8 A method for generating model accuracy information 105 for a model is illustrated, wherein the model is associated with a model ID and / or an analysis ID. Method 800 is performed by a first network entity 100 (e.g., a source NWDAF with AnLF). Method 800 includes step 801: obtaining a first instruction 101 for modifying the provided model accuracy information 105. The provision of model accuracy information 105 is used by a third network entity 103 (e.g., an NWDAF with MTLF) for using the model accuracy information 105 and is associated with the model and one or more parameter information; the model accuracy information 105 represents the quality information of the model. Method 800 further includes step 802: based on the first instruction 101, providing the third network entity 103 with a second instruction 104 related to a modification of the provision of model accuracy information 105. The second instruction 104 includes at least one of the following: The first information indicates that the provision of model accuracy information 105 for model ID and / or analysis ID is terminated at the first network entity 100.

[0177] The second information indicates that the provision of model accuracy information 105, which is used for model ID and / or analysis ID, will be redirected from the first network entity 100 to the second network entity 102 (e.g., a source NWDAF with AnLF).

[0178] The third information indicates one or more changes related to the provision of model accuracy information 105 used for model ID and / or analysis ID.

[0179] The fourth information instruction, based on the first instruction 101, registers the second network entity 102 as a provider of model accuracy information 105 for model ID and / or analysis ID.

[0180] Figure 9 A method 900 of this disclosure is illustrated using model accuracy information 105 of a model, wherein the model is associated with a model ID and / or an analysis ID, and the model accuracy information 105 represents quality information of the model. Method 900 is performed by a third network entity 103 (e.g., an NWDAF with an MTLF) and includes step 901: using the model accuracy information 105 associated with the model ID and / or analysis ID from a first network entity 100 (e.g., a source NWDAF with an AnLF), wherein the first network entity 100 is used to generate the model accuracy information 105 associated with the analysis ID. Method 900 further includes step 902: obtaining a second instruction 104 from the first network entity 100 or from a second network entity 102 (e.g., a target NWDAF with an AnLF), wherein the second instruction 104 is as shown above for method 800.

[0181] In summary, this disclosure offers several advantages. For example, when NWDAFs using ML models may be involved in the analysis migration process and MTLF cannot fully grasp which NWDAFs with AnLF are using the same ML model, this disclosure can reduce the uncertainty risk for NWDAF 103 with MTLF in determining the quality / correctness of the ML model used for analysis ID. This is achieved by indicating the need to change (or redirect) the subscription to ML model accuracy information.

[0182] In addition, the NWDAF 103 with MTLF can pause subscription to ML model accuracy information based on configuration and / or instructions obtained from the NWDAF 100, 102 with AnLF.

[0183] Furthermore, NWDAF 103 with MTLF can resume previously suspended subscriptions to ML model accuracy information based on new registrations and / or notifications from a new (target) NWDAF 102 with AnLF, with or without indication of changes related to ML model accuracy generation.

[0184] Furthermore, it reduces the risk of configuration failures where, among all NWDAFs initially using the same ML model for the analytics ID, only a subset of NWDAFs are updated to use a new (expectedly better) ML model for the analytics ID. This can be achieved by having an NWAF 103 with MTLF provide a termination, redirection, or migration indication based on ML model accuracy to another NWDAF with AnLF, offering a subscription request for ML model accuracy monitoring.

[0185] This disclosure has been described in conjunction with various embodiments as examples and implementations. However, based on a study of the drawings, this disclosure, and the independent claims, those skilled in the art will be able to understand and implement other variations in carrying out the claimed subject matter. In the claims and the description, the word "comprising" does not exclude other elements or steps, and the indefinite article "a" does not exclude a plurality. A single element or other unit may fulfill the function of several entities or items set forth in the claims. The enumeration of certain measures in dissimilar dependent claims does not imply that a combination of these measures cannot be used efficiently in advantageous implementations.

Claims

1. A first network entity (100) for generating model accuracy information of a model, characterized in that, The model is associated with a model identifier (ID) and / or analysis ID, and the first network entity (100) is used for: Obtain a first instruction (101) for providing model accuracy information (105) for modification, wherein the provision of the model accuracy information (105) is used by a third network entity (103) for using the model accuracy information (105) of the model, and is associated with the model and one or more parameter information; the model accuracy information (105) represents the quality information of the model; Based on the first instruction (101), a second instruction (104) related to the change in the provision of the model accuracy information (105) is provided to the third network entity (103), wherein the second instruction (104) includes at least one of the following: - First information, indicating that the provision of the model accuracy information (105) for the model ID and / or the analysis ID is terminated at the first network entity (100); - Second information, indicating that the provision of the model accuracy information (105) used for the model ID and / or analysis ID will be relocated from the first network entity to the second network entity (102). -Third information, indicating one or more changes related to the provision of the model accuracy information (105) for the model ID and / or analysis ID; - Fourth information, indicating that, in accordance with the first instruction (101), the second network entity (102) is registered as a provider of model accuracy information (105) for the model ID and / or analysis ID.

2. The first network entity (100) according to claim 1, characterized in that, It is also used to: obtain the first instruction (101) provided for changing the model accuracy information (105) based on at least one of the following: - By identifying the model ID and / or analysis ID associated with the provision of the model accuracy information (105) and related to the analysis migration from the first network entity (100) to the second network entity (102); - By receiving any of the following from the second network entity (102): - Model accuracy information context type, indicating that migration information related to the provision of the model accuracy information (105) should be included in the analysis context information; - Model accuracy information context flag, indicating the type of context for which the model accuracy information is requested; - One or more parameters related to the provision of the model accuracy information (105), wherein the provision of the model accuracy information (105) is associated with the model.

3. The first network entity (100) according to claim 2, characterized in that, It is also used to: if the first indication (101) is obtained by receiving the model accuracy information context type or the model accuracy information context flag from the second network entity (102), request the second network entity (102) one or more parameters related to the provision of the model accuracy information (105), wherein the provision of the model accuracy information (105) is associated with the model.

4. The first network entity (100) according to any one of claims 1 to 3, characterized in that it is used for: providing the second instruction (104) to the third network entity (103), wherein, The second instruction is included in the cancellation message or in a notification message with or without model accuracy information (105).

5. The first network entity (100) according to any one of claims 1 to 4, characterized in that, The second instruction (104) also includes at least one of the following: - The registration ID of the first network entity (100), which is related to the provision of the model accuracy information (105) performed by the first network entity (100), wherein the provision of the model accuracy information (105) is associated with the model; - The subscription ID of the third network entity (103) at the first network entity (100), the subscription ID being used for providing the model accuracy information (105) performed by the first network entity (100), wherein the provision of the model accuracy information (105) is associated with the model.

6. The first network entity (100) according to any one of claims 1 to 5, characterized in that, When the second network entity (102) registers as a provider of model accuracy information (105) for the model ID and / or analysis ID, the fourth information includes the ID of the second network entity (102).

7. A third network entity (103) for using model accuracy information (105) of a model, characterized in that, The model is associated with a model identifier (ID) and / or analysis ID, and the model accuracy information (105) represents the quality information of the model; the third network entity (103) is used for: Model accuracy information (105) from a first network entity (100) used to generate a model is used, wherein the model accuracy information (105) is associated with the model ID and / or the analysis ID; A second instruction (104) is obtained from the first network entity (100) or the second network entity (102), wherein the second instruction (104) includes at least one of the following: - First information, indicating that the provision of model accuracy information (105) for the model ID and / or analysis ID is terminated at the first network entity (100); - Second information, indicating that the provision of the model accuracy information (105) for the model ID and / or analysis ID is redirected from the first network entity (100) to the second network entity (102). - Third information, indicating one or more changes related to the provision of the model accuracy information (105) for the model ID and / or analysis ID from the second network entity (102); - Fourth information, indicating that the second network entity (102) is registered as a provider of model accuracy information (105) for the model ID and / or analysis ID.

8. The third network entity (103) according to claim 7, characterized in that, Also used for: Determine the relationship and / or mapping between the acquired second instruction (104) and the third network entity's use of the model accuracy information (105) for the model ID and / or analysis ID, wherein, The use of the model accuracy information (105) is associated with the model and one or more parameter information.

9. The third network entity (103) according to claim 7 or 8, characterized in that, It is also used to: if the second instruction (104) includes the first information or the second information, then based on local configuration and / or based on the second instruction (104), suspend the use of data and / or one or more processes related to the use of the model accuracy information (105) from the first network entity (100).

10. The third network entity (103) according to claim 9, characterized in that, In order to suspend one or more processes associated with the use of the model accuracy information (105), the third network entity (103) uses: (a) Stop changing or deleting any of the following for a specified period of time: - Data associated with the use of the model accuracy information (105) from the first network entity (100); - The training or retraining process associated with the model, which is related to the use of the model accuracy information (105) from the first network entity (100); or (b) Suspend the use of the model accuracy information (105) for a defined period of time without altering or deleting any of the following: - Subscription to the use of the model accuracy information (105) from the first network entity (100); - Data associated with the use of the model accuracy information (105) from the first network entity (100); - The process related to the use of the model accuracy information (105) from the first network entity (100).

11. The third network entity (103) according to any one of claims 7 to 10, characterized in that, It is also used to: provide the second network entity (102) with a request for a new provision of model accuracy information (105) to generate model accuracy information (105), wherein the request is based on the second instruction (104) if the second instruction (104) includes the fourth information.

12. The third network entity (103) according to any one of claims 7, 8 or 11, characterized in that, The third network entity (103) is further configured to: reuse or redirect or re-associate data and / or information from the one or more processes to a new provision of the model accuracy information (105), wherein the one or more processes are associated with the use of the model accuracy information (105) from the first network entity (100), and the new provision of the model accuracy information (105) is associated with the model ID and / or analysis ID from the second network entity (102).

13. The third network entity (103) according to any one of claims 7 to 12, characterized in that, It is also used to: if the second instruction (104) includes the third information, update the information at the third network entity (103) based on the second instruction (104), wherein the information is related to the use of the model accuracy information (105), the use of the model accuracy information (105) being associated with the model ID and / or analysis ID from the second network entity.

14. The third network entity (103) according to any one of claims 7, 8 or 12, characterized in that, The third network entity (103) is further configured to: reuse or redirect or re-associate data and / or information from the one or more processes to the updated information related to the use of the model accuracy information (105), wherein the one or more processes are associated with the use of the model accuracy information (105) from the first network entity (100), and the use of the model accuracy information (105) is associated with the model ID and / or analysis ID from the second network entity (102).

15. The third network entity (103) according to any one of claims 7 to 14, characterized in that, It is also used to: based on the second instruction (104), restore the data and / or one or more processes related to the use of the model accuracy information (105).

16. The third network entity (103) according to claims 7 to 15, characterized in that, Used for: determining that the second network entity (102) is associated with the suspended data and / or one or more processes related to the use of the model accuracy information (105), wherein the use of the model accuracy information (105) is associated with the model ID and / or analysis ID provided by the first network entity (100); resuming the use of the data and / or one or more processes related to the use of the model accuracy information (105) from the first network entity (100).

17. The third network entity (103) according to claim 15 or 16, characterized in that, In order to recover the data and / or one or more processes related to the use of the model accuracy information (105), the third network entity (103) is used to: The data and / or information from the one or more suspended processes are reused or redirected to or reassociated with a new provision of the model accuracy information (105), wherein the one or more suspended processes are associated with the use of the model accuracy information (105) from the first network entity (100), and the new provision of the model accuracy information (105) is associated with the model ID and / or analysis ID from the second network entity (102).

18. The third network entity (103) according to any one of claims 10 to 17, characterized in that, It is also configured to: if the time period associated with the suspended one or more processes times out before the third network entity (103) receives the third or fourth information from the second network entity (102), wherein the suspended one or more processes are associated with the use of the model accuracy information (105), then the third network entity (103) is configured to: deactivate the suspended one or more processes associated with the use of the model accuracy information (105), and / or delete data associated with the suspended one or more processes, wherein the suspended one or more processes are associated with the use of the model accuracy information (105).

19. A method (800) for generating model accuracy information (105) for a model, characterized in that, The model is associated with a model identifier (ID) and / or an analysis ID, and the method (00) is performed by a first network entity (100) and includes: Obtain (801) a first instruction (101) for providing model accuracy information (105), wherein the provision of the model accuracy information (105) is used by a third network entity (103) for training the model and is associated with the model and one or more parameter information; the model accuracy information (105) represents the quality information of the model; Based on the first instruction (101), a second instruction (104) related to the change in the provision of the model accuracy information (105) is provided (802) to the third network entity (103), wherein the second instruction (104) includes at least one of the following: - First information, indicating that the provision of the model accuracy information (105) for the model ID and / or the analysis ID is terminated at the first network entity (100); - Second information, indicating that the provision of the model accuracy information (105) for the model ID and / or analysis ID is redirected from the first network entity (100) to the second network entity (102). -Third information, indicating one or more changes related to the provision of the model accuracy information (105) for the model ID and / or analysis ID; - Fourth information, indicating that, in accordance with the first instruction (101), the second network entity (102) is registered as a provider of model accuracy information (105) for the model ID and / or analysis ID.

20. A method (900) for using model accuracy information (105) of a model, characterized in that, The model is associated with a model identifier (ID) and / or analysis ID, and the model accuracy information (105) represents the quality information of the model; the method (900) is performed by a third network entity (103) and includes: Using (901) the model accuracy information (105) from the first network entity (100) for generating the model, wherein the model accuracy information (105) is associated with the model ID and / or the analysis ID; Obtain (902) a second instruction (104) from the first network entity (100) or the second network entity (102), wherein the second instruction (104) includes at least one of the following: - First information, indicating that the provision of model accuracy information (105) for the model ID and / or analysis ID is terminated at the first network entity (100); - Second information, indicating that the provision of the model accuracy information (105) for the model ID and / or analysis ID is redirected from the first network entity (100) to the second network entity (102). - Third information, indicating one or more changes related to the provision of the model accuracy information (105) for the model ID and / or analysis ID from the second network entity (102); - Fourth information, indicating that the second network entity (102) is registered as a provider of model accuracy information (105) for the model ID and / or analysis ID.

21. A computer program comprising instructions, characterized in that, When the program is executed by a computer, it causes the computer to perform the method (800, 900) according to claim 19 or 20.

Images