Network analytics tracing, and rollback for stable consumption of analytics output

Abstract

The present disclosure relates to new generation mobile networks, e.g., 5G mobile networks, and generation of analytics information in mobile networks. In particular, the present disclosure relates to tracing an analytics ID or one or more analytics outputs for an analytics ID. Furthermore, the present disclosure relates to inferring or training a rollback for an unstable analytics ID or for at least one unstable analytics output for an analytics ID. To this end, the disclosure presents a network analytics tracing entity configured to obtain a tracing of one or more analytics outputs for an analytics ID and / or an indication having information for initiating tracing of the analytics ID, and configured to provide a rollback notification including one or more rollback actions associated with the analytics ID if at least one output for the analytics ID is unstable and / or if the analytics ID is unstable. The present disclosure further provides a network analytics tracing entity and a network analytics training entity configured to obtain and perform one or more rollback actions.

Description

Technical Field

[0001] The present disclosure relates to a new generation mobile network, such as a fifth generation (5G) mobile network, and to the generation of analytics information in a mobile network. In particular, the present disclosure relates to tracing one or more analytics outputs for an analytics identifier (ID), or analytics ID. Further, the present disclosure relates to inferring or training a rollback for an unstable analytics ID, or at least one unstable analytics output for an analytics ID. For this purpose, This the disclosure presents a network analytics tracing entity, a network analytics inference entity, a network analytics training entity, a network analytics consumer entity, a network analytics management entity, and methods for these entities.

Background Art

[0002] In current mobile networks, the network data analytics function (NWDAF) provides various analytics functions, and the various analytics functions can be used by some network functions (NFs) to improve or make their decisions (e.g., analytics information that can be provided by the NWDAF to support the NFs for radio access technology (RAT) and frequency selection). Each analytics function 、N requested by the WDAF being has its own ID (analytics ID) that can be used by the NF function to indicate the analytics (output). As shown in FIG. 1, the NWDAF can be part of the same NWDAF entity or can be placed in different NWDAF entities, two functionalities, namely, a) using a trained machine learning model for analytics and / orIt consists of a) an inference that provides a prediction and b) a training that generates a machine learning model using the collected data and / or the training data set. FIG. 1 shows, in particular, an example of analytics ID consumption for session management (SM).

[0003] The quality of NWDAF analytics is affected by various factors, such as the quality and quantity of the collected data for training the model, the configuration of the machine learning model, etc. As a result, the quality of the NWDAF machine learning model may affect the network performance or network status depending on the use of NWDAF analytics. A successful (or efficient or reliable) analytics ID can bring about or maintain a stable network status, while a low success rate (or inefficient or unreliable) analytics ID can bring about or produce an unstable network status. In the case of a stable network status, the system key performance indicators (KPIs) and / or metrics are maintained (or improved) within the expected pattern of use. In the case of an unstable network status, the KPIs and / or metrics of the system load remain within the expected pattern, but the KPIs indicating specific situations are unstable (or degrade from the expected pattern).

[0004] Therefore, it is important to detect and address cases of analytics IDs that result in an unstable network status. However, the time to detect and correct an analytics ID that results in an unstable network status is a variable that depends on the NWDAF black box logic. It may take seconds (e.g., if it is a model re-selection), or it may take days (e.g., it requires data collection with a sufficient amount of data or has human intervention for model tuning by an ML analyst). SUMMARY OF THE INVENTION

[0005] The present disclosure and its embodiments are based on the following further considerations with reference to FIG. 2, which shows an example of an unstable network status due to the impact of NF determination using analytics ID.

[0006] Currently, if an NF stops consuming an analytics ID (e.g., while the NWDAF is modifying it), this creates a major gap in the logic of NF operation, which in turn creates a major contradiction in its decision-making, and since the decision affecting the network status does not reflect the entire logic of the NF, it may worsen the degradation of the network status or the NWDAF may collect distorted data. If the NF continues to consume the same analytics ID with problems, the NF will continue to make decisions that result in an unstable network status. If the NWDAF inference simply reselects the ML model (e.g., from its local information or by querying NWDAF training), it can only recognize problems with the analytics ID through the data collection cycle. Therefore, none of the above alternative options constitute an effective or efficient solution to the problems described above.

[0007] That is, there is a problem that the NWDAF inference may not be able to continue providing an analytics ID output that results in a stable network status. Furthermore, the NWDAF training may not recognize problems in the analytics ID fast enough and thus may not be able to support the modification of the configuration / generation of the analytics ID output that results in an unstable network status.

[0008] The current lack of a solution to this problem gives rise to the following important technical problems. There are bound to be inconsistencies between the network status considered in the repaired analytics ID and the network status when the NF resumes consuming such repaired analytics after the NWDAF has repaired the unstable analytics ID and the NF has stopped consuming the analytics output for the analytics ID considered unstable. When the NF stops consuming the analytics ID (due to the need for repair), the NF will change its decision logic (by no longer considering the analytics ID output), which may have an effect on the network status. In parallel, since the data that the NWDAF can use to repair the analytics ID does not reflect the same scenario when the analytics ID is consumed again by the NF, if the network status data reflects only part of the actual behavior of the network status affected by the NF, the NWDAF is using this data to repair the unstable analytics ID.

[0009] Previous conventional solutions focus on (a) detecting the reliability of labels in training samples. However, this is a problem specific only to the training phase, and this alone does not address the issue of the trained model in the system KPI (i.e., the analytics KusNeither can the effect of the [[ID=]] be detected nor can incorrect labels for changes in the network status be traced. Furthermore, the solution focuses on collecting batches of data from multiple sources to allow for a rollback to batches of previously collected data. However, such a solution does not consider a mechanism for specifically detecting unreliable / unstable analytics [[ID=]] or for controlling the composition of the analytics [[ID=]]. Additionally, the solution focuses on performing a rollback to a previously known good state when a request for resources within the computer system is received (where the resources are defined as a database, load balancer, scaling loop machine). However, no specific solution for the rollback configuration of the analytics [[ID=]] is considered. Moreover, the analytics [[ID=]] is not a computer resource. The analytics [[ID=]] can be broadly defined as a system property.

[0010] In 3GPP, no specific solution is defined, but the specification suggests the possibility that the NWDAF can receive direct feedback from an NF that consumes the analytics [[ID=]] output. This can enable the NWDAF to detect performance degradation without requiring a significant amount of data collection. However, the issue of what happens to the NF during an ongoing analytics subscription period when the NWDAF attempts to detect performance degradation and improve the analytics [[ID=]] output is not considered.

[0011] In view of the above, embodiments of the present disclosure aim to solve the identified problems and provide an improved solution that avoids the drawbacks described above. The aim is to ensure that the analytics ID, or one or more analytics outputs for the analytics ID, results in a stable network status. Another aim is to enable the analytics training to quickly recognize problems in any unstable analytics ID or in at least one unstable analytics output for the analytics ID.

[0012] These and other aims are achieved by embodiments of the present disclosure as described in the appended independent claims. Advantageous implementations of the embodiments are further defined in the dependent claims.

[0013] In particular, the present disclosure introduces a method, an interface, and an entity for enabling the tracing of an analytics ID or one or more analytics outputs for an analytics ID. The tracing may be performed by a network analytics tracing entity and can enable the detection of an unstable analytics ID or an unstable analytics output for an analytics ID. This can further enable the restoration (or rollback) of the configuration of an unstable analytics ID or an unstable analytics output for analytics (in inference and / or training), particularly a rollback to the last known stable network state within a mobile system. This enables a network analytics inference entity (e.g., NWDAF inference) to continue to provide an analytics ID output to one or more network analytics consumer entities (e.g., NWDAF consumers), which results in a stable network status. Further, this enables the network analytics training entity (e.g., NWDAF training) to recognize issues in the configuration and / or generation of an analytics ID or one or more analytics outputs for an analytics ID that may result in an unstable network status.

[0014] A first aspect of the present disclosure is a network analytics tracing entity configured to obtain a display having information for initiating tracing of one or more analytics outputs for an analytics identifier (ID) and / or tracing of the analytics ID, and to provide a rollback notification related to the analytics ID when at least one output for the analytics ID is unstable and / or the analytics ID is unstable. The rollback notification includes one or more of at least one unstable analytics output for the analytics ID and / or the analytics ID, which is provided to a network analytics consumer entity, a network analytics inference entity, or a network analytics training entity; an inference rollback action for at least one unstable analytics output for the analytics ID and / or for the analytics ID, which is provided to the network analytics inference entity; and a training rollback action for at least one unstable analytics output for the analytics ID and / or for the analytics ID, which is provided to the network analytics training entity. The network analytics tracing entity is provided.

[0015] The network analytics tracing entity can initiate tracing of an analytics ID and / or one or more analytics outputs of the analytics ID, and thus can identify unstable settings. The tracing entity can further provide rollback notifications along with respective rollback actions, which enable returning the analytics ID or analytics output to a stable setting. Thus, the tracing entity helps ensure that the analytics ID and / or one or more analytics outputs for the analytics ID result in a stable network status. Further, it can help the network analytics training entity recognize issues regarding an unstable analytics ID and / or at least one unstable analytics output for the analytics ID more quickly.

[0016] In one implementation of the first aspect, the inference rollback action is an action in the network analytics inference entity to change inference settings regarding at least one unstable analytics output and / or regarding the analytics ID, and / or the training rollback action is an action in the network analytics training entity to change training settings regarding at least one unstable analytics output and / or regarding the analytics ID.

[0017] Changing the inference settings and / or the training settings can return at least one unstable analytics output for the unstable analytics ID and / or the analytics ID to a stable state.

[0018] In one implementation of the first aspect, the network analytics tracing entity traces one or more analytics outputs for an analytics ID and / or traces the analytics ID based on a display having information for initiating the tracing, and / or provides a rollback notification based on the tracing of one or more analytics outputs for the analytics ID and / or the tracing of the analytics ID.

[0019] The tracing entity can identify an unstable analytics ID and / or at least one unstable analytics output for the analytics ID. The tracing entity can also identify the reason for the instability. The rollback action in the rollback notification can be selected based on the tracing, i.e., based on the identification of the instability and optionally the reason. Thus, the best possible rollback action can be provided by the rollback notification.

[0020] In one implementation of the first aspect, the network analytics tracing entity is further configured to receive a rollback status notification, which includes at least one of the status of an inference rollback action executed by the network analytics inference entity and the status of a training rollback action executed by the network analytics training entity.

[0021] Thus, the tracing entity of the first aspect can be informed whether the rollback action has been successful, for example. Further, the tracing entity can thus gather information for future decisions. For example, if another instability occurs for the same analytics ID and / or at least one unstable output for the analytics ID, it can provide a further rollback notification taking into account the previously received rollback status.

[0022] In one implementation of the first aspect, the network analytics tracing entity provides at least one of an analytics status notification indicating that at least one of one or more analytics outputs for the analytics ID and the analytics ID is stable, a confirmation that at least one unstable analytics output for the analytics ID is unstable, an inference tracing activation display having information for activating tracing of an inference setting for at least one of one or more analytics outputs for the analytics ID and the analytics ID, and a training tracing activation display for activating tracing of a training setting for at least one of one or more analytics outputs for the analytics ID and the analytics ID.

[0023] This may enable other entities to adapt to unstable analytics IDs and / or unstable analytics outputs. Further, this may initiate tracing of inference and / or training settings in, for example, a network analytics training entity and a network analytics inference entity, respectively.

[0024] In one implementation of the first aspect, the network analytics tracing entity is further configured to generate analytics tracing information for one or more analytics outputs for an analytics ID and / or for an analytics ID, and based on the analytics tracing information, determine an inference rollback action and / or a training rollback action, the analytics tracing information including any of an analytics ID, an association of the analytics ID with one or more analytics outputs for the analytics ID, one or more quality indicators for at least one of one or more analytics outputs and the analytics ID for the analytics ID, inference settings for the analytics ID and / or for one or more analytics outputs for the analytics ID, and training settings for the analytics ID and / or for one or more analytics outputs for the analytics ID.

[0025] The analytics tracing information is also referred to in the present disclosure as an "analytics tracing data structure". The analytics tracing information can be used in the tracing entity to maintain all information available regarding an analytics ID and / or one or more analytics outputs for the analytics ID. It may maintain each training setting and / or inference setting and thus support the selection of the best rollback action by the tracing entity. The tracing entity may have analytics tracing information for a number of analytics IDs and respective analytics outputs for these analytics IDs.

[0026] In one implementation of the first aspect, the analytics tracing information further includes the status of an inference rollback action, the status of a training rollback action, one or more inference rollback actions, and at least one of one or more training rollback actions, which are associated with an analytics ID and / or with one or more analytics outputs for the analytics ID.

[0027] This further supports the selection of rollback actions by the tracing entity, especially when the same analytics ID and / or at least one analytics output for the analytics ID becomes unstable again.

[0028] In one implementation of the first aspect, the analytics tracing information further includes an association of the analytics ID and / or one or more analytics outputs for the analytics ID with at least one of a timestamp, an identification of a network analytics inference entity, an identification of a network analytics training entity, an identification of a network analytics consumer entity, and an identification of one or more analytics outputs for the analytics ID.

[0029] This further facilitates the operation of the tracing entity, especially the tracing of various analytics IDs and analytics outputs for the analytics IDs, as well as the selection of rollback actions in case of instability.

[0030] In one implementation of the first aspect, the step of obtaining a display having information for initiating tracing of one or more analytics outputs for an analytics ID and / or tracing of the analytics ID includes receiving, from at least one of a network analytics consumer entity, a network analytics inference entity, and a network analytics training entity, a message including a display having information for initiating tracing of one or more outputs for an analytics ID and / or tracing of the analytics ID, and receiving a setting including a display having information for initiating tracing of one or more analytics outputs for an analytics ID and / or tracing of the analytics ID.

[0031] That is, the tracing entity can be sent information for initiating tracing by another entity or can be configured with this information to initiate tracing.

[0032] In one implementation of the first aspect, the information for initiating tracing of one or more analytics outputs for an analytics ID and / or tracing of the analytics ID is a flag, and the flag includes a flag that defines when to start tracing of one or more analytics outputs for an analytics ID and / or tracing of the analytics ID, and at least one of an analytics ID and a flag that defines when to start tracing of one or more analytics outputs for an analytics ID and / or tracing of the analytics ID.

[0033] Thus, tracing can be triggered for a specific analytics ID and / or at a specific time.

[0034] A second aspect of the present disclosure is a network analytics inference entity configured to obtain at least one analytics output for an analytics ID and / or an inference rollback action for the analytics ID, and to perform the inference rollback action, where performing the inference rollback action includes at least one of changing an inference setting for at least one analytics output for the analytics ID and / or for the analytics ID, determining and setting a new inference setting for at least one analytics output for the analytics ID and / or for the analytics ID, selecting a new analytics model for at least one analytics output for the analytics ID and / or for the analytics ID, and stopping an analytics model for at least one analytics output for the analytics ID and / or for the analytics ID, and provides a network analytics inference entity.

[0035] The network analytics inference entity of the second aspect is an entity configured to perform analytics inference (e.g., NWDAF inference). Thereby, it can output one or more analytics outputs for one or more analytics IDs respectively.

[0036] By performing the inference rollback action, the inference entity helps to restore the network to a stable status, in particular, to ensure that stable analytics IDs and / or stable analytics outputs for the analytics IDs are provided. Thereby, the inference entity can select from various options to return to a stable status.

[0037] In one implementation of the second aspect, obtaining an inference rollback action is receiving, from a network analytics tracing entity, a rollback notification related to an analytics ID, where the rollback notification includes at least one analytics output for the analytics ID and / or an inference rollback action for the analytics ID, receiving, and being set by the inference rollback action, and receiving, from the network analytics tracing entity, a rollback notification, where the rollback notification includes at least one analytics output for the analytics ID and / or the analytics ID, and based on the rollback notification, determining an inference rollback action, and includes one of them.

[0038] That is, there are various possibilities regarding how the inference entity recognizes the inference rollback action to be executed. The tracing entity of the first aspect may configure the inference entity with the inference rollback action.

[0039] In one implementation of the second aspect, the network analytics inference entity is further configured to provide at least one of an analytics status notification indicating at least one of: for a network analytics consumer entity, a network analytics training entity, or another network analytics inference entity, that at least one analytics output for the analytics ID is unstable and / or the analytics ID is unstable; a confirmation that at least one unstable analytics output for the analytics ID is unstable and / or the analytics ID is unstable; that one or more analytics outputs for the analytics ID are stable and the analytics ID is stable, based on the obtained or executed inference rollback action.

[0040] In this way, one or more other entities can be analytics outputs for recognized or stable and / or unstable analytics IDs and analytics IDs, and can appropriately adapt their behavior or the use of the analytics outputs.

[0041] In one implementation of the second aspect, the network analytics inference entity is further configured to provide a display having information for initiating tracing of one or more analytics outputs for an analytics ID and / or tracing of the analytics ID to the network analytics tracing entity.

[0042] Thus, the inference entity can be the entity that triggers the network analytics tracing entity. The inference entity can select the analytics IDs to be traced and / or the analytics outputs for these analytics IDs. The inference entity can do this, for example, when it suspects that there may be unstable analytics IDs or analytics outputs, for example, when the inference result is not as expected.

[0043] In one implementation of the second aspect, selecting at least one analytics output for an analytics ID and / or a new analytics model for the analytics ID includes providing a selection display related to an unstable analytics ID and / or related to at least one unstable analytics output for the analytics ID to a network analytics training entity, the selection display including a notification that at least one analytics output for the analytics ID is unstable and / or the analytics ID is unstable, and a request to select a new analytics model, the request including a reason for selecting the new analytics model, and a request to retrain the analytics model to generate a new analytics model, the request including a reason for retraining the analytics model, including at least one of the requests.

[0044] The stability of the analytics ID and / or the analytics output for the analytics ID can be restored by the new analytics model.

[0045] In one implementation of the second aspect, the network analytics inference entity is further configured to receive from the network analytics consumer entity a display for restarting at least one analytics output for the analytics ID and / or a previously suspended subscription for the analytics ID.

[0046] This may apply, for example, when one or more analytics outputs for an analytics ID and / or an analytics ID identified as unstable have been rolled back to a stable status (e.g., the last stable version).

[0047] In one implementation of the second aspect, the network analytics inference entity is further configured to provide a rollback status notification to the network analytics tracing entity, where the rollback status notification includes the status of the inference rollback action performed by the network analytics inference entity.

[0048] Accordingly, the tracing entity may recognize the status of the inference rollback action and may take it into account as described above.

[0049] In one implementation of the second aspect, the network analytics inference entity is further configured to obtain a display having information for initiating tracing of one or more analytics outputs for the analytics ID and / or tracing of the analytics ID, and based on the display having information for initiating tracing, generate inference settings regarding the analytics ID and / or regarding at least one analytics output for the analytics ID.

[0050] In one implementation of the second aspect, the network analytics inference entity is further configured to provide inference settings regarding the analytics ID and / or regarding at least one analytics output for the analytics ID to the network analytics tracing entity.

[0051] Since the inference settings can be provided to the tracing entity, it may facilitate the selection of an inference rollback action to address the instability of the analytics ID and / or the analytics output for the analytics ID.

[0052] A third aspect of the present disclosure is a network analytics training entity configured to obtain at least one analytics output for an analytics ID and / or a training rollback action for the analytics ID, and to execute the training rollback action, where executing the training rollback action includes at least one of changing training settings for at least one analytics output for the analytics ID and / or for the analytics ID, selecting and setting new training settings for at least one analytics output for the analytics ID and / or for the analytics ID, retraining or reselecting an analytics model for at least one analytics output for the analytics ID and / or for the analytics ID, and stopping an analytics model for at least one analytics output for the analytics ID and / or for the analytics ID, and provides a network analytics training entity.

[0053] The network analytics training entity of the third aspect is an entity configured to perform analytics training (e.g., NWDAF training). Thereby, it can train one or more analytics models, which can be used by a network analytics inference entity to provide analytics outputs. The network analytics training entity and the network analytics inference entity can be combined into one entity (e.g., NWDAF).

[0054] By executing a training rollback action, the training entity helps to restore the network to a stable status, in particular, to ensure that a stable analytics ID and / or a stable analytics output for the analytics ID are provided. Thereby, the training entity can select from various options to return to a stable status.

[0055] In one implementation of the third aspect, obtaining a training rollback action is receiving, from a network analytics tracing entity, at least one analytics output for an analytics ID and / or a rollback notification for the analytics ID, the rollback notification including at least one analytics output for the analytics ID and / or a training rollback action for the analytics ID, and including one of being set by the training rollback action.

[0056] That is, there are various possibilities regarding how the training entity recognizes the training rollback action to be executed. The tracing entity of the first aspect may configure the training entity by the training rollback action.

[0057] In one implementation of the third aspect, the network analytics tracing entity, based on the acquired or executed training rollback action, provides at least one of a notification to a network analytics consumer entity, a network analytics inference entity, or another network analytics training entity that at least one analytics output for an analytics ID is unstable and / or the analytics ID is unstable, a confirmation that at least one unstable analytics output for the analytics ID is unstable and / or the analytics ID is unstable, an analytics status notification indicating that at least one of one or more analytics outputs for the analytics ID is stable and / or the analytics ID is stable, information regarding the training rollback action, and a display of the reason for retraining, reselecting, or stopping the analytics model.

[0058] In this way, one or more other entities safe can obtain the defined and / or unstable analytics IDs and analytics outputs for the analytics IDs recognize and can appropriately adapt their behavior or the use of the analytics outputs.

[0059] In one implementation of the third aspect, the network analytics training entity is further configured to obtain a display of the reason for retraining or reselecting the analytics model from a network analytics inference entity or another network analytics training entity, and the display of the reason includes at least one of a notification that at least one analytics output for the analytics ID is unstable and / or the analytics ID is unstable, a request to select a new analytics model, and a request to retrain the analytics model to generate a new analytics model.

[0060] With the new analytics model, the stability of the analytics ID and / or the analytics output for the analytics ID can be restored.

[0061] In one implementation of the third aspect, the network analytics training entity is further configured to determine training settings for at least one analytics output for the analytics ID and / or for the analytics model regarding the analytics ID, based on the obtained display of reasons.

[0062] In one implementation of the third aspect, The network analytics training entity is it is further configured to provide a display having information for initiating the tracing of one or more analytics outputs for the analytics ID and / or the tracing of the analytics ID to a network analytics inference entity or another network analytics training entity.

[0063] In one implementation of the third aspect, the network analytics training entity is further configured to provide a rollback status notification to the network analytics tracing entity, and the rollback status notification includes the status of a training rollback action executed by the network analytics inference entity.

[0064] Accordingly, the tracing entity can recognize the status of the training rollback action and can take it into account as described above.

[0065] In one implementation of the third aspect, the network analytics training entity obtains a display having information for initiating the tracing of one or more analytics outputs for the analytics ID and / or the tracing of the analytics ID, and generates, based on the display having information for initiating the tracing, one or more analytics outputs for the analytics ID and / or training settings for the analytics ID.

[0066] In one implementation of the third aspect, the network analytics training entity is further configured to provide, to the network analytics tracing entity, one or more analytics outputs for the analytics ID and / or training settings for the analytics ID.

[0067] Since the training settings can be provided to the tracing entity, the training settings can facilitate the selection of an inference rollback action for handling the analytics ID and / or the instability of the analytics outputs for the analytics ID.

[0068] A fourth aspect of the present disclosure provides a network data analytics consumer entity configured to provide a display having information for initiating the tracing of one or more analytics outputs for the analytics ID and / or the tracing of the analytics ID to the network analytics tracing entity or the network analytics inference entity.

[0069] Accordingly, a consumer entity (e.g., NWDAF consumer) can trigger the tracing of the analytics ID and / or the analytics outputs for the analytics ID, and thus can support rollback in case of instability.

[0070] In one implementation of the fourth aspect, the network data analytics consumer entity is further configured to provide a display for one or more analytics outputs for an analytics ID and / or to restart a previously paused subscription for the analytics ID.

[0071] For example, when the previously unstable analytics ID or the output for the analytics ID has become stable again.

[0072] In one implementation of the fourth aspect, the network analytics consumer entity obtains from a network analytics tracing entity or a network analytics inference entity at least one of a notification that at least one analytics output for an analytics ID is unstable and / or the analytics ID is unstable, a confirmation that at least one unstable analytics output for the analytics ID is unstable, and an analytics status notification indicating that at least one of one or more analytics outputs for the analytics ID and / or the analytics ID is stable.

[0073] Accordingly, the network analytics consumer entity can appropriately adapt its behavior, particularly the use of the analytics ID and / or the analytics output for the analytics ID. For example, it can pause the consumption and / or subscription of one or more unstable analytics IDs and / or one or more unstable analytics outputs for one or more analytics IDs.

[0074] In one implementation of the fourth aspect, the network analytics consumer entity is further configured to provide one or more quality indications to the network analytics tracing entity regarding at least one of one or more analytics outputs of the analytics ID and / or regarding the analytics ID.

[0075] This may enable the tracing entity to determine whether one or more analytics IDs and / or one or more analytics outputs for one or more analytics IDs are unstable.

[0076] A fifth aspect of the present disclosure provides a network analytics management entity configured to make settings for initiating tracing of one or more analytics outputs for an analytics ID and / or tracing of the analytics ID, and / or to set analytics tracing information for the analytics ID, and / or to make settings for initiating collection of data on one or more quality indications regarding one or more analytics outputs for the analytics ID and / or regarding the analytics ID.

[0077] Thus, the network analytics management entity may support the tracing entity in its tracing and thus support the stability of analytics outputs in the network.

[0078] A sixth aspect of the present disclosure is a method for a network analytics tracing entity, the method comprising obtaining a display having information for initiating tracing of one or more analytics outputs for an analytics identifier (ID) and / or tracing of the analytics ID; and providing a rollback notification related to the analytics ID if at least one output for the analytics ID is unstable and / or if the analytics ID is unstable, the rollback notification being at least one unstable analytics output and / or analytics ID for the analytics ID, the rollback notification being provided to a network analytics consumer entity, a network analytics inference entity, or a network analytics training entity, and an inference rollback action for at least one unstable analytics output for the analytics ID and / or for the analytics ID, the rollback notification being provided to the network analytics inference entity, and a training rollback action for at least one unstable analytics output for the analytics ID and / or for the analytics ID, the rollback notification being provided to the network analytics training entity, the method comprising one or more of the foregoing.

[0079] The method of the sixth aspect corresponds to the tracing entity of the first aspect and may have an implementation form corresponding to the implementation form of the first aspect. Accordingly, the method of the sixth aspect and its implementation form may achieve the same advantages as those described above for the tracing entity of the first aspect and its implementation form.

[0080] A seventh aspect of the present disclosure is a method for a network analytics inference entity, the method comprising obtaining at least one analytics output for an analytics ID and / or an inference rollback action for the analytics ID, and executing the inference rollback action, the step of executing the inference rollback action including at least one of changing an inference setting for at least one analytics output for the analytics ID and / or for the analytics ID, determining and setting a new inference setting for at least one output for the analytics ID and / or for the analytics ID, selecting a new analytics model for at least one analytics output for the analytics ID and / or for the analytics ID, and stopping the analytics model for at least one analytics output for the analytics ID and / or for the analytics ID.

[0081] The method of the seventh aspect corresponds to the inference entity of the second aspect and may have an implementation format corresponding to the implementation format of the second aspect. Accordingly, the method of the seventh aspect and its implementation format may achieve the same advantages as those described above for the inference entity of the second aspect and its implementation format.

[0082] An eighth aspect of the present disclosure is a method for a network analytics training entity, the method comprising obtaining at least one analytics output for an analytics ID and / or a training rollback action for the analytics ID, and executing the training rollback action, the step of executing the training rollback action including changing training settings for at least one analytics output for the analytics ID and / or for the analytics ID, selecting and setting new training settings for at least one analytics output for the analytics ID and / or for the analytics ID, retraining or reselecting an analytics model for at least one analytics output for the analytics ID and / or for the analytics ID, and stopping an analytics model for at least one analytics output for the analytics ID and / or for the analytics ID, the method comprising at least one of the above.

[0083] The method of the eighth aspect corresponds to the training entity of the third aspect and may have an implementation form corresponding to the implementation form of the third aspect. Therefore, the method of the eighth aspect and its implementation form may achieve the same advantages as those described above for the training entity of the third aspect and its implementation form.

[0084] A ninth aspect of the present disclosure is a method for a network data analytics consumer entity, the method comprising providing a display having information for initiating tracing of one or more analytics outputs for an analytics ID and / or tracing of the analytics ID to a network analytics tracing entity or a network analytics inference entity.

[0085] The method of the ninth aspect corresponds to the consumer entity of the fourth aspect and may have an implementation form corresponding to the implementation form of the fourth aspect. Therefore, the method of the ninth aspect and its implementation form may achieve the same advantages as those described above for the consumer entity of the fourth aspect and its implementation form.

[0086] A tenth aspect of the present disclosure is a method for a network analytics management entity, the method including the steps of making settings for initiating tracing of one or more analytics outputs and / or tracing of an analytics ID for the analytics ID, and / or setting analytics tracing information for the analytics ID, and / or making settings for initiating collection of data for one or more quality indicators related to one or more analytics outputs for the analytics ID and / or related to the analytics ID.

[0087] The method of the tenth aspect corresponds to the management entity of the fifth aspect and may have an implementation form corresponding to the implementation form of the fifth aspect. Therefore, the method of the tenth aspect and its implementation form may achieve the same advantages as those described above for the management entity of the tenth aspect and its implementation form.

[0088] An eleventh aspect of the present disclosure is a computer program comprising program code for performing a method according to the sixth aspect, the seventh aspect, the eighth aspect, the ninth aspect or the tenth aspect, or any of these implementation forms, when executed on a computer or a processor.

[0089] A twelfth aspect of the present disclosure is a non-transitory storage medium storing executable program code, which, when executed by a processor, causes the method according to the sixth aspect, the seventh aspect, the eighth aspect, the ninth aspect or the tenth aspect, or any of these implementation forms, to be performed.

[0090] It should be noted that all entities, elements, units and means described in the present disclosure can be implemented in software elements or hardware elements, or any combination of these. Any steps performed by the various entities described in this application, and the functionality described as being to be performed by the various entities, are intended to mean that each entity is adapted or configured to perform its respective steps and functionality. In the following description of specific embodiments, even if a specific functionality or step to be performed by an external entity is not reflected in the description of the specific detailed elements of the entity performing that specific step or functionality, it should be apparent to those skilled in the art that these methods and functionalities can be implemented in respective software elements or hardware elements, or any combination of these.

Brief Description of the Drawings

[0091] The aspects and implementation forms described above will be explained in the following description of specific embodiments in relation to the accompanying drawings.

Figure 1

Figure 2

Figure 3

Figure 4

Figure 5

Figure 6

Figure 7

Figure 8

Figure 9

Figure 10

Figure 11

Figure 12

DETAILED DESCRIPTION OF THE INVENTION

[0092] definition In this section of the present disclosure, various terms used in the summary section and the detailed description of the embodiments of the present disclosure are explained and / or defined. The explanations and / or definitions are valid throughout the present disclosure.

[0093] Unstable Analytics ID: One or more analytics outputs and / or analytics IDs of an analytics ID (e.g., analytics type) associated with an unstable network status. Synonyms for the term unstable analytics ID include unstable analytics output, inefficient analytics output (or ID, or type), unreliable analytics output (or ID, or type), and low-accuracy analytics output (or ID, or type). One or more analytics outputs and / or analytics IDs of an analytics ID considered to be an unstable analytics ID are associated with an indication of low network performance. Examples of these indications are feedback from NF consumers (NF feedback), an analytics ID grade, and a notification regarding an analytics ID grade that deviates from a defined / configured / expected threshold.

[0094] Stable Analytics ID: One or more analytics outputs (e.g., analytics type) and / or analytics IDs of an analytics ID that results in (e.g., is related to, or associated with) a stable network status. Synonyms for the term stable analytics ID are stable analytics output, efficient analytics output (or ID, or type), reliable analytics output (or ID, or type), and high-accuracy analytics output (or ID, or type).

[0095] Network Status: One or more information that quantifies network performance. Examples of information that can be used to represent network status are KPIs and / or metrics associated with the network.

[0096] Stable Network Status: Defines a network status in which KPIs and / or metrics associated with the network are maintained (or improved) within an expected usage pattern.

[0097] Unstable network status: KPIs and / or metrics related to the overall performance of the network define a network status that stays within the expected pattern, but specific KPIs and / or metrics indicating specific situations deviate from or decrease from the expected pattern. Examples of overall performance metrics are the total number of received PDU sessions, average link throughput, and total number of UE registrations per slice. Examples of specific KPIs or metrics are the number of rejected sessions per application type and link utilization per UE group. KPIs or metrics related to the overall performance of the network can generally be related to performance information that can be obtained from OAM, management entities. On the other hand, specific KPIs and / or metrics can generally be related to performance information that can be obtained from control plane entities (e.g., session management function (SMF), policy control function (PCF), access and mobility management function (AMF)).

[0098] Information regarding unstable analytics ID: Defines a set of one or more data and / or parameters and / or properties, and / or the composition of the analytics ID (e.g., analytics type, analytics output) in the inference and / or training where this set is associated with an unstable analytics ID.

[0099] Information on how to repair an unstable analytics ID: Defines a set of data and / or parameters and / or properties, and / or the composition of the analytics ID (e.g., analytics type, analytics output) in the inference and / or training associated with the last known stable network state in the mobile system.

[0100] Tracing of Analytics ID: Represents a process executed to associate an Analytics ID with analytics output, indication of the quality of use of analytics (ID and / or output) in a mobile network, and / or configuration in inference and / or training for such an Analytics ID. In other words, it is a process of creating an Analytics Tracing Data Structure (ATDS) for a given Analytics ID and creating ATDS records for such an Analytics ID over time.

[0101] Indication of Analytics ID Tracing Activation: A message containing activation of Analytics ID tracing.

[0102] Analytics ID Tracing Activation: One or more parameters or information indicating that tracing of an Analytics ID needs to be started. Examples of these parameters are flags (e.g., tracing flag or rollback flag), Analytics ID, and tuples with flags.

[0103] Indication of Inference Tracing Activation: One or more parameters or information representing tracing of inference settings for which analytics (ID and / or output) needs to be started ) For example, this one or more parameters indicate for an inference NF that it needs to create Analytics Information Configuration Information (AICI) for the indicated analytics and provide such information to a network analytics tracing entity.

[0104] Indication of Training Tracing Activation: Analytics (ID and / or output) needs to be started )One or more parameters or information representing the tracing of training settings for. For example, this one or more parameters indicate for the training NF that it needs to create ATCI information for the indicated analytics and provide such information to the analytics tracing entity.

[0105] Analytics ID tracing activation settings: One or more information provided by a management entity (e.g., Operations, Administration, and Maintenance (OAM)) that defines that the tracing of the analytics ID needs to be started.

[0106] Analytics output: One or more information representing the results of the generated, calculated, predicted instances of the analytics ID or type. Different analytics IDs have different sets of information representing the analytics output. For example, an analytics ID such as "Observed service experience information" as defined in 3GPP TS 23.288 Release 17 defines that the output for such an analytics ID is the following information, namely, a set of S-NSSAI, slice instance service experience (0... maximum), and for each NSI ID related to the S-NSSAI, there is other information such as NSI ID, slice instance service experience (0... maximum), SUPI list, (0..SUPImax), the estimated percentage of UEs with similar service experience, etc.

[0107] Analytics ID or analytics type: The type of analytics that can be generated by an inference NF (e.g., NWDAF). For example, observed service experience information, slice load level information, network performance information, etc.

[0108] ATDS Record: An Analytics Tracing Data Structure (ATDS) record defines the network state at a given point in time when using a given analytics (e.g., analytics ID or analytics type or analytics output). An ATDS record can be a tuple indicating a mapping at a given point in time of an analytics ID to at least one of one or more analytics output identifications and / or one or more analytics outputs, a display of the quality of use of analytics (ID and / or output) in a mobile network (e.g., analytics output instance or analytics ID grade information (AidGI) or unstable analytics ID information (UAiDI) for network function (NF) feedback), and inference and / or training setting information (e.g., AICI and / or ATCI) for the analytics output or analytics ID. Optionally, the ATDS record can also include a rollback status and analytics rollback actions.

[0109] Last Known Stable Network State: The network state (e.g., ATDS record) at a given point in time when using a given analytics (e.g., analytics ID or analytics type or analytics output) that has the most appropriate analytics inference setting information (AICI) and / or analytics training setting information (ATCI) for the analytics (ID and / or output).

[0110] Analytics Rollback Action: One or more information that defines possible actions that can be taken to change, revert, or reset settings associated with an analytics (e.g., analytics output or analytics ID, or analytics type). Examples of this information and how they represent actions are described below. · One set of information relates to old / current settings, parameters, properties associated with analytics (ID and / or output), and a second set of information relates to new possible settings, parameters, properties associated with analytics (ID and / or output). These two sets of information define that an action to replace old information with new information should be performed. · Only one set of information Relates to new settings, parameters, and properties associated with analytics (ID and / or output). being This indicates that new information should be used to replace existing information associated with analytics (ID and / or output). · Only one set of information Relates to old / current settings. being This indicates that no specific action for analytics rollback or reset or repair was identified.

[0111] Rollback status notification: One or more information explaining the result of performing an analytics rollback action. Examples of sets of information explaining the result of the action are as follows. · A successful rollback, optionally indicating whether actions and / or settings included in an inference rollback notification or a training rollback notification were implemented (e.g., used to change settings), or, if different actions and / or settings from those included in the inference rollback notification or the training rollback notification were used, optionally indicating that different actions and / or settings were used. actions and / or settings · A failed rollback, optionally indicating whether the display of actions and / or settings included in an inference rollback notification or a training rollback notification was not implemented (e.g., the proposed settings could not be used). · · No rollback, which indicates that one or more proposed analytics rollback actions were not used by the analytics tracing entity.

[0112] Analytics status notification: Information defining the analytics (ID and / or analytics output) is considered a stable analytics ID, which means there are no issues with the analytics or that the analytics is no longer considered a stable analytics ID, which means there are issues with the analytics.

[0113] Indication of the quality of use of analytics: Information defining the quality of the analytics ID (or analytics type) and / or analytics output in relation to the network of such analytics (analytics ID or analytics type, or analytics output).

[0114] Analytics ID performance information (API): Optionally, information including KPIs and metrics to be monitored and associated with the consumed analytics (ID and / or analytics output) for a particular consumer of such analytics (e.g., analytics consumer). The API serves as an input for AidGI calculation and supports the identification of the effect of analytics ID consumption in network status changes.

[0115] Analytics ID grade information (AidGI): One or more information quantifying the effect of the consumed analytics (ID and / or output) on changes in network status after consumption of the analytics ID, for example in the format of a grade. An example of a set of information including AidGI is any of the following. ) · Identification of AidGI · Analytics ID · Analytics ID ofType identification: Analytics ID of the type for which the grade must be calculated · Identification of analytics ID output (list): Identification of one or more analytics output information for the analytics ID consumed in the time interval for grade calculation · Grade value: A single value representing how the monitored API for the consumed analytics ID diverges from the expected pattern of network status. For example, ■ The grade value can be a real number between 1 and -1, provided that - 0 indicates that the consumption of the analytics ID had no significant effect in the expected network status pattern - 1 indicates that the consumption of the analytics ID had a significant positive effect in the expected network status pattern (e.g., improved the pattern of KPIs) - -1 indicates that the consumption of the analytics ID had a significant negative effect in the expected network status pattern (e.g., degraded the pattern of KPIs) Unstable analytics ID information (UAidI): One or more information that identifies the crossing and deviation of the grade associated with the consumed analytics (ID and / or output) with a threshold representing that the analytics was identified as causing an unstable network status. Examples of information including UAidI are any of the following. · Identification of UAidI · Analytics ID of Type identification: Analytics ID of the type for which the grade must be calculated · Identification of analytics ID output (list): Identification of one or more analytics output information for the analytics ID consumed in the time interval for grade calculation · Calculated grade · Deviation of grade value from the threshold NF Feedback: Information that identifies the consumption of analytics (ID and / or output) from the perspective of the consumers of such analytics. Examples of NF feedback information can be any of the following. · NF feedback can be a set of information equivalent to analytics ID grade information (AidGI), provided that the NF (e.g., analytics consumer) defines a grade for the consumed analytics (ID and / or output) based on its internal information. · NF feedback can be a set of information equivalent to unstable analytics ID information (UAidI), provided that the NF (e.g., analytics consumer) identifies a grade for the consumed analytics (ID and / or output) based on its internal information, and the crossing of a threshold indicates a problem with the analytics. · NF feedback can be a tuple having information about the analytics (ID and / or output) and a flag indicating a problem with the analytics.

[0116] Analytics Tracing Data Structure (ATDS): One or more network states regarding analytics (e.g., analytics ID, analytics type, analytics output). The ATDS has an association with the analytics ID, the analytics output, an indication of the quality of use of the analytics in the mobile network, and settings in the inference and / or training regarding the analytics (which can mean the analytics output or the analytics ID or the analytics type), and optionally, an analytics rollback action and / or rollback status ATDSIt is a set of record histories. With this data structure, analytics ID, different associations of analytics outputs for this analytics ID, analytics ID grade value (AidGI) and / or UAidI warnings and / or NF feedback, inference settings and / or training settings information (e.g., AICI and / or ATCI) for analytics outputs or analytics ID, and optionally, if an analytics rollback has been performed and is associated with the ATDS record, it is possible to trace the rollback status over time. The analytics ID grade value (AidGI) and / or UAidI warnings are for the analytics ID in the mobile network of It is an example of the indication of the quality of use.

[0117] Analytics ID inference settings information (AICI): One or more information related to the settings and / or parameterizations used by the inference NF to generate analytics (ID and / or output). AICI includes any of the examples of information listed below. · The settings are the interval for data collection to generate analytics, whether preprocessing of the collected data for analytics generation is applied, the type of the collected data for analytics generation, the mapping of access network properties (TA, cell, radio type, radio frequency, etc.) to core network entities (e.g., associating TA with AMF) or properties (e.g., restricted or unrestricted S-NSSAI per TA), the geographical aggregation level (e.g., per UE, per AoI), and the temporal aggregation (i.e., per minute, per hour). · The parameterizations are related to analytics filter information and analytics reporting information (as defined in 3GPP TS 23.288 Release 17) for analytics generation, or, when collecting data for analytics generation, are related to data specifications or formats and processing (as defined in 3GPP TS 23.288 Release 17) definitions Analytics ID Training Configuration Information (ATCI): One or more information related to the settings and / or parameterizations of a machine learning (ML) model or simply a model (e.g., an optimization model) used for analytics (ID and / or output) at a given point in time. ATCI includes any of the examples of information listed below. · Settings are the interval for data collection to train an ML model or a model for analytics, whether preprocessing of the collected data for analytics training was applied, the type of the collected data for analytics training, the mapping of access network properties (TA, cell, radio type, radio frequency, etc.) to core network entities (e.g., associating TA with AMF) or properties (e.g., restricted or unrestricted S-NSSAI per TA), geographical aggregation level (e.g., per UE, per AoI), and temporal aggregation (i.e., per minute, per hour). · Parameterization related to the requested analytics ID to be trained or modeled: Analytics filter information and analytics reporting information (as defined in 3GPP TS 23.288 Release 17 for analytics), or, when collecting data for analytics training or statistical calculation, related to data specifications or formats and processing definitions (as defined in 3GPP TS 23.288 Release 17). · Parameterization related to the ML model or the model: Type of algorithm, gradient, weight, architecture information related to the model (e.g., neural network, number of layers, amount of neurons per layer), type of activation function, type of loss function, number of epochs used for training, settings for model validation (e.g., 5-fold cross-validation), distribution of the amount of data for various phases (training, validation test), scores for training, validation, and test for all phases, type of metric used (e.g., mean absolute error, mean squared error, coefficient of determination R^2, F1 score, precision, recall, accuracy, etc.).

[0118] Inference Rollback Notification (IRN): One or more pieces of information, perhaps enclosed within a message, that define rollback actions related to existing analytics IDs that can be performed by an inference NF. An inference rollback notification can, for example, be the following. · A message containing one ATDS record with unstable analytics ID information and another ATDS record with an ATDS record associated with the last known stable network state for the analytics ID. · Another possible implementation of an inference rollback notification is to have parameters such as a tuple with an analytics ID and / or identification of analytics output and a flag, where this flag indicates that there is a problem (or error) with the analytics (ID and / or output). Examples of flags that can be used are any of unstable analytics, analytics suspension, invalid analytics, temporarily invalid analytics, and inference rollback trigger.

[0119] Training Rollback Notification (TRN): One or more pieces of information, perhaps enclosed within a message, related to rollback actions associated with existing analytics (ID and / or) output) that can be performed by a training NF. A training rollback notification can, for example, be the following. · A message containing training information (e.g., ATCI) of an ATDS record with an ML model setting or parameterization for an unstable analytics ID and another training information (ATCI) of an ATDS record with an ATDS record associated with the last known stable network state for the ML model for the analytics ID · Another possible embodiment for the training rollback notification is to have parameters such as a tuple having an analytics ID and / or an identifier and a flag of the analytics output, provided that this flag indicates that there is a problem (or error) with the analytics (ID and / or output). Examples of flags that can be used are any of unstable analytics, analytics suspension, invalid analytics, temporarily invalid analytics, and training rollback trigger.

[0120] Unstable analytics notification (UN): One or more information that identifies an analytics ID and / or an analytics output as an unstable analytics ID, that is, an analytics (ID and / or output) related to a problem in the network status. The unstable analytics notification includes any of the information listed as examples below. · An ATDS record having analytics inference setting information (AICI) and / or analytics training setting information (ATCI) for the analytics ID of the ATDS record regarded as an unstable analytics ID. This information will indicate that there is a problem (or error) with the analytics ID and / or the analytics output. · Parameters such as a tuple having an identifier and a flag of the analytics ID and / or the analytics output, provided that this flag indicates that there is a problem (or error) with the analytics (ID and / or output). Examples of flags that can be used are unstable analytics, analytics suspension, invalid analytics, and temporarily invalid analytics.

[0121] Display for re-selection: One or more information that associates an ML model and / or a model with an analytics ID and / or an analytics output regarded as an unstable analytics ID, that is, an analytics (ID and / or output) related to a problem in the network status.

[0122] Indications regarding retraining: One or more pieces of information that associate the need (e.g., request) regarding retraining of an ML model and / or model training (and / or retraining) with an analytics ID and / or analytics output that is considered an unstable analytics ID, i.e., an analytics (ID and / or output) related to a problem in the network status.

[0123] Training rollback information: One or more pieces of information indicating changes in an ML model and / or model related to an analytics ID and / or analytics output, provided that the ML model and / or model is identified as being associated with an unstable analytics ID, i.e., an analytics (ID and / or output) related to a problem in the network status. Examples of one or more pieces of information including the training rollback information are · Identification of the ML model and / or model related to the unstable analytics ID · Explanation of the changed information related to the ML model and / or model related to the unstable analytics ID · One or more pieces of changed information (e.g., gradients, versions, architectures) related to the ML model and / or model related to the unstable analytics ID · ML model and / or model script and / or file and / or settings having the changes are as follows.

[0124] Detailed description of the embodiments Figure 3 shows various entities 300, 310, 320, 330, 340 according to an embodiment of the present disclosure, and the entities participate in the procedures proposed in the present disclosure in some form. Specifically, FIG. 3 shows a network analytics tracing entity 300 (also referred to as "analytics tracing NF"), a network analytics inference entity 310 (e.g., NWDAF inference, also referred to as "inference NF"), a network analytics training entity 320 (e.g., NWDAF training, also referred to as "training NF"), a network analytics consumer entity 330 (e.g., NWDAF consumer, also referred to as "consumer" or "consumer NF"), and a network analytics management entity 340.

[0125] Each entity 300, 310, 320, 330, 340 may include a processor or processing circuit (not shown) configured to perform, implement, or initiate the various operations of the respective entities 300, 310, 320, 330, 340 described herein. The processing circuit may comprise hardware and / or the processing circuit may be controlled by software. The hardware may comprise an analog circuit configuration or a digital circuit configuration, or both an analog circuit configuration and a digital circuit configuration. The digital circuit configuration may be an application specific integrated circuit (ASIC), a field programmable gateIt may include components such as an array (FPGA), a digital signal processor (DSP), or a multi-purpose processor. Each entity 300, 310, 320, 330, 340 may further include a memory circuit configuration, which stores one or more instructions, and the one or more instructions can be executed by a processor or a processing circuit, especially under the control of software. For example, the memory circuit configuration may include a non-transitory storage medium that stores executable software code, and when the executable software code is executed by a processor or a processing circuit, it causes various operations of each entity 300, 310, 320, 330, 340. In one embodiment, the processing circuit includes one or more processors and a non-transitory memory connected to the one or more processors. The non-transitory memory may carry executable program code, and when the executable program code is executed by one or more processors, it causes, implements, or initiates the operations or methods described herein for each entity 300, 310, 320, 330, 340.

[0126] Each of the entities 300, 310, 320, 330, and 340 may be part of the same mobile network, such as a 5G mobile network. The entities 300, 310, 320, 330, 340 may be further configured to interact with each other, especially via the mobile network of which they are a part. Specifically, the entities 300, 310, 320, 330, 340 may be configured to communicate with each other, that is, to exchange information by providing information to and / or receiving information from another entity. Any communication between two of the entities 300, 310, 320, 330, 340 may further include additional entities (including entities not described) within the mobile network, that is, the communication may be direct or indirect.

[0127] The network analytics tracing entity 300 is configured to obtain a display 301 having information for tracing one or more analytics outputs for an analytics ID and / or for initiating tracing of the analytics ID. For example, it may receive the display 301 from a consumer entity 330, an inference entity 310 (which may receive it from the consumer entity 330) from , or a training entity 320. Further, the tracing entity 300 may be configured using the display 301, for example, via a setting 341 by an administrative entity 340, to obtain it.

[0128] The tracing entity 300 is further configured to provide a rollback notification 302 related to the analytics ID if at least one output for the analytics ID is unstable and / or if the analytics ID is unstable. The rollback notification 302 may include at least one unstable analytics output for the analytics ID and / or the analytics ID and may be provided to the consumer entity 330, the inference entity 320, or the training entity 320. Further, it may include an inference rollback action 311 for at least one unstable analytics output for the analytics ID and / or for the analytics ID, in which case the rollback notification 302 is provided to the network analytics inference entity 310. It may also include a training rollback action 321 for at least one unstable analytics output for the analytics ID and / or for the analytics ID, in which case the rollback notification 302 is provided to the network analytics training entity 320.

[0129] The network analytics inference entity 310 is configured to obtain at least one analytics output for an analytics ID and / or an inference rollback action 311 for the analytics ID. For example, it may obtain the inference rollback action 311 by receiving a rollback notification 302 related to the analytics ID from the network analytics tracing entity 300. It may extract the inference rollback action 311 from the rollback notification 302 or determine it based on the rollback notification 302. Another possibility is, for example, that the inference entity 310 is configured with the inference rollback action 311 by the tracing entity 300 or the management entity 340.

[0130] Furthermore, the inference entity 310 is configured to execute the inference rollback action 311, and executing the inference rollback action 311 includes at least one of changing inference settings for at least one analytics output for the analytics ID and / or related to the analytics ID, determining and setting new inference settings for at least one analytics output for the analytics ID and / or related to the analytics ID, selecting a new analytics model for at least one analytics output for the analytics ID and / or for the analytics ID, and stopping the analytics model for at least one analytics output for the analytics ID and / or for the analytics ID. The rollback notification 302 may indicate which of the above should be done.

[0131] The network analytics training entity 320 is configured to obtain a training rollback action 321 for at least one analytics output regarding the analytics ID and / or for the analytics ID. For example, it may obtain the training rollback action 321 by receiving a rollback notification 302 related to the analytics ID from the network analytics tracing entity 300. It may extract the training rollback action 3 from the rollback notification 302 2 21 or determine it based on the rollback notification 302. Another possibility is, for example, that the training entity 320 is configured with the training rollback action 3 using the tracing entity 300 or the management entity 340 2 21.

[0132] Furthermore, the training entity 320 is configured to execute the training rollback action 321, and executing the training rollback action 321 includes at least one of changing the training settings regarding at least one analytics output for the analytics ID and / or for the analytics ID, selecting and setting new training settings regarding at least one analytics output for the analytics ID and / or for the analytics ID, retraining or reselecting the analytics model for at least one analytics output for the analytics ID and / or for the analytics ID, and stopping the analytics model for at least one analytics output for the analytics ID and / or for the analytics ID. The rollback notification 302 may indicate which of the above should be done.

[0133] The network data analytics consumer entity 330 is configured to provide a display 301 having information for initiating the tracing of one or more analytics outputs for an analytics ID and / or the tracing of the analytics ID to either the network analytics tracing entity 300 or the network analytics inference entity 310.

[0134] The network analytics management entity 340 is configured to, for example, in the tracing entity 300, perform settings 341 for initiating the tracing of one or more analytics outputs for an analytics ID and / or the tracing of the analytics ID. Further, the management entity 340 can be configured to, for example, in the tracing entity 300, set the analytics tracing information 403 for the analytics ID. Further, the management entity 340 is configured to, for example, in the tracing entity 300 or the consumer entity 330, perform settings 341 for initiating the collection of data regarding one or more quality displays regarding one or more analytics outputs for an analytics ID and / or regarding the analytics ID.

[0135] Figures 4 and 5 show exemplary detailed procedures proposed by the present disclosure as solutions to the problems mentioned in the overview section. The procedures include various entities 310, 320, 330 according to embodiments of the present disclosure, as already explained with respect to Figure 3 for example. The proposed solution is constructed in at least one of five stages. Figure 4 shows stages 1 and 4, while Figure 5 shows stages 2, 3, and 5. Thereby, · Stage 1 includes tracing analytics information, determining unstable analytics (output and / or ID), and triggering analytics rollback actions 311, 321. · Step 2 includes handling analytics (inference) rollback actions 311, 321 in the network analytics inference entity 310. · Step 3 includes handling analytics (training) rollback actions 311, 321 in the network analytics training entity 320. · Step 4 includes associating analytics consumption with the network status. · Step 5 includes actions of the analytics consumer entity 330 upon unstable analytics notification.

[0136] Steps 1 and 4 are essential in this exemplary procedure as shown in FIGS. 4 and 5. Steps 2 and 3 include alternative actions that can optionally be incorporated into the solution at Step 1. Step 5 is optional and can be triggered at Steps 1, 2, and 3.

[0137] As shown in FIG. 4, in stage 1, a new entity called the network analytics tracing entity 300 obtains an analytics ID tracing activation, for example, by receiving a display 301 (in particular, a synonym for the analytics ID is the analytics type or simply analytics) (step 0, 1). Based on this obtained analytics ID tracing activation, the tracing entity 300 creates analytics tracing information 403 (i.e., ATDS) associated with the analytics ID, for example, ATDS for the analytics ID, or ATDS of the analytics ID (step 4). The created ATDS can maintain and store a set of ATDS records over time. Therefore, the ATDS, in this case, is a set of histories of ATDS records having an association of the analytics ID with the analytics output, a display of the quality of use of analytics (ID and / or output) in the mobile network, settings in inference and / or training for the analytics (which can mean the analytics output or the analytics ID or the analytics type), and, optionally, the analytics rollback action and / or rollback status. Each ATDS record can define the network status when using the analytics ID within the mobile network and can include a tuple having a mapping at a given point in time of the analytics ID to at least one of the following parameters. · Timestamp of record creation. · One or more analytics output identifications and / or one or more analytics outputs. · Analytics ID grade information (AidGI) and / or unstable analytics ID information (UAiDI) for an analytics output instance from the analytics monitoring entity 400 (e.g., NWDAF or another NF), or a display of the quality of use of analytics (ID and / or output) in the mobile network, including NF feedback from the NF or the consumer entity 330. · AIC from the inference entity 310 (e.g., NWDAF containing inference NF, Analytics Logic Function (AnLF)). I、 And / or, the training entity 320 (e.g., NWDAF containing training NF or Model Training Logic Function (MTLF)) for analytics (which can mean analytics output or analytics ID or analytics type) provides an ATCI. · (Optionally) analytics rollback actions 311, 321. · (Optionally) the rollback status (or rollback status notification) is a set of information that explains the result of performing the analytics rollback actions 311, 321.

[0138] The activation of analytics ID tracing (e.g., creation of an ATDS associated with the analytics ID) can be obtained by the tracing entity 300 based on at least one of the following. · An NF (e.g., a NWDAF consumer or consumer entity 330 such as an NF, a NWDAF containing an Analytics Logic Function (AnLF), or a NWDAF containing a Model Training Logic Function (MTLF)) provides an indication 301 for activating analytics ID tracing to the tracing entity 300 (steps 0a - e). · A management entity 340 (e.g., OAM) provides a setting for activating analytics ID tracing to the tracing entity 300 (step 1).

[0139] Depending on how the activation of analytics ID tracing is obtained by the tracing entity 300, the tracing entity 300 may have to provide an indication of inference tracing activation 402 and / or an indication of training tracing activation 401 to the inference entity 310 and / or the training entity 320 respectively. For example, · When the tracing entity 300 receives an indication 301 from the consumer entity 330 to initiate analytics ID tracing (step 0a), or when it is configured to initiate tracing (step 1), the tracing entity 300 needs to indicate to the inference entity 310 and / or the training entity 320 that a tracing process and a rollback process related to analytics (ID and / or output) will be performed. Then, the inference entity 310 and / or the training entity 320 are triggered by an inference tracing initiation 402 display or a training tracing initiation 401 display to perform tracing and, if necessary, analytics rollback actions 311, 321 as described below. · When the tracing entity 300 receives an indication from the inference entity 310 to initiate analytics ID tracing (step 0d), the tracing entity 300 optionally needs to indicate to the training entity 320 that a tracing process rollback process related to analytics (ID and / or output) will be performed. · When the tracing entity 300 receives an indication from the training entity 320 to initiate analytics ID tracing (step 0e), the tracing entity 300 optionally needs to indicate to the inference entity 310 that a tracing process and a rollback process related to analytics (ID and / or output) will be performed.

[0140] The tracing entity 300 regularly obtains information for creating, constructing, and assembling ATDS records (step 5-9 ), which may include at least one of the following. · Analytics output generation (e.g., analytics output identification or the analytics output itself). · Analytics ID grade information (AidGI) and / or unstable Analytics ID information (UAiDI) and / or NF feedback regarding Analytics output. · AICI and / or ATCI for Analytics output and / or Analytics ID.

[0141] Tracing entity 300 creates, constructs, and assembles ATDS records based on regularly acquired information for creating ATDS records, includes this ATDS record in the ATDS for the Analytics ID, adds it, and stores it (step 11a).

[0142] Tracing entity 300 periodically checks the ATDS of the Analytics ID to determine if there is an unstable Analytics associated with the Analytics ID (step 11b). If the tracing entity 300 identifies an ATDS record with an unstable Analytics ID, it inserts this information (e.g., as a flag) ATDS into the record. A Identification of Unstable Analytics in the TDS Record for the determination of , the tracing entity 300 can perform any of the following processes. · Identify that the ATDS record for the Analytics ID contains feedback from the NF (e.g., NF feedback) that has an indication of a problem caused by one or more Analytics outputs for the Analytics ID. · Make a determination of unstable Analytics. There are various ways in which the tracing entity 300 can make a determination of unstable Analytics.

[0143] ■ One possible way is by verifying that the ATDS record has unstable Analytics ID information (UAiDI), where the unstable Analytics ID information indicates that the Analytics ID results in an unstable network status.

[0144] ■Another possible alternative is when the tracing entity 300 checks a subset of the ATDS records for the analytics ID and identifies that the grade information associated with the analytics ID grade information (AidGI) follows a trend indicating a degradation of the network status that could potentially result in an unstable network status. For example, one example of the check is that the tracing entity 300 applies a moving average to the grades of the last 10 ATDS records and, based on this check, detects a downward trend in the grades.

[0145] If the tracing entity 300 identifies unstable analytics (e.g., the presence of ATDS records associated with an unstable analytics ID being considered), it checks other ATDS records for the analytics ID (without the information of the unstable analytics ID) (step 11c) to determine the last known stable network state for the analytics. The last known stable network state for the analytics is the ATDS record having the most appropriate analytics inference setting information (AICI) and / or analytics training setting information (ATCI) for the analytics output and / or the analytics ID. There are various possible alternatives for determining the last known stable network state for the analytics. · The tracing entity 300 selects from the ATDS for the analytics the ATDS record that stores and has the best analytics grade value recorded in the analytics ID grade information (AidGI). · The tracing entity 300 selects the ATDS record having the most frequently used AICI and / or ATCI for the analytics ID. · The tracing entity 300 stores in the analytics ID grade information (AidGI) of the ATDS record for the analytics ID, calculates the average of the recorded grade values, and identifies the analytics output and / or the most frequently used AICI and / or ATCI for the analytics ID within the standard deviation from the calculated average of the grade values.

[0146] If the tracing entity 300 identifies the last known stable network state for the analytics (i.e., identifies the ATDS record with the most appropriate AICI and / or ATCI for the analytics ID), the tracing entity 300 determines whether analytics rollback actions 311, 312 should be triggered to revert, change settings and parameters related to the analytics ID used by, for example, the consumer entity 330 and / or the inference entity 310 and / or the training entity 320. Examples of identifying the need for analytics rollback actions 311, 321, and the identified analytics rollback actions 311, 321 (also simply referred to as "rollback actions") are described as follows. · Option 1: If the AICI and / or ATCI for the analytics ID of the ATDS record associated with the unstable analytics is the same as the AICI and / or ATCI for the analytics ID of the ATDS record identified as the last known stable network state for the analytics, since there is no alternative AICI and / or ATCI for the analytics ID to be used, the tracing entity 300 cannot perform a rollback. In this case, the tracing entity 300 can provide an unstable analytics notification 504 to the consumer entity 330 (step 13a in stage 2) and / or the inference entity 310 (step 12a in stage 2) and / or the training entity 320 (step 12b in stage 3). In this case, determining whether anything can be done to resolve the issue is delegated to the entity that received such an unstable analytics notification 504. · The AIC for the analytics ID of the ATDS record associated with the unstable analytics I and and / or ATCI is different from the AICI and / or ATCI for the analytics ID of the ATDS record identified as the last known stable network state for the analytics, the tracing entity 300 can decide to take the following actions. · Option 2: If the AICI is different information between the two compared ATDS records, provide an inference rollback notification 302 (which may include a rollback action 311) containing the AICI associated with the unstable analytics ID and / or the AICI associated with the last known stable network state for the analytics to the inference entity 310 (step 12a in stage 2). This action will trigger stage 2 in the proposed solution. · Option 3: If the ATCI is different information between the two compared ATDS records, provide a training rollback notification 302 (which may include a rollback action 321) containing the ATCI associated with the unstable analytics ID and / or the ATCI associated with the last known stable network state of the analytics to the training entity 320 (step 12b). This action will trigger stage 3 in the proposed solution. · Option 4: If both the AICI and the ATCI are different information between the two compared ATDS records, provide an inference rollback notification 302 / 311 containing the AICI and ATCI associated with the unstable analytics ID and / or the AICI and ATCI associated with the last known stable network state of the analytics to the inference entity 310 (step 12a in stage 2). This action will trigger stage 2 in the proposed solution. · Option 5: If both the AICI and the ATCI are different information between the two compared ATDS records, the tracing entity 300 can trigger stages 2 and 3 of the solution simultaneously. When the tracing entity 300 triggers stage 2, it provides an inference rollback notification 302 / 311 containing the AICI associated with the unstable analytics ID and / or the AICI associated with the last known stable network state of the analytics to the inference entity 310 (step 12a in stage 2). When the tracing entity 300 triggers stage 3, it provides a training rollback notification 302 / 321 containing the ATCI associated with the unstable analytics ID and / or the ATCI associated with the last known stable network state of the analytics to the training entity 320 (step 12b).

[0147] As shown in FIG. 5, in step 2, when the inference entity 310 obtains the analytics rollback action 311, the inference entity 310 verifies the information including the analytics rollback action 311. · When the information represents an unstable analytics notification 504, the inference entity 310 can perform the following actions.

[0148] ■ Provide the unstable analytics notification 504 to the consumer entity 330 of the analytics (ID and / or output) indicated in the unstable analytics notification 504 (step 13a). For example, the inference entity 310 may transfer the exact same unstable analytics notification 504 to the consumer entity 330, or the inference entity 310 may process the information from the obtained unstable analytics notification 504 and generate another unstable analytics notification 504 to send to one or more consumer entities 330. Examples of processing include removing information regarding AICI or ATCI from the unstable analytics notification 504, as well as converting this message into a tuple of analytics (ID and / or output) and a flag indicating the unstable analytics ID. · When the information represents an inference rollback notification 302, the inference entity 310 can perform the following actions.

[0149] ■If the information associated with the rollback action 311 (optionally) represents an unstable analytics notification 504, following the same process described for the unstable analytics notification 504, provide or transfer the unstable analytics notification 504 to the consumer entity 330 (step 13a). Also, if the processing of the unstable analytics notification 504 is performed by the inference entity 310, the processed unstable analytics notification 504 can also contain additional information, such as a timer indicating the estimated time interval for the rollback action 311 to be performed. This enables the consumer entity 330 to receive from the inference entity 310 (and / or the tracing entity 300) after this time period, a confirmation of the change in the analytics status (e.g., a change from an unstable analytics ID to a stable analytics ID if the rollback was successful, a confirmation of the unstable analytics ID).

[0150] ■Analyze the acquired information and implement changes in the analytics (id and / or output) related to the analytics rollback action 311 (step 14a). The following actions can be performed depending on the acquired information.

[0151] - (Option 2.1 / 5.1) If the analytics rollback action 311 includes two sets of information, both related to AICI, one set of information is related to the current AICI associated with the analytics (ID and / or output), and the second set of information is related to a new possible AICI. As a result, the inference entity 310 will replace the old set of AICI with the new set.

[0152] - (Option 2.2 / 5.2) If the analytics rollback action 311 includes a set of information related to AICI, one set of information is related to the current AICI associated with analytics (ID and / or output) considered as an unstable analytics ID, and as a result, the inference entity 310 can determine a new AICI and use this information for the generation of analytics (ID and / or output).

[0153] - (Option 2.3 / 5.3) If the analytics rollback action 311 includes a set of information related to AICI, one set of information is related to the new AICI associated with analytics (ID and / or output) considered as a stable analytics ID, and as a result, the inference entity 310 can replace its own local current settings for the analytics with the new AICI and use this information for the generation of analytics (ID and / or output).

[0154] - (Option 4.1) If the analytics rollback action 311 includes four sets of information, two sets of information are related to the current AICI and ATCI associated with analytics (ID and / or output), and the other two sets of information are related to the new AICI and ATCI associated with analytics (ID and / or output), the inference entity 310 can do the following.

[0155] 〇 Decide to replace only the old set of AICI with the new set and use it for analytics generation.

[0156] Decide to replace the old set of AICI with a new set and the old set of ATCI with a new set and use them for analytics generation. In this case, if the inference entity 310 does not have an ML model and / or a model within the ATCI, the inference entity 310 may discover and request the indicated ML model and / or model related to the ATCI from the training entity 320 (i.e., perform model reselection).

[0157] Decide to replace only the old set of ATCI with a new set and use it for analytics generation again. If the ML model and / or model within the new ATCI is not available in the inference entity 310, the inference entity 310 may need to perform model reselection.

[0158] - (Option 4.2) If the analytics rollback action 311 includes two sets of information and the two sets of information are related to the current AICI and ATCI associated with analytics (ID and / or output), the inference entity 310 can do the following.

[0159] Decide to determine only the new set of AICI and use it for analytics generation.

[0160] Decide to determine (e.g., reselect) a new ML model and / or model for analytics generation without informing any training entity 320 of the reason for reselection.

[0161] Decide to determine (e.g., reselect or request retraining) the new ML model and / or model to be used for analytics generation and provide the indication for reselection and / or the indication for retraining to the training entity 320 (Step 15a, Figure 4).

[0162] 〇 It is determined to notify the training entity 320 regarding problems in analytics (ID and / or output) (step 16a, Figure 4), in which case the inference entity 310 can send an unstable analytics notification 504 to the training entity 320.

[0163] When the inference entity 310 has finished executing the analytics rollback action 311, it can provide any of the following information. · Rollback status notification 501 to the tracing entity 300 (step 17a) · Analytics status notification 502 to the consumer entity 330 (e.g., when the execution of the analytics rollback action 311 is successful) (step 18a) · Confirmation 503 of an unstable analytics notification to the consumer entity 330 (e.g., when the execution of the analytics rollback action 311 fails) (step 18a) As shown in Figure 5, in stage 3, when the training entity 320 obtains the analytics rollback action 321, the training entity 320 verifies the information including the analytics rollback action 321 (step 15b). · If the information represents an unstable analytics notification 504, the training entity 320 can perform the following actions.

[0164] Provide and / or forward the unstable analytics notification 504 to the ML model associated with the analytics (ID and / or output) shown in the unstable analytics notification 504 and / or any other training entity 320 related to the training of the model or the model tuning process. This will enable all training entities 320 involved in the process of the ML model associated with the analytics and / or the training of the model or the model tuning process to recognize the problem. If a (first) training entity 320 provides the unstable analytics notification 504 to another training entity 320, the (first) training entity 320 may apply some further processing within the received unstable analytics notification 504 before sending it to the other training entity 320. For example, it can include information related to the shared training, such as partial ML models and / or model weights, parameters received in collaborative training, etc.

[0165] Provide and / or forward the unstable analytics notification 504 to one (or more) inference entities 310 that consumed the ML model and / or model related to the unstable analytics notification 504 (step 14b, Figure 4). For example, the analytics ID and / or output identification is included in the unstable analytics notification, and the training entity 320 identifies the list of inference entities 310 that requested, subscribed to, or were provided with the ML model and / or model associated with such analytics. · If the information represents a training rollback notification 302, the training entity 320 can perform the following actions.

[0166] ■(Option 3.1 / 5.1) If the analytics rollback action 321 includes two sets of information both related to the ATCI, where one set of information is related to the current (also referred to as old) ATCI associated with the analytics (ID and / or output), and the second set of information is related to the new possible ATCI, the training entity 320 can replace the old set of ATCI with the new set. The training entity 320 can optionally perform the following actions.

[0167] - Identify all consumers of the ML model and / or model associated with the current ATCI and provide the ML model and / or model associated with the new ATCI to these entities. This can include pushing, notifying, or sending the updated ML model and / or model to the inference entity 310 that consumes such a model, or during the training phase, sharing the changed ML model and / or model, including the reason for the change, to other training entities 320 that require the change, indicating that the current (pre-change) ML model and / or model used by the consumer is related to an unstable analytics ID. Including this reason will enable the consumers of the ML model and / or model to understand that this is a necessary change not only to optimize the generation of the analytics ID but also to bring it to a stable state.

[0168] - Optionally mark the ML model and / or model associated with the analytics (ID and / or) output) of the old acquired ATCI as stopped.

[0169] ■(Option 3.2 / 5.2) If the analytics rollback action 321 includes a set of information related to the ATCI, one set of information is related to the current ATCI associated with analytics (ID and / or output) considered to be an unstable analytics ID, and as a result, the training entity 320 can perform any of the following.

[0170] - Determine an ML model for analytics (ID and / or output) related to the current ATCI and / or a new ATCI for the model via training and / or model tuning, and optionally, re-identify all consumers of the ML model and / or model associated with the new ATCI, and provide the training rollback information 302 to these entities (as described above) along with the new updated ML model and / or model.

[0171] - Optionally mark the ML model and / or model associated with the analytics (ID and / or output) of the old acquired ATCI as stopped.

[0172] ■(Option 3.3 / 5.3) If the analytics rollback action 321 includes a set of information related to the ATCI, one set of information is related to a new ATCI associated with analytics (ID and / or output) considered to be a stable analytics ID, and as a result, the training entity 320 can replace its own local current settings for the analytics with the new ATCI for the ML model and / or model for the analytics (ID and / or output), and optionally, re-identify all consumers of the modified ML model and / or model, and provide the training rollback information 302 to these entities (as described above) along with the new updated ML model and / or model.

[0173] When determining a new ATCI for an ML model and / or model for analytics (ID and / or output), or when the ML model and / or model requires shared training, the training entity 320 may need to interact with other training entities 320. In this case, the training entity 320 identifies the additional training entities 320 required and sends any of the following information that triggers the training entities 320 to cooperate with each other to determine a new ML model and / or model for analytics. - Display of reasons for re-selection related to unstable analytics ID - Display of reasons for retraining related to unstable analytics ID

[0174] ■ The training entity 320 may not be able to determine a new ATCI for an ML model and / or model for analytics (ID and / or output) associated with the obtained analytics rollback action 321, or replace the current ATCI with a new ATCI. In this case, if the training entity 320 has previously provided an unstable analytics notification 504 (step 16b), it can provide the inference entity 310 with confirmation of the unstable analytics notification 504, or provide the inference entity 310, which is a consumer of the ML model and / or model related to the ATCI, with the unstable analytics notification 504.

[0175] When the training entity 320 has finished executing the analytics rollback action 321, it can provide any of the following. · Confirmation 503 of the unstable analytics notification, or training rollback information 302, or analytics status notification 504 (step 16b) for the inference entity 310 · Rollback status notification 501 (step 17b) for the tracing entity 300.

[0176] When the inference entity 310 receives an unstable analytics notification 504 (step 14b, FIG. 4) from the training entity 320 and transfers such information to the consumer entity 330, when the inference entity 310 receives a confirmation 503 of the unstable analytics notification or training rollback information 302 or an analytics status notification 502 from the training entity 320, the inference entity 310 may optionally provide the consumer entity 330 with the analytics status notification 502 or the unstable analytics notification confirmation of 503 or transfer it (step 18b, FIG. 4).

[0177] As shown in FIG. 4, in stage 4, A new entity called the network analytics monitoring entity 400 (AMon entity) is Optionally obtain analytics performance information (API) about a particular consumer of analytics for a particular consumed analytics (ID and / or output) from other entities (e.g., the tracing entity 300, the consumer entity 330) (step 5a or 5b )。 The goal of the API is to capture KPIs and / or metrics that are not related as input data for the generation of analytics (ID and / or output), but are related to identifying how the consumed analytics change the network status. Based on the API, the monitoring entity 400 identifies the sources of data collection for the KPIs and / or metrics related to the API and initiates data collection from these data sources.

[0178] The API can be obtained using any of the following possibilities. · Settings 341 from the network analytics management entity 340 · Settings 341 from the management entity 340 and from another entity including a particular analytics (ID and / or output), and / or from the analytics consumer entity 330 to which the configured API should be associatedreceived Message gi and Combination · Messages received from other entities (e.g., tracing entity 300, training entity 320, inference entity 310) and / or consumer entity 330, including an API, specific analytics (ID and / or output).

[0179] Based on the collected data related to the API, the monitoring entity 400 monitors the effect of the analytics (ID and / or output) on the change in network status after consumption of the analytics ID. This monitoring process can result in the generation of two pieces of information. · A set of information derived from the calculation of a grade related to the effect of the analytics on the change in network status after consumption of the analytics, the analytics ID grade information (AidGI). · A set of information derived from the calculation of a grade related to the effect of the analytics on the change in network status after consumption of the analytics, crossing a threshold (e.g., a threshold set by an operator) identifying this grade as leading to an unstable network status for the analytics, and identifying it as a deviation, the unstable analytics ID information (UAiDI) The monitoring entity 400 provides any of the following information to other entities (e.g., the tracing entity 300). · Analytics ID grade information (AidGI) · Unstable analytics ID information (UAiDI)

[0180] In Figure 5, at stage 5, when the consumer entity 330 (e.g., NF consumer) receives an unstable analytics notification 504 (steps 12, 13a, 18a, 13b, 17b), the consumer entity 330 can take various actions. For example, · By canceling or unsubscribing from analytics in the inference entity 310, consumption of the analytics associated with the received unstable analytics notification 504 is stopped. When canceling or unsubscribing, the consumer entity 330 can indicate that it wishes to be notified if the analytics are reconsidered to be a stable analytics ID, thereby enabling restart of the previous subscription to the analytics ID (step 19a in figure 5 ). · Continue consuming the analytics and not place much emphasis on the information of such analytics for its internal decision-making · Continue consuming the analytics and request to be notified if the analytics are reconsidered to be a stable analytics ID (step 19b in figure 5 ).

[0181] When the consumer entity 330 receives the analytics status notification 502, the consumer entity 330 can take various actions. For example, · If the consumer entity 330 stops consuming the analytics, it can resubscribe to the analytics. In this case, the consumer entity 330 indicates to the inference entity 310 that there was a previous subscription to the analytics ID and that this previous subscription should be restarted, optionally for a specific analytics ID.

[0182] In FIGS. 4 and 5, in common for steps 1, 2, 3, and 5, when the tracing entity 300 identifies the need for analytics rollback actions 311, 321 and triggers step 2 and / or step 3, the tracing entity 330 may obtain a rollback status notification 501 from the inference entity 310 (step 17a) and / or the training entity 320 (step 17b). The rollback status notification 501 may represent a conclusion regarding the execution of the analytics rollback actions 311, 321.

[0183] When the tracing entity 300 detects the end of the analytics rollback actions 311, 321, the tracing entity 300 may optionally provide a confirmation of the analytics status notification 502 or the unstable analytics notification 504 to the entity that provided the indication for the start of the analytics ID tracing. The difference between the first unstable analytics notification 502 (steps 12, 13a, 13b) that can be provided by the tracing entity 330 and the confirmation 503 of the unstable analytics notification, i.e., the second unstable analytics notification (steps 18a, 17b), is that the first unstable analytics notification 502 indicates that the analytics rollback actions 311, 321 are being performed, while the second unstable analytics notification 503 indicates that the analytics rollback actions 311, 321 may be successfully performed, for example, in that the rollback, reset, or repair of the analytics (ID and / or output) could not be performed within at least a short time period (e.g., below the minute level or within a few minutes).

[0184] The benefits of this exemplary embodiment shown in FIGS. 4 and 5 include the following. ·The (new) consumer entity 330 (especially NF) of one or more analytics IDs consumes analytics output for those analytics IDs with a guarantee to continue consuming in order to provide a stable network status (even if any NF sends feedback indicating problems with previous analytics output for the analytics ID). ·There is no interruption or disruption to the criteria used by an entity (NF), such as the consumer entity 330, for decision-making (e.g., by removing analytics output that is output as a reference when an NF sends feedback on a problem to the NWDAF, such as the inference entity 310). ·There is a guarantee that the analytics ID usage degradation system KPI (i.e., resulting in an unstable network status) becomes visible and traceable to the inference entity 310 and / or the training entity 320 (e.g., NWDAF with inference and training capabilities without delay caused by data collection). ) 。 ·The mobile operator can automatically revert the use of unstable analytics IDs (i.e., analytics IDs that result in an unstable network status) while the inference entity 310 and / or the training entity (e.g., one or more NWDAFs) are making improvements or fixing unstable analytics IDs or one or more unstable analytics outputs for the analytics ID. ·There is no need for manual triggering of settings in the inference entity 310 or the training entity 320, specifically the NWDAF, nor is there a need for logarithmic checks in the MP. ·The chain / tree of the training entity 320 (e.g., NWDAF with training capabilities) that shares the ML model associated with the unstable analytics ID enables such relationships to be automatically recognized. ·The ML model designer can use this information to make decisions regarding the next development of the ML model.

[0185] Furthermore, in the present disclosure, two possible alternatives for the proposed solutions and the implementation of the described entities according to the embodiments of the present disclosure are described, all based on entities defined in the 3GPP 5G architecture in TS 23.501 R17 and enhanced network analytics defined in 3GPP TS 23.288 R17. The implementations are listed below.

[0186] The first alternative is shown in FIG. 6 and relates to tracing and rollback support embedded in the existing NWDAF service. In this first alternative, the tracing entity 300, the inference entity 310, and the monitoring entity 400 are embedded within and / or hosted by an NWDAF having inference capabilities (i.e., Analytics Logic Function (AnLF ) -NWDAF(AnLF) ) . In this case, the existing service of inference of the NWDAF (inference entity 310) is extended to support the tracing and rollback capabilities of the tracing entity 300 and the monitoring entity 400. The extended services are as follows. · Nnwdaf_AnalyticsSubscription service · Nnwdaf_AnalyticsInfo service Furthermore 、 The training capability, i.e., an NWDAF having a Model Training Logic Function (MTLF), which may be referred to as NWDAF(MTLF) (i.e., training entity 320), is enhanced to support interactions for enabling tracing and rollback actions 311, 321 when interacting with the NWDAF(AnLF) enhanced by / hosted by the tracing entity 300 and the monitoring entity 400. Also in this case, the services of the NWDAF(AnLF) (e.g., training entity 320) are enhanced to support the tracing and rollback functionality proposed in the present disclosure. The enhanced services are as follows. · Nnwdaf_MLModelProvision service The second alternative is shown in FIG. 7 and relates to tracing and rollback support as a new NWDAF service. In this second alternative, illustrated in FIG. 6, the tracing entity 300 and the monitoring entity 400 are embedded and hosted by the NWDAF, and individual new services are defined to manifest the functionality of the tracing entity 300 and the monitoring entity 400 to support the tracing and rollback proposed in this disclosure. The new service is defined as follows. · Nnwdaf_AnalyticsTracing service The service of the NWDAF with inference capabilities, i.e., NWDAF(AnLF), is also extended to support the principles proposed in this disclosure. The services to be extended are as follows. · Nnwdaf_AnalyticsSubscription service · Nnwdaf_AnalyticsInfo service The service of the NWDAF with training capabilities, i.e., NWDAF(MTLF), is also enhanced to support interactions to enable tracing and rollback actions 311, 321 when interacting with the NWDAF that supports the functionality of the tracing entity 300 and the monitoring entity 400. The services to be enhanced are as follows. · Nnwdaf_MLModelProvision service The following description provides details of the service extensions for each of the alternatives, as well as possible procedures based on these alternatives that reflect stages 1 - 5 defined in this disclosure with respect to FIGS. 4 and 5.

[0187] Second alternative: A new interface is manifested by the NWDAF only for the analytics tracing entity 300 capabilities and the analytics monitoring entity 400 capabilities

[0188] (1) Analytics ID Tracing Activation: In this case, the new service needs to be defined in NWDAF to enable the consumer of this service to indicate that Analytics ID tracing needs to be activated within the system. The new service, which can be called Nnwdaf_AnalyticsTracing, examples of its operations and parameters are defined as follows. · Nnwdaf_AnalyticsTracing_Subscribe:

[0189] ■ Input Parameters: - Identification of the consumer entity 330 of the Analytics to be traced, for example, this identification can be the NF ID or the notification correlation ID of the consumer of the Analytics ID (i.e., the consumer who called Nnwdaf_AnalyticsSubscription_Subscribe or Nnwdaf_AnalyticsInfo_Request). - Notification correlation ID, which enables the consumer entity 330 of Nnwdaf_AnalyticsTracing_Subscribe to identify its own subscription to Analytics tracing. - Identification of one or more Analytics ID subscriptions with associated flags having start / stop values, which means that the NWDAF with the tracing entity 300 functionality or capability always starts (and / or stops depending on the flag value) the tracing of the indicated one or more subscriptions to the Analytics ID when it receives a call for Nnwdaf_AnalyticsTracing_Subscribe. - One or more analytics IDs with associated flags having start / stop values, which means that the NWDAF having the tracing entity 300 functionality or capability will always start (and / or stop depending on the flag value) the tracing of the indicated one or more analytics IDs when receiving a call for Nnwdaf_AnalyticsTracing_Subscribe. - Further 、 Information related to the API can also be included in the call to Nnwdaf_AnalyticsTracing_Subscribe. Examples of such information are any of those listed below.

[0190] 〇 One possible alternative is to reuse the fields already defined for the Nnwdaf_DataManagement service defined in clause 7.4.2 of TS 23.288 (V17.0.0.1), not for defining API information, but for including those fields in the subscription / request. These fields can be any of service operation, data specification, formatting instructions, processing instructions, NF (or NF configured) ID, ADRF information as defined in clause 6.2.6.1 of TS 23.288 R17 (V17.0.01).

[0191] □ Service operation: Define the service operation to be used by the NWDAF, DCCF, MFAF, or ADRF to request data (e.g., Namf_EventExposure_Subscribe or OAM subscribe) □ Data specification: Define any of event ID, event report target and event filter information as defined in clause 4.15.1 of TS 23.502 [3], and / or identification of information to be retrieved from OAM, area of interest for API data collection. □ Formatting instruction: The parameters defined in clause 4.15.1 of TS 23.502[3] for event report information are also part of the possible formatting instructions and processing instructions, and additionally 、 The following parameters may be included, namely, periodic bulk data notification, time window, notification event clipping, processing rules. □ NF (or NF-configured) ID: Defines a specific NF or NF set that is the source for API information defined in the data specification.

[0192] 〇 Another possible alternative is to include a new set of API parameters, which may be any of the following.

[0193] □ Area of interest for API data collection □ NF (or NF-configured) ID: Defines a specific NF or NF set that is the source for API information defined in the data specification. □ List of KPIs and / or metrics to be collected □ NF (or NF-configured) ID: Defines a specific NF or NF set that is the source for API information defined in the data specification.

[0194] ■ Output parameters: Subscription correlation ID, confirmation if activation was possible. · Nnwdaf_AnalyticsTracing_Notify:

[0195] ■ Input parameters: - Notification correlation ID, as a result, the consumer of Nnwdaf_AnalyticsTracing_Notify can identify a specific subscription to trace the analytics ID. - Unstable analytics notifications, and / or inference rollback notifications, and / or training rollback notifications :Notification correlation information when an entity receiving a notification from service operation Nnwdaf_AnalyticsTracing_Notify needs to send a rollback status notification to an NWDAF with analytics tracing capabilities

[0196] ■ Output parameter: None · Nnwdaf_AnalyticsTracing_RollbackStatusNotify

[0197] ■ Input parameters: Subscription correlation ID and / or notification correlation ID, and / or rollback status notification

[0198] ■ Output parameter: None

[0199] (2) Extension in the NWDAF service to enable an NWDAF instance using only analytics monitoring entity capabilities: In this case, the new service needs to be defined in the NWDAF so that a consumer entity 300 of this service can indicate an API for analytics IDs and obtain AiDGI and / or UAidI. Examples of this new service, operations, and parameters are defined as follows. · Nnwdaf_AnalyticsMonitoring_Subscribe:

[0200] ■ Input parameters: - Notification correlation ID, which enables the consumer of Nnwdaf_AnalyticsMonitoring_Subscribe to identify its own subscription to analytics-related APIs. - Analytics identification (e.g., analytics ID) - API information, and the same possible alternatives for the API embodiments described in service Nnwdaf_AnalyticsTracing_Subscribe are also applicable in this case.

[0201] ■ Output parameter: Subscription correlation ID, confirmation if startup monitoring was possible. · Nnwdaf_AnalyticsMonitoring_Notify:

[0202] ■ Input parameters: - Notification correlation ID, as a result, the consumer of Nnwdaf_AnalyticsMonitoring_Notify can identify a specific subscription to monitoring information regarding the analytics ID. - AidGI and / or UAidI notifications

[0203] ■ Output parameter: None Furthermore, the changes in NWDAF (AnLF) to support the embodiments of the inference entity 310 in stages 1, 2, 3, 4, and 5, which are common to the first alternative and the second alternative, are as follows.

[0204] (1) Settings for analytics ID tracing startup: In this case, NWDAF is set using analytics ID tracing startup. The settings for analytics ID tracing startup include any of the following examples. · A tuple having an analytics ID and a flag with a start / stop value, which means that the NWDAF having the analytics tracing entity functionality or capability will always start tracing the analytics ID when it receives a message related to the analytics ID (e.g., a subscription to the analytics ID). · An analytics ID, a flag having start / stop values, and a tuple having specific trigger conditions, which means that the NWDAF having analytics tracing entity functionality or capabilities starts tracing of the analytics ID only when the specific trigger condition exists within the received message related to the analytics ID (e.g., a subscription to the analytics ID). Any of the examples listed below can be a specific trigger condition. · Identification of a consumer (e.g., NF type) · Identification of the target of analytics (e.g., UE, group of UEs) · Identification of the area of interest (e.g., list of TAs or cells) · Identification of a network slice (e.g., S-NSSAI) · Identification of an application (e.g., application ID) · Identification of a data network (e.g., DNN or DNAI)

[0205] (2) Extension of the NWDAF subscribe / request service operation for implementing a subscription with tracing activation: · Display of analytics ID tracing activation: In this case, the display of analytics ID tracing activation is implemented as part of the existing NWDAF service operation for subscribing (or requesting) to the analytics ID. · Analytics ID tracing activation: Any of the examples listed below are new parameters used for analytics ID tracing activation.

[0206] ■ Option A: An input parameter in the NWDAF service indicating to return unstable analytics notifications without performing an analytics rollback action. For example, this would enable an analytics consumer to decide to stop subscribing to the analytics ID manifested by the NWDAF, or to continue consuming it but place less importance on this information in its internal decision-making.

[0207] ■ Option B: An input parameter in the NWDAF service indicating to perform an analytics rollback action if possible, optionally with a notification if the rollback occurs

[0208] ■ Option C: An input parameter for the NWDAF for performing a rollback process if possible, optionally with an API of interest to be used for the consumer of the analytics ID

[0209] (3) Extension of the NWDAF notification service for implementing unstable analytics notifications and / or analytics status notifications and / or confirmation of unstable analytics notifications: · The input parameters of the Nnwdaf_AnalyticsSubscription_Notify service operation can be extended by flags that can be set to represent unstable analytics notifications and / or analytics status notifications and / or confirmation of unstable analytics notifications. This flag can be called "Analytics Status" and can have the following values, namely, unstable, stable, and unstable and confirmed. · The same type of extension can also be provided for the NWDAF AnalyticsInfo_Request response, provided that the output parameters of this service operation are extended by the flags discussed above.

[0210] (4) Extension of the NWDAF notification service to implement restart : · The output parameters of the Nnwdaf_AnalyticsSubscription_Notify service operation can be extended to enable the NWDAF to indicate whether the NF consumer desires to be notified regarding changes in "Analytics Status" even after the NF consumer unsubscribes from the analytics ID after receiving "Analytics Status = Unstable" or "Analytics Status = Confirmed Unstable". The new output parameters of the Nnwdaf_AnalyticsSubscription_Notify service operation can be as follows.

[0211] ■ A flag that enables the NF consumer to initiate a change in status notification: for example, true or false "Status Change Notification" with a value of

[0212] ■ (Optional) A notification target address (+ notification correlation ID) that indicates to the NWDAF the address to be used to send the next Nnwdaf_AnalyticsSubscription_Notify when the analytics status changes. In this case, this next Nnwdaf_AnalyticsSubscription_Notify should include, as input parameters, "Analytics Status = Stable" and the notification correlation ID provided by the NF consumer in the output parameters of the Nnwdaf_AnalyticsSubscription_Notify. changing Furthermore, the NF consumer can analytics ID be enabled to unsubscribe.

[0213] (5) Extension of the NWDAF subscribe (or request) service to implement restart of the subscription when analytics changes its status from unstable to stable: · This is another option for implementing the restart of a subscription, in contrast to the extension highlighted in (5). · The input parameters of Nnwdaf_AnalyticsSubscription_Subscribe (and / or Nnwdaf_AnalyticsInfo_Request) are extended to include new parameters:

[0214] ■ A subscription correlation ID indicating that the previous subscription should be restarted

[0215] ■ A list of analytics IDs with an indication for restart. Such an indication can be, for example, a flag called "restart". · Since this is a restart of analytics IDs, there is no need to include additional required or optional parameters in the subscription, such as the target of the analytics report, the notification target address (+ notification correlation ID), the analytics report parameters, the analytics target period, etc.

[0216] (6) Extension of the NWDAF non - subscribe service to enable the NF consumer to perform a subscription restart when analytics changes its status from unstable to stable: · The input parameters of Nnwdaf_AnalyticsSubscription_Unsubscribe are extended to include new parameters:

[0217] ■ A flag, for example called "analytics status change notification", indicating that the consumer entity 330 wishes to be notified regarding changes in the status of analytics

[0218] ■ A list of analytics IDs indicating that the NF consumer is interested in having notifications regarding that status change

[0219] Notification target address (+ notification correlation ID) that enables the NWDAF to notify consumers regarding a change in analytics status from an unstable analytics ID to a stable analytics ID. · Also, an extension in the Nnwdaf_AnalyticsSubscription_Notify service operation needs to be made. In this case, the purpose is to indicate to NF consumers that have unsubscribed from the analytics ID that since the status of the analytics ID has changed from an unstable analytics ID to a stable analytics ID, this NF consumer should be able to restart the subscription to the analytics ID, by using the same existing service Nnwdaf_AnalyticsSubscription_Notify. In this case, when the status of the analytics ID shown in the Nnwdaf_AnalyticsSubscription_Unsubscribe service operation changes, the NWDAF will use Nnwdaf_AnalyticsSubscription_Notify with the following parameter extensions:

[0220] ■ Notification correlation ID as a required parameter in the case of notification of a change in analytics ID status

[0221] ■ List of analytics IDs with the changed status.

[0222] (7) Extension of the NWDAF subscribe (or request) service to implement the definition of analytics performance information to be monitored: · The input parameters of Nnwdaf_AnalyticsSubscription_Subscribe (and / or Nnwdaf_AnalyticsInfo_Request) are extended to include new parameters:

[0223] ■One possible alternative is to reuse the fields already defined for the Nnwdaf_DataManagement service defined in clause 7.4.2 of TS 23.288 (V17.0.0.1), not for defining API information, but for including those fields within subscriptions / requests. These fields can be any of service operation, data specification, formatting instructions, processing instructions, NF (or NF - configured) ID, ADRF information as defined in clause 6.2.6.1 of TS 23.288 R17 (V17.0.01). - Service operation: Define the service operation to be used by NWDAF, DCCF, MFAF, or ADRF to request data (e.g., Namf_EventExposure_Subscribe or OAM subscribe). - Data specification: Define any of event ID, event report target and event filter information as defined in clause 4.15.1 of TS 23.502 [3], and / or identification of information to be retrieved from OAM, area of interest for API data collection. - Formatting instructions: The parameters defined in clause 4.15.1 of TS 23.502 [3] for event report information are also part of possible formatting instructions and processing instructions, and additionally 、 the following parameters may be included: periodic bulk data notification, time window, notification event clustering, processing rules. - NF (or NF - configured) ID: Define a specific NF or NF set that is the source for the API information defined in the data specification.

[0224] ■Another possible alternative may include a new set of API parameters, any of the following: - Area of interest for API data collection - NF (or NF-configured) ID: Defines a specific NF or NF set that is the source for API information defined in the data specification - List of KPIs and / or metrics to be collected - NF (or NF-configured) ID: Defines a specific NF or NF set that is the source for API information defined in the data specification

[0225] (8) Extension of NWDAF service operation to support a NWDAF that receives an indication to start inference tracing: · One alternative is that the NWDAF is enhanced by a new service operation called Nnwdaf_AnalyticsSubscription_AnalyticsTraceSubscription. This service enables the NWDAF to generate and provide AICIs for the NWDAF using the analytics tracing entity capabilities.

[0226] ■ Input parameters: (any of those listed below) - Identification of the consumer of the analytics to be traced, e.g., this identification can be the NF ID or notification correlation ID of the consumer of the analytics ID (i.e., the consumer who called Nnwdaf_AnalyticsSubscription_Subscribe or Nnwdaf_AnalyticsInfo_Request). - Identification of the analytics ID subscription - Identification of the analytics ID - Notification correlation ID of the consumer for starting tracing in the NWDAF, which enables the consumer of Nnwdaf_AnalyticsSubscription_ActivateTrace to identify its own subscription to analytics tracing information in the NWDAF.

[0227] ■ Output parameter: Subscription correlation ID. ·Nnwdaf_AnalyticsSubscription_AnalyticsTraceNotify:

[0228] ■Input parameter (any of the following): - Notification correlation ID, identification of analytics ID subscription, AICI Changes in NWDAF (MTLF) to support the implementation of trained NFs in stages 1, 2, and 3:

[0229] (1) Common to all possible extensions discussed in this part of the implementation: · A potential alternative for representing a set of information about the display of re-selection or re-training is to use a flag such as "model tracing status" that can be set to "re-train" or "re-select". Furthermore 、 The set of information from both displays may also include the reason for "re-train" or "re-select". This can be implemented as another parameter, for example, a flag set to "analytics stable" or "analytics unstable".

[0230] (2) Possible alternatives for the extension of NWDAF (MTLF) to receive unstable analytics notifications and / or training rollback notifications from NWDAF (AnLF) or a stand-alone NWDAF with an analytics tracing entity are as follows: · Extension of the service provided by NWDAF (MTLF). The new service will be the Nnwdaf_MLModelTracing service. Possible operations are as follows:

[0231] ■GetTracingNotifications (Request / Response): When a consumer can include in the input parameters unstable analytics notifications and / or training rollback notifications for NWDAF (MTLF), along with additional information (any of the following), namely, analytics ID, analytics stage (inference and / or training). 。 · Extension of services provided by NWDAF (AnLF) or a stand-alone NWDAF having an analytics tracing entity 300, provided that NWDAF (MTLF) subscribes to receive unstable analytics notifications and / or training rollback notifications for a given ML model. The service can be Nnwdaf_AnaltyicsTracing, and the operation can be related to subscribe / notification, provided that the operation and associated parameters can be as follows:

[0232] ■SubscribeTracingInfo (Subscribe operation), - Input parameters (any of the following): analytics ID, analytics stage (inference and / or training), ML model and / or model identification, notification address (to enable NWDAF (MTLF) to receive notifications), notification correlation ID. Also, the input parameters can specify which particular type of notification the consumer is interested in receiving: for example, analytics notifications, training rollback notifications. If a particular type of notification is not indicated in the subscription, the consumer will receive any type of notification related to the indicated ML model and / or the model indicated in the subscription. - Output parameter: subscription correlation ID.

[0233] ■NotifyTracingInfo (Notification operation) - Input parameters: notification correlation ID (which enables the consumer to know which ML model and / or models this notification is associated with), and notifications, e.g., unstable analytics notifications and / or training rollback notifications

[0234] (3) Possible alternatives for the extension of NWDAF(MTLF) to receive indications for reselection and / or indications for retraining from NWDAF(AnLF) are as follows: · One alternative is to define an extension of the existing service Nnwdaf_MLModelProvision_Subscribe to include new input parameters, namely, a subscription correlation ID, a list of analytics IDs, and indications for reselection and / or indications for retraining for each of the analytics IDs. Since this is a retraining or reselection of the model for an existing subscription to an ML model for an analytics ID, there is no need to include additional required or optional parameters in the subscription as defined in clause 7.5.2 in TS 23.288 (V17.0.1). · Another alternative is to also use the service Nnwdaf_MLModelTacing_GetTracingNotifications (request / response) described above to enable NWDAF(AnLF) to send indications for reselection and / or indications for retraining to NWDAF(MTLF).

[0235] (4) NWDAF(MTLF) provides confirmation or training rollback information or analytics status notifications to NWDAF(AnLF) or another NWDAF(MTLF), an extension in the Nnwdaf_MLModelProvision service from NWDAF(MTLF), which is related to the ML model and / or models previously consumed by such NWDAF(AnLF). of ​: ·One possible extension is to provide unstable analytics notifications and / or unstable analytics confirmation or training rollback information or analytics status notifications to consumers of the Nnwdaf_MLModelProvision_Subscribe service, NWDAF (AnLF) and / or other NWDAF (MTLF), in the context of ML models for analytics and / or subscriptions to models. of This can be achieved by enabling the indication, along with the notification, of the intention to receive confirmation or training rollback information or analytics status notifications. In this case, the Nnwdaf_MLModelProvision_Subscribe service input parameters can be extended by any of the following parameters:

[0236] ■ Current ML model and / or model identification (i.e., the model currently in use) ■ Analytics identification associated with the ML model and / or model being used (e.g., analytics ID) ■ A flag indicating that the current model is associated with unstable analytics IDs ■ A flag indicating that the confirmation of the current model is associated with unstable analytics IDs ■ A flag indicating that a new model is required ■ A flag indicating that the confirmation of the current model is stable ■ Rollback training information (e.g., information including rollback training information) ·Another alternative is to create an individual service.

[0237] (5)Possible alternatives for embodiments for providing training rollback information to the inference NF and / or other training NFs that consume the ML model and / or model are as follows: ·Extension of the service operation for notifying regarding the subscribed / requested model to include any of the following parameters as additional information related to the training rollback information:

[0238] ■ Identification of the current ML model being used and / or the model ■ Analytics ID ■ Identification of the changed ML model and / or the model ■ Changed ML model and / or model information (e.g., any of the following information, namely, among others, gradient, algorithm, weight, model architecture) ■ Flag indicating that the current model is related to unstable analytics IDs ■ ML model with changes and / or model script and / or file and / or settings

[0239] (6) Extension of NWDAF service operations to support NWDAF receiving a display of the start of training tracing: · One alternative is that the NWDAF is enhanced by a new service operation called Nnwdaf_MLModelProvision_AnalyticsTraceSubscription. This service enables the NWDAF to generate and provide AICIs for the NWDAF using the analytics tracing entity capabilities.

[0240] ■ Input parameters: (any of those listed below): - Identification of the ML model and / or the consumer of the model for the analytics to be traced (e.g., analytics ID), for example, this identification can be the NF ID or the notification correlation ID of the consumer of the ML model and / or the model supplied for the analytics ID (i.e., the consumer who called Nnwdaf_MLModelProvision_Subscribe). - Identification of the ML model and / or the model subscription - Identification of the analytics ID - Identification of the ML model and / or the model - The consumer notification correlation ID for tracing activation in NWDAF, which enables the consumer of Nnwdaf_MLModelProvision_AnalyticsTraceSubscription to identify their subscription to analytics tracing information in NWDAF.

[0241] ■ Output parameter: Subscription correlation ID · Nnwdaf_MLModelProvision_AnalyticsTraceNotify:

[0242] ■ Input parameter (any of the following): - Notification correlation ID, ML model and / or identification of the model for analytics ID subscription, ATCI

[0243] ■ Output parameter: None

[0244] Figure 8 shows a method 800 according to an embodiment of the present disclosure. The method 800 is for the network analytics tracing entity 300 as described above and can be performed by the tracing entity 300. The method 800 includes step 801 of obtaining a display 301 having information for initiating tracing of one or more analytics outputs for an analytics ID and / or tracing of the analytics ID. Further, Method 800 is step 802 of providing a rollback notification 302 related to the analytics ID if at least one output for the analytics ID is unstable and / or if the analytics ID is unstable including .

[0245] The rollback notification 302 includes one or more of the following: · At least one unstable analytics output and / or analytics ID for an analytics ID, where the rollback notification 302 is provided to the network analytics consumer entity 330, the network analytics inference entity 310, or the network analytics training entity 320, at least one unstable analytics output and / or analytics ID for an analytics ID · An inference rollback action 311 for at least one unstable analytics output for an analytics ID and / or for an analytics ID, where the rollback notification 302 is provided to the network analytics inference entity 310, the inference rollback action 311 · A training rollback action 321 for at least one unstable analytics output for an analytics ID and / or for an analytics ID, where the rollback notification 302 is provided to the network analytics training entity 320, the training rollback action 321 Figure 9 shows a method 900 according to an embodiment of the present disclosure. The method 900 is for the network analytics inference entity 310 and can be performed by the inference entity 310. The method 900 includes a step 901 of obtaining an inference rollback action 311 for an analytics ID and / or for at least one analytics output for the analytics ID. Further, the method 900 includes a step 902 of executing the inference rollback action 311.

[0246] Executing the inference rollback action 311 includes at least one of the following: · Changing inference settings regarding at least one analytics output for an analytics ID and / or regarding the analytics ID Determining and setting at least one output regarding an analytics ID and / or a new inference setting regarding the analytics ID Selecting a new analytics model for at least one analytics output regarding an analytics ID and / or for the analytics ID Halting an analytics model for at least one analytics output regarding an analytics ID and / or for the analytics ID

[0247] FIG. 10 shows a method 1000 according to an embodiment of the present disclosure. The method 1000 is for a network analytics training entity 320 and can be performed by the training entity 320. The method 1000 includes a step 1001 of obtaining a training rollback action 321 for at least one analytics output regarding an analytics ID and / or for the analytics ID. Further, the method 1000 includes a step 1002 of executing the training rollback action 321.

[0248] Executing the training rollback action 321 in 1002 includes at least one of the following: Changing a training setting for at least one analytics output regarding an analytics ID and / or for the analytics ID Selecting and setting a new training setting for at least one analytics output regarding an analytics ID and / or for the analytics ID Retraining or reselecting an analytics model for at least one analytics output regarding an analytics ID and / or for the analytics ID Halting an analytics model for at least one analytics output regarding an analytics ID and / or for the analytics ID

[0249] Figure 11 shows a method 1100 according to an embodiment of the present disclosure. The method 1100 is for a network data analytics consumer entity 330 and can be performed by the consumer entity 330. The method 1100 includes step 1101 of providing a display 301 having information for initiating tracing of one or more analytics outputs for an analytics ID and / or tracing of the analytics ID to a network analytics tracing entity 300 or a network analytics inference entity 310.

[0250] Figure 12 shows a method 1200 according to an embodiment of the present disclosure. The method 1200 is for a network analytics management entity 340 and can be performed by the management entity 340. The method 1200 includes step 1201 of making a setting 341 for initiating tracing of one or more analytics outputs for an analytics ID and / or tracing of the analytics ID. Additionally, or alternatively, the method 1200 includes step 1202 of setting analytics tracing information 403 for the analytics ID. Additionally, or alternatively, the method 1200 includes step 1203 of making a setting 341 for initiating collection of data on one or more quality indicators regarding one or more analytics outputs for the analytics ID and / or regarding the analytics ID.

[0251] The present disclosure has been described with a variety of embodiments as examples and implementations. However, other variations can be understood and achieved by those skilled in the art and those implementing the claimed subject matter from a consideration of the drawings, the present disclosure, and the independent claims. In the claims and the description, the word "comprising" does not exclude other elements or steps, and the indefinite article "a" or "an" does not exclude a plurality. A single element or other unit may realize the functions of several entities or items recited in the claims. The mere fact that certain measures are recited in mutually different dependent claims does not indicate that a combination of these measures cannot be used in an advantageous implementation.

Claims

A method applied to a network data analytics function (NWDAF) including an analytics logic function (AnLF), the method being executed by an entity of a network, comprising: obtaining a display (301) having information for initiating tracing of one or more analytics outputs for an analytics identifier (ID) and / or tracing of the analytics ID; providing a notification (302) related to the analytics ID if at least one output for the analytics ID is unstable and / or if the analytics ID is unstable; wherein the notification (302) includes: - the at least one unstable analytics output for the analytics ID and / or the analytics ID, the notification being provided to a network analytics consumer entity (330), a network analytics inference entity (310), or a network analytics training entity (320); - an inference notification (311) for the at least one unstable analytics output for the analytics ID and / or for the analytics ID, the notification (302) being provided to the network analytics inference entity (310); - a training notification (321) for the at least one unstable analytics output for the analytics ID and / or for the analytics ID, the notification (302) being provided to the network analytics training entity (320); A method comprising one or more of the above. **Claim 2** The method according to claim 1, further comprising creating analytics tracing information associated with the analytics ID. **Claim 3** The method according to claim 2, wherein the analytics tracing information associated with the analytics ID includes an analytics tracing data structure (ATDS). **Claim 4** The method according to claim 3, wherein the ATS includes the ATS of the analytics ID or the ATS for the analytics ID. **Claim 5** The method according to claim 3 or 4, wherein the ATS is associated with the recording of time analytics tracing information for the analytics ID. **Claim 6** The method according to any one of claims 3 to 5, wherein the ATS includes a historical set of ATS records. **Claim 7** The method according to any one of claims 3 to 6, wherein the ATS includes an ATS record indicating the network state at a given point in time when using a given analytics. **Claim 8** The given analytics includes an analytics ID, an analytics type, or one or more analytics outputs The method according to claim 7, including one or more of the foregoing. **Claim 9** The notification includes an unstable analytics ID, one or more pieces of information associated with one or more of the unstable analytics outputs associated with the analytics ID The method according to any one of claims 1 to 8, including one or more of the foregoing. **Claim 10** The notification includes - An ATS record having analytics inference setting information (AICI) and / or analytics training setting information (ATCI) for the analytics ID of the ATS record regarded as an unstable analytics ID, - A parameter such as a tuple having an identification and a flag of an analytics ID and / or an analytics output, the flag indicating that there is a problem (or error) with the analytics (ID and / or output) The method according to claim 9, including one or more of the foregoing. **Claim 11** The flag includes an unstable analytics, suspension of analytics use, an invalid analytics, or a temporarily invalid analytics The method according to claim 10, including one or more of the foregoing. **Claim 12** The method according to claim 10 or 11, wherein the unstable analytics ID indicates that there is a problem (and / or error) with the analytics ID and / or the analytics output. **Claim 13** A computer program comprising program code for performing the method according to any one of claims 1 to 12 when executed on a computer.

14. An apparatus comprising a processor, wherein when the processor calls program code, the apparatus is enabled to execute the method according to any one of claims 1 to 12.

Images