Method and apparatus for traceablity of data analytics result

Abstract

Embodiments of the present disclosure provide a method and an apparatus for traceablity of data analytics result. An apparatus (10) for an analytics server comrprises: at least one processor (102); and at least one memory (104) including computer program code. The at least one memory (104) and the computer program code are configured to, with the at least one processor (102), cause the apparatus (10) for the analytics server at least to perform: receiving (S202), from an analytics consumer, a request for analytics metadata related to a data analytics determination; determining (S204) to apply models trained by vertical federated learning for the data analytics determination; determining (S206) at least one analytics client for the data analytics determination; transmitting (S208), to the at least one analytics client, a request for a subset of analytics metadata related to a subtask of the data analytics determination; receiving (S210), from the at least one analytics client, the subset of analytics metadata; generating (S212) the analytics metadata based at least on the received subset of analytics metadata; and transmitting (S214), to the analytics consumer, a response including a first part of the analytics metadata.

Description

METHOD AND APPARATUS FOR TRACEABLITY OF DATA ANALYTICS RESULTTECHNICAL FIELD

[0001] Various example embodiments of the present disclosure relate generally to the technology of communication, and in particular to a method and apparatus for traceablity of data analytics result.BACKGROUND

[0002] In communication networks, data analytics are used more and more. As part of the output Analytics, Metadata about the analytics such as number of data samples used for the generation of the output analytics and Data source(s) of the data used for the generation of the output analytics may be provided. It is unclear how the inference server can determine this information.SUMMARY

[0003] This summary is provided to introduce some aspects in a simplified form that are further described below in the detailed description. This summary is not intended to identify key features or essential features of the claimed subject matter, nor is it intended to be used to limit the scope of the claimed subject matter.

[0004] Certain aspects of the present disclosure and their embodiments may provide solutions to these or other challenges. There are, proposed herein, various embodiments which address one or more of the issues disclosed herein. Specific method and apparatus for traceablity of data analytics result may be provided.

[0005] A first aspect of the present disclosure provides an apparatus for an analytics server. The apparatus for the analytics server comprises at least one processor; and at least one memory including computer program code. The at least one memory and the computer program code are configured to, with the at least one processor, cause the apparatus for the analytics server at least to perform: receiving, from an analytics consumer, a request for analytics metadata related to a data analytics determination; determining to apply models trained by vertical federated learning for the data analytics determination; determining at least one analytics client for the data analytics determination; transmitting, to the at least one analytics client, a request for a subset of analytics metadata related to a subtask of the data analytics determination; receiving, from the at least one analytics client, the subset of analytics metadata; generating the analytics metadata based at least on the received subset of analytics metadata; and transmitting, to the analytics consumer, a response including a first part of the analytics metadata.

[0006] In exemplary embodiments of the present disclosure, the apparatus for the analytics server is further caused to perform: storing at least a second part of the analytics metadata ina data storage network node. The first part of the analytics metadata is the same with, or is different from the second part of the analytics metadata.

[0007] In exemplary embodiments of the present disclosure, the response further comprises: a data set tag indicating at least the second part of the analytics metadata stored as data set in the data storage network node.

[0008] In exemplary embodiments of the present disclosure, the apparatus for the analytics server is further caused to perform: receiving, from the analytics consumer, a report about an issue in an analytics output and / or the analytics metadata of the data analytics determination comprising the data set tag; transmitting, to the data storage network node, the comprised data set tag to retrieve the analytics metadata from the data storage network node; and determining based on the retrieved analytics metadata, whether to use additional analytics clients of vertical federated features for subsequent data analytics determination.

[0009] In exemplary embodiments of the present disclosure, the apparatus for the analytics server is further caused to perform: determining that there is an issue in an analytics output and / or the analytics metadata of the data analytics determination; and transmitting, to the data storage network node, the data set tag to retrieve the analytics metadata from the data storage network node.

[0010] In exemplary embodiments of the present disclosure, the apparatus for the analytics server is further caused to perform: determining based on the retrieved analytics metadata whether to use additional analytics clients of vertical federated features for subsequent data analytics determination and or whether to trigger of models to be used for subsequent data analytics data determination.

[0011] In exemplary embodiments of the present disclosure, the second part of the analytics metadata stored in the data storage network node comprise at least one of the following: at least one data set tag of a data set used to determine analytics data; at least one identifier of a model used to determine the analytics data; an indication that the analytics data were determined using models trained by vertical federated learning; a description of features or vertical federated clients used to determine the analytics data; an identifier of the related model for each feature or vertical federated client used to determine intermediate results for the analytics data; an identifier of the related number of data samples for each feature or vertical federated client used to determine intermediate results for the analytics data; one or more identifiers of related data sources for data samples for each feature or vertical federated client used to determine intermediate results for the analytics data; or an identifier of one or more data set tags of related used data samples for each feature or vertical federated client used to determine intermediate results for the analytics data.

[0012] In exemplary embodiments of the present disclosure, the first part of the analytics metadata comprises at least one of: a number of data samples used for a generation of an analytics output of the data analytics determination; a data time window of the data samples; dataset statistical properties of the analytics output; one or more data sources of the datasamples used for the generation of the analytics output; a data formatting and processing applied on the data samples; or an output strategy used for reporting of the analytics output.

[0013] In exemplary embodiments of the present disclosure, the subset of analytics metadata received from the at least one analytics client comprises at least one of the following: a data set tag indicating the subset of analytics metadata stored as data set in the data storage network node; a number of data samples used by the subtask of the data analytics determination; a data time window of the data samples used for the subtask of the data analytics determination; dataset statistical properties of data samples used by the subtask of the data analytics determination; one or more data sources of the data samples used by the subtask of the data analytics determination; at least one data set tag of a data set used by the subtask of the data analytics determination; or at least one identifier of a model used by the subtask of the data analytics determination.

[0014] In exemplary embodiments of the present disclosure, the request for analytics metadata is combined with a request for data analytics determination; and the request for a subset of analytics metadata transmitted to the analytics client is combined with a request to perform a subtask of the data analytics determination.

[0015] In exemplary embodiments of the present disclosure, the request for analytics metadata is indicated by a flag in the request for data analytics determination; and the request for the subset of analytics metadata is indicated by a flag in the request to perform the subtask of the data analytics determination.

[0016] In exemplary embodiments of the present disclosure, the requested data analytics determination is indicated by an analytics identifier, ID.

[0017] In exemplary embodiments of the present disclosure, the analytics client comprises a Vertical Federated Learning, VFL, inference client in a Network Data Analytics Function, NWDAF, or Analytics Logic Function, AnLF; the analytics server comprises a Vertical Federated Learning, VFL, inference server in a Network Data Analytics Function, NWDAF, or Analytics Logic Function, AnLF; and the data storage network node comprises an Analytics Data Repository Function, ADRF.

[0018] A second aspect of the present disclosure provides an apparatus for an analytics client. The apparatus for the analytics client comprises at least one processor; and at least one memory including computer program code. The at least one memory and the computer program code are configured to, with the at least one processor, cause the apparatus for the analytics client at least to perform: receiving, from an analytics server, a request for a subset of analytics metadata related to a subtask of a data analytics determination; transmitting, to the analytics server, the subset of analytics metadata. The subtask is allocated to the analytics client. The subset of analytics metadata is for the analytics server to generate an analytics metadata.

[0019] In exemplary embodiments of the present disclosure, the analytics metadata comprises at least one of: a data set tag indicating the subset of analytics metadata stored asdata set in the data storage network node; a number of data samples used by the subtask of the data analytics determination; a data time window of the data samples used for the subtask of the data analytics determination; dataset statistical properties of data samples used by the subtask of the data analytics determination; one or more data sources of the data samples used by the subtask of the data analytics determination; at least one data set tag of a data set used used by the subtask of the data analytics determination; or at least one identifier of a model used by the subtask of the data analytics determination.

[0020] In exemplary embodiments of the present disclosure, the analytics client comprises a Vertical Federated Learning, VFL, inference client in a Network Data Analytics Function, NWDAF, or Analytics Logic Function, AnLF; and the analytics server comprises a Vertical Federated Learning, VFL, inference server in a Network Data Analytics Function, NWDAF, or Analytics Logic Function, AnLF.

[0021] A third aspect of the present disclosure provides an apparatus for an analytics consumer. The apparatus for the analytics consumer comprises at least one processor; and at least one memory including computer program code. The at least one memory and the computer program code are configured to, with the at least one processor, cause the apparatus for the analytics consumer at least to perform: transmitting, to an analytics server, a request for analytics metadata related to a data analytics determination; and receiving, from the analytics server, a response including a first part of the analytics metadata. The response further comprises a data set tag indicating that at least a second part of the analytics metadata are stored as data set in a data storage network node.

[0022] In exemplary embodiments of the present disclosure, the request for analytics metadata is combined with a request for data analytics determination. The request for analytics metadata is indicated by a flag in the request for data analytics determination. The first part of the analytics metadata is the same with, or is different from the second part of the analytics metadata.

[0023] In exemplary embodiments of the present disclosure, the requested analytics determination is indicated by an analytics identifier, ID.

[0024] In exemplary embodiments of the present disclosure, the apparatus for the analytics consumer is further caused to perform: transmitting, to the analytics server or a monitoring node, a report about an issue in an analytics output and / or the analytics metadata of the data analytics determination comprising the data set tag.

[0025] In exemplary embodiments of the present disclosure, the apparatus for the analytics consumer is further caused to perform: transmitting, to the data storage network node, the data set tag to retrieve the second part of analytics metadata from the data storage network node.

[0026] In exemplary embodiments of the present disclosure, the first part of analytics metadata comprises at least one of: a number of data samples used for a generation of an analytics output of the data analytics determination; a data time window of the data samples;dataset statistical properties of the analytics output; one or more data sources of the data samples used for the generation of the analytics output; a data formatting and processing applied on the data samples; or an output strategy used for reporting of the analytics output.

[0027] In exemplary embodiments of the present disclosure, the second part of the analytics metadata comprise at least one of the following: any of the analytics metadata; at least one data set tag of a data set used to determine analytics data; at least one identifier of a model used to determine the analytics data; an indication that the analytics data were determined using models trained by vertical federated learning; a description of features or vertical federated clients used to determine the analytics data; an identifier of a related model for each feature or vertical federated client used to determine intermediate results for the analytics data; an identifier of related number of data samples for each feature or vertical federated client used to determine intermediate results for the analytics data; one or more identifiers of related data sources for data samples for each feature or vertical federated client used to determine intermediate results for the analytics data; or an identifier of one or more data set tags of related used data samples for each feature or vertical federated client used to determine intermediate results for the analytics data.

[0028] In exemplary embodiments of the present disclosure, the analytics server comprises a Vertical Federated Learning, VFL, inference server in a Network Data Analytics Function, NWDAF, or in an Analytics Logic Function, AnLF; and the data storage network node comprises an Analytics Data Repository Function, ADRF.

[0029] A fourth aspect of the present disclosure provides a method performed by an apparatus for an analytics server, according to any of embodiments of the first aspect.

[0030] A fifth aspect of the present disclosure provides a method performed by an apparatus for an analytics client, according to any of embodiments of the first aspect.

[0031] A sixth aspect of the present disclosure provides a method performed by an apparatus for an analytics consumer, according to any of embodiments of the first aspect.

[0032] A seventh aspect of the present disclosure provides a computer-readable storage medium storing instructions, which when executed by at least one processor of an apparatus, cause the at least one processor of the apparatus to perform at least the method according to any of the embodiments above mentioned.

[0033] According to embodiments of the present disclosure, the exemplary embodiments of the present disclosure propose a mechanism that provides specifical procedures for the inference server to provide information about metadata. With such information, the traceablity of data analytics result may be improved.BRIEF DESCRIPTION OF DRAWINGS

[0034] The above and other aspects, features, and benefits of various embodiments of the present disclosure will become more fully apparent, by way of example, from the following detailed description with reference to the accompanying drawings, in which like referencenumerals or letters are used to designate like or equivalent elements. The drawings are illustrated for facilitating better understanding of the embodiments of the disclosure and not necessarily drawn to scale, in which:

[0035] FIG. l is a block diagram showing an exemplary structure for an apparatus for an analytics server, according to exemplary embodiments of the present disclosure.

[0036] FIG. 2A is a flow chart showing a method performed by an apparatus for an analytics server.

[0037] FIG. 2B is a flow chart showing further steps of the method as shown in FIG. 2A, according to exemplary embodiments of the present disclosure.

[0038] FIG. 3 is a block diagram showing an exemplary structure for an apparatus for an analytics client, according to exemplary embodiments of the present disclosure.

[0039] FIG. 4 is a flow chart showing a method performed by an apparatus for an analytics client.

[0040] FIG. 5 is a block diagram showing an exemplary structure for an apparatus for an analytics consumer, according to exemplary embodiments of the present disclosure.

[0041] FIG. 6A is a flow chart showing a method performed by an apparatus for an analytics consumer.

[0042] FIG. 6B is a flow chart showing further steps of the method as shown in FIG. 6A, according to exemplary embodiments of the present disclosure.

[0043] FIG. 6C is a flow chart showing further steps of the method as shown in FIG. 6A, according to exemplary embodiments of the present disclosure.

[0044] FIG. 7 is a block diagram showing an apparatus / computer readable storage medium, according to embodiments of the present disclosure.

[0045] FIG. 8 is a block diagram showing exemplary apparatus units for an analytics server, which is suitable for performing the method according to embodiments of the disclosure.

[0046] FIG. 9 is a block diagram showing exemplary apparatus units for an analytics client, which is suitable for performing the method according to embodiments of the disclosure.

[0047] FIG. 10 is a block diagram showing exemplary apparatus units for an analytics consumer, which is suitable for performing the method according to embodiments of the disclosure.

[0048] FIG. 11 A, FIG. 1 IB, FIG. 11C are diagrams showing an exemplary signalling flow of embodiments of the present disclosure.DETAILED DESCRIPTION

[0049] The embodiments of the present disclosure are described in detail with reference to the accompanying drawings. It should be understood that these embodiments are discussed only for better understanding, rather than limitations on the scope of the present disclosure. The described features, advantages, and characteristics of the disclosure may be combined in any suitable manner in one or more embodiments.

[0050] Generally, all terms used herein are to be interpreted according to their ordinary meaning in the relevant technical field, unless a different meaning is clearly given and / or is implied from the context in which it is used. The steps of any methods disclosed herein do not have to be performed in the exact order disclosed, unless clearly given and / or implied from the context. Any feature of any of the embodiments disclosed herein may be applied to any other embodiment, wherever appropriate.

[0051] As used herein, the term “network” or “communication network” refers to a network following any suitable communication standards (such for an internet network, or any wireless network). For example, wireless communication standards may comprise WLAN (Wireless Local Area Network), new radio (NR), long term evolution (LTE), LTE- Advanced, 5G NR, 6G etc. In the following description, the terms “network” and “system” can be used interchangeably.

[0052] The term “node / network node” refers to a computing device or computing entity or computing function or any other devices (physical or virtual) in a communication network. For example, the node in the network may include a base station (BS), an access point (AP), or any other suitable device in a wireless communication network. The BS may be, for example, a node B (NodeB or NB), an evolved NodeB (eNodeB or eNB), a next generation NodeB (gNodeB or gNB), a remote radio unit (RRU), a radio header (RH), a remote radio head (RRH), a relay, a low power node such as a femto, a pico, and so forth. Further, the node may include other core network node, such as an Access and Mobility Management Function, AMF, a Session Management Function, SMF, a User Plane Function, UPF, a mobility management entity, MME, or a serving gateway, S-GW, etc.

[0053] The term “terminal device” refers to any end device that can access a communication network and receive services therefrom. By way of example and not limitation, the terminal device refers to a mobile terminal, user equipment (UE), a non- AP device (such as a non-AP Station (STA)), or other suitable devices. The terminal device may include, but not limited to, a mobile phone, a cellular phone, a smart phone, a wearable device, a vehicle-mounted wireless terminal device, a vehicle, and the like.

[0054] As one example, a terminal device may represent a device configured for communication in accordance with one or more communication standards promulgated by any standard organization, such as 3rdgeneration partnership project, 3 GPP.

[0055] As yet another example, in an Internet of Things (loT) scenario, a terminal device may represent a machine or other device that performs monitoring and / or measurements, and transmits the results of such monitoring and / or measurements to another terminal device and / or network equipment. Particular examples of such machines or devices are sensors, metering devices such as power meters, industrial machinery, or home or personal appliances, for example refrigerators, televisions, personal wearables such as watches etc. In other scenarios, a terminal device may represent a vehicle or other equipment that is capable of monitoring and / or reporting on its operational status or other functions associatedwith its operation.

[0056] It shall be understood that although the terms “first” and “second” etc. may be used herein to describe various elements, these elements should not be limited by these terms. These terms are only used to distinguish one element from another. For example, a first element could be termed a second element, and similarly, a second element could be termed a first element, without departing from the scope of example embodiments. As used herein, the term “and / or” includes any and all combinations of one or more of the associated listed terms.

[0057] As used herein, “at least one of the following: ” and “at least one of ” and similar wording, where the list of two or more elements are joined by “and” or “or”, mean at least any one of the elements, or at least any two or more of the elements, or at least all the elements.

[0058] For Release (Rel)-19, Service & System Aspects Working Group 2 (SA2) has studied Artificial Intelligence Markup language (AIML) enhancements for the core network in 3GPP TR 23.700-84 V2.0.0 (2024-09), among other including key issue for vertical federated learning (VFL). The study was concluded with the following related conclusions and normative work is about to begin:5.2.2 Key Issue #2: 5GC Support for Vertical Federated LearningThis key issue aims to provide solutions for enabling 5GC support for vertical federated learning (VFL) involving NWDAF and / or AF, where no raw data need to be exchanged but some level of coordination is still required when training and inference are performed on local models. In particular, datasets used for each local model need to share the same samples while holding different features.In Rel-18, ML model sharing between NWDAFs has been studied as a part of Horizontal Federated Learning. However, federated learning between NWDAF and AF has not been studied (e.g. when the NWDAFs and / or AFs are in different domains, locations, regions etc).Vertical Federated Learning (VFL) can be considered as an alternative mechanism for distributed functionalities of an ML model. Note that, as scoped in Rel-19, NWDAF and / or AF may be involved for VFL.This Key Issue aims to study architecture enhancement to support VFL, which allows the cooperative AI / ML training and inference with the following aspects:- Identify VFL use cases and under which conditions, and for which entities these VFL use cases show that VFL is justified to train ML models.- Whether and how to support architecture enhancement for supporting VFL for model training and / or inference. In particular:- Whether and how the existing NF discovery and selection needs to be enhanced.- Whether and how ML Model training and / or inference related procedures need to be enhanced to support VFL.- Whether and how to do performance monitoring for the ML model trained via VFL.- Whether and how to provide ML Models to the participants in the VFL training process.- How to support sample and feature alignment among the participating network entities when performing VFL.NOTE 1: Application layer-based VFL requiring communication between AFs and / or UEs application client, is out of scope.NOTE 2: During the study on this KI, consultation with SA WG3 is required for handling security aspects.NOTE 3: RAN and UE aspects are out of scope.NOTE 4: The existing procedures defined for Horizontal FL in TS 23.288

[0005] will be taken into account when studying the procedure for VFL.

[0059] The study was concluded with the following related conclusions in TR 23.700-84. and normative work is about to begin.8.2 Conclusions for KI#2: 5GC Support for Vertical Federated LearningP#2.5: For VFL inference process:P#2.5.1: NWDAF acting as VFL server scenario, the NF consumer of analytics will obtain the required output based on the VFL inference process coordinated by VFL server, which is generated via the VFL inference process between VFL sever and corresponding AFs or NWDAF acting as VFL client(s):- For NWDAF acting as VFL server, NWDAF triggers the VFL inference phase after receiving the subscription or request for analytics and delivers the analytics to the NF consumer.P 2.5.2: For the AF acting as a VFL server, the AF acting as VFL server can also start an inference process with corresponding NWDAF acting as VFL client(s). The AF may be triggered to start the inference by a 5GC consumer (i.e. NWDAF containing AnLF). Interactions are via NEF if AF is untrusted.P 2.5.3: Before performing the VFL inference, the NWDAF acting as VFL server determines corresponding AF(s) and / or NWDAF (s) as VFL client(s) for the inference process based on the same identifier used in the VFL training process. Related details will be specified in the normative phase.P 2.5.4: The VFL inference process may be controlled by a set of requirements, i.e. whether the determination if all VFL participants associated with the same identifier are needed in the VFL inference process may be based e.g. on accuracyrequirements, the VFL signalling and load cost, contribution weights of each client, and temporal availability of output from VFL participants.P#2.5.5: When performing VFL model performance monitoring, inference data can be used for model retraining, which is aligned with R18.NOTE 9: Whether and how P#2.5.4 is supported will be determined in normative phase.NOTE 10: Whether the NWDAF supports both AnLF and MTLF or not during VFL training and inference is to be discussed in normative phase.NOTE 11: How multiple NWDAFs are involved in VFL when AF is acting as VFL server is defined in normative phase.NOTE 12: The details of the accuracy monitoring related to VFL process will be defined in the normative phase.NOTE 13: The details of (optionally new) services and detailed list of parameters to enable the VFL processes will be defined in the normative phase.NOTE 14: Details of the ML model storage will be defined in the normative phase.

[0060] Detailed call flows for VFL are provided, for instance the flow in Figure 6.22.2.1- 1: VFL model training and inference involving MTLF, AnLF, ADRF and AFm S2-2405938.The AnLF acting as VFL inference server registers its capability to act as VFL inference server in the NRF together with Analytics IDs it supports and a vendor ID.1. The AnLF acting as VFL inference client registers its capability to act as VFL inference client in the NRF together with Analytics ID(s) and related Feature ED(s) it supports.2. The MTLF acting as VFL training client registers its capability to act as VFL training client in the NRF together with Analytics ID(s) and related Feature ED(s) it supports.3. Based on configured information, an NEF registers on behalf of an AF the capability of the AF to act as VFL client in the NRF together with Analytics IDs and related Feature ID(s) the AF supports.4. The MTLF acting as VFL training server receives or observes a trigger to start a VFL model training for an Analytics ID.5. The MTLF acting as VFL training server selects the required features for the model training based on implementation.6. For each feature, the MTLF acting as VFL training server sends a discovery request to the NRF for an MTLF acting as VFL training client or for an AF acting as VFL client and provides the Analytics ID and Feature ID.7. For each feature, the NRF provides profiles of candidate NFs matching the discovery request and the MTLF acting as VFL training server selects one such NF8. The MTLF acting as VFL training server may perform sample alignment with the VFL training clients. If UEs are used as samples, the procedure in Figure 6.22.2.2-1 applies.9. The MTLF acting as VFL training server assigns a unique VFL session ID. Steps 11 to 14 are repeated until the MTLF acting as VFL training server determines that the training is complete.10. The MTLF acting as VFL training server sends a request to start a VFL training iteration to each MTLF and AF acting as VFL training client provides the Analytics ID, Feature ID, and VFL session ID.NOTE 1: Details of the service operations for steps 11, 13 and 14 will be determined during the normative work.11. Each MTLF or AF acting as VFL training client collect input data, updates the own model based on the input data, and calculates so-called “gradients ” as intermediate results based on the updated model.12. Each MTLF or AF acting as VFL training client sends the intermediate results to the MTLF acting as VFL training server13. The MTLF acting as VFL training server updates the own model based on the received intermediate results. The MTLF may in addition collect own input data and also use those data to update the model. For each VFL client, it determines intermediate results based on the updated model and provides those intermediate results to the VFL clients in the request to start the next training itteration.14. Once the MTLF acting as VFL training server determines that the trained model is sufficiently stable to terminate the training, it informs the MTLFs acting as VFL training clients that the VFL training is completed.15. The MTLF acting as VFL training server and the VFL training clients perform VFL model training. The VFL training server trains a model which is able to combine information related to several features. Each VFL training client trains a model relating to a special feature. Those models are trained together over several cycles. Each Cycle is triggered by the MTLF acting as VFL training server sending a request as in step 11 to each VFL training client. After each cycle, each VFL training client provides so-called “gradients ” as intermediate results to the VFL training server that the server uses to update its model, and the server also provides intermediate results to each VFL training client that the clients use to update their feature-related models.16. The VFL training server may store the model it trained in the ADRF together with the Analytics ID, VFL session ID, and Feature IDs of all features used to train the model and an indication that the model is for an VFL server.17. Each VFL training client may store the model it trained in the ADRF together with the Analytics ID, VFL session ID and the Feature ID of the feature the model relates to and an indication that the model is for an VFL client.18. An AnLF acting as VFL inference server receives a request or subscription for analytics or statistics for an Analytics ID.19. The AnLF acting as VFL inference server queries the NRF for an VFL training server for the Analytics ID and its own vendor ID.20. The NRF provides profdes of candidate NFs matching the discovery request and the AnLF acting as VFL inference server selects one such NF.21. The AnLF acting as VFL inference server requests a model from the selected MTLF acting as VFL training server providing the Analytics ID.22. The MTLF acting as VFL training server may retrieve the requested model from the ADRF. The ADRF provides together with the model the VFL session identifier and the Feature IDs of all features used to train the model.23. The MTLF acting as VFL training server provides information related to the requested model it trained (either the model or information how to retrieve it).The VFL training server also provides the VFL session ID and the Feature IDs of all features used to train the model.24. If the MTLF acting as VFL training server provided information how to retrieve the model, the AnLF acting as VFL inference server retrieves the model.25. For each Feature, the AnLF acting as VFL inference server queries the NRF for an AnLF acting as VFL inference client or an AF acting as VFL client for the Analytics ID and related Feature ID.26. For each Feature, the NRF provides profiles of candidate NFs matching the discovery request and the AnLF acting as VFL inference server selects one such NF.27. For each selected AnLF acting as VFL inference client, the AnLF acting as VFL inference server sends a VFL inference request or subscription and provides the Analytics ID, VFL session ID and Feature ID within the request.NOTE 2: Details of the service operations for steps 28 and 36 will be determined during the normative work.28. Each AnLF acting as VFL inference client queries the NRF for an MTLF acting as VFL training client for the Analytics ID and Feature ID.29. Towards each AnLF acting as VFL inference client, the NRF provides profiles of candidate NFs matching the discovery request and the AnLF acting as VFL inference client selects one such NF.30. Each AnLF acting as VFL inference client requests the model from the selected MTLF acting as VFL training client providing the Analytics ID, Feature ID and VFL session identifier.31. Each MTLF acting as VFL training client may retrieve the requested model from the ADRF, providing the Analytics ID, Feature ID and VFL session identifier.32. Each MTLF acting as VFL training client provides information related to the requested model it trained (either the model or information how to retrieve it).33. If the MTLF acting as VFL training client provided information how to retrieve the model, the AnLF acting as VFL inference client retrieves the model.34. Each AnLF acting as VFL inference client retrieves input data and calculates output data related to the corresponding feature and Analytics ID using the retrieved model for that feature.35. Each AnLF acting as VFL inference client provides the output data to the AnLF acting as VFL inference server.36. For each AF acting as VFL client selected in step 23, the AnLF acting as VFL inference server sends a VFL inference request or subscription and provides the Analytics ID, VFL session ID and Feature ID within the request.NOTE 3: Details of the service operations for steps 37 and 39 will be determined during the normative work.37. Each AF acting as VFL client selects the applicable model for the inference based on the Analytics ID, VFL session ID and Feature ID, retrieves input data, and calculates output data using the selected model.38. Each AF acting as VFL client provide the output data to the AnLF acting as VFL inference server.39. The AnLF acting as VFL inference server calculates analytics or predictions using the retrieved model and the received feature-related output data.

[0061] Procedures for storing analytics as so-called historical data are already defined in 3GPP TS 23.288 V19.0.0 (2024-09), such as in Figure 6.2B.2-1: Historical Data and Analytics storage, and Figure 6.2B.3-1: Historical Data and Analytics Storage via Notifications. However, those procedures do not provide means about how the analytics results were derived.6.2B.2 Historical Data and Analytics storageThe procedure depicted in figure 6.2B.2-1 is used by consumers (e.g. NWDAF, DCCF or MF AF) to store historical data and / or analytics, i.e. data and / or analytics related to past time period that has been obtained by the consumer. Afterthe consumer obtains data and / or analytics, consumer may store historical data and / or analytics in an ADRF. Whether the consumer directly contacts the ADRF or goes via the DCCF or via the Messaging Framework is based on configuration. The consumer may include in the storage request the DataSetTag attribute which the data records are to be associated to when stored by ADRF. The DataSetTag attribute is defined in Table 6.2B-1. Data records can be associated to multiple DataSetTag attributes.Table 6.2B-1: DataSetTag attributeThe consumer may include in the storage request the Data Synthesis and Compression (DSC) information. The detail of DSC information is up to implementation, which is out of 3 GPP scope.NOTE: DCS information can include the following information: indication that the data have been generated using a data synthesis tool; indication that the data have been generated using a data compression tool; the information about the data synthesis and / or compression technique. (Omitted here)Figure 6.2B.2-1: Historical Data and Analytics storageOa-c. NWDAF, DCCF or ADRF are configured with default operator storage policies as described in clause 5B.1.1. The consumer sends data and / or analytics to the ADRF by invoking the Nadrf DataManagement StorageRequest (collected data with time stamp, analytics with timestamp, Service Operation, Analytics Specification or Data Specification, Storage Handling information, optionally DataSetTag, optionally DSC information) service operation. The NWDAF or DCCF may provide notification endpoint information to the ADRF for use by the ADRF to send notifications (implicit subscription) alerting the DCCF or NWDAF that data are about to be deleted (see step 6).2.a-c Based on Storage Handling information (if available) and StoragePolicy, the ADRF, DCCF or NWDAF determines the Storage Approach (lifetime for storing data and whether consumer is notified prior to data deletion).3. The ADRF stores the data and / or analytics sent by the consumer. The ADRF may, based on implementation, determine whether the same data and / or analytics is already stored or being stored based on the information sent in step 1 by the consumer NF and, if the data and / or analytics is already stored or being stored in the ADRF, the ADRF decides to not store again the data and / or analytics sent by the consumer. If the DataSetTag attribute is included for data and / or analytics already stored or being stored, then ADRF associates the data records with such DataSetTag.4. The ADRF sends Nadrf DataManagement StorageRequest Response message to the consumer indicating that data and / or analytics is stored, whether the ADRF determined at step 3 that data or analytics is already stored and the Storage Approach.Conditional on ADRF Managing the Storage Approach5 The ADRF determines that the lifetime of the stored data or analytics has expired (according to the Storage Approach).6 If indicated by the Storage Approach, the ADRF sends a notification alerting the DCCF or NWDAF that data are about to be deleted.NOTE: This is an implicit subscription.7. The DCCF or NWDAF indicate in the response to the ADRF whether they will retrieve the Data or Analytics8 The DCCF or NWDAF may retrieve the Data or Analytics from the ADRF Conditional on the DCCF or NWDAF Managing the Storage Approach9 The NWDAF or DCCF determine that the lifetime of the stored data or analytics has expired (according to the Storage Approach).10. The NWDAF or DCCF optionally retrieve the data or analytics from the ADRF11. The NWDAF or DCCF request that the data or analytics be deleted from the ADRF12. The ADRF deletes the data or analytics if:- the Storage Approach in the ADRF indicates alerting the consumer is not required prior to data or analytics deletion;- in step 7 the DCCF or NWDAF indicated data or analytics will not be retrieved prior to deletion;- data or analytics retrieval in step 8 has completed; or- A request to delete data or analytics was received in step 11.If the ADRF received a response from the NWDAF or DCCF in step 7 indicating data or analytics will be retrieved but retrieval is not initiated before an adequate fixed time has elapsed, the ADRF may autonomously delete the data or analytics.• 6.2B.3 Historical Data and Analytics Storage via NotificationsThe procedure depicted in figure 6.2B.3-1 is used by consumers (NWDAF, DCCF) to store received notifications in the ADRF The consumer requests the ADRF to initiate a subscription for data and / or analytics. Data and / or analytics provided in notifications as a result of the subsequent subscription by the ADRF are stored in the ADRF.The consumer may include in the subscription request the DataSetTag attribute defined in Table 6.2B-1.(Omitted here)Figure 6.2B.3-1: Historical Data and Analytics Storage via NotificationsOa-c NWDAF, DCCF or ADRF are configured with default operator storage policies as described in clause 5B.1. la-d. Based on provisioning or based on reception of a DataManagement subscription request (e.g. see clause 6.2.6.3.2), the DCCF or the NWDAF determines that notifications are to be stored in an ADRF. The subscription request may contain Storage Handling information with a requested ADRF storage lifetime and a request to be notified before data are deleted from the ADRF.2a-b. The DCCF or the NWDAF determines the ADRF where data and / or analytics needs to be stored and requests that the ADRF subscribes to receive notifications. The determination may be made based on configuration or information supplied by the data consumer as described in clauses 6.1.4 and 6.2.6.3. The request to the ADRF specifies the data and / or analytics to which the ADRF will subscribe by invoking theNadrf DataManagement StorageSubscriptionRequest service operation. The request may include the DataSetTag attribute which the data records are to be associated to when stored by ADRF. If the Storage Policy is not configured on the NWDAF or DCCF, the NWDAF or DCCF sends the Storage Handling Request to the ADRF. The NWDAF or DCCF may provide notification endpoint information to the ADRF for use by the ADRF to send notifications (implicit subscription) alerting the DCCF or NWDAF that data are about to be deleted (see step 12).3. [Optional] The ADRF may, based on implementation, determine whether the same data and / or analytics is already stored or being stored, based on the information sent in step 2 by the consumer. If the DataSetTag attribute is included for data and / or analytics already stored or being stored, then ADRF associates the data records with such DataSetTag.3a-c. [ Optional ] Based on Storage Handling information and Storage Policy, the ADRF, DCCF or NWDAF determines the Storage Approach (lifetime for storing data and whether consumer is notified prior to data deletion).4. [Optional] If the data and / or analytics is already stored and / or being stored in the ADRF, the ADRF sendsNadrf DataManagement StorageSubscriptionRequest Response message to the consumer indicating that data and / or analytics is stored. The ADRF includes the Storage Approach if determined in step 3c.5a-b. The DCCF or NWDAF sends a subscription response to the Data or Analytics Consumer. The response may contain the Storage Approach determined by the ADRF, NWDAF or DCCF.6a-b. ADRF subscribes to the DCCF or the NWDAF to receive notifications, providing its notification endpoint address and a notification correlation ID.7. The DCCF, the MFAF or the NWDAF sends Analytics or Data notifications containing the notification correlation ID provided by the ADRF to ADRF notification endpoint address. The Analytics or Data notifications shall contain timestamp. The ADRF stores the notifications.Conditional on DCCF or NWDAF determining to stop storing data or analytics8a-b. The DCCF or the NWDAF determines to stop storing notifications in the ADRF.9a-b. The DCCF or the NWDAF requests that the ADRF unsubscribes to receive notifications. lOa-b. The ADRF sends a request to the DCCF or the NWDAF to unsubscribe to data notifications.The NWDAF may interact with the Data Source and the DCCF may interact with the Data Source and / or MFAF. Delivery of notifications from the DCCF / MFAF or NWDAF to the ADRF are subsequently halted.Conditional on ADRF Managing the Storage Approach11. The ADRF determines that the lifetime of the stored data or analytics has expired (according to the Storage Approach).12. If indicated by the Storage Approach, the ADRF sends a notification alerting the DCCF or NWDAF that data are about to be deleted.NOTE: This is an implicit subscription.Conditional if the Storage Appoach is based on Default Operator Policies provisioned on the NWDAF or DCCF (see step 3a-c and clause 5B.1)13. The DCCF or NWDAF indicate in the response to the ADRF whether they will retrieve the Data or Analytics14. The DCCF or NWDAF may retrieve the Data or Analytics from the ADRFConditional if the Storage Appoach is based on a Storage Handling Request received from the Data or Analytics Consumer (see step 3a-c and clause 5B.1) 15a-b. The NWDAF or DCCF sends a notification to the Data or AnalyticsConsumer alerting it that the data or analytics are about to be deleted.16a-b. The Data or Analytics Consumer indicates in the response to the NWDAF or DCCF whether it will retrieve the Data or Analytics17. The DCCF or NWDAF indicate in the response to the ADRF whether the consumer will retrieve the Data or Analytics18. The Data or Analytics Consumer retrieves the stored data or analytics Conditional on DCCF or NWDAF Managing the Storage Approach19a-b. The NWDAF or DCCF determines that the lifetime of the stored data has expired (according to the Storage Approach).Conditional if the Storage Appoach is based on Default Operator Policies provisioned on the NWDAF or DCCF (see step 3a-c and clause 5B.1)20. The NWDAF or DCCF optionally retrieve the data or analytics from the ADRF21. The NWDAF or DCCF request that the data or analytics be deleted from the ADRFConditional if the Storage Appoach is based on a Storage Handling Request received from the Data or Analytics Consumer (see step 3a-c and clause 5B.1) 22a-b. The NWDAF or DCCF sends a notification to the Data or AnalyticsConsumer alerting it that the data or analytics are about to be deleted.23a-b. The Data or Analytics Consumer indicates in the response to the NWDAF or DCCF whether it will retrieve the Data or Analytics24. The Data or Analytics Consumer retrieves the stored data or analytics25. The NWDAF or DCCF request that the data or analytics be deleted from the ADRF26. The ADRF deletes the data or analytics if:- the Storage Approach in the ADRF indicates alerting the consumer is not required prior to data or analytics deletion;- in Steps 13 or 17 the DCCF or NWDAF indicated data or analytics will not be retrieved prior to deletion;- data or analytics retrieval in steps 14 or 18 has completed or- A request to delete data or analytics was received in steps 21 or 25.If the ADRF received a response from the NWDAF or DCCF in steps 14 or 17 indicating data or analytics will be retrieved but retrieval is not initiated before an adequate fixed time has elapsed, the ADRF may autonomously delete the data or analytics.10.2.2 Nadrf DataManagement StorageRequest service operation Service operation name: Nadrf DataManagement StorageRequest Description: The consumer NF uses this service operation to request the ADRF to store data or analytics. Data or analytics are provided to the ADRF in the request message.Inputs, Required: Data with timestamp or Analytics with timestamp to be stored, Service operation, Analytics Specification or Data Specification."Service Operation" identifies the service used to obtain the data or analytics from a Data Source (e.g. Namf EventExposure Subscribe or Nrrwdaf AnalyticsSubscription Subscribe)."Analytics Specification or Data Specification" is the "Service Operation" specific required and optional input parameters that identify the data that was stored (e.g. Analytics ID(s) / Event ID (s), Target of Analytics Reporting or Target of Event Reporting, Analytics Filter or Event Filter, etc.). Service Operations and input parameters are defined in clause 7 for NWDAF and in clause 5.2 of TS 23.502 [3] for the other NFs.Inputs, Optional: DataSetTag, DSC information, Storage Handling Information, Data Deletion Notification Endpoint (see clause 6.2B.2).Outputs Required: Result Indication.Outputs, Optional: Storage Transaction Identifier, DataSetTag(s), Storage Approach.

[0062] TS 23.288 defines the contents of analytics exposure as follows and allows that Analytics metadata are exposed together with the analytics, and that the analytics consumer provides feedback about the accuracy of the analytics.6.1.3 Contents of Analytics ExposureThe consumers of the Nrrwdaf AnalyticsSubscription Subscribe or Nrrwdaf Analyticsinfo Request service operations described in clause 7 provide the input parameters listed below.- A list of Analytics IDs: identifies the requested analytics.- Analytics Filter Information: indicates the conditions to be fulfilled for reporting Analytics Information. This set of optional parameter types and values enables to select which type of analytics information is requested. Analytics Filter Information is defined in the analytics related clauses.- Target of Analytics Reporting: indicates the object(s) for which Analytics information is requested, entities such as specific list of UEs, i.e. a list of SUPIs, group of UEs, i.e. a list of Internal-Group-Ids, or any UE (i.e. all UEs).- (Only for Nnwdaf AnalyticsSubscription Subscribe) A Notification Target Address (+ Notification Correlation ID) as defined in clause 4.15.1 ofTS 23.502 [3], allowing to correlate notifications received from NWDAF with this subscription.- (Only for Nnwdaf AnalyticsSubscription Subscribe) Subscription Correlation ID: identifies an existing analytics subscription that is to be modified.- Related to analytic consumers that aggregate analytics from multiple NWDAF subscriptions:- [ OPTIONAL ] (Set of) NWDAF identifiers of NWDAF instances used by the NWDAF service consumer when aggregating multiple analytics subscriptions. See clause 6.1 A.- Analytics Reporting Information with the following parameters:- ( Only for Nnwdaf AnalyticsSubscription Subscribe ) Analytics Reporting Parameters as per Event Reporting parameters defined in Table 4.15.1-1 ofTS 23.502 [3].NOTE 1: When the Analytics Reporting Parameters indicates a periodic reporting mode and the periodicity of the report is equal to or greater than the Supported Analytics Delay associated with the Analytics ID (if available) defined in clause 6.2.6.2 ofTS 23.501 [2], it is expected that the periodic reporting can be provided by the NWDAF as requested.- (Only for Nnwdaf AnalyticsSubscription Subscribe) Reporting Thresholds, which indicate conditions on the level of each requested analytics that when reached shall be notified by the NWDAF.- [ OPTIONAL ] Matching direction: A matching direction may be provided such as below, above, or crossed. If no matching direction is provided, the default direction is crossed.- [OPTIONAL] Acceptable deviation: An acceptable deviation from the threshold level in the non-critical direction (i.e. in which the QoS is improving) may be set to limit the amount of signalling.- Analytics target period: time interval [start.. end], either in the past (both start time and end time in the past) or in the future (both start time and end time in the future). An Analytics target period in the past is a request or subscription for statistics. An Analytics target period in the future is a request or subscription for predictions. The time interval is expressed with actual start time and actual end time (e.g. via UTC time). When the Analytics Reporting Parameters indicate a periodic reporting mode, the time interval can also be expressed as positive or negative offsets to the reporting time, which indicates a subscription for predictions or statisticsrespectively. By setting start time and end time to the same value, the consumer of the analytics can request analytics or subscribe to analytics for a specific time rather than for a time interval.- Time window for historical analytics: time interval [start.. end]. The time window for historical analytics indicates the time interval during which the historical analytics was generated. If the time window for historical analytics is included, the NWDAF only needs to provide the existing analytics, and does not need to generate new analytics.- [OPTIONAL] Data time window: if specified, only events that have been created in the specified time interval are considered for the analytics generation.- [ OPTIONAL ] Preferred level of accuracy of the analytics ("Low ", "Medium", "High" or "Highest").- [ OPTIONAL ] Preferred level of accuracy per analytics subset ("Low ", "Medium", "High" or "Highest"). When a preferred level of accuracy is expressed for a given analytics subset, it takes precedence for this subset over the above preferred level of accuracy of the analytics. Analytics subsets are defined in the "Output Analytics" clause of applicable analytics.- [OPTIONAL] Dataset Statistical Properties: information in order to influence the data selection mechanisms to be used for the generation of an Analytics ID, assuring that the generated Analytics ID reflects the statistical characteristics of the data that are relevant for the NWDAF consumer. The following dataset statistical properties are allowed:- Uniformly distributed datasets, which indicates the use of data samples that are uniformly distributed according to the different aspects of the requested analytics (e.g. equivalent data samples for each UE listed as a Target of Analytics Reporting or for S-NSSAIs included in the Analytics Filter Information).- Datasets with or without outliers, which indicates that the data samples shall consider or disregard data samples that are at the extreme boundaries of the value range.- Time when analytics information is needed (if applicable): indicates to the NWDAF the latest time the analytics consumer expects to receive analytics data provided by the NWDAF. It should not be set to a value less than the Supported Analytics Delay of the selected NWDAF if applicable. If the time is reached the consumer does not need to wait for the analytics information any longer, yet the NWDAF may send an error response or error notification to the consumer. "Time when analytics information isneeded" is a relative time interval as the gap with respect to analytics request / subscription (e.g. "in 10 minutes").NOTE 2: If the analytics request contains the parameter "Time when analytics information is needed" for Analytics ID(s), this parameter takes precedence over the requested periodicity, if a periodic reporting mode is requested.NOTE 3: If the Time when analytics information is needed is provided and it is less than the Supported Analytics Delay per Analytics ID (if available) defined in clause 6.2.6.2 ofTS 23.501 [2], it is expected that the NWDAF might not be able to treat the Analytics ID on time.- [OPTIONAL] Maximum number of objects requested by the consumer (max) to limit the number of objects in a list of analytics perNrrwdaf AnalyticsSubscription Notify or Nrrwdaf Analyticsinfo Request response.- [OPTIONAL] Preferred granularity of location information: "TA level", "cell level" or "longitude and latitude level".NOTE 4: As defined in clause 4 of TS 23.032

[0034] , longitude and latitude level means the location information is expressed as longitude and latitude in geographical coordinate instead of TA ID or cell ID that is only known in 3 GPP system. It also stands for the location information that is expressed as a reference point in local co-ordinate.- [OPTIONAL] Spatial granularity size: maximum number of TA or cells used to define an area for which analytics are provided. When this parameter is provided, the NWDAF should provide analytics per group of TA of cells accordingly.- [OPTIONAL] Temporal granularity size: minimum duration of each time slot for which analytics are provide. When this parameter is provided, the NWDAF should provide analytics per elementary time slot accordingly.NOTE 5: It is up to NWDAF implementation to determine whether the data is taken into account that the UE locates in an area for a shorter time than the Temporal granularity size.- [OPTIONAL] Preferred orientation of location information: ("horizontal", "vertical", "both").- [ OPTIONAL ] Preferred order of results when a list of analytics is returned, possibly with a criterion for identifying the property of the results to which the preferred ordering is applied.- [OPTIONAL] Maximum number of SUPIs (SUPImax) requested by the consumer to limit the number of SUPIs in an object. When SUPImax is not provided, the NWDAF shall return all SUPIs concerned by the analyticsobject. When SUPImax is set to 0, the NWDAF shall not provide any SUPI.- [OPTIONAL] Output strategy: indicates the relevant factors for determining when the analytics reported. The following values are allowed:- Binary output strategy: indicates that the analytics shall only be reported when the preferred level of accuracy is reached within a cycle of periodic notification as defined in the Analytics Reporting Parameters.NOTE 6: If preferred level of accuracy is more important than providing an output, then the binary strategy is used so that all analytics outputs have equivalent confidence in the prediction.- Gradient output strategy: indicates that the analytics shall be reported according to the periodicity defined in the Analytics Reporting Parameters irrespective if the preferred level of accuracy has been reached.NOTE 7: If having an analytics output is more important than reaching the preferred level of accuracy, then the gradient output strategy is used so that each NWDAF will timely provide the output indicating the confidence of the prediction at the moment of the output generation.NOTE 8: When no output strategy is included in the subscription, the analytics output will be generated based on the gradient strategy and includes the confidence of the prediction for the reporting period.- [OPTIONAL] Analytics metadata request: indicates a request from one NWDAF to another NWDAF to provide the "analytics metadata information" related to the produced output analytics. This input parameter indicates which parameters in "analytics metadata information" are required to aggregate the output analytics for the requested Analytics ID(s).- (Only for Nrrwdaf AnalyticsSubscription Subscribe) Consumer NF's serving area or NF ID. During a pending analytics subscription transfer, this information can be used by the NWDAF to find out if the analytics consumers may change as described in clause 6. IB.2.- (Only for Nrrwdaf AnalyticsSubscription Subscribe) Information of previous analytics subscription. When setting up the analytics generation, this information may be used to retrieve analytics context from the previous NWDAF in order to build upon the context that is already related to this subscription as described in clause 6. IB.2.1.- [OPTIONAL] Use case context: indicates the context of use of the analytics to select the most relevant ML Model.NOTE 9: The NWDAF can use the parameter "Use case context" to select the most relevant ML Model, when several ML Models are available for the requested Analytics ID(s). NWDAF containing AnLF can additionally provide the parameter "Use case context" when requesting an ML Model from an NWDAF containing MTLF. The values of this parameter are not standardized. For example, the AMF can use a given value of "Use case context" when requesting UE Mobility analytics for optimizing the definition of a Registration Area, and a different value of "Use case context" when requesting UE Mobility analytics for determines a paging strategy.- (Only for Nrrwdaf AnalyticsSubscription Subscribe) [OPTIONAL] Analytics Feedback Information: indicates that the consumer NF has taken an action(s) influenced by the previously provided analytics, which may or may not affect the ground truth data corresponding to analytic ID requested at the time which the prediction refers to, and consequently affect the ML Model Accuracy Monitoring by the subscription with following parameter (s) :- Corresponding Analytics ID(s) which has been used for taking an action(s);- Indication whether the action will affect ground truth data (if available);- Time stamp(s) of the action(s) taken.NOTE 10: The consumer NF cannot include Analytics Feedback Information in initial subscription request. Analytics Feedback Information can be included in modification request for the existing analytics subscription.- [ OPTIONAL ] Analytics Accuracy Request information with the following parameters:- Analytics accuracy request: indicates NWDAF to provide accuracy information to the analytics consumer.- [ OPTIONAL ] Analytics Accuracy Information time window : time interval [start, end], which indicates that analytics consumers only consider the accuracy information which is generated within this time interval.- [ OPTIONAL ] Analytics Accuracy Information periodicity: time period, which indicates periodic reporting of accuracy information for the corresponding Analytics ID(s).- [ OPTIONAL ] Analytics Accuracy threshold: a reporting threshold accuracy value, which indicates that:- The NWDAF can provide analytics output and optionally analytics accuracy value to the analytics consumer(s) when the accuracy value isabove this Analytics Accuracy threshold (i.e. the accuracy is sufficient according to the threshold);- The NWDAF can provide "Stop Analytics Output Consumption indication ", "Updated Analytics " or the Analytics Accuracy Information to the analytics consumer(s) when the accuracy value is under this threshold (This indicates the deviation of the predictions from the actual network data does not meet analytics accuracy requirement, i.e. the accuracy is not sufficient according to the threshold).- [ OPTIONAL ] Minimal number of analytics output occurrences: determines the minimal number of analytics outputs provided by NWDAF that have to be considered in the determination of the accuracy information.- [OPTIONAL] Updated Analytics flag: indicates that the NWDAF can provide updated analytics for provided Analytics ID(s), if updated analytics can be generated within Analytics Accuracy Information time window.- [OPTIONAL] Correction time period: a relative time interval as the gap with respect to analytics is provided, which is indicated the time interval during which the updated analytics can be accepted by the analytics consumer.- [OPTIONAL] Pause analytics consumption flag: is a flag indicating to NWDAF to stop sending the notifications of analytics outputs for a subscribed analytics ID, without unsubscribing to such analytics ID.- [OPTIONAL] Resume Analytics Subscription request: is a flag indicating to NWDAF to resume the notification of analytics outputs for an existing analytics ID(s) subscription(s) that have been previously paused.The NWDAF provides to the consumer of theNrrwdaf AnalyticsSubscription Subscribe or Nrrwdaf Analyticsinfo Request service operations described in clause 7, the output information listed below, using a Nrrwdaf AnalyticsSubscription Notify service operation or theNrrwdaf Analyticsinfo Request response, respectively:- (Only for Nrrwdaf AnalyticsSubscription Notify) The Notification Correlation Information.- For each Analytics ID, the analytics information in the requested Analytics target period. If the analytics subset is subscribed or requested, then the corresponding analytics information shall be provided.- Timestamp of analytics generation: allows consumers to decide until when the received information shall be used. For instance, an NF can deem areceived notification from NWDAF for a given feedback as invalid based on this timestamp;- Validity period: defines the time period for which the analytics information is valid.NOTE 11: Validity period is determined by NWDAF internal logic and it is a subset of Analytics target period.- Confidence: probability assertion, i.e. confidence in the prediction.- [OPTIONAL] For each Analytics ID the Termination Request, which notifies the consumer that the subscription is requested to be cancelled as the NWDAF can no longer serve this subscription, e.g. due to user consent revoked, NWDAF overload, UE moved out of NWDAF serving area, etc.- [OPTIONAL] Analytics metadata information: additional information required to aggregate the output analytics for the requested Analytics ID(s). This parameter shall be provided if the "Analytics metadata request" parameter was provided in the correspondingNrrwdaf AnalyticsSubscription Subscribe or Nrrwdaf Analyticsinfo Request service operation.- Number of data samples used for the generation of the output analytics;- Data time window of the data samples;- Dataset Statistical Properties of the analytics output used for the generation of the analytics;- [OPTIONAL] Data source(s) of the data used for the generation of the output analytics;- [ OPTIONAL ] Data Formatting and Processing applied on the data;- Output strategy (i.e. gradient output strategy or binary output strategy) used for the reporting of the analytics.- (Only for error response or error notification) Revised waiting time: indicates to the consumer a revised waiting value for "Time when analytics information is needed". Each NWDAF may include this as part of error response or error notification to "Time when analytics information is needed" as described in clause 6.2.5. Revised waiting time is the minimum time interval recommended by NWDAF to use as "Time when analytics information is needed" for similar future analytics requests / subscriptions.- [ OPTIONAL ] Analytics Accuracy Information generated for each analytics ID, including:- Analytics accuracy value for requested Analytics ID(s): a value shall be provided if "Analytics accuracy request" parameter was provided in the corresponding Nrrwdaf AnalyticsSubscription Subscribe service operation. This parameter may be provided if the value crosses theanalytics accuracy threshold(s) which is indicated in the subscribe request or locally configured, or the Analytics Accuracy Information periodicity indicated in the subscribe request is reached.- [OPTIONAL] An indication that the determined accuracy value for the analytics ID does not meet the analytics accuracy threshold requested for the analytics ID.- [ OPTIONAL ] Updated Analytics: NWDAF provides updated Analytics, which is generated within Analytics Accuracy Information time window, for provided Analytics ID(s), if "Updated Analytics flag" parameter was indicated in the corresponding Nrrwdaf AnalyticsSubscription Subscribe service operation.- [OPTIONAL] Stop Analytics Output Consumption indication: NWDAF provides to the consumer an indication to stop the consumption of the Analytics ID(s) related to the subscription ID based on NWDAF internal logic or specified analytics accuracy threshold.- [OPTIONAL] Stop Analytics Output Consumption time window: NWDAF provides to the consumer a time window to stop the consumption of the Analytics ID(s) related to the subscription ID based on NWDAF internal logic or specified analytics accuracy threshold.- [OPTIONAL] Resume Analytics Output Consumption indication: NWDAF provides to consumer an indication to resume the consumption of analytics output for existing subscription to the analytics ID(s) that was previously paused.- [ OPTIONAL ] Accuracy Information Termination: NWDAF notifies the consumer that the subscription to the accuracy information for an analytics ID has been cancelled as the NWDAF does not support the accuracy checking capability, e.g. as an indication that a new target serving NWDAF supporting accuracy checking capability could not have been selected during the analytics transfer procedures.NOTE 12: It is left to Stage 3 to decide whether the Accuracy Information Termination is a cause related to the Termination Request or not.

[0063] As part of the output Analytics, Metadata about the analytics such as number of data samples used for the generation of the output analytics and Data source(s) of the data used for the generation of the output analytics may be provided. It is unclear how the inference server can determine this information.

[0064] Further, it has been concluded that not all VFL participants used in the training phase may be involved in the VFL inference process, and VFL participants may be selected based e.g. on accuracy requirements, the VFL signaling and load cost, contribution weightsof each client, and temporal availability of output from VFL participants. If not all VFL clients and related models used during the training phase are used during the inference phase, the accuracy may suffer. It is important to keep track of the models used to derive inference results to achieve an explainability and traceability of results and determine whether a retraining of model becomes necessary if inference results are deemed not to have the desired accuracy.

[0065] Embodiments of the present disclosure provide specifical procedures for the inference server to provide information about metadata of data analytics. Such information may be used to keep track of the models used to derive inference results to achieve an explainability and traceability of results and determine whether a retraining of model becomes necessary.

[0066] FIG. l is a block diagram showing an exemplary structure for an apparatus for an analytics server, according to exemplary embodiments of the present disclosure.

[0067] As shown in FIG. 1, the apparatus 10 for the analytics server comprises at least one processor 102; and at least one memory 104 including computer program code. The at least one memory 104 and the computer program code are configured to, with the at least one processor 102, cause the apparatus for the analytics server at least to perform the method as shown in FIG. 2A-FIG. 2E, FIG. 11 A-FIG. 1 IB.

[0068] FIG. 2A is a flow chart showing a method performed by an apparatus for an analytics server.

[0069] As shown in FIG. 2A, the method 200 comprises: a step S202, receiving, from an analytics consumer, a request for analytics metadata related to a data analytics determination; a step S204, determining to apply models trained by vertical federated learning for the data analytics determination; a step S206, determining at least one analytics client for the data analytics determination; a step S208, transmitting, to the at least one analytics client, a request for a subset of analytics metadata related to a subtask of the data analytics determination; a step S210, receiving, from the at least one analytics client, the subset of analytics metadata; a step S212, generating the analytics metadata based at least on the received subset of analytics metadata; and a step S214, transmitting, to the analytics consumer, a response including a first part of the analytics metadata.

[0070] According to embodiments of the present disclosure, the exemplary embodiments of the present disclosure propose a mechanism that provides specifical procedures for the inference server to provide information about metadata. With such information, the traceablity of data analytics result may be improved.

[0071] FIG. 2B is a flow chart showing further steps of the method as shown in FIG. 2A, according to exemplary embodiments of the present disclosure.

[0072] As shown in FIG. 2B, the method 200 further comprises: a step S216, storing at least a second part of the analytics metadata in a data storage network node. The first part of the analytics metadata is the same with, or is different from the second part of the analyticsmetadata.

[0073] In exemplary embodiments of the present disclosure, the response further comprises: a data set tag indicating at least the second part of the analytics metadata stored as data set in the data storage network node.

[0074] According to embodiments of the present disclosure, at least some part of the analytics metadata may be stored, and thus such analytics metadata may be retrieved whenever necessary. The traceablity of data analytics result may be further improved. For example, a data set tag may be used for retrieval of such metadata. The analytics metadata thus can be clearly indicated, and the security may be also improved compared to providing the metadata itself directly.

[0075] FIG. 2C is a flow chart showing further steps of the method as shown in FIG. 2A, according to exemplary embodiments of the present disclosure.

[0076] As shown in FIG. 2C, the method 200 further comprises: a step S218, receiving, from the analytics consumer, a report about an issue in an analytics output and / or the analytics metadata of the data analytics determination comprising the data set tag; a step S220, transmitting, to the data storage network node, the comprised data set tag to retrieve the analytics metadata from the data storage network node; and a step S222, determining based on the retrieved analytics metadata, whether to use additional analytics clients of vertical federated features for subsequent data analytics determination.

[0077] According to embodiments of the present disclosure, an exemplary implementation scenario for retrieval of the metadata is that, an issue (e.g. about accuracy or creditability) in an analytics output and / or the analytics metadata is reported by the analytics consumer, and the metadata needs to be retrieved to find out where the problem is. By using such historical metadata, the accuracy, or creditability etc. may be improved.

[0078] FIG. 2D is a flow chart showing further steps of the method as shown in FIG. 2A, according to exemplary embodiments of the present disclosure.

[0079] As shown in FIG. 2D, the method 200 further comprises: a step S224, determining that there is an issue in an analytics output and / or the analytics metadata of the data analytics determination; and a step S226, transmitting, to the data storage network node, the data set tag to retrieve the analytics metadata from the data storage network node.

[0080] According to embodiments of the present disclosure, the issue may be also found out by the analytics server itself.

[0081] FIG. 2E is a flow chart showing further steps of the method as shown in FIG. 2A, according to exemplary embodiments of the present disclosure.

[0082] As shown in FIG. 2E, the method 200 further comprises: a step S228, determining based on the retrieved analytics metadata whether to use additional analytics clients of vertical federated features for subsequent data analytics determination and or whether to trigger of models to be used for subsequent data analytics data determination.

[0083] According to embodiments of the present disclosure, based on retrieved analyticsmetadata, various manners may be used to improve the data analytics determination, such as using additional analytics client or changing models, etc.

[0084] In exemplary embodiments of the present disclosure, the second part of the analytics metadata stored in the data storage network node comprise at least one of the following: at least one data set tag of a data set used to determine analytics data; at least one identifier of a model used to determine the analytics data; an indication that the analytics data were determined using models trained by vertical federated learning; a description of features or vertical federated clients used to determine the analytics data; an identifier of the related model for each feature or vertical federated client used to determine intermediate results for the analytics data; an identifier of the related number of data samples for each feature or vertical federated client used to determine intermediate results for the analytics data; one or more identifiers of related data sources for data samples for each feature or vertical federated client used to determine intermediate results for the analytics data; or an identifier of one or more data set tags of related used data samples for each feature or vertical federated client used to determine intermediate results for the analytics data.

[0085] In exemplary embodiments of the present disclosure, the first part of the analytics metadata comprises at least one of: a number of data samples used for a generation of an analytics output of the data analytics determination; a data time window of the data samples; dataset statistical properties of the analytics output; one or more data sources of the data samples used for the generation of the analytics output; a data formatting and processing applied on the data samples; or an output strategy used for reporting of the analytics output.

[0086] In exemplary embodiments of the present disclosure, the subset of analytics metadata received from the at least one analytics client comprises at least one of the following: a data set tag indicating the subset of analytics metadata stored as data set in the data storage network node; a number of data samples used by the subtask of the data analytics determination; a data time window of the data samples used for the subtask of the data analytics determination; dataset statistical properties of data samples used by the subtask of the data analytics determination; one or more data sources of the data samples used by the subtask of the data analytics determination; at least one data set tag of a data set used by the subtask of the data analytics determination; or at least one identifier of a model used by the subtask of the data analytics determination.

[0087] According to embodiments of the present disclosure, some parameters of the metadata are listed above as example, but not limitation.

[0088] In exemplary embodiments of the present disclosure, the request for analytics metadata is combined with a request for data analytics determination; and the request for a subset of analytics metadata transmitted to the analytics client is combined with a request to perform a subtask of the data analytics determination.

[0089] In exemplary embodiments of the present disclosure, the request for analytics metadata is indicated by a flag in the request for data analytics determination; and the requestfor the subset of analytics metadata is indicated by a flag in the request to perform the subtask of the data analytics determination.

[0090] According to embodiments of the present disclosure, the request for analytics metadata may be a separate request, or may be combined with another request. For example, it may be indicated by a flag in another request. And thus, the change to the interface protocol or signaling definition may be reduced and compatibility may be also improved.

[0091] In exemplary embodiments of the present disclosure, the requested data analytics determination is indicated by an analytics identifier, ID.

[0092] In exemplary embodiments of the present disclosure, the analytics client comprises a Vertical Federated Learning, VFL, inference client in a Network Data Analytics Function, NWDAF, or Analytics Logic Function, AnLF; the analytics server comprises a Vertical Federated Learning, VFL, inference server in a Network Data Analytics Function, NWDAF, or Analytics Logic Function, AnLF; and the data storage network node comprises an Analytics Data Repository Function, ADRF.

[0093] In exemplary embodiments of the present disclosure, such apparatus and method may be applied to, but not limited to a Vertical Federated Learning structure.

[0094] FIG. 3 is a block diagram showing an exemplary structure for an apparatus for an analytics client, according to exemplary embodiments of the present disclosure.

[0095] As shown in FIG. 3, the apparatus 30 for the analytics client comprises at least one processor 302; and at least one memory 304 including computer program code. The at least one memory 304 and the computer program code are configured to, with the at least one processor 302, cause the apparatus for the analytics client at least to perform the method as shown in FIG. 4, FIG. 11 A-FIG. 1 IB.

[0096] FIG. 4 is a flow chart showing a method performed by an apparatus for an analytics client.

[0097] As shown in FIG. 4, the method 400 comprises: a step S402, receiving, from an analytics server, a request for a subset of analytics metadata related to a subtask of a data analytics determination; a step S404, transmitting, to the analytics server, the subset of analytics metadata. The subtask is allocated to the analytics client. The subset of analytics metadata is for the analytics server to generate an analytics metadata.

[0098] In exemplary embodiments of the present disclosure, the analytics metadata comprises at least one of: a data set tag indicating the subset of analytics metadata stored as data set in the data storage network node; a number of data samples used by the subtask of the data analytics determination; a data time window of the data samples used for the subtask of the data analytics determination; dataset statistical properties of data samples used by the subtask of the data analytics determination; one or more data sources of the data samples used by the subtask of the data analytics determination; at least one data set tag of a data set used used by the subtask of the data analytics determination; or at least one identifier of a model used by the subtask of the data analytics determination.

[0099] In exemplary embodiments of the present disclosure, the analytics client comprises a Vertical Federated Learning, VFL, inference client in a Network Data Analytics Function, NWDAF, or Analytics Logic Function, AnLF; and the analytics server comprises a Vertical Federated Learning, VFL, inference server in a Network Data Analytics Function, NWDAF, or Analytics Logic Function, AnLF.

[0100] FIG. 5 is a block diagram showing an exemplary structure for an apparatus for an analytics consumer, according to exemplary embodiments of the present disclosure.

[0101] As shown in FIG. 5, the apparatus 50 for the analytics consumer comprises at least one processor 502; and at least one memory 504 including computer program code. The at least one memory 504 and the computer program code are configured to, with the at least one processor 502, cause the apparatus 50 for the analytics consumer at least to perform the method as shown in FIG. 6A-FIG. 6C, FIG. 11 A-FIG. 1 IB.

[0102] FIG. 6A is a flow chart showing a method performed by an apparatus for an analytics consumer.

[0103] As shown in FIG. 6A, the method 600 comprises: a step S602, transmitting, to an analytics server, a request for analytics metadata related to a data analytics determination; and a step S604, receiving, from the analytics server, a response including a first part of the analytics metadata. The response further comprises a data set tag indicating that at least a second part of the analytics metadata are stored as data set in a data storage network node.

[0104] In exemplary embodiments of the present disclosure, the request for analytics metadata is combined with a request for data analytics determination. The request for analytics metadata is indicated by a flag in the request for data analytics determination. The first part of the analytics metadata is the same with, or is different from the second part of the analytics metadata.

[0105] In exemplary embodiments of the present disclosure, the requested analytics determination is indicated by an analytics identifier, ID.

[0106] FIG. 6B is a flow chart showing further steps of the method as shown in FIG. 6A, according to exemplary embodiments of the present disclosure.

[0107] As shown in FIG. 6B, the method 600 further comprises: a step S606, transmitting, to the analytics server or a monitoring node, a report about an issue in an analytics output and / or the analytics metadata of the data analytics determination comprising the data set tag.

[0108] FIG. 6C is a flow chart showing further steps of the method as shown in FIG. 6A, according to exemplary embodiments of the present disclosure.

[0109] As shown in FIG. 6A, the method 600 further comprises: a step S608, transmitting, to the data storage network node, the data set tag to retrieve the second part of analytics metadata from the data storage network node.

[0110] In exemplary embodiments of the present disclosure, the first part of analytics metadata comprises at least one of: a number of data samples used for a generation of an analytics output of the data analytics determination; a data time window of the data samples;dataset statistical properties of the analytics output; one or more data sources of the data samples used for the generation of the analytics output; a data formatting and processing applied on the data samples; or an output strategy used for reporting of the analytics output.

[0111] In exemplary embodiments of the present disclosure, the second part of the analytics metadata comprise at least one of the following: any of the analytics metadata; at least one data set tag of a data set used to determine analytics data; at least one identifier of a model used to determine the analytics data; an indication that the analytics data were determined using models trained by vertical federated learning; a description of features or vertical federated clients used to determine the analytics data; an identifier of a related model for each feature or vertical federated client used to determine intermediate results for the analytics data; an identifier of related number of data samples for each feature or vertical federated client used to determine intermediate results for the analytics data; one or more identifiers of related data sources for data samples for each feature or vertical federated client used to determine intermediate results for the analytics data; or an identifier of one or more data set tags of related used data samples for each feature or vertical federated client used to determine intermediate results for the analytics data.

[0112] In exemplary embodiments of the present disclosure, the analytics server comprises a Vertical Federated Learning, VFL, inference server in a Network Data Analytics Function, NWDAF, or in an Analytics Logic Function, AnLF; and the data storage network node comprises an Analytics Data Repository Function, ADRF.

[0113] The processor 102, 302, 502 may be any kind of processing component, such as one or more microprocessor or microcontrollers, as well as other digital hardware, which may include digital signal processors (DSPs), special-purpose digital logic, and the like. The memory 104, 304, 504 may be any kind of storage component, such as read-only memory (ROM), random-access memory, cache memory, flash memory devices, optical storage devices, etc.

[0114] FIG. 7 is a block diagram showing an apparatus / computer readable storage medium, according to embodiments of the present disclosure.

[0115] As shown in FIG. 7, a computer-readable storage medium 70 storing instructions 71, which when executed by at least one processor of an apparatus, cause the at least one processor of the apparatus to perform the method according to any of the embodiments herein mentioned, such as shown in FIG. 2A-FIG. 2E, FIG. 4, FIG. 6A-FIG. 6C, FIG. 11 A- FIG. 11B.

[0116] In addition, the present disclosure may also provide a carrier containing the computer program / instructions as mentioned above. The carrier is one of an electronic signal, optical signal, radio signal, or the above computer readable storage medium. The computer readable storage medium can be, for example, an optical compact disk or an electronic memory device like a RAM (random access memory), a ROM (read only memory), Flash memory, magnetic tape, CD-ROM, DVD, Blue-ray disc and the like.

[0117] FIG. 8 is a block diagram showing exemplary apparatus units for an analytics server, which is suitable for performing the method according to embodiments of the disclosure.

[0118] As shown in FIG. 8, the analytics server 80 may include: a first receiving unit 802, configured for receiving, from an analytics consumer, a request for analytics metadata related to a data analytics determination; a first determining unit 804, configured for determining to apply models trained by vertical federated learning for the data analytics determination; a second determining unit 806, configured for determining at least one analytics client for the data analytics determination; a first transmitting unit 808, configured for transmitting, to the at least one analytics client, a request for a subset of analytics metadata related to a subtask of the data analytics determination; a second receiving unit 810, configured for receiving, from the at least one analytics client, the subset of analytics metadata; a generating unit 812, configured for generating the analytics metadata based at least on the received subset of analytics metadata; and a second transmitting unit 814, configured for transmitting, to the analytics consumer, a response including a first part of the analytics metadata.

[0119] The first, second receiving units may be the same one or not. The first, second determining units may be the same one or not. The first, second transmitting units may be the same one or not.

[0120] In exemplary embodiments of the present disclosure, the analytics server 80 is further configured for performing the method according to any of the embodiments above mentioned, such as shown in FIG. 2A-FIG. 2E, 11 A-l IB.

[0121] FIG. 9 is a block diagram showing exemplary apparatus units for analytics client, which is suitable for performing the method according to embodiments of the disclosure.

[0122] As shown in FIG. 9, the analytics client 90 may include: a receiving unit 902, configured for receiving, from an analytics server, a request for a subset of analytics metadata related to a subtask of a data analytics determination; a transmitting unit 904, configured for transmitting, to the analytics server, the subset of analytics metadata. The subtask is allocated to the analytics client. The subset of analytics metadata is for the analytics server to generate an analytics metadata.

[0123] In exemplary embodiments of the present disclosure, the analytics client 90 is further configured for performing the method according to any of the embodiments above mentioned, such as shown in FIG. 4, 11 A-l IB.

[0124] FIG. 10 is a block diagram showing exemplary apparatus units for an analytics consumer, which is suitable for performing the method according to embodiments of the disclosure.

[0125] As shown in FIG. 10, the analytics consumer 100 may include: a transmitting unit 1002, configured for transmitting, to an analytics server, a request for analytics metadata related to a data analytics determination; and a receiving unit 1004, configured for receiving,from the analytics server, a response including a first part of the analytics metadata. The response further comprises a data set tag indicating that at least a second part of the analytics metadata are stored as data set in a data storage network node.

[0126] In exemplary embodiments of the present disclosure, the analytics consumer 100 is further configured for performing the method according to any of the embodiments above mentioned, such as shown in FIG. 6A-FIG. 6C, 11 A-l IB.

[0127] The term ‘unit’ may have conventional meaning in the field of electronics, electrical devices and / or electronic devices and may include, for example, electrical and / or electronic circuitry, devices, modules, processors, memories, logic solid state and / or discrete devices, computer programs or instructions for carrying out respective tasks, procedures, computations, outputs, and / or displaying functions, and so on, as such as those that are described herein.

[0128] As used in the present disclosure, the term “circuitry” may refer to one or more or all of the following:(a) hardware-only circuit implementations (such as implementations in only analogy and / or digital circuitry) and(b) combinations of hardware circuits and software, such as (as applicable):(i) a combination of analogy and / or digital hardware circuit(s) with software / firmware and(ii) any portions of hardware processor(s) with software (including digital signal processor(s)), software, and memory(ies) that work together to cause an apparatus, such as a mobile phone or server, to perform various functions) and(c) hardware circuit(s) and or processor(s), such as a microprocessor(s) or a portion of a microprocessor(s), that requires software (e.g., firmware) for operation, but the software may not be present when it is not needed for operation.”

[0129] This definition of circuitry applies to all uses of this term in the present disclosure, including in any claims. As a further example, as used in the present disclosure, the term circuitry also covers an implementation of merely a hardware circuit or processor (or multiple processors) or portion of a hardware circuit or processor and its (or their) accompanying software and / or firmware. The term circuitry also covers, for example and if applicable to the particular claim element, a baseband integrated circuit or processor integrated circuit for a mobile device or a similar integrated circuit in server, a cellular network device, or other computing or network device.

[0130] With these units, the apparatus may not need a fixed processor or memory, any kind of computing resource and storage resource may be arranged from at least one node / device / entity / apparatus relating to the communication system. The virtualization technology and network computing technology (e.g., cloud computing) may be further introduced, so as to improve the usage efficiency of the network resources and the flexibility of the network.

[0131] The techniques described herein may be implemented by various means so that an apparatus implementing one or more functions of a corresponding apparatus described with an embodiment comprises not only prior art means, but also means for implementing the one or more functions of the corresponding apparatus described with the embodiment and it may comprise separate means for each separate function, or means that may be configured to perform two or more functions. For example, these techniques may be implemented in hardware (one or more apparatuses), firmware (one or more apparatuses), software (one or more modules / units), or combinations thereof. For a firmware or software, implementation may be made through modules (e.g., procedures, functions, and so on) that perform the functions described herein.

[0132] In certain embodiments, some or all of the functionality described herein may be provided by processing circuitry executing instructions stored on in memory, which in certain embodiments may be a computer program product in the form of a non-transitory computer-readable storage medium. In alternative embodiments, some or all of the functionalities may be provided by the processing circuitry without executing instructions stored on a separate or discrete device-readable storage medium, such as in a hard-wired manner. In any of those particular embodiments, whether executing instructions stored on a non-transitory computer-readable storage medium or not, the processing circuitry can be configured to perform the described functionality. The benefits provided by such functionality are not limited to the processing circuitry alone or to other components of the computing device, but are enjoyed by the computing device as a whole, and / or by end users and a wireless network generally.

[0133] The term “non-transitory,” as used herein, is a limitation of the medium itself (i.e., tangible, not a signal) as opposed to a limitation on data storage persistency (e.g., RAM vs. ROM).

[0134] According to above embodiments, some further detailed solutions may be provided as follows.

[0135] In some exemplary scenarios, the VFL inference clients report information about analytics metadata such as number of input data samples used for the generation of the intermediate inference results and data source(s) of the data used for the generation of the intermediate inference results, possible information identifying stored or retrieved input data sets at the Analytics Data Repository Function (ADRF) used for the generation of intermediate results, dataset statistical properties, a time window used to collect the input data, and possible information about models used for the generation of intermediate results together with the intermediate results. The VFL inference server aggregates that information into the analytics metadata it generates.

[0136] In a preferred embodiment, when performing VFL inference to derive analytics output for an Analytics ID, the VFL inference server also provides information identifying the models and / or the features used for the VFL inference and / or identifying stored inputdata sets as part of the analytics metadata and / or stores this information in a database such as the ADRF together with an internal inference token or together with the related Analytics ID and a timestamp identifying the time when the output data were provided or an inference token. To hide information related to features and models from analytics consumers, the VFL inference server preferably includes the internal inference token in the analytics metadata instead of that information, and stores that information in separate records accessible via the inference token. This enables to explain the origin of analytics at a later stage, for instance if an analytics consumer reports accuracy issues.

[0137] FIG. 11 A, FIG. 11B, FIG. 11C are diagrams showing an exemplary signalling flows of embodiments of the present disclosure.

[0138] As shown in FIG. 11 A-FIG. 11C, the procedure includes following main steps.

[0139] 1. An Analytics Concumer requests Analytics from the VFL Inference server (for instance an NWDAF or AnLF). It provides the requested Analytics ID and filter parameters for the requested analytics, and may also request related analytics metadata.

[0140] 2. The VFL Inference server decides to use VFL for inference of analytical statistics or predictions for the requested analytics ID. It selects Features used for the analytics inference and for each selected feature a related VFL inference client. The VFL inference server may determine not to use some features used in the model training phase for the inference.

[0141] Steps 3 to 9 are executed separately for each selected VFL inference client (for instance an NWDAF or AnLF). Namely, steps 3a-9a are for oen client, and the setps 3b-9b are for another. The following description will be general just for clearity and simplicity.

[0142] 3. The VFL inference server requests the VFL inference client to perform inference for the Analytics ID and the related feature. It may provide a VFL session ID to guide the VFL client to select an appropriate model. The request may be a one-time request or a subscription. The VFL inference server may request the VFL client to provide analytics metadata, for instance if the VFL server received a request for analytics metadata from the analytics consumer.

[0143] 4. The VFL inference client determines a local model based on the received Analytics ID, Feature Information, and VFL session ID. It determines data source for required input data for the inference and retrieves those input data from the determined data sources.

[0144] 5. The VFL inference client may store the retrieved input data at the ADRF together with a data set tag it assigns.

[0145] 6-7. The VFL inference server may also retrieve data sets from the ADRF using a data set tag and use them as input data (through a request and a response).

[0146] 8. Using the retrieved input data and the determined model, the VFL inference client calculates local intermediate inference results. If requested in step 3, the VFL inference client also determines local analytics metadata such as number of input data samples usedfor the generation of the local intermediate inference results, and data source(s) of the input data used for the generation of the intermediate results, Possible data set tags used for storing and / or retrieving input data at / from the ADRF, dataset statistical properties, a time window used to collect the input data, and possible information about the used model.

[0147] 9. The VFL inference client sends the local intermediate inference results together with the possible local analytics metadata to the VFL inference server.

[0148] 10. Using the information obtained from all VFL inference clients, the VFL inference server determines global inference results (i.e. statistics or predictions for the analytics ID and filters received in step 1). It also aggregates the received local analytics metadata into the global analytics metadata it generates. For instance, the server may take the sum, average, maximum or minimum of received local number of data samples as the global number of data samples. Taking an average or minimum may for instance be appropriate if the data samples relate to UEs. The VFL inference server may determine a time window that includes all received local time windows. The VFL inference server may remove network internal information such as model IDs, data source(s) and dataset tags from the global analytics metadata, but may instead add an inference dataset tag under which such information can be retrieved from the ADRF.

[0149] 11. The VFL inference server may store information about the inference operation at the ADRF together with an inference data set tag. This information may comprise any local analytics metadata received in step 9 / see description about local analytics metadata in step 8), information about features used for the inference operation, and the VFL session ID as provided in step 3. In a preferred embodiment, the feature information and the related local analytics metadata are associated with each other in the storage format. The information may also comprise information about features used during the model training which the inference server decided to omit in step 2. The information about the inference operation may include network-internal information which is not suitable to be exposed to an external analytics consumer.

[0150] 12. The VFL inference server provides as analytics output the global inference results (i.e. statistics or predictions for the analytics ID and filters received in step 1) together with the global analytics metadata generated in step 10 to the analytics consumer. The global analytics metadata may comprise the inference data set tag.

[0151] 13. The VFL inference server may store the analytics output it provided in step 12 at the ADRF. The global analytics metadata within may comprise the inference data set tag.

[0152] 14. If the analytics consumer discovers that the analytics output is not of sufficient quality or incorrect, it may report those problems to the inference server or another monitoring entity and may provide the received inference data set tag. Alternatively, the inference server or another monitoring entity (e.g. AnLF, MTLF, NWDAF) that monitors analytics accuracy may detect a problem.

[0153] 15.-16. The VFL inference server or the other monitoring entity may retrieve theanalytics output from the ADRF. The global analytics metadata within may comprise the inference data set tag.

[0154] 17.-18. Using the inference dataset tag received in step 14 or step 16, the VFL inference server or the other monitoring entity may retrieve the stored information about the information about the inference operation from the ADRF.

[0155] 19. If the VFL inference server decided in step 2 not to use some used features during the model training phase, and the VFL inference server or other monitoring entity discovers from the inference data set received in step 18 that some features used for the model training were omitted during the inference operation, the VFL inference server or the other monitoring entity may determine to use such additional features for subsequent inference operations. If no features used during the model training were omitted for the inference, the VFL inference server or the other monitoring entity may determine to no longer use the models as identified by the VFL session ID or the model IDs in the inference data set and may instead trigger a new VFL model training.

[0156] According to above exemplary embodiments of the present disclosure, A mechanism is proposed for the inference server to provide information about metadata. With such information, the traceablity of data analytics result may be improved.

[0157] For example, a VFL server may know how to specifically provide analytics metadata to a consumer.

[0158] It should be understood that the above embodiments are only for illustration but not limitation. The present disclosure may be carried out in other ways than those specifically set forth herein without departing from essential characteristics of the disclosure. All changes to these embodiments not departing from the meaning and equivalency of the appended claims are intended to be comprised herein.

[0159] The following documents may be incorporated in entirety by reference.3rd generation partnership project technical report (3GPP TR) 23.700-84 V2.0.0 (2024-09)3GPP TS 23.288 V19.0.0 (2024-09)3GPP TS 23.501 V19.1.0 (2024-09)S2-2405938, SA WG2 Meeting #163, 27 May-31 May 2024, Jeju island, Korea

[0160] ABBREVIATION EXPLANATION

[0161] ADRF Analytics Data Repository FunctionAnLF Analytics Logical FunctionNWDAF Network Data Analytics FunctionVFL Vertical Federated LearningAIML Artificial Intelligence Markup languageRel ReleaseSA2 Service & System Aspects Working Group 2MTLF Model Training Logic FunctionNF Network FunctionUE User EquipmentNR New Radio3GPP 3rd Generation Partnership ProjectTR Technical ReportNW NetworkUE User Equipment3GPP 3rd generation partnership project5GC 5th Generation Core Network5G fifth generationNR New Radio6G sixth generationFor other abbreviation, see 3GPP TS 23.501.

Claims

CLAIMS1. An apparatus (10) for an analytics server, comprising: at least one processor (102); and at least one memory (104) including computer program code; the at least one memory (104) and the computer program code configured to, with the at least one processor (102), cause the apparatus (10) for the analytics server at least to perform: receiving (S202), from an analytics consumer, a request for analytics metadata related to a data analytics determination; determining (S204) to apply models trained by vertical federated learning for the data analytics determination; determining (S206) at least one analytics client for the data analytics determination; transmitting (S208), to the at least one analytics client, a request for a subset of analytics metadata related to a subtask of the data analytics determination; receiving (S210), from the at least one analytics client, the subset of analytics metadata; generating (S212) the analytics metadata based at least on the received subset of analytics metadata; and transmitting (S214), to the analytics consumer, a response including a first part of the analytics metadata.

2. The apparatus (10) according to claim 1, further caused to perform: storing (S216) at least a second part of the analytics metadata in a data storage network node; wherein the first part of the analytics metadata is the same with, or is different from the second part of the analytics metadata.

3. The apparatus (10) according to claim 2, wherein the response further comprises: a data set tag indicating at least the second part of the analytics metadata stored as data set in the data storage network node.

4. The apparatus (10) according to claim 3, further caused to perform: receiving (S218), from the analytics consumer, a report about an issue in an analytics output and / or the analytics metadata of the data analytics determination comprising the data set tag; transmitting (S220), to the data storage network node, the comprised data set tag to retrieve the analytics metadata from the data storage network node; and determining (S222) based on the retrieved analytics metadata, whether to use additional analytics clients of vertical federated features for subsequent data analytics determination.

5. The apparatus (10) according to claim 3, further caused to perform: determining (S224) that there is an issue in an analytics output and / or the analytics metadata of the data analytics determination; and transmitting (S226), to the data storage network node, the data set tag to retrieve the analytics metadata from the data storage network node.

6. The apparatus (10) according to claim 4 or 5, further caused to perform: determining (S228) based on the retrieved analytics metadata whether to use additional analytics clients of vertical federated features for subsequent data analytics determination and or whether to trigger of models to be used for subsequent data analytics data determination.

7. The apparatus (10) according to any of claims 2 to 6, wherein the second part of the analytics metadata stored in the data storage network node comprise at least one of the following: at least one data set tag of a data set used to determine analytics data; at least one identifier of a model used to determine the analytics data; an indication that the analytics data were determined using models trained by vertical federated learning; a description of features or vertical federated clients used to determine the analytics data; an identifier of the related model for each feature or vertical federated client used to determine intermediate results for the analytics data; an identifier of the related number of data samples for each feature or vertical federated client used to determine intermediate results for the analytics data; one or more identifiers of related data sources for data samples for each feature or vertical federated client used to determine intermediate results for the analytics data; or an identifier of one or more data set tags of related used data samples for each feature or vertical federated client used to determine intermediate results for the analytics data.

8. The apparatus (10) according to any of claims 1 to 6, wherein the first part of the analytics metadata comprises at least one of: a number of data samples used for a generation of an analytics output of the data analytics determination; a data time window of the data samples; dataset statistical properties of the analytics output;one or more data sources of the data samples used for the generation of the analytics output; a data formatting and processing applied on the data samples; or an output strategy used for reporting of the analytics output.

9. The apparatus (10) according to any of claims 1 to 8, wherein the subset of analytics metadata received from the at least one analytics client comprises at least one of the following: a data set tag indicating the subset of analytics metadata stored as data set in the data storage network node; a number of data samples used by the subtask of the data analytics determination; a data time window of the data samples used for the subtask of the data analytics determination; dataset statistical properties of data samples used by the subtask of the data analytics determination; one or more data sources of the data samples used by the subtask of the data analytics determination; at least one data set tag of a data set used by the subtask of the data analytics determination; or at least one identifier of a model used by the subtask of the data analytics determination.

10. The apparatus (10) according to any of claims 1 to 9, wherein the request for analytics metadata is combined with a request for data analytics determination; and wherein the request for a subset of analytics metadata transmitted to the analytics client is combined with a request to perform a subtask of the data analytics determination.

11. The apparatus (10) according to claim 10, wherein the request for analytics metadata is indicated by a flag in the request for data analytics determination; and wherein the request for the subset of analytics metadata is indicated by a flag in the request to perform the subtask of the data analytics determination.

12. The apparatus (10) according to any of claims 1 to 11, wherein the requested data analytics determination is indicated by an analytics identifier, ID.

13. The apparatus (10) according to any of claims 1 to 12, wherein the analytics client comprises a Vertical Federated Learning, VFL, inferenceclient in a Network Data Analytics Function, NWDAF, or Analytics Logic Function, AnLF; wherein the analytics server comprises a Vertical Federated Learning, VFL, inference server in a Network Data Analytics Function, NWDAF, or Analytics Logic Function, AnLF; and wherein the data storage network node comprises an Analytics Data Repository Function, ADRF.

14. An apparatus (30) for an analytics client, comprising: at least one processor (302); and at least one memory (304) including computer program code; the at least one memory (304) and the computer program code configured to, with the at least one processor (302), cause the apparatus (30) for the analytics client at least to perform: receiving (S402), from an analytics server, a request for a subset of analytics metadata related to a subtask of a data analytics determination; transmitting (S404), to the analytics server, the subset of analytics metadata; wherein the subtask is allocated to the analytics client; and wherein the subset of analytics metadata is for the analytics server to generate an analytics metadata.

15. The apparatus (30) according to claim 14, wherein the analytics metadata comprises at least one of: a data set tag indicating the subset of analytics metadata stored as data set in the data storage network node; a number of data samples used by the subtask of the data analytics determination; a data time window of the data samples used for the subtask of the data analytics determination; dataset statistical properties of data samples used by the subtask of the data analytics determination; one or more data sources of the data samples used by the subtask of the data analytics determination; at least one data set tag of a data set used used by the subtask of the data analytics determination; or at least one identifier of a model used by the subtask of the data analytics determination.

16. The apparatus (30) according to claim 14 or 15, wherein the analytics client comprises a Vertical Federated Learning, VFL, inference client in a Network Data Analytics Function, NWDAF, or Analytics Logic Function, AnLF; andwherein the analytics server comprises a Vertical Federated Learning, VFL, inference server in a Network Data Analytics Function, NWDAF, or Analytics Logic Function, AnLF.

17. An apparatus (50) for an analytics consumer, comprising: at least one processor (502); and at least one memory (504) including computer program code; the at least one memory (504) and the computer program code configured to, with the at least one processor (502), cause the apparatus (50) for the analytics consumer at least to perform: transmitting (S602), to an analytics server, a request for analytics metadata related to a data analytics determination; and receiving (S604), from the analytics server, a response including a first part of the analytics metadata; wherein the response further comprises a data set tag indicating that at least a second part of the analytics metadata are stored as data set in a data storage network node.

18. The apparatus (50) according to claim 17, wherein the request for analytics metadata is combined with a request for data analytics determination; wherein the request for analytics metadata is indicated by a flag in the request for data analytics determination; and wherein the first part of the analytics metadata is the same with, or is different from the second part of the analytics metadata.

19. The apparatus (50) according to claim 18, wherein the requested analytics determination is indicated by an analytics identifier, ID.

20. The apparatus (50) according to any of claims 17 to 19, further caused to perform: transmitting (S606), to the analytics server or a monitoring node, a report about an issue in an analytics output and / or the analytics metadata of the data analytics determination comprising the data set tag.

21. The apparatus (50) according to any of claims 17 to 20, further caused to perform: transmitting (S608), to the data storage network node, the data set tag to retrieve the second part of analytics metadata from the data storage network node.

22. The apparatus (50) according to any of claims 17 to 21, wherein the first part of analytics metadata comprises at least one of: a number of data samples used for a generation of an analytics output of the data analytics determination; a data time window of the data samples; dataset statistical properties of the analytics output; one or more data sources of the data samples used for the generation of the analytics output; a data formatting and processing applied on the data samples; or an output strategy used for reporting of the analytics output.

23. The apparatus (50) according to any of claims 17 to 22, wherein the second part of the analytics metadata comprise at least one of the following: any of the analytics metadata; at least one data set tag of a data set used to determine analytics data; at least one identifier of a model used to determine the analytics data; an indication that the analytics data were determined using models trained by vertical federated learning; a description of features or vertical federated clients used to determine the analytics data; an identifier of a related model for each feature or vertical federated client used to determine intermediate results for the analytics data; an identifier of related number of data samples for each feature or vertical federated client used to determine intermediate results for the analytics data; one or more identifiers of related data sources for data samples for each feature or vertical federated client used to determine intermediate results for the analytics data; or an identifier of one or more data set tags of related used data samples for each feature or vertical federated client used to determine intermediate results for the analytics data.

24. The apparatus (50) according to any of claims 17 to 23, wherein the analytics server comprises a Vertical Federated Learning, VFL, inference server in a Network Data Analytics Function, NWDAF, or in an Analytics Logic Function, AnLF; and wherein the data storage network node comprises an Analytics Data Repository Function, ADRF.

25. A method (200) performed by an apparatus for an analytics server, comprising: receiving (S202), from an analytics consumer, a request for analytics metadata related to a data analytics determination;determining (S204) to apply models trained by vertical federated learning for the data analytics determination; determining (S206) at least one analytics client for the data analytics determination; transmitting (S208), to the at least one analytics client, a request for a subset of analytics metadata related to a subtask of the data analytics determination; receiving (S210), from the analytics client, a subset of analytics metadata; generating (S212) the analytics metadata based at least on the received subset of analytics metadata; and transmitting (S214), to the analytics consumer, a response including a first part of the analytics metadata.

26. A method (400) performed by an apparatus for an analytics client, comprising: receiving (S402), from an analytics server, a request for a subset of analytics metadata related to a subtask of a data analytics determination; transmitting (S404), to the analytics server, the subset of analytics metadata; wherein the subtask is allocated to the analytics client; and wherein the subset of analytics metadata is for the analytics server to generate an analytics metadata.

27. A method (600) performed by an apparatus for an analytics consumer, comprising: transmitting (S602), to an analytics server, a request for analytics metadata related to a data analytics determination; and receiving (S604), from the analytics server, a response including a first part of the analytics metadata; wherein the response further comprises a data set tag indicating that at least a second part of the analytics metadata are stored as data set in a data storage network node.

28. A computer-readable storage medium (70) storing instructions (71), which when executed by at least one processor of an apparatus, cause the at least one processor of the apparatus to at least perform the method according to any of claims 25 to 27.

Images