Method, apparatus and computer program

Abstract

An apparatus comprising: means for comparing, for a machine learning model for a user equipment, at least one of: a value of a first loss function for first data and a value of a second loss function for second data, and an accuracy of prediction using the first data and an accuracy of prediction using the second data; wherein the first data has not been processed to enhance privacy to protect an identity of the user equipment and has not been used to train the machine learning model, and wherein the second data has been used to train the machine learning model; the apparatus comprising: means for determining, based on the comparison, a level of privacy enhancement of the identity of the user equipment in the second data.

Description

[0001] METHOD, APPARATUS AND COMPUTER PROGRAM

[0002] TECHNICAL FIELD

[0003] Various example embodiments of this disclosure relate to a method, apparatus, system and computer program for a communications network. Some examples can be used to determine whether a privacy level for an identity of a User Equipment (UE) has been respected by one or more network entities.

[0004] BACKGROUND

[0005] A communication network can be seen as a facility that enables communications between two or more communication devices, or provides communication devices access to a data network. A mobile or wireless communication network is one example of a communication network. A communication device may be provided with a service by an application server.

[0006] Such communication networks operate in according with standards such as those provided by 3GPP (Third Generation Partnership Project) or ETSI (European Telecommunications Standards Institute). Examples of standards are the so-called 5G (5th Generation) standards provided by 3GPP.

[0007] SUMMARY

[0008] Some example embodiments of this disclosure will be described with respect to certain aspects. These aspects are not intended to indicate key or essential features of the embodiments of this disclosure, nor are they intended to be used to limit the scope of thereof. Other features, aspects, and elements will be readily apparent to a person skilled in the art in view of this disclosure.

[0009] According to an aspect, there is provided an apparatus comprising: means for comparing, for a machine learning model for a user equipment, at least one of: a value of a first loss function for first data and a value of a second loss function for second data, and an accuracy of prediction using the first data and an accuracy of prediction using the second data; wherein the first data has not been processed to enhance privacy to protect an identity of the user equipment and has not been used to train the machine learning model, and wherein the second data has been used to train the machine learning model; the apparatus comprising: means for determining, based on the comparison, a level of privacy enhancement of the identity of the user equipment in the second data.

[0010] According to some examples, the first data has not been processed to anonymise the identity of the user equipment.

[0011] According to some examples, the apparatus comprises: means for sending, to the user equipment, information indicative of the level of privacy enhancement of the identity of the user equipment in the second data.

[0012] According to some examples, the apparatus comprises: means for receiving, from the user equipment, a request for auditing of privacy enhancement of the identity of the user equipment in the second data; and wherein the comparison is performed in response to receiving the request for auditing.

[0013] According to some examples, the apparatus comprises: means for sending a request to the user equipment for the second data; means for receiving the second data from the user equipment.

[0014] According to some examples, the apparatus comprises: means for determining, based on the second data, a network function or a radio access network node where the second data is used to train the machine learning model; means for sending, to the network function or the radio access network node, a request for information describing the machine learning model; means for receiving, from the network function or the radio access network node, the information describing the machine learning model; means for determining the value of the first loss function for the first data and the value of the second loss function for the second data using the information describing the machine learning model. According to some examples, the apparatus comprises: means for receiving the first data from the user equipment; means for receiving third data for at least one other user equipment, wherein the third data has not been processed to enhance privacy to protect an identity of each of the at least one user equipment; means for determining a first accuracy of prediction of the machine learning model using the first data and the third data; means for sending, to a network function where the second data is used to train the machine learning model, a request for information indicative of a second accuracy of prediction of the machine learning model, the second accuracy of prediction determined using the second data for the user equipment and for the at least one user equipment; means for receiving, from the network function, the information indicative of the second accuracy of prediction; means for comparing the first accuracy of prediction of the machine learning model with the second accuracy of prediction of the machine learning model.

[0015] According to some examples, the apparatus comprises: means for receiving, from the user equipment, information indicative of an accuracy of prediction of the machine learning model using the second data; means for receiving, from the user equipment, the first data; means for determining the accuracy of prediction of the machine learning model using the first data.

[0016] According to some examples, the apparatus comprises a user equipment, and the apparatus comprises: means for sending, to a network function, the second data; means for receiving, from the network function, information indicating the value of the second loss function for the second data; means for sending, to a network function, the first data; means for receiving, from the network function, information indicating the value of the first loss function for the first data.

[0017] According to some examples, the apparatus comprises: means for sending, to a 5G core network node, information indicative of the level of anonymization of the user equipment in the second data.

[0018] According to some examples, the first data and / or the second data comprises at least one of: temperature measurements from the user equipment during a first time period; location data for the user equipment during a second time period. According to an aspect, there is provided an apparatus comprising at least one processor and at least one memory storing instructions that, when executed by the at least one processor, cause the apparatus at least to: comparing, for a machine learning model for a user equipment, at least one of: a value of a first loss function for first data and a value of a second loss function for second data, and an accuracy of prediction using the first data and an accuracy of prediction using the second data; wherein the first data has not been processed to enhance privacy to protect an identity of the user equipment and has not been used to train the machine learning model, and wherein the second data has been used to train the machine learning model; determining, based on the comparison, a level of privacy enhancement of the identity of the user equipment in the second data.

[0019] According to some examples, the first data has not been processed to anonymise the identity of the user equipment.

[0020] According to some examples, the at least one processor may be configured to cause the apparatus to perform: sending, to the user equipment, information indicative of the level of privacy enhancement of the identity of the user equipment in the second data.

[0021] According to some examples, the at least one processor may be configured to cause the apparatus to perform: receiving, from the user equipment, a request for auditing of privacy enhancement of the identity of the user equipment in the second data; and wherein the comparison is performed in response to receiving the request for auditing.

[0022] According to some examples, the at least one processor may be configured to cause the apparatus to perform: sending a request to the user equipment for the second data; receiving the second data from the user equipment.

[0023] According to some examples, the at least one processor may be configured to cause the apparatus to perform: determining, based on the second data, a network function or a radio access network node where the second data is used to train the machine learning model; sending, to the network function or the radio access network node, a request for information describing the machine learning model; receiving, from the network function or the radio access network node, the information describing the machine learning model; determining the value of the first loss function for the first data and the value of the second loss function for the second data using the information describing the machine learning model.

[0024] According to some examples, the at least one processor may be configured to cause the apparatus to perform: receiving the first data from the user equipment; receiving third data for at least one other user equipment, wherein the third data has not been processed to enhance privacy to protect an identity of each of the at least one user equipment; determining a first accuracy of prediction of the machine learning model using the first data and the third data; sending, to a network function where the second data is used to train the machine learning model, a request for information indicative of a second accuracy of prediction of the machine learning model, the second accuracy of prediction determined using the second data for the user equipment and for the at least one user equipment; receiving, from the network function, the information indicative of the second accuracy of prediction; comparing the first accuracy of prediction of the machine learning model with the second accuracy of prediction of the machine learning model.

[0025] According to some examples, the at least one processor may be configured to cause the apparatus to perform: receiving, from the user equipment, information indicative of an accuracy of prediction of the machine learning model using the second data; receiving, from the user equipment, the first data; determining the accuracy of prediction of the machine learning model using the first data.

[0026] According to some examples, the apparatus comprises a user equipment, and the at least one processor may be configured to cause the apparatus to perform: sending, to a network function, the second data; receiving, from the network function, information indicating the value of the second loss function for the second data; sending, to a network function, the first data; receiving, from the network function, information indicating the value of the first loss function for the first data. According to some examples, the at least one processor may be configured to cause the apparatus to perform: sending, to a 5G core network node, information indicative of the level of anonymization of the user equipment in the second data.

[0027] According to some examples, the first data and / or the second data comprises at least one of: temperature measurements from the user equipment during a first time period; location data for the user equipment during a second time period.

[0028] According to an aspect, there is provided a method comprising: comparing, for a machine learning model for a user equipment, at least one of: a value of a first loss function for first data and a value of a second loss function for second data, and an accuracy of prediction using the first data and an accuracy of prediction using the second data; wherein the first data has not been processed to enhance privacy to protect an identity of the user equipment and has not been used to train the machine learning model, and wherein the second data has been used to train the machine learning model; determining, based on the comparison, a level of privacy enhancement of the identity of the user equipment in the second data.

[0029] According to some examples, the first data has not been processed to anonymise the identity of the user equipment.

[0030] According to some examples, the method comprises: sending, to the user equipment, information indicative of the level of privacy enhancement of the identity of the user equipment in the second data.

[0031] According to some examples, the method comprises: receiving, from the user equipment, a request for auditing of privacy enhancement of the identity of the user equipment in the second data; and wherein the comparison is performed in response to receiving the request for auditing.

[0032] According to some examples, the method comprises: sending a request to the user equipment for the second data; receiving the second data from the user equipment. According to some examples, the method comprises: determining, based on the second data, a network function or a radio access network node where the second data is used to train the machine learning model; sending, to the network function or the radio access network node, a request for information describing the machine learning model; receiving, from the network function or the radio access network node, the information describing the machine learning model; determining the value of the first loss function for the first data and the value of the second loss function for the second data using the information describing the machine learning model.

[0033] According to some examples, the method comprises: receiving the first data from the user equipment; receiving third data for at least one other user equipment, wherein the third data has not been processed to enhance privacy to protect an identity of each of the at least one user equipment; determining a first accuracy of prediction of the machine learning model using the first data and the third data; sending, to a network function where the second data is used to train the machine learning model, a request for information indicative of a second accuracy of prediction of the machine learning model, the second accuracy of prediction determined using the second data for the user equipment and for the at least one user equipment; receiving, from the network function, the information indicative of the second accuracy of prediction; comparing the first accuracy of prediction of the machine learning model with the second accuracy of prediction of the machine learning model.

[0034] According to some examples, the method comprises: receiving, from the user equipment, information indicative of an accuracy of prediction of the machine learning model using the second data; receiving, from the user equipment, the first data; determining the accuracy of prediction of the machine learning model using the first data.

[0035] According to some examples, method is performed by a user equipment, and the at method comprises: sending, to a network function, the second data; receiving, from the network function, information indicating the value of the second loss function for the second data; sending, to a network function, the first data; receiving, from the network function, information indicating the value of the first loss function for the first data.

[0036] According to some examples, the method comprises: sending, to a 5G core network node, information indicative of the level of anonymization of the user equipment in the second data.

[0037] According to some examples, the first data and / or the second data comprises at least one of: temperature measurements from the user equipment during a first time period; location data for the user equipment during a second time period.

[0038] According to an aspect, there is provided a computer readable medium comprising instructions which, when executed by an apparatus, cause the apparatus to perform at least the following: comparing, for a machine learning model for a user equipment, at least one of: a value of a first loss function for first data and a value of a second loss function for second data, and an accuracy of prediction using the first data and an accuracy of prediction using the second data; wherein the first data has not been processed to enhance privacy to protect an identity of the user equipment and has not been used to train the machine learning model, and wherein the second data has been used to train the machine learning model; determining, based on the comparison, a level of privacy enhancement of the identity of the user equipment in the second data.

[0039] According to an aspect, there is provided an apparatus comprising: means for storing, at the apparatus, first data and second data; means for sending, to a first network function, the second data, wherein the second data is used at the network function for training a machine learning model for the apparatus; means for sending, to a second network function, the first data, wherein the first data has not been processed to enhance privacy to protect an identity of the apparatus.

[0040] According to some examples, the apparatus comprises: the first data has not been processed to anonymise the identity of the user equipment. According to some examples, the apparatus comprises: means for receiving, from the second network function, information indicative of the level of privacy enhancement of the identity of the apparatus in the second data.

[0041] According to some examples, the apparatus comprises: means for sending, to the second network function, a request for auditing of privacy enhancement of the identity of the apparatus in the second data.

[0042] According to an aspect, there is provided an apparatus comprising at least one processor and at least one memory storing instructions that, when executed by the at least one processor, cause the apparatus at least to perform: storing, at the apparatus, first data and second data; sending, to a first network function, the second data, wherein the second data is used at the network function for training a machine learning model for the apparatus; sending, to a second network function, the first data, wherein the first data has not been processed to enhance privacy to protect an identity of the apparatus.

[0043] According to some examples, the first data has not been processed to anonymise the identity of the user equipment.

[0044] According to some examples, the at least one processor and at least one memory storing instructions that, when executed by the at least one processor, cause the apparatus at least to perform: receiving, from the second network function, information indicative of the level of privacy enhancement of the identity of the apparatus in the second data.

[0045] According to some examples, at least one processor and at least one memory storing instructions that, when executed by the at least one processor, cause the apparatus at least to perform sending, to the second network function, a request for auditing of privacy enhancement of the identity of the apparatus in the second data.

[0046] According to an aspect, there is provided a method comprising: storing, at the apparatus, first data and second data; sending, to a first network function, the second data, wherein the second data is used at the network function for training a machine learning model for the apparatus; sending, to a second network function, the first data, wherein the first data has not been processed to enhance privacy to protect an identity of the apparatus.

[0047] According to some examples, the first data has not been processed to anonymise the identity of the user equipment.

[0048] According to some examples, the method comprises: receiving, from the second network function, information indicative of the level of privacy enhancement of the identity of the apparatus in the second data.

[0049] According to some examples, the method comprises: sending, to the second network function, a request for auditing of privacy enhancement of the identity of the apparatus in the second data.

[0050] According to an aspect, there is provided a computer readable medium comprising instructions which, when executed by an apparatus, cause the apparatus to perform at least the following: storing, at the apparatus, first data and second data; sending, to a first network function, the second data, wherein the second data is used at the network function for training a machine learning model for the apparatus; sending, to a second network function, the first data, wherein the first data has not been processed to enhance privacy to protect an identity of the apparatus.

[0051] According to an aspect, there is provided a non-transitory computer readable medium comprising program instructions that, when executed by an apparatus, cause the apparatus to perform at least the method according to any of the preceding aspects.

[0052] In the above, many different embodiments have been described. It should be appreciated that further embodiments may be provided by the combination of any two or more of the embodiments described above. DESCRIPTION OF FIGURES

[0053] Some example embodiments will now be described, by way of non-limiting and illustrative example only, with reference to the accompanying Figures in which:

[0054] FIG. 1 shows a representation of a 5thgeneration communication system;

[0055] FIG. 2 shows a first example message flow for determining a privacy level of a UE at a network function;

[0056] FIG. 3 shows a second example message flow for determining a privacy level of one or more UEs at a network function;

[0057] FIG. 4 shows a third example message flow for determining a privacy level of a UE at a network function;

[0058] FIG. 5 shows an example message flow for determining a privacy level of a UE at the UE;

[0059] FIG. 6 shows an example method;

[0060] FIG. 7 shows an example method;

[0061] FIG. 8 shows a representation of an apparatus for the communication system of FIG. 1 according to some example embodiments;

[0062] FIG. 9 shows a representation of an apparatus according to some example embodiments; and

[0063] FIG. 10 shows a schematic representation of a non-volatile memory medium storing instructions which when executed by a processor allow a processor to perform one or more of the steps of the methods disclosed herein.

[0064] DETAILED DESCRIPTION

[0065] In communications networks, the network may be requested to maintain a certain level of privacy of an identity of a User Equipment (UE). The request may be sent from the UE, for example, or could be requested by another entity in the network. The request may be to preserve the privacy of the UE, at least partially, using Privacy Enhancing Technologies (PET) or by anonymizing data from the UE. Some examples described herein provide a method, system and computer program for verifying whether the privacy request for the UE is respected in the network. For example, the network may respect the request by receiving the privacy request, processing the privacy request and successful implementing the condition(s) in the privacy request.

[0066] After verifying whether the privacy request has been respected or not, the verification results can be provided to the UE. In some examples, the verification results can be provided to other network entities. This can be useful in communications networks (e.g., 5G, 6G networks) comprising software or components from different operators, and an inter-operator UE privacy verification is required (e.g., so that one operator can determine that another operator is correctly respecting UE privacy requests).

[0067] In the following various example embodiments are explained with reference to communication devices capable of communication with a communication system. Before explaining in detail the embodiments of the methods and apparatuses of the present disclosure, a 5thgeneration communication system (5GS), an access network and a core network (5GC) thereof, and communication devices are briefly explained with reference to FIG. 1.

[0068] FIG 1 shows a schematic representation of a 5G communication system (5GS). The 5GS may comprise a user equipment (UE), an access network such as a 5G radio access network (5G-RAN) or next generation radio access network (NG- RAN), a 5G core network (5GC), , and one or more application functions. An application function may be deployed in the 5GS as trusted application function or may be deployed or host on one or more application servers of the data network. Such application functions are untrusted application functions. The 5GS connects the UE to a data network the access network and the 5GC (e.g., a UPF of the 5GC).

[0069] The 5G-RAN may comprise one or more radio access nodes, such as gNodeB (GNB). A gNB may include one or more gNodeB (GNB) distributed units connected to one or more gNodeB (GNB) centralized units .

[0070] The 5GC may comprise the following network functions: Network Slice Selection Function (NSSF); Network Exposure Function; Network Repository Function (NRF); Policy Control Function (PCF); Unified Data Management (UDM); Application Function (AF); Authentication Server Function (AUSF); an Access and Mobility Management Function (AMF); and Session Management Function (SMF), and a user plane function (UPF). FIG. 1 also shows the various interfaces (N1 , N2 etc.) that may be implemented between the various elements of the system.

[0071] Some examples described herein perform auditing of UE privacy to determine whether a privacy request for a UE identity has been respected by a network or not. According to some examples (FIGS. 2 to 4), the auditing is performed at a Privacy Auditing Network Function / Application Function (PANF). The PANF may be considered to comprise a network Function (NF) or Application Function (AF). According to some examples, the PANF may be a part of the 5GC or may comprise an external AF. According to some examples (FIG. 5) the auditing is performed at a UE.

[0072] Examples perform auditing by determining whether a machine learning model used in the network has been trained using training data that respects a privacy request for a UE identity. This can be performed by comparing loss values or prediction accuracy measurements for the trained machine learning model when using the training data and when using test data. The test data has not been processed using PET or to anonymise the test data. When the training data has not been processed using PET (e.g., to anonymise the data or to remove at least some user identifiable data), when the trained ML model performs predictions using the training data and the test data, the trained ML model should perform better on the training data than on the test data. Threshold / statistical testing schemes can therefore analyse how much better the trained ML model performs making predictions on the training data than on the test data, to determine whether the training data has been processed using PET as requested.

[0073] A non-limiting example of an auditing process using a loss function can be considered as follows:

[0074] At the auditor (UE or PANF): 1. Using the N Raw test data (Xr, Yr) to infer from the Neural Network (NN) model M(-)

[0075] 1. Run the inference: M(Xr) = Yr

[0076] 2. Compute the loss: L(Yr, Yr) = lr

[0077] 2. Using the N Raw training data to infer from the NN model M(-).

[0078] 1. Run the inference: M(t) = Yt

[0079] 2. Compute the loss: L(Yt, Yt) = lt

[0080] 3. Sorting test:

[0081] 1. Merge the vectors of size N ltand lr[lt: lr]=LTotai

[0082] 2. Sort the combined vector LTotaiof size 2N Lsort

[0083] 3. If Lsort=LTotaithen no Data Privatization (DP) has been used Else: DP is fine.

[0084] The parameters above are defined as in the following:

[0085] The loss function of an NN may be determined using any know loss function. Loss function L(Yr, Yr) > 0 is any function that measure the difference between the NN’s predicted value M(r) = Yrand the labels. This difference could be in terms of e.g., absolute values, Mean Squared Errors, cross entropy, etc.

[0086] The whole trained Neural Network could be represented as a function that admits the possibly multiple dimensional tensors dataset input Xr(e.g. Network Analytics Data, User private data, images of faces, ...) and outputs its prediction possibly scalars, class numbers, or multiple dimensional tensors, which could be e.g. Anomalous or normal Network traffic, User location estimation, detection of a human from another one,... . M(r) = Yr.

[0087] In step 3 (the sorting test), the difference between the two dataset’s loss is analysed. If there is not DP, then the training dataset will result in a smaller loss (because the NN has seen the raw training data directly). On the other hand, if the two vectors are different, then the NN has not seen the raw training data directly but with addition of DP noise. Therefore, NN does not perform as well on the raw training data as when there is no DP. In other examples, instead of using a loss function, analytics indicating an accuracy of prediction for a trained ML model can be compared when using training data and test data, where the test data has not been processed using PET. By determining that the accuracy of the predictions or the trained ML model is significantly higher when using the training data than the test data, it can be determined that PET was used on the training data. In contrast, if the accuracy of predictions for the trained ML model is not significantly higher when using the training data than the test data, it can be determined that PET was not used on the training data.

[0088] FIG. 2 shows an example message flow where UE 200 sends raw data (comprising raw test data and raw training data) for the UE 200 to PANF 208, the PANF 208 retrieves the ML model trained using training data from UE 200 and then PANF 208 performs privacy auditing for the UE’s data using the retrieved ML model and comparing loss / accuracy when using the raw test data and the raw training data.

[0089] At 201a, UE 200 requests auditing from NF 208 (referred to as “PANF” herein, but may be any other suitable network function). Alternatively, at 201 b an external entity (e.g., a governmental agency) or Operations, Administration and Maintenance (OAM) requests privacy auditing for UE 200.

[0090] PANF 208 may be part of 5GC 206 . In other examples, PANF 208 may be an AF deployed by .the operator, or by an independent party.

[0091] At 203a, in response to either 201a or 201 b, PANF 208 sends a request for raw data to UE 200. The raw data comprises raw test data that has not been processed using PET. The raw data may comprise data recorded at UE 200. The raw data may also comprise raw training data that is sent to a network function for training a machine learning model (e.g., network training management function 202). The raw training data also has not been processed using PET by UE 200. The training data may have had PET applied at the network function training the mode. The raw data may comprise sensor measurements (e.g., temperature measurements) for a certain time period (e.g., 1 day) or location data / history for a certain time period (e.g., 3 days), for example. The raw test data may not be shared with a network training management function 202, and shared only with PANF 208 during auditing.

[0092] The request sent at 203a may also include a request for an expected privacy level for the raw data (e.g., low, medium, high). The privacy level may be defined according to thresholds used by the PANF during auditing analysis.

[0093] Alternatively to 203a, or when the information requested at 203a is not available at UE 200, PANF 208 may send the request to Analytics Data Repository Function (ADRF) 204. In such examples, the ADRF may provide a response (not shown in FIG. 2) with the requested information.

[0094] At 205, in response the request at 203a, UE 200 sends the requested information (raw data and optionally, the expected privacy level for UE 200).

[0095] At 207, using the raw data for UE 200, PANF 208 can determine the NFs or Radio Access Network (RAN) node where the training data for UE 200 is used. This can be performed by querying ADRF 204 or querying all NFs where an ML model could be trained.

[0096] At 209, PANF 208 sends a request to network training management function 202 (for example: 5GC Model Training Management Function such as NWDAF; a NF; a RAN node) for the specific ML model that is trained using training data from UE 200. The request may also include a request for analytics performed by the ML model.

[0097] At 211 , the network training management function 202 sends information describing the requested ML model and in some examples, the analytics performed by the ML model to PANF 208.

[0098] At 213, PANF 208 performs a privacy auditing analysis of UE 200. PANF 208 may use the received ML model and the raw training data and raw test data (the raw training and test data not having undergone any PET processing). PANF 208 can either compare loss function values determined using the ML model with the raw training data and raw test data or compare prediction accuracy determined using the ML model with the raw training data and raw test data to determine whether the ML model was trained on the raw training data or on PET-enhanced training data.

[0099] If the model was trained on the raw training data, it would be expected that the model would perform significantly better on the raw training data than on the raw test data. This would indicate that the level of privacy protection for UE 200 is low. If the model was trained on the raw training data with a relatively low amount of PET processing (a low amount of DP noise), it would be expected that the difference between the model’s performance on the raw training data and the raw test data would not be as large as if no PET processing was performed. If the model was trained on the raw training data with a relatively high amount of PET processing (a high amount of DP noise), it would be expected that the difference between the model’s performance on the raw training data and the raw test data would be even smaller. This would indicate that the level of privacy protection for UE 200 is high. Threshold levels of difference between the trained ML model’s performance (based on loss functions, prediction accuracy) for the raw training data and the raw test data can be compared to determine the level of privacy protection for UE 200.

[0100] At 215a and / or 215b PANF 208 sends the result of the auditing analysis at 213 to the requesting entity (UE 200, OAM, AF, etc.). The auditing analysis may provide an indication of whether the expected privacy level for UE 200 is met.

[0101] FIG. 3 shows an example message flow where UE 300 sends raw data (comprising raw test data and raw training data) for the UE 300 to PANF 308. PANF 308 may batch raw data over a plurality of UEs. PANF 308 then requests analytics for the batched UEs from network function 302. PANF 308 the compares loss / accuracy of the ML model using the raw test data and the raw training data based on the analytics received.

[0102] At 301 , UE 300 requests privacy auditing from PANF 308. At 303, in response to 301 , PANF 308 sends a request for raw data to UE 300. The raw data comprises raw test data that has not been processed using PET. The raw data may comprise data recorded at UE 200. The raw data may also comprise raw training data that is sent to a network function for training a machine learning model (e.g., network training management function 302). The raw training data also has not been processed using PET by UE 200. The training data may have had PET applied at the network function training the mode. The raw data may comprise sensor measurements (e.g., temperature measurements) for a certain time period (e.g., 1 day) or location data / history for a certain time period (e.g., 3 days), for example.

[0103] The request sent at 203 may also include a request for an expected privacy level for the raw data (e.g., low, medium, high). The privacy level may be defined according to thresholds used by the PANF during auditing analysis.

[0104] At 305, in response the request at 303a, UE 300 sends the requested information (raw data and optionally, the expected privacy level for UE 300).

[0105] At 307, PANF 308 receives raw data from multiple UEs for which privacy auditing can be performed for. By batching together raw data for multiple UEs, raw test data can be obfuscated during auditing, which protects a UE’s identity. Further, the accuracy of the auditing analysis is increased.

[0106] At 309, PANF 309 requests from NF 302 inference and / or analysis over the raw data from the batched UEs. At 311 , network function 302 returns analytics to PANF 308. The analytics comprises loss function results of the trained ML model for UE 300 using the raw training data and / or inference accuracy measurements of the trained ML model for UE 300 using the raw training data. The analytics also comprises loss function results of the trained ML model for UE 300 using the raw test data and / or inference accuracy measurements of the trained ML model for UE 300 using the raw test data.

[0107] At 313, PANF 308 can compare the accuracy / loss of the trained ML model using the analytics received at 311 . Based on this comparison, a level of PET processing used on the training data at network function 302 can be determined as described above.

[0108] At 315 and / or 317 PANF 308 sends the result of the auditing analysis at 213 to the requesting entity (UE 200, OAM, AF, etc.). The auditing analysis may provide an indication of whether the expected privacy level for UE 300 is met.

[0109] FIG. 4 shows an example message flow where UE 400 sends raw data (comprising raw test data and raw training data) for the UE 400 to network function 402, where the network function 402 trains an ML model for UE 400. The network function 402 sends analytics for the raw training data back to UE 400. UE 400 then shares the analytics with PANF 408 along with raw test data. PANF 408 can then determine the analytics results for the raw test data and compare with the analytics results received for the raw training data.

[0110] At 401 , UE 400 requests privacy auditing from PANF 408. PANF 408 may then optionally perform a confirmation that UE 400 is able to consume the service of the method shown in FIG. 4 (for example, by checking if UE 400 is registered for the service).

[0111] At 403, PANF 408 sends a request for analytics results and for raw data to audit a privacy level for UE 400.

[0112] At 405, UE 400 sends a request for analytics to network function 402. The request comprises the raw data (raw training data and optionally raw test data).

[0113] Network function 402 then determines analytics (e.g., loss / inference accuracy) of the raw training data using the trained ML model. At 407, the analytics results are sent to UE 400.

[0114] At 409, the analytics results corresponding to the raw training data and the raw data (including the raw test data and optionally the raw training data) is sent to PANF 408. If a loss function is used by network function 402 to determine the analytics results, this loss function may be included in the analytics results. At 411 , PANF 408 determines analytics results for the raw test data corresponding to the analytics results received for the raw training data. For examples, if a loss function is determined for the raw training data, a loss function can be determined for the raw test data. If accuracy of inference is determined using another metric for the raw training data, the same metric can be determined for the raw test data. The accuracy / loss for the ML model on the raw training data and the raw test data can then be compared to determine whether privacy for the identify of UE 400 is respected or not by NF 402.

[0115] At 412 and / or 413 PANF 408 sends the result of the auditing analysis at 411 to the requesting entity (UE 200, 0AM, AF, etc.). The auditing analysis may provide an indication of whether the expected privacy level for UE 400 is met.

[0116] FIG. 5 shows an example where auditing is performed at UE 500.

[0117] At 501 , UE 500 sends raw training data to network function 502. At 503, network function 502 sends analytics results returns for the raw training data. The analytics results may be determined based on a trained ML model and the raw training data. The analytics results may comprise loss values and / or inference accuracy values.

[0118] At 505, UE 500 sends raw test data to network function 502. The raw test data is not in the raw training data. At 507, network function 502 sends analytics results returns for the raw test data. The analytics results may be determined based on a trained ML model and the raw test data. The analytics results may comprise loss values and / or inference accuracy values.

[0119] At 509, UE 500 compares the analytics results for the raw test data and the raw training data. Based on the comparison, the privacy of the UE can be estimated as described above. At 511 , the privacy results may be shared to the core network.

[0120] FIG. 6 shows an example method flow. The method may be performed, for example, by a network function (e.g., PANF 208, 308, 408) or UE (e.g., UE 500). At 600, the method comprises comparing, for a machine learning model for a user equipment, at least one of: a value of a first loss function for first data and a value of a second loss function for second data, and an accuracy of prediction using the first data and an accuracy of prediction using the second data. The first data may not have been processed to enhance privacy to protect an identity of the user equipment and has not been used to train the machine learning model, and the second data may have been used to train the machine learning mode.

[0121] At 602, the method comprises determining, based on the comparison, a level of privacy enhancement of the identity of the user equipment in the second data

[0122] FIG. 7 shows an example method flow. The method may be performed, for example, by a UE such as UE 200, UE 300, UE 400.

[0123] At 700, the method comprises storing, at a user equipment, first data and second data.

[0124] At 702, the method comprises sending, to a first network function, the second data, wherein the second data is used at the network function for training a machine learning model for the user equipment.

[0125] At 704, the method comprises sending, to a second network function, the first data, wherein the first data has not been processed to enhance privacy to protect an identity of the user equipment.

[0126] FIG. 8 illustrates an example of an apparatus 800 for one or more network functions illustrated in FIG. 1-5. . The apparatus 800 may comprise at least one random access memory (RAM) 811a, at least on read only memory (ROM) 811 b, at least one processor 812, 813 and a network interface 814. The at least one processor 812, 813 may be coupled to the RAM 811a and the ROM 811b. The at least one processor 812, 813 may be configured to execute an appropriate software code 815. Execution of the software code 815 may for example may cause the apparatus to perform operations for one or more network functions. The software code 815 may be stored in the ROM 811 b. The apparatus 800 may be interconnected with another apparatus 800 for controlling other network functions of the 5GC. In some embodiments, one or more network functions of the 5GC is deployed or hosted on an apparatus 800. In alternative embodiments, two or more functions of the 5GC may share an apparatus 800.

[0127] FIG. 9 illustrates an example of a communication device 900, such as the terminal illustrated on FIG. 1. The communication device 900 may be provided by any device capable of sending and receiving radio signals. Non-limiting examples of a communication device 900 comprise a user equipment, a mobile station (MS) or mobile device such as a mobile phone or what is known as a ’smart phone’, a computer provided with a wireless interface card or other wireless interface facility (e.g., USB dongle), a personal data assistant (PDA) or a tablet provided with wireless communication capabilities, a machine-type communications (MTC) device, an Internet of things (loT) type communication device or any combinations of these or the like. The communication device 900 may comprise a transceiver for transmitting and / or receiving, for example, wireless signals carrying communications, for example radio signals. The communications may be one or more of voice, electronic mail (email), text messages, multimedia data, machine data and so on.

[0128] The communication device 900 may receive wireless signals (e.g., radio signals) over an air or radio interface 907 via appropriate apparatus for receiving and may transmit wireless signals via appropriate apparatus for transmitting radio signals. In FIG. 9 transceiver is designated schematically by block 906. The transceiver 906 may comprise, for example, a radio part and associated antenna arrangement. The antenna arrangement may be arranged internally or externally to the mobile device and may comprise one or more antenna elements. The antenna arrangement may be a multi-input multi output (MIMO) antenna.

[0129] The communication device 900 may be provided with at least one processor 901 , at least one memory ROM 902a, at least one RAM 902b and other possible components 903 for use in software and hardware aided execution of tasks it is designed to perform, including control of access to and communications with access networks (e.g., the 5G-RAN or NG-RAN illustrated in FIG. 1) and other communication devices. The at least one processor 901 is coupled to the RAM 902b and the ROM 902a. The at least one processor 901 may be configured to execute an appropriate software code 908. The software code 908 may for example allow to perform one or more operations of the communication device. The software code 908 may be stored in the ROM 902a.

[0130] The processor, the ROM, and the RAM, the transceiver and other circuitry of the communication device (e.g., a modem) can be provided on a circuit board, in chipsets, or in a system on chip. The circuit board, chipsets or system on chip is denoted by reference 904. The communication device 900 may optionally have a user interface such as key pad 905, touch sensitive screen or pad, combinations thereof or the like. Optionally one or more of a display, a speaker and a microphone may be provided depending on the type of communication device.

[0131] FIG. 10 shows a schematic representation of non-volatile memory media 1000a (e.g. computer disc (CD) or digital versatile disc (DVD)) and 1000b (e.g. universal serial bus (USB) memory stick) storing instructions and / or parameters 1002 which when executed by a processor allow the processor to perform one or more of the steps of any method flow described herein.

[0132] It is understood that references in the above to various network functions (e.g., to an AMF, an SMF, TNF etc.) may comprise apparatus that perform at least some of the functionality associated with those network functions. Further, an apparatus comprising a network function may comprise a virtual network function instance of that network function.

[0133] It should be understood that the apparatuses may comprise or be coupled to other units or modules etc., such as radio parts or radio heads, used in or for transmission and / or reception. Although the apparatuses have been described as one entity, different modules and memory may be implemented in one or more physical or logical entities.

[0134] It is noted that whilst some embodiments have been described in relation to 5G networks, similar principles can be applied in relation to other networks and communication systems. Therefore, although certain embodiments were described above by way of example with reference to certain example architectures for wireless networks, technologies and standards, embodiments may be applied to any other suitable forms of communication systems than those illustrated and described herein.

[0135] It is also noted herein that while the above describes example embodiments, there are several variations and modifications which may be made to the disclosed solution without departing from the scope of the present invention.

[0136] As used herein, “at least one of the following: ” and “at least one of ” and similar wording, where the list of two or more elements are joined by “and” or “or”, mean at least any one of the elements, or at least any two or more of the elements, or at least all the elements.

[0137] In general, the various embodiments may be implemented in hardware or special purpose circuitry, software, logic or any combination thereof. Some aspects of the disclosure may be implemented in hardware, while other aspects may be implemented in firmware or software which may be executed by a controller, microprocessor or other computing device, although the disclosure is not limited thereto. While various aspects of the disclosure may be illustrated and described as block diagrams, flow charts, or using some other pictorial representation, it is well understood that these blocks, apparatus, systems, techniques or methods described herein may be implemented in, as non-limiting examples, hardware, software, firmware, special purpose circuits or logic, general purpose hardware or controller or other computing devices, or some combination thereof.

[0138] As used herein, the term “circuitry” may refer to one or more or all of the following:

[0139] (a) hardware-only circuit implementations (such as implementations in only analog and / or digital circuitry) and

[0140] (b) combinations of hardware circuits and software, such as (as applicable):

[0141] (i) a combination of analog and / or digital hardware circuit(s) with software / firmware and (ii) any portions of hardware processor(s) with software (including digital signal processor(s)), software, and memory(ies) that work together to cause an apparatus, such as a mobile phone or server, to perform various functions) and

[0142] (c) hardware circuit(s) and or processor(s), such as a microprocessor(s) or a portion of a microprocessor(s), that requires software (e.g., firmware) for operation, but the software may not be present when it is not needed for operation.”

[0143] This definition of circuitry applies to all uses of the term “means” herein, including in any claims. As a further example, as used herein, the term circuitry also covers an implementation of merely a hardware circuit or processor (or multiple processors) or portion of a hardware circuit or processor and its (or their) accompanying software and / or firmware. The term circuitry also covers, for example and if applicable to the particular claim element, a baseband integrated circuit or processor integrated circuit for a mobile device or a similar integrated circuit in server, a cellular network device, or other computing or network device.

[0144] The embodiments of this disclosure may be implemented by computer software executable by a data processor of the mobile device, such as in the processor entity, or by hardware, or by a combination of software and hardware. Computer software or program, also called program product, including software routines, applets and / or macros, may be stored in any apparatus-readable data storage medium and they comprise program instructions to perform particular tasks. A computer program product may comprise one or more computer-executable components which, when the program is run, are configured to carry out embodiments. The one or more computer-executable components may be at least one software code or portions of it.

[0145] Further in this regard it should be noted that any blocks of the logic flow as in the Figures may represent program steps, or interconnected logic circuits, blocks and functions, or a combination of program steps and logic circuits, blocks and functions. The software may be stored on such physical media as memory chips, or memory blocks implemented within the processor, magnetic media such as hard disk or floppy disks, and optical media such as for example DVD and the data variants thereof, CD. The physical media is a non-transitory media. The term “non-transitory,” as used herein, is a limitation of the medium itself (i.e. , tangible, not a signal ) as opposed to a limitation on data storage persistency (e.g., RAM vs. ROM).

[0146] The memory may be of any type suitable to the local technical environment and may be implemented using any suitable data storage technology, such as semiconductor based memory devices, magnetic memory devices and systems, optical memory devices and systems, fixed memory and removable memory. The data processors may be of any type suitable to the local technical environment, and may comprise one or more of general purpose computers, special purpose computers, microprocessors, digital signal processors (DSPs), application specific integrated circuits (ASIC), FPGA, gate level circuits and processors based on multi core processor architecture, as non-limiting examples.

[0147] Various example embodiments of the disclosure may be practiced in various components such as integrated circuit modules. The design of integrated circuits is by and large a highly automated process. Complex and powerful software tools are available for converting a logic level design into a semiconductor circuit design ready to be etched and formed on a semiconductor substrate.

[0148] The scope of protection sought for various example embodiments of the disclosure is set out by the independent claims. The example embodiments and features thereof, if any, described in this disclosure that do not fall under the scope of the independent claims are to be interpreted as examples useful for understanding various example embodiments of the disclosure.

[0149] The foregoing description has provided, by way of non-limiting and illustrative examples, a full and informative description of the various example embodiments of this disclosure. However, various modifications and adaptations may become apparent to those skilled in the relevant arts in view of the foregoing description, when read in conjunction with the accompanying drawings and the claims. However, all such and similar modifications of the teachings will still fall within the various example embodiments of the disclosure as set forth in the claims. By way of non-limiting and illustrative example, there is a further example embodiment comprising a combination of one or more example embodiments with any of the other example embodiments previously discussed.

Claims

Claims:

1. An apparatus comprising: means for comparing, for a machine learning model for a user equipment, at least one of: a value of a first loss function for first data and a value of a second loss function for second data, and an accuracy of prediction using the first data and an accuracy of prediction using the second data; wherein the first data has not been processed to enhance privacy to protect an identity of the user equipment and has not been used to train the machine learning model, and wherein the second data has been used to train the machine learning model; the apparatus comprising: means for determining, based on the comparison, a level of privacy enhancement of the identity of the user equipment in the second data.

2. An apparatus according to claim 1 , wherein the first data has not been processed to anonymise the identity of the user equipment.

3. An apparatus according to claim 1 or claim 2, the apparatus comprising: means for sending, to the user equipment, information indicative of the level of privacy enhancement of the identity of the user equipment in the second data.

4. An apparatus according to any preceding claim, the apparatus comprising: means for receiving, from the user equipment, a request for auditing of privacy enhancement of the identity of the user equipment in the second data; and wherein the comparison is performed in response to receiving the request for auditing.

5. An apparatus according to any preceding claim, the apparatus comprising: means for sending a request to the user equipment for the second data;means for receiving the second data from the user equipment.

6. An apparatus according to claim 1 , the apparatus comprising: means for determining, based on the second data, a network function or a radio access network node where the second data is used to train the machine learning model; means for sending, to the network function or the radio access network node, a request for information describing the machine learning model; means for receiving, from the network function or the radio access network node, the information describing the machine learning model; means for determining the value of the first loss function for the first data and the value of the second loss function for the second data using the information describing the machine learning model.

7. An apparatus according to any of claims 1 to 4, the apparatus comprising: means for receiving the first data from the user equipment; means for receiving third data for at least one other user equipment, wherein the third data has not been processed to enhance privacy to protect an identity of each of the at least one user equipment; means for determining a first accuracy of prediction of the machine learning model using the first data and the third data; means for sending, to a network function where the second data is used to train the machine learning model, a request for information indicative of a second accuracy of prediction of the machine learning model, the second accuracy of prediction determined using the second data for the user equipment and for the at least one user equipment; means for receiving, from the network function, the information indicative of the second accuracy of prediction; means for comparing the first accuracy of prediction of the machine learning model with the second accuracy of prediction of the machine learning model.

8. An apparatus according to claims 1 to 4, the apparatus comprising: means for receiving, from the user equipment, information indicative of an accuracy of prediction of the machine learning model using the second data; means for receiving, from the user equipment, the first data; means for determining the accuracy of prediction of the machine learning model using the first data.

9. An apparatus according to claim 1 , wherein the apparatus comprises a user equipment, the apparatus comprising: means for sending, to a network function, the second data; means for receiving, from the network function, information indicating the value of the second loss function for the second data; means for sending, to a network function, the first data; means for receiving, from the network function, information indicating the value of the first loss function for the first data.

10. An apparatus according to claim 9, the apparatus comprising: means for sending, to a 5G core network node, information indicative of the level of anonymization of the user equipment in the second data.

11. An apparatus according to any preceding claim, wherein the first data and / or the second data comprises at least one of: temperature measurements from the user equipment during a first time period; location data for the user equipment during a second time period.

12. A method comprising: comparing, for a machine learning model for a user equipment, at least one of: a value of a first loss function for first data and a value of a second loss function for second data, andan accuracy of prediction using the first data and an accuracy of prediction using the second data; wherein the first data has not been processed to enhance privacy to protect an identity of the user equipment and has not been used to train the machine learning model, and wherein the second data has been used to train the machine learning model; the method comprising: determining, based on the comparison, a level of privacy enhancement of the identity of the user equipment in the second data.

13. A computer program comprising instructions stored thereon for performing at least the following: comparing, for a machine learning model for a user equipment, at least one of: a value of a first loss function for first data and a value of a second loss function for second data, and an accuracy of prediction using the first data and an accuracy of prediction using the second data; wherein the first data has not been processed to enhance privacy to protect an identity of the user equipment and has not been used to train the machine learning model, and wherein the second data has been used to train the machine learning model; and determining, based on the comparison, a level of privacy enhancement of the identity of the user equipment in the second data.

14. An apparatus comprising: means for storing, at the apparatus, first data and second data; means for sending, to a first network function, the second data, wherein the second data is used at the network function for training a machine learning model for the apparatus; means for sending, to a second network function, the first data, wherein the first data has not been processed to enhance privacy to protect an identity of the apparatus.

15. An apparatus according to claim 14, wherein the first data has not been processed to anonymise the identity of the user equipment.

16. An apparatus according to claim 14 or claim 15, the apparatus comprising: means for receiving, from the second network function, information indicative of the level of privacy enhancement of the identity of the apparatus in the second data.

17. An apparatus according to any of claims 14 to 16, the apparatus comprising: means for sending, to the second network function, a request for auditing of privacy enhancement of the identity of the apparatus in the second data.

18. A method comprising: storing, at a user equipment, first data and second data; sending, to a first network function, the second data, wherein the second data is used at the network function for training a machine learning model for the user equipment; sending, to a second network function, the first data, wherein the first data has not been processed to enhance privacy to protect an identity of the user equipment.

19. A computer program comprising instructions stored thereon for performing at least the following: storing, at a user equipment, first data and second data; sending, to a first network function, the second data, wherein the second data is used at the network function for training a machine learning model for the user equipment;sending, to a second network function, the first data, wherein the first data has not been processed to enhance privacy to protect an identity of the user equipment.

Images