Defending against patient re-identification attacks

Abstract

Disclosed is a method for de-identification of an image of a specific type indicating a type of imaged objects. The method comprises providing a first machine learning model to perform an object identification task and a second machine learning model to perform an image analysis task, using the first machine learning model for determining for the image a first rate of change of a first loss related to the first machine learning model; using the second machine learning model for determining for the image a second rate of change of an output of the second machine learning model; using the first rate and the second rate for determining a perturbation of the image such that a change in an output of the second machine learning model between the image and the image perturbed with the perturbation is mitigated; modifying the image using the perturbation.

Description

[0001] DEFENDING AGAINST PATIENT RE-IDENTIFICATION ATTACKS

[0002] FIELD OF THE INVENTION

[0003] The invention relates to a system for processing of data, to a related arrangement and method, to a computer program element, and to a computer readable medium.

[0004] BACKGROUND OF THE INVENTION

[0005] Person re-identification in camera images or video streams is an active field of research with many applications, specifically in the security domain.

[0006] Recently, such techniques have been successfully employed to chest X-Ray images, such as reported by K Packhaus er et al in "Deep learning-based patient re-identification is able to exploit the biometric nature of medical chest X-ray data", published in NATURE, scientific reports. Vol 12: 14851, 2022. The authors utilized a Siamese Neural Network (SNN) with Contrastive Loss to generate a lowdimensional mapping of the input data. Using this setup, the network learns to map images from the same patient to nearby points in the low-dimensional space, while images from different patients are mapped to distant points. Using different datasets, they were able to consistently achieve very high re-identification scores (Precision@l between 0.882 to 0.996).

[0007] A more detailed analysis further indicates a strong robustness with respect to the presence of abnormalities and the time span between the acquisitions. These results indicate that common deidentification techniques for medical image (covering Digital Imaging and Communications in Medicine (DICOM) tags or burned in annotations), might not be sufficient to protect the identity of a patient.

[0008] SUMMARY OF THE INVENTION

[0009] There may therefore be a need for improved methods and systems for defending against re-identification attacks, in particular in respect of personal data, such as medical or other.

[0010] An object of the present invention is achieved by the subject matter of the independent claims where further embodiments are incorporated in the dependent claims. It should be noted that the following described aspect of the invention equally applies to the related arrangement and method, to the computer program element and to the computer readable medium.

[0011] Example embodiments provide a method for de -identification of an image (referred to as target image) of a specific type, the specific type indicating a type of imaged objects, the method comprising: providing a first machine learning model to perform an object identification task and a second machine learning model to perform an image analysis task, the first machine learning model having been trained on images of the specific type, the second machine learning model having been trained on images of the specific type; using the first machine learning model for determining a first rate of change of a metric loss with respect to the target image; using the second machine learning model for determining a second rate of change of an output of the second machine learning model with respect to the target image; using the first rate and the second rate for determining a perturbation of the target image such that a change in an output of the second machine learning model, between the target image and the target image perturbed with the perturbation, is mitigated; modifying the target image using the perturbation.

[0012] Example embodiments provide a computer program product comprising a computer readable medium, the computer readable medium having computer readable code embodied therein, the computer readable code being configured such that, on execution by a suitable computer or processor, the computer or processor is caused to perform the method.

[0013] Example embodiments provide an apparatus comprising at least one processor; and at least one memory storing instructions that, when executed by the at least one processor, cause the apparatus at least to perform the method.

[0014] BRIEF DESCRIPTION OF THE DRAWINGS

[0015] Examples of the invention will now be described with reference to the following drawings, which, unless stated otherwise, are not to scale, wherein:

[0016] Fig. 1 is a flowchart of a method for de-identification of an image according to one example of the present subject matter;

[0017] Fig. 2 shows a block diagram of a data processing system;

[0018] Fig. 3 shows a block diagram of a data processing component configured for defending against a re-identification attack;

[0019] Fig. 4 illustrates operation of the re-identification defender component in Fig. 3 according to one example of the present subject matter; and

[0020] Fig. 5 shows a flowchart of a computer implemented method for defending against reidentification attacks.

[0021] DETAILED DESCRIPTION

[0022] “First,” “Second,” etc. as used herein, these terms are used as labels for nouns that they precede, and do not imply any type of ordering (e.g., spatial, temporal, logical) unless explicitly defined as such.

[0023] The first machine learning model may be configured to perform an object identification task. The first machine learning model may have been trained on images of the specific type to minimize a first loss function of the first machine learning model. The first machine learning model may either be a third-party model with a known or assumed loss function or may be a proprietary model specifically developed for this purpose.

[0024] The first machine learning model may be configured for object identification to recognize and classify objects within images of the specific type. This first machine learning model may have undergone a training process. During training, the first machine learning model may have been exposed to a dataset of images, all of the specific type relevant to the object identification task (e.g., images of cars, animals, medical scans, etc.). The training process involved feeding these images into the first machine learning model and adjusting the first machine learning model's parameters to improve its accuracy. This adjustment may be guided by the first loss function. The first loss function may indicate the error for a specific prediction of the first machine learning model.

[0025] The training of the first machine learning model, along with the choice of its first loss function, may depend on the specific type of object identification task the first machine learning model is designed to perform.

[0026] In one example, the object identification task may be a re-identification (re-ID) task. The first loss function may, for example, comprise a metric-based loss like Triplet Loss and Contrastive Loss, which may aim to minimize the distance between similar objects while maximizing it between different ones. For example, in each training iteration, a pair of images may be fed through the first machine learning model to generate their respective feature embeddings, and the first loss function may be computed using the feature embeddings. The pair of images may include the image of the specific type and one or more reference images of the object. In the context of re-identification tasks, the reference image may correspond to a probe image, while the target image may correspond to a query or gallery image. The goal of training may, for example, be to minimize the first loss function, which may mean that the first machine learning model learns to make more accurate predictions over time.

[0027] In another example, the object identification task may involve a standard object identification, where the objective is to assign a class label to the imaged object. In this scenario, the first loss function may, for example, be a Cross-Entropy Loss. For example, in each training iteration an image is fed through the first machine leaning model to generate its feature embedding, and the first loss function is computed based on how well the first machine learning model distinguishes between different object classes, without the need for paired images. The first machine learning model may learn to correctly identify and classify objects based on their unique features. The goal of training may, for example, be to minimize the first loss function, which may mean that the first machine learning model learns to make more accurate predictions over time.

[0028] The second machine learning model may be configured to carry out the image analysis task, which may involve various objectives such as detecting patterns, segmenting parts of an image, extracting features, or analyzing the content of images in more detail. The second machine learning model is not configured for re-identification purpose, that is, the medical image analysis task is not a reidentification task. The second machine learning model may have been trained using a collection of images that are of the specific type relevant to the image analysis task (e.g., medical images like X-rays, satellite images, images of handwritten text, etc.). The training process for the second machine learning model may involve inputting these types of images into the second machine learning model and iteratively adjusting its parameters to improve its performance on the image analysis task. This adjustment may be guided by a second loss function, which may represent the difference between the second machine model's output and the expected or correct output (such as labels, masks, or other forms of annotations depending on the image analysis task). By minimizing the second loss function, the second machine learning model may become proficient at the specific image analysis task.

[0029] The "type of imaged objects" may refer to the specific category of items captured in images e.g., which may be used for training the first and second machine learning models. This could include various domains, such as medical images (e.g., chest X-rays or brain MRIs), natural objects (e.g., animals or plants), manufactured items (e.g., vehicles or buildings), human faces, or microscopic subjects like cells. The first and second machine learning models may have been trained using images of the specific type. For example, the type of imaged objects may be chest medical images. In this case, the first and second machine learning models may have been trained using medical images of the chests of different patients. In another example, the type of imaged objects may be brain medical images. In this case, the first and second machine learning models may have been trained using medical images of the brains of different patients.

[0030] The first machine learning model may be used for determining for the target image a first rate of change of the metric loss. The metric loss may, for example, be a loss function. The metric loss may or may not be the first loss function. In one example, the metric loss may be used to measure the similarity or dissimilarity between the output of the first machine learning model for the target image and a reference. The reference would vary depending on the specific object identification task that the first machine learning model is designed to perform. For a re-identification task, the reference might be the output of the first machine learning model for a reference image of the same object depicted in the target image. For a standard identification task, the reference may be a label or ground truth definition of the object in the target image. The metric loss may refer to distance-based loss, which uses measures like Euclidean distance between embeddings to assess similarity or dissimilarity; embedding loss, which is applied to the learned feature vectors or embeddings of objects etc. In one example, the metric loss may be the first loss function.

[0031] The first rate may measure how the metric loss changes as elements of the target image change. The first rate of change of the metric loss may, for example, be with respect to the target image. The first rate of change of the metric loss with respect to the target image may be referred to as the first rate. The first rate may be calculated based on the type of object identification task, whether it is for reidentification or standard identification. The first rate may, for example, be computed by using numerical techniques, with or without discretization. For example, the target image, and optionally the reference image depending on the object identification task type, may be input into the first machine learning model. In response, the first machine learning model may generate an output. The metric loss may be calculated using the output. If the first machine learning model is a neural network type model, the gradient of the metric loss with respect to the target image may be computed by backpropagating through the network e.g., using the chain rule. The gradient may be an example implementation of the first rate of change of the metric loss. In another example, given the first machine learning model, the derivative of the metric loss with respect to the target image may be modeled or defined analytically. This modeled derivative may be evaluated by using the target image to obtain the first rate of change of the metric loss.

[0032] The second machine learning model may be used for determining for the target image a second rate of change of an output of the second machine learning model. The second rate of change of the output of the second machine learning model may, for example, be with respect to the target image. The second rate of change of the output of the second machine learning model may be referred to as the second rate. The second rate may measure how the output changes as elements of the target image change. Indeed, although trained with the second loss function based on comparisons between the predicted outputs and the correct outputs, the present subject matter may use, for the image deidentification, the second rate of change of the output alone with respect to the target image to find the direction in which to modify the target image e.g., to mitigate large deviations in the output or maintain the output unchanged. The second rate may, for example, be computed by using numerical techniques, with or without discretization. For example, the target image may be provided as input to the second machine learning model. In response, the second machine learning model may generate an output. If the second machine learning model is a neural network type model, the gradient of the output with respect to the target image may be calculated by backpropagating through the network e.g., using the chain rule. The gradient may be an example implementation of the second rate of change of the output. In another example, given the second machine learning model, the derivative of the second machine learning model’s output with respect to the target image may be modeled or defined analytically. This modeled derivative may be evaluated by using the target image to obtain the second rate of change of the output.

[0033] The first rate of change of the metric loss with respect to the target image may indicate for each element of the target image a perturbation value in the element that can foul the first machine learning model into making incorrect prediction different from the output obtained for the target image. For example, if the first rate with respect to a specific element of the target image is positive, increasing the element’s value may may increase the metric loss. Conversely, if the first rate with respect to a specific element of the target image is negative, decreasing the element’s value may may increase the metric loss. For example, if I is the target image and Iadvis the target image after the perturbation is applied to it, the first machine learning model may make incorrect prediction means that the prediction of the first machine learning model for the target image I is different from the prediction of the first machine learning model for the perturbed image ladv. The second rate may indicate for each element of the target image how much a change in that element would affect the present output obtained for the target image by the second machine learning model. For example, a positive value of the second rate in the element may mean that increasing the value of that element may increase the second machine learning model’s confidence in the output obtained for the target image thereby maintaining the output unchanged. Conversely, a negative value may mean that increasing the value of that element may decrease the second machine learning model's confidence in the output.

[0034] Hence, the present subject matter may make use of these features of the first and second rates to perform a reliable de -identification of the target image. For example, the modified target image may be provided e.g., stored, so that it may be used for the image analysis task by the second machine learning model, whilst the modified target image still capable of defending against a re-identification ("re- ID ") attack. For example, after determining the first rate and the second rate, an operation may be performed, which may be referred to as perturbation generation operation. The perturbation may, for example, comprise a perturbation value per element of the target image. The perturbation generation operation may comprise using the first rate and the second rate for determining a perturbation of the target image, wherein the perturbation is determined such that a change in an output of the second machine learning model between the target image and the target image perturbed with the perturbation is mitigated. That is, the difference between a first output of the second machine learning model for the target image and a second output of the second machine learning model for the target image, perturbed with the perturbation, is mitigated. For example, the difference between the first output of the second machine learning model for the target image and the second output of the second machine learning model for the target image, perturbed with the perturbation, is mitigated under a certain maximum difference threshold. For example, if I is the target image and Iadvis the target image after the perturbation is applied to it, then the perturbation may be determined such that the difference | C (Z) — C (adv) | is smaller than the maximum difference threshold, where C represents the second machine learning model. This example may allow to control the response of the second machine learning model to perturbations, while the impact on the first machine learning model may remain uncertain. This may be based on the assumption that the first machine learning model may inevitably be affected by perturbations. Indeed, models for object identification tasks such as re-identification tasks may be ill-conditioned, so a small modification is enough to compromise their results. This assumption can save resources that would otherwise be needed for multi -objective optimization, while also producing a second machine learning model that may remain unaffected by the generated perturbations. Additionally, it provides a de-identified target image with a higher likelihood of not being recognized by the first machine learning model compared to the unmodified target image.

[0035] In one example implementation of the perturbation generation operation, the perturbation may be determined such that change in an output of the second machine learning model between the target image and the target image perturbed with the perturbation is mitigated and that the metric loss is increased. Increasing the metric loss may enable to foul the first machine learning model into making incorrect prediction for the perturbed target image compared to the prediction for the target image. For example, if I is the target image and Iadvis the target image after the perturbation of this example is applied to it, the metric loss may be increased so that the first machine learning model may make incorrect prediction e.g., so that the prediction of the first machine learning model for the target image I is different from the prediction of the first machine learning model for the perturbed image Iadv.

[0036] In one example implementation of the perturbation generation operation, the first rate and the second rate may be used for determining a perturbation of the target image such that an output of the second machine learning model between the target image and the target image perturbed with the perturbation is unchanged. Following the above notation, according to this example, the perturbation may be determined such that the difference is zero, | C(T)~ C(adv' | = 0 or substantially zero.

[0037] In one example implementation of the perturbation generation operation, a first individual perturbation comprising first changes for the elements of the target image respectively may be determined using the first rate. The first individual perturbation may, for example, be the first rate scaled by a scale factor. The second rate may be used for determining a second individual perturbation. The second individual perturbation comprises second changes for the elements of the target image respectively. The second individual perturbation may, for example, be the second rate scaled by the scale factor. The determined perturbation may be a combination of the first individual perturbation and the second individual perturbation. The combined perturbation may, for example, be obtained such that the difference between the first and second individual perturbations is maximized. This may, for example, be obtained by using a technique such as vector projection. Alternatively, the first individual perturbation and the second individual perturbation may form the determined perturbation, wherein the target image is modified by applying the first individual perturbation and the second individual perturbation to respective parts of the target image. For example, the first individual perturbation may be applied to parts which are not relevant for both the object identification task and the image analysis task while the second individual perturbation may be applied to parts which are relevant for the object identification task but not relevant for the image analysis task.

[0038] According to one example implementation of the perturbation generation operation, the first rate is a first vector whose elements represent the target image. The second rate is a second vector whose elements represent the target image. The perturbation is a vector rejection of the first vector from the second vector, wherein the vector rejection is scaled by a scale factor. The elements of the first vector may correspond with elements of the target image respectively. The elements of the second vector may correspond with elements of the target image respectively.

[0039] According to one example implementation of the perturbation generation operation, the perturbation may be defined as follows for the first machine learning model being a re-ID model. First, an individual perturbation VPto alter the target image I may be provided such that the perturbed target image Iadvmay be obtained as follows: Iadv= I + e ■ VP. where VP= sign where D is the metric loss providing a similarity measure between embeddings of the outputs which are provided by the first machine learning model in response to receiving a probe image P and target image / , respectively, and sign( ) denotes the signum function. The probe image P and target image I may be images of an object such as a patient e.g., from different perspectives. The target image I may be an array, while ' + ' refers to an array operation. The perturbed target image may be used to disturb the response of the first machine learning model e.g., being for example a deep learning model for re-identification. However, even though the perturbed target image is designed to disturb the response of first machine learning model, it is unclear to which extent the changes would affect the second machine learning model when using that perturbed image, e.g., for pathology classification. The present example may solve this as follows. Given the second machine learning model as a deep learning model for image classification C(7), the aim may be a similar response is obtained by the second machine learning model for the perturbed target image ladvand the target image I. Therefore, the gradient of the metric loss D with respect to the target image I may not be used directly. Rather, the projection onto a vector that is orthogonal to the gradient of the image classification C with respect to the target image I may be used. Since the gradient of the image classification C with respect to the target image I is orthogonal to the level set of the classification function C, such a change may minimize the effect of the perturbation e • ( VPwhich is used to modify the target image I as follows: Iadv= I + e • (vP K where VPis the perturbation towards the image P, K represents the gradient of the classification function of the target image I, K = e is a scale factor, T is a transpose operator and VP-

[0040] K1K K may be a vector rejection of the first vector VPfrom the second vector K.

[0041] According to one example implementation of the perturbation generation operation, the perturbation comprises a first perturbation that increases the metric loss and a second perturbation, wherein the second perturbation is determined such that a change in an output of the second machine learning model, between the target image and the target image perturbed with the second perturbation, is mitigated, wherein the modification comprises: using the first perturbation to modify a first part of the target image and using the second perturbation to modify a second part of the target image. The first part may, for example, be not relevant for both the object identification task and the image analysis task while the second part may be relevant for the object identification task but not relevant for the image analysis task.

[0042] The modification of the target image using the perturbation may involve altering every element of the target image with the respective value of the perturbation. Alternatively, the modification may be applied selectively to specific elements or regions of the target image, depending on the desired outcome or level of precision required. For that, the perturbation values that correspond with the selected elements or regions may be used to modify those selected elements or regions respectively.

[0043] The target image may be an image of an object whose type is indicated by the specific type. The target image may comprise image data such as image pixel values. According to one example, the target image is provided in a format representing the image data. The format may allow the image data to be input into the first machine learning model and / or second machine learning model. For example, the specific type of the target image may further indicate the format of the target image. That is, the training and the inference of the first and second machine learning models use images in the same format. In one example, the format may be an array. The array may be a multi-dimensional array or onedimensional array. Regardless of the array's dimensions, the array may represent the target image and reflect the spatial structure of the pixels within that target image.

[0044] In one array generation example, the elements of the array may be obtained from the target image. For example, the target image, which may originally be structured as a grid of pixels, may be provided as a muti -dimensional array. Each element of this array may be the value of a pixel or a set of values in the case of color images. In this example, the array may comprise the target image.

[0045] In another array generation example, the array may be obtained from the target image by pre-processing the target image to obtain a feature vector representing the target image. That is, the array of the target image may be the feature vector. The feature vector may, for example, be obtained using a machine learning based encoder such as an auto-encoder "(AE"), or a variational encoder ("VAE"). In this example, the array is said to represent the target image.

[0046] In another example, the target image may comprise image data and metadata. According to one example, the target image is provided in a format representing image data and metadata descriptive of the imaged object and / or the target image. For example, the specific type of the target image may further indicate the format of the target image. That is, the training and the inference of the first and second machine learning models use the images in the same format. For example, the first and second loss functions may consider the image data and / or the metadata. This may provide a comprehensive representation of the target image in a machine learning context. The image data and the metadata may be represented with respective individual formats which may or may not be the same. For example, the image data may be represented as a multi-dimensional array while the metadata may be represented as one dimensional array. The format of the target image may, for example, be obtained by combining the individual formats. For example, to combine different array types (e.g., one -dimensional and multidimensional arrays) into a single format suitable for input into the first and second machine learning model, techniques such as flattening multi-dimensional arrays to one-dimensional ones and then concatenating them, may ensure a consistent input structure. Alternatively, different arrays may be embedded into a structured array or list, allowing the model to process each array within a unified format while preserving their original shapes. Hence, the target image is represented in a format, which may not only represent the raw pixel values (image data) but also encapsulate additional information (metadata) that describes aspects of the target image or the object depicted within it. This metadata may include details such as the object's dimensions, color profde, context, or annotations about the image's content. When the first and second machine learning models process this format of images, they may use at least one of the image data and the accompanying metadata to calculate the first and second loss functions respectively. These losses may guide the model's learning process, allowing it to evaluate its predictions by considering the combined influence of both the visual content and the descriptive metadata, leading to a more nuanced and informed adjustment of the model's parameters.

[0047] The target image may be provided in the same format for both the first and second machine learning models. Alternatively, the format of the target image may be adjusted differently for each model, allowing each to process the image according to its specific requirements. For example, the two models may use the target image to determine their respective rates of change, with each model utilizing an array generated by its own array generation method. In this case, the formats of the first rate and the second rate may be different. Thus, to provide the perturbation using both the first rate and the second rate, one or both rates may be transformed into a compatible format. For example, this transformation may comprise dimensionality reduction (e.g., flattening) or expansion (e.g., upscaling or embedding) or resampling to align their dimensions before computing the perturbation.

[0048] The elements of the target image refer to the elements of the array that comprises or represents the target image. For example, an element of the target image may comprise a feature of the feature vector, a pixel value or an attribute of the metadata (e.g., object ID).

[0049] Regardless of the array generation method, the array of the target image comprises elements, wherein the perturbation comprises a perturbation for each element of the array. According to one example, the perturbation is used to modify the image data of the target image and / or the metadata of the target image. This may provide a deliberate alteration made to either the visual content of the target image, the associated descriptive information, or both. When modifying the image data, the perturbation might involve subtle changes to the pixel values, such as adjusting colors, brightness, or introducing noise, to affect how the image is perceived or processed by a machine learning model. If the perturbation targets the metadata, it could involve altering the descriptive attributes of the image, such as changing the labels, annotations, or any contextual information that accompanies the image. This modification can influence how the model interprets and learns from the image, potentially testing the model's robustness or steering its learning process in a specific direction. By perturbing either or both aspects, the aim could be to assess the model's sensitivity to changes by refining how it handles the combined input of visual and contextual data.

[0050] According to one example, before the determination of the first rate and second rate, the metadata of the target image is de-identified using a predefined de -identification method. The deidentification method may, for example, comprise removing or anonymizing any personally identifiable information (PII) or sensitive information within the metadata. These removed or anonymized information may potentially link the target image to an individual or specific context. In one example, the de -identification method may include techniques such as redacting names, removing geolocation data, altering timestamps, or encoding sensitive attributes in a way that preserves the utility of the metadata while protecting privacy. This step may enable the target image to be safely used in further processing or analysis by the first and second machine learning models without compromising the security of disclosed information on the imaged objects.

[0051] According to one example, the target image is provided as an array, wherein the perturbation is an array of a same size as the array of the target image, wherein the modification of the target image comprises: scaling each element of the array of the target image by a corresponding value of the perturbation array.

[0052] For example, the target image may be represented as a vector, a one-dimensional array of numerical values, where each element corresponds to specific pixel data from the target image. The perturbation is another vector of the same size as the image vector, designed to introduce controlled changes to the target image. The modification process involves element-wise scaling, where each element of the image vector is multiplied by the corresponding element in the perturbation vector. This may mean that for every pixel value in the image vector, there is a corresponding perturbation factor that alters that value, either amplifying, reducing, or subtly adjusting it based on the perturbation vector's design. This method of modification may allow for precise control over the changes made to the target image, ensuring that the perturbation can influence the target image in a targeted manner.

[0053] According to one example, the first machine learning model is a re-identification machine learning model that is configured to perform the object identification task using as input an image of the specific type and one or more reference images. The first machine learning model is a re-identification model designed specifically to recognize and match individuals or objects across different images, even if they appear in varying conditions or from different perspectives. The first machine learning model may take as input not just a single image of the specific type, but also one or more reference images, which serve as known examples of the subject or object being identified. The purpose of this re-identification model is to accurately recognize or verify a subject's identity across different images, though it could be vulnerable to misuse in re-identification attacks.

[0054] According to one example, the specific type of the target image indicates a medical image type. The specific type may refer to the nature of the image being processed, indicating that it is a medical image type. These images may come from various modalities such as X-rays, Magnetic Resonance Imaging (MRI), Computed Tomography (CT) scans, ultrasound, or Positron Emission Tomography (PET) scans. Each type of medical image may provide unique insights into different aspects of the body's anatomy or function. Medical images are typically used by healthcare professionals to diagnose conditions, monitor disease progression, guide surgical procedures, or plan treatments. The specific characteristics of medical images, such as high resolution, detailed anatomical structures, and inclusion of metadata like patient information, make them distinct from other types of images. When a machine learning model is designed to work with medical images, it should be capable of accurately interpreting complex structures and anomalies within the images, which may be critical for accurate diagnosis and effective treatment planning.

[0055] According to one example, the specific type indicates an image format, wherein the image format comprises DICOM format. The specific type refers to the format of the target image, and in this case, it indicates that the target image is in the DICOM format. DICOM is a widely used standard in the medical field for storing, transmitting, and sharing medical images and related data. It supports various types of medical imaging, such as X-rays, MRIs, CT scans, and ultrasound images, and includes not only the image data but also metadata, such as patient information, imaging parameters, and diagnostic details. The DICOM format may ensure that medical images are stored with high fidelity and can be easily exchanged between healthcare systems and professionals while maintaining the integrity and confidentiality of the data. The present machine learning models are configured to work with images in the DICOM format and are capable of handling both the image data and the embedded metadata, which may be advantageous fortasks like diagnosis, treatment planning, or patient monitoring.

[0056] According to one example, the first rate is a gradient of the metric loss with respect to the target image, and the second rate is a gradient of the output with respect to the target image. "Gradient" as used herein is a way / computational device / algorithm for quantifying rate of change of the model’s output or of the model’s loss function, with respect to the model's input data. The model’s output or the model’s loss function may be represented as a function and the gradient of the model’s output or of the model’s loss function is the gradient of said function. Gradient may be computed by using numerical techniques, with or without discretization.

[0057] According to one example, the first machine learning model may be a deep neural network. The second machine learning model may be a deep neural network.

[0058] Fig. 1 is a flowchart of a method for de-identification of an image of a specific type, according to an example of the present subject matter. The specific type indicates a type of imaged objects. For the purpose of explanation, the method described in reference to Fig. 1 may be implemented in a computer system. The computer system may comprise at least one processor and at least one memory storing instructions that, when executed by the at least one processor, cause the computer system at least to perform the method.

[0059] A first machine learning model and a second machine learning model may be provided in step 101. The providing may, for example, be performed as follows. For example, the first machine learning model and / or the second machine learning model may be received by the computer system from a model provider. In one example, the first machine learning model and / or the second machine learning model may be generated by the computer system. The first machine learning model may be configured to perform an object identification task. The second machine learning model may be configured to perform an image analysis task. The first machine learning model has been trained on images of the specific type. The second machine learning model has been trained on images of the specific type.

[0060] The first machine learning model may be used in step 103 for determining for the image a first rate of change of a metric loss. The first rate may be with respect to the image.

[0061] The second machine learning model may be used in step 105 for determining for the image a second rate of change of an output of the second machine learning model. The second rate may be with respect to the image.

[0062] The first rate and the second rate may be used in step 107 for determining a perturbation of the image such that a change in an output of the second machine learning model, between the image and the image perturbed with the perturbation, is mitigated.

[0063] The image may be modified in step 109 using the perturbation. This may enable to deidentify the image.

[0064] The present subject matter may comprise the following subject matter that may be useful for the understanding of some of the described examples, where the target image may provide an example of input data based on personal data, the first machine learning model may provide an example of the adverse model and the second machine learning model may provide an example of the target model.

[0065] According to a first example there is provided a system for processing of data, comprising: an interface capable to receive input data based on personal data processable by a target machine learning model for producing a result; and a modifier capable of modifying the input data into modified input data, such that the said result corresponds to a result obtainable by having the modified input data processed by the target machine learning model, whilst the modified data is still capable of defending against a re-identification ("re-ID ") attack. The data may describe one of more aspects of an individual (a person). The said personal data may include patient data for example, such a medical imagery. In particular the said correspondence is such that the target model as applied to the (original) input data yields a result that is equal to the said result obtainable by having the modified input data processed by the target machine learning model. In addition, and as a distinction, a defense against the re- ID attack is such, that responses of the re-ID attack are different for the original data than it for the modified data. In particular, a fiirther / different, ML model ("adverse model") may be used to implement the re-ID attack. In such scenario, a result of applying the adverse model to the original input data is different from a result obtainable when applying the adverse mode to the modified data. Thus, the input data is so modified herein by the proposed system that only partial invariance is achieved with respect the two models, the target and the adverse model. The modified data may at least reduce likelihood for a given re-ID attack to succeed, as compared to when processing the original data in the re-ID attack. In one example, the modifier is capable of so modifying the input data based on the gradient of the target model. In one example, the modifier is capable of so modifying along a direction other than a direction of the said gradient of the target model. The data may be represented, eg by embedding, as elements of a suitably structured data space, such as a vector space, preferably with a defined concept of distance (function) or similarity measure / function, such as a normed vector space, such as a Euclidean vector space, or merely a vector space with a metric. The said direction may thus be defined in said space, such as in said vector space. The modification operation by modifier may be implemented by application of a modifier vector to the input data, wherein each, the data, the modifier vector and the modified data are elements in said data space. The modifier vector may be the computed / determined by modifier based on said model's gradient at the considered data (point / vector). In one example, the modifier is capable of so modifying along a direction that is essentially perpendicular to the direction of the said gradient of model. As mentioned above, the re-identification attack is performable by the further (adverse) machine leaning model. In one example, the modification is (further) based on a gradient of said further model. In one example, the modification is based on a gradient of the distance function, where such distance function measures the distance between the results obtainable by the further machine leaning model. In one example, the said result corresponds to a domain task. In one example, the said task is medical. In one example, the task is based on any one of a regression or classification and such is performed by the target model. Such task may include semantic segmentation, instance segmentation or detection / localization, either of which may be conceptualized as classification or regression. The operation of the further model is a re-ID attempt (attack) which may be conceptualized as a classification task. The classification task, if any, performable by the target model is different from the classification task of further mode (adverse model). The classification as per target model is a domain task, such as disease detection based on input patient imagery, or other such as in medical domain. Thus, target model is not configured for re-ID purpose, whilst adverse model is so configured. In one example, the at least one of the target model or the further model is based on an artificial neural network. In one example, the artificial neural network has multiple hidden layers. In one example, the personal data includes patient data, such as medical imagery, lab tests, reports, etc. Medical imagery (of internals of different respective patients) is of main interest herein. Thus, the personal data includes image data. Such image data may include still imagery, or video. It may be 2D. 3D or 4D, with time being one of the dimensions in the 4D=(3D,t) imagery, and, optionally, in the 3D = (2D,t) imagery, besides the 2 or 3 spatial dimensions. Alternatively, the 3D is fully spatial, such as a CT (computed tomography) volume.

[0066] In another example there is provided an arrangement including the said system of any one of the above described examples, and one or more of: i) one or more memories on which is stored the input data, ii) the target model, iii) the further model, iv) a data representation transformer capable of providing the input data based on the personal data. In yet another example there is provided a method of data processing, comprising: receiving input data based on personal data, processable by a target machine learning model for producing a result; and modifying the input data into modified input data, such that the said result corresponds to a result obtainable by having the modified input data rocessed by the target machine learning model, whilst the modified data is still capable of defending against a re-identification attack. In yet another example there is provided at least one computer program element, which, when being executed by at least one processing system, is adapted to cause the processing system to perform the said method. In yet another example there is provided at least one computer readable medium having stored thereon the at least one program element, and / or on which is stored the target model or the further model, or an approximation of the said further model.

[0067] Thus, what is proposed herein as example is a novel system and method to defend against image-based re-identification attacks. The method is designed to alter the original image so that risk of a Deep Learning-based re-identification is minimized while the response of the target model, such as diagnostic Deep Learning algorithm / model, is preserved. Thereby the approach is not restricted to X-Ray images but could be employed to data from other modalities as well. With the proposed modification, the results of the target model may be the same for both, the original data and the modified data. However, the modification also ensures that with high likelihood the processing by the attacker of the modified data yield different results as compared to a processing of the original data. Thus, the proposed modification may amount to a scrambling having a "partial invariance property": veracity of results are preserved for the target model, but compromised for the adverse re-ID model of a re-ID attacker. The proposed technology could be integrated into data storage systems (e.g., PACS) to enhance existing anonymization / de-identification functionality. The proposed image-based de-identification techniques could be employed for data acquired in the context of research / product development activities in order ensure higher data protection standards. Application of the proposed system and method in the medical domain is mainly envisage herein. That being said, again, such medical applications, whilst mainly envisaged, are not at the exclusion of other, non-medical applications also envisaged herein in alternative examples.

[0068] Reference is now made to the block diagram of Fig. 2 which illustrates components of a data processing system SYS. The system SYS may process data A'. Data A may be held in one or more data memories MEM. The data A may be personal data pertaining to one or more individuals x. Thus, the data A may be personal data such as medical data as held in one or more databases at medical facilities, hospitals, GP practices etc. Indeed, operation of the data processing system SYS as envisaged herein in examples will be explained with main reference to the medical domain with the understanding that other (non-medical) applications are not excluded herein. Broadly, the system is configured with a defense mechanism DF against abuse of data, such as in one or more re-identification attacks that may be perpetrated by one (or more) attacker(s).

[0069] In such re-identification attacks, the attacker may use computational techniques one or more computer systems A (which will also be referred to herein as the "attacker (system)". The system A may use computing resources (such as run a software) and may be desirable to process the data X1in order to determine the individual (the person) to which the data X1belongs or otherwise pertains. The data X’ may be accessible via one or more computing networks NT which may be wired, wireless or hybrid as required. The system may be arranged as shown in cloud setting where the data is held in database or others one or more storages, remote from the attacker A and / or a legitimate user CS of such data.

[0070] Thus, in addition to attacker A, there may be a legitimate data consumer CS, such as a computer system operable by a medical practitioner or medical professional, in order to process a patient’s data X . Whilst the systems A, CS are shown as separate computing entities, such is not necessarily so in all examples. Both may be the very same system, but used at different times or simultaneously by different users, legitimate user and attacker, that each run different software or code for different purposes, for example such as a computing system in a pay-for-access computing center, such as an "internet cafe" establishment, etc. Also, the designation "legitimate" user may or may not imply that such user is authorized (eg, has valid user credentials) to access the data X', although in most cases such user CS will indeed be so authorized. What matters for present purposes herein is that the legitimate user CS does preferably not process the data for re-ID, whilst the attacker system A does. The attacker system may or may not be authorized for data access. Attacker may penetrate the system instead to gain access by "hacking" their way into data storage MEM. Also, as user herein the term "attacker" may or may not imply malicious intent. In some cases, such intent may not as such be malicious , or when an attacker merely seeks to obtain re-ID information for commercial or research purposes, of example. In other cases, sadly, intention may indeed be malicious, such as in obtaining information for blackmailing or other. However, malicious or not, what matters herein for present purposes is that there is personal data and information derivable therefrom which the data owner (e.g., patient or other entity who holds the data) may not wish to divulge or have disclosed, either explicitly or implicitly, for whichever reason.

[0071] Relating now, for the sake of example and definiteness, to the medical domain in more detail, the said data X may relate to medical records. The data X' may include for example medical imagery. Medical imagery can be understood as measurements in respect of a given induvial patient x, obtained or acquired by a data source IA, such as an imaging apparatus. Imaging apparatus may include a signal source SS that is operable to emit signals to interact with patient tissue. The resulting interaction event is detected by a detector device D as the said measurements to obtain the imagery X . Such medical imagery so measured represents internal aspects of patient, such as structure of internal tissues, organs, etc, in distinction to optical imagery of a patient's face, outer body (parts), etc. Such medical imagery X' may be stored in order to support therapy and / or diagnosis. Examples of such imagery envisaged herein as personal data X include X-ray, radiographs, such as chest X-rays, CT imagery, other tomographic imageries such PECT, MRI, Ultrasound or any other.

[0072] Generally, the said X data is personal data in that it pertains to a respective individual x. Thus, the data is personal as it may allow establishing aspects of a respective patient or individual x to which the data, such as the said image measurements, pertains. In extreme cases, such personal data may allow even unique identification of the respective individual. The processing of data with the aim for such identification may be called re-identification ("re-ID") attack. Anonymizing techniques have been solely used in the past to prevent re-identification attacks which may be undesirable as the patient may not want to have certain personal information divulged to parties A, CS. Some data, although personal, may in fact not allow unique identification of the patients. Such data are sometimes referred to as quasi-identifier, in distinction to other data items, called identifiers, that do allow such unique identification. For example, such identifier data may include a person’s passport number, or other ID number etc. The place (a city for example) of residence, or place of birth, on the other hand, on its own, is usually considered a quasi- identifier as it does not allow unique identification of the individual x. Medical imagery, as opposed to optical imagery of a face for example, has been previously thought of as a quasi-identifier as it was thought for example that a patient's chest X-ray as such does not allow the unique identification of the patient. Recent tests with machine learning model-based processing have shown this not to be true (see for example the Packhauser paper cited above). Thus, it has been shown that an individual x may well be identified by using a suitably configured machine learning ("ML") model to process such medical imagery X'. De -identification models may allow computing a probability that a certain medical image (X- ray, CT, MRI etc) belongs to a certain individual patient. Thus, such techniques may allow the attacker to establish for example whether a patient has in fact visited a certain medical facility when processing the data X' of the facility as held in data memory MEM. Such re-ID attack may be undesirable. Thus, in order to avoid, or at least reduce the likelihood of such re-identification attacks succeeding, the system SYS as proposed herein includes a re-identification defender system DF. This system DF may be arranged at the data source, such as at the memory MEM where the data is held, or may be otherwise arranged, at a place where the data is generated, such as at the imager IA, or elsewhere / other data source. The re-identification defender DF may be understood as a go-in-between component, which is arranged and / or operative anywhere between data generator / source IA or data storage MEM on the one hand, and parties A, CS who wish to access the data.

[0073] The defender DF, for such it will be referred to herein for the sake of brevity, may act prospectively or retrospectively. In the prospective option, the original data X , once obtained, is processed by the defender DF before there is a request for this data. The processed data is then stored and is available for a requester. Each data item X' for different patients for example may be so processed in series or in parallel. In the retrospective option, the defender DF is operable only once a request for data, for example by attacker, comes in. Thus, the defender may include an event handler functionality to intercept such data request. The defender then processes the data, and it is the so processed data Xathat is then made available for the requester CS, A, instead of the original data X'. In either case, prospective or retrospective, the defender operates to modify data X', as will be explained in more detail below. The original data X is thus (slightly) modified, and it only this modified data Xathat is held in memory / storage, etc MEM accessible to requester CS, A, in particular attacker A. The original data X' itself may not be made available, such as by encryption, keeping same offline, etc. Thus, whilst the legitimate user CS is usually not thought to have malicious intentions, they are also not given access to the original data X', but it is preferably only the modified data Xathat is so made accessible. This affords a further layer or security, as the original data X' may still be leaked, inadvertently, even by the legitimate user CS.

[0074] As will be explained in more detail below, the modified data Xaas produced by the defender DF, still allows legitimate use thereof, for example by the data consumer CS, such as a medical user, who wishes to process patient data X for medical purposes. Alas, the modified data Xais constructed so that it leads, with high likelihood, the re-identification ("re-ID")-attack by attacker awry. In distinction, the legitimate user may still obtain the same results when processing the modified data Xa, as they had obtained, had user CS been allowed to process the original data X'. But the attacker A will be left with a wrong re-identification result when processing the modified Xa, as opposed to the correct re-ID result, if attacker had been allowed access to the original (unmodified) data X'.

[0075] For example, legitimate use by the medical user may include the computing system CS processing the accessed (modified) data Xaby a suitably configured (target) machine learning model M for domain knowledge purposes, such as in the medical realm. For example, the legitimate system CS may use the machine learning model M to establish, based on patient imagery Xa, whether a certain disease is present, such as in mammography screening, whether a certain treatment is likely to be successful, or in order to segment such imagery, or indeed to establish any other result in support of a medical task, or other task, whichever the domain at issue. Thus, even the legitimate user is only allowed to operate on the modified data Xaproduced by the defender DF. But, as explored below in more detail, the so modified imagery Xais modified in a specific way that does not compromise accuracy and operation of the ML model M used by legitimate user / system CS, whilst it does interfere with, and compromises results of, the attacker A's re-identifier model M*.

[0076] Both, attacker A and the legitimate user CS system, may each use respective, differently configured, machine learning models, such as in deep learning or any other, to processes the data X for their purposes. Deep learning models are a specific kind of artificial neural network("NN") type models. In NN models, computational nodes with weights are arranged in layers, starting from input layer proceeding through one or more hidden layers, and resulting in an output layer. In deep learning models, there is a plurality of such hidden layers which has been found to lead to good performance. Some NNs are convolutional, thus includes one or more convolutional layers that implements convolutional operators. Such CNNs (convolutional NNs) are advantageous for processing spatially correlated data, such as imagery. The layer may include different types of layers such as the said convolutional layer or pooling layers, etc. The multiple layers in DL models may preferably include multiple convolutional layers.

[0077] Data, such as modified image data X, is processed at input layer, passes as intermediate data (also referred to as feature maps) through the one or more hidden layers, and is then produced at output layer as an output result. Such result may include, in case of the legitimate user CS, a sought medical result, such as a prediction / estimate. Such prediction / estimate may include a classification result or a regression result. Thus, the result may indicate presence of a medical condition or prediction of an outcome for a radiotherapy treatment plan, a segmented image with the input image X segmented for certain regions of interest such as tissue, organ, organ part, tumor, etc. In case of the adverse user model M*, geared for re-identification attack the result is a re-ID result, such as a probability / score whether the processed image does or does not belong to a set of known individuals (sometimes referred to as the "gallery" in re-identification technology). Both the attacker and / or the legitimate (e.g., medical) user CS system may use machine learning models of the neural network types, or the two may use different ML models. Gradient-based models are preferred herein. The models M, M* are generally different in makeup and purpose as will be explained below in more detail.

[0078] In either case, the respective model M, M*has been trained previously on suitable training data. In such training, weights or other parameters of the models M, M* are adjusted, based on the training data. The training is controlled by an objective function, such a loss or utility function. Any training algorithm may be used, but preferably gradient based. For example, back-propagation techniques may be used. The training data may use a supervised learning setup. In setups, the training data may include input training data, some or each associated with a respective target ("label"). The objective function measures how well the model maps the training input data to its respective target. Parameters are adjusted in one or more iterations for example in the iterative scheme in order to improve the objective function. Once a pre-defined quality threshold is reached, the respective model is considered trained and can be used in deployment, such as for the intended medial task purposes by the medical consumer CS, or for re-ID attack purposes by the attacker A. Again, completely different types of models may be used by an attacker A and the legitimate user CS. Also, different types and kinds of models and / or training schemes may be used by each party CS, A. Non-supervised or semi-supervised training methods are not excluded herein.

[0079] As schematically indicated in Fig. 2, Attacker A and / or legitimate user CS may each include respective user interfaces UI, UI* which allows manipulating the received data, for example for processing by respective machine learning model M, M* as explained above. Attacker or legitimate user may wish to have the imagery and / or the result of their respective machine learning processing displayed on a respective display device DD.

[0080] Reference is now made to the block diagram of Fig. 3, which provides more details on operation and components of the defender DF. As broadly explained above, the defender DF is configured herein to defend against re-identification attacks by an attacker A, using the suitably configured machine learning M* . Some or all of the original data X is retrieved from memory MEM and received at input port IN. In some cases, operation on the original (raw) data X' as such may not be advisable, so here is a transformer or mapper component TF used, in order to first transform the original data X to a suitable data representation first, and this is the processed by the defender DF. Thus, transformer TF may instantiate a mapping that maps the original data X to the transformed data X=T(X'). However, such a transformer TF may be optional, as in some cases operation on the original data X' may be indeed done. Operation of the transformer TF in some instances is such that it produces a lower dimensional ("dim") representation of the original data X . Such transformer may be thought to effect an embedding of the data A as points in a suitably structured and dimensioned data space. Such transformation TF may itself be referred to as "embedding". Such transformation, or re -representation of the data to the processed down the line, may allow faster and more memory efficient processing. Any one of a number of dimensional reduction techniques, such as principal components, or other may be used. In fact, the representation transformer TF may itself be arranged as a machine learning model such as an auto-encoder "(AE"), or a variational encoder ("VAE") as needed. In the earlier case, the auto-encoder is trained so that the original data X is applied to the model which is then transformed to a lower dimensional representation. Such lower dimensional representation is then re-transformed back (hence auto-encoder) into the original input data. For example, the lower dimensional data X may be obtained as a latent representation within the auto-encoder's mode architecture. For example, the AE may be arranged as an NN. The lower representation may be accessed as a feature map in one of the hidden layers of the auto-encoder such as the one located centrally between input and output layer. The auto-encoder may itself be trained by a separate objective function, different from the objective function used for training the model M, M* . As said, use of such any such or other transformer TF may be optional, Thus, in such cases, the transformer may be thought of as the identity operator, when such transformer is not in fact used.

[0081] In the following we will refer to the data processed by data defender as "X", irrespective of whether this is the original X' or transformed data X=T(X'), unless such distinction is needed. This shorthand allows disencumbering notation, but is also conceptually justified as in most cases the transformer TF maps uniquely into data points so is configured (at least) as an injective operation. Thus, different data items X' for different individuals x is mapped to different embedded points X=T(X').

[0082] The input data, embedded / transformed or not, is modified by modifier MOD to produce the modified data Xa, such as modified imagery, which is then made available for access by any party, such as the legitimate user or the attacker, for processing, such as ML-processing by models M, M* .

[0083] As will be explained in more detail below, the modification operation by modifier MOD is so geared that it preserves the operational accuracy of the legitimate model M, whilst it distorts or otherwise interferes, at least with high probability, output of the attacker model M*. It is of note that if the transformer TF is used, its operation is different and in addition to the operation of the modifier MOD. Thus, the transformer TF merely allows a more efficient representation of the data A and such transformed data does not have the property (referred to below as "partial invariance") of the modified data A' as produced by the modifier MOD. Thus, the transformed data TfAj^Amay not allow defending against re-identification attacks as compared to the modified data A' as produced herein by operation of the modifier MOD. The modifier MOD may apply its modification operation component-wise for example per pixel or other component of the data. For example, in case the input data X,T(X) is an image, the modifier MOD may operate to per-pixel on the input data image X or T(X). Not all pixels / component of a data item X, T(X) may need to be so modified. Modification of a subset may suffice. Some or all components / pixels may be modified in the same manner, e.g. by the same amount a, or some or each such components / pixels is modified differently. For example, such modification may include adding and / or multiplying a pixel with a number. Any other algebraic operation may be used. In generally and preferably, the modified data X3is, least for the human user, barely, or not, distinguishable from the input image / data X by the average human user. But even if it is, the modification is generally chosen to as to preserve aspects of global and local patterns. The right amount of modification that is not visually perceptible by most users may be found by tests run with a selection of the target audience (e.g., Radiologists, etc). The modification level / magnitude 8 may be chosen in line with certain noise characteristics of the data source IA, explained in more detail below.

[0084] Reference is now made to Figs 4A, B which illustrate in yet more detail operation of the modifier MOD. Whilst reference to transformer TF is made, again, this is optional: Thus, transformer TF may be conceptualized as the identity operator, and thus the below and above is of equal application in case no transformer TF is in fact used, thus when operation of modifier MOD on the original data is preferred.

[0085] Different data items X and P , for example from individuals x, p, may be processed by the transformer TF mapped into a data space DS such as a vector space with a suitable norm, or metric more generally, to yield elements X, P as illustrated in Fig. 4A. In an affine space DS, such elements may be visualized as data points as illustrated in Fig. 4A. Preferably, no such transformer is used for target model M. But may be used by adverse model M*. Data space DS may be Euclidean space, with dimension mxn, with m and n being numbers of rows and columns of data X, such as when this is an image X. However, data may be any data, such as reports or other, Thus, the herein-described in not confined to image data, although this is indeed envisaged in the main.

[0086] A distance function D() (or metric loss) may be defined in data space DS that allows measuring how similar or close those data points (and hence the underlying data X',P', and hence by further extension the individuals x,p) are. Thus, the metric or distance function D() is configured so that if the data pertains to the same individual, such data in general will yield a small distance, ideally zero, and different individuals get mapped to points in the data space DS, with a distance as measured by D() that is larger than the threshold, but in any case, non-zero. Euclidean distance D() for example may be used, in any required dimension. Thus, the distance measure D may be understood as a similarity measure as it measures the similarity of the mapped data points X, P. Thus, the distance function may itself be understood as a mapping from points of data space into the non-negative reals.

[0087] The modifier applies a modification A to the data which is shown only for X to yield the data Xathat protects against adverse re-identification attacks. Such an operation may be understood as a vector operation, such as "shifting" of data point X:

[0088] X1X A (1) The re-ID defence as afforded by defender DF is thus that the assumed known model M of the legitimate user still yields the same results for X3= X+ , as it would for the unmodified data X. Alas, the attacker's re-identification model M* will be led away by yielding different results for the modified data A7as compared to the original data X. This is shown in the lower right of Fig. 4B. This re- ID defence conferring property may be referred to herein as "partial invariance". Thus, invariance in respect of results is preserved for the legitimate model M, whilst such invariance is not preserved for the re-ID model M* . This partial invariance is achieved by configuring the modification vector A in a specific manner. This is also illustrated in Fig. 4B, to which reference is now made in more detail. Any model, such as legitimate model M of consumer system CS or attacker A 's model M* may be understood as a mapping of data X to the respective result, such as regression or classification. Specifically, in case of target model M, this may be a medical result. In case of adverse model M*, the result is different from the results M delivers, in particular M*'s output may be the re-ID result. Thus, each model may give rise to a model function, or predictor function C, R, such as C: X -> M(X), R: X -> M(X). Each such function C,R may have a gradient grad (C) / grad (R), sometimes written herein without the brackets to disencumber notation.

[0089] Specifically, the gradient of predictor function C is illustrated in Fig. 4B as having a particular direction in the data space DS. It has been found herein that by choosing the modification vector A to be running in a different direction than the gradient of the legitimate predictor function C may yield modified data Xathat has the partial invariance property. Specifically, it has been found that modifier vector , X3= X / 1 may be chosen to be orthogonal to the gradient grad (C) of function C. However, any other direction, as long as it is not along grad(C) (parallel or anti-parallel), may also be used. As will be explained in more detail below, a geometrical construction mechanism to construct orthogonal modification vectors A may be used. A randomized mechanism suitably constrained may be used instead of such a geometrical construction mechanism to yield suitable modification vectors A. As envisaged herein, to be "not orthogonal" to gradient C may include herein being parallel and / or antiparallel to gradient grad(C).

[0090] In some examples of construction, the modifier vector A may be understood in terms of a perturbation Up towards a chosen point P as shown schematically in Figs 4A, B. The point P in data space DS may represent another individual (e.g., patient p other than patient x, who is represented by data X instead).

[0091] The perturbation vector to such other point P may be projected onto the gradient of C at point X, and then the difference vector of the projection along gradient C and the perturbation vector Up is taken to yield modifier vector A, now guaranteed to be orthogonal to gradient. Points X, P may be understood to form an associative pair with is completely arbitrary. Such pairs may be established automatically by system through a randomizer in the set of all patients {x} or whichever set of data is to be processed for re-ID hardening by defender DF. Optionally, a user interface UI may be used where user can interactively establish such pairs. Later, when processing point P accordingly, computing time can be saved by simply taking A(P) = - A(X). Thus, for the other pair member, in this case P in the associative pair (X,P), the modifier vector may be chosen as the negative (anti -parallel) of the modifier vector A(X) earlier computed for the member X processed first. Data point / vector P may be chosen at random. In some use cases, P may be chosen to have some similarity to X as measured by a distance function D().

[0092] In some cases, no further knowledge of the adverse model M* may be needed, as some models are ill-conditioned, so a small modification is enough to compromise their results as in re-ID model M*, whilst the proposed gradient based approach ensures, at least with sufficient likelihood, the desired invariance of known model M against such modification / perturbation A. Indeed, as illustrated by Goodfellow et al (in "Explaining and Harnessing Adversarial Examples", available online at arXiv service, reference code arXiv: 1412. 6572v3 [stat.ML], published 2015), even an added human- imperceptibly small perturbation may mislead a classifier to misclassify a slightly perturbed image of a Panda (Ailuropoda melanoleuca)' as one of a Gibbon (Hylobatidae), although the classification was correct if no such perturbation is applied. See page 3, Fig. 1 in said cited Goodfellow et al.

[0093] It will be understood herein that for each point X that is to be modified, the gradient grad (C) will need to be evaluated at position X, grad (C) = grad x (C). However, publicly available knowledge, or indeed leaked knowledge, may be used to estimate re-ID action of adverse model M*. A prototype model M* * may be build and trained on data X, in order to so approximate or estimate re-ID action of the adverse model M*. In some examples, such knowledge may be used herein to estimate Xa. In particular, a gradient of model M* may be used by modifier MOD, in addition to grad C of model M. In some examples, the gradient of grad D° M* « grad D° M** may be used by modifier MOD.

[0094] A specific, vector-geometry based construction of such modifier vector A is now described in more detail. For example, a projection-based modification construction may be used. More specifically, a Gram-Schmidt-type vector-based, or other similar modification, may be used in examples. The construction may be described based on the perturbation Up towards random or pre -determined point P. P may vary or may remain fixed, as needed. Varying P may be preferred in some examples, but fixed P constructions (for some or all X) may be done instead in some contexts.

[0095] Specially, Given the target model M (such as with deep learning (DL) architecture or any other), configured / trained for image classification C(X), it is desirable that a similar response for Xaand X is obtained. Therefore, what is proposed herein is not to use the projection onto a vector that is orthogonal to the gradient of C with respect to X. Since the gradient of C with respect to X is orthogonal to the level set of the classification function C, such a change will minimize the effect of the perturbation: where “7” is a transpose operator, U any vector U in data space DS, not parallel / anti- parallel to G. In some examples, U may be chosen as the said perturbation vector Up. That is, in some examples, UPis the perturbation towards the other data point P (e.g., an image of patient p) as introduced above. G represents the gradient of the classification function C of the image X as per target model M:-

[0096] > ac( ) dX (3)

[0097] As said, in (2), and other vector U not parallel / anti-parallel to G may be used instead of perturbation Up. Choosing U= Up being more one example, which, however, has been found to yield good results and allows a canonical, straightforward implementation. It may have the further benefit that such configuration U= Up may better “mislead” M* by leading to a misclassification of X into P. This may conceal to A that the modified data X is actually misleading its model * . However, in the general case, perturbation U may not need to point to another such point P that represents an existing patient P.

[0098] In some examples, effect of the distance function / )() on the output of M* may be accounted for in the above computing of modified data A'. Thus, in some examples, good results can be obtained by choosing / configuring the distance function so that distances between points X, P are magnified. Other distance functions with such magnifying property can be use instead. In examples, in order to account for unwanted effects of the distance function D in data space D S, and to avoid undue response by the predictor function C, in some examples, effects of the distance function D on changes in C are included in the construction of the modifier A. Thus, the effect on how much the re-ID model M* “spreads” its outputs can be taken into account, when defining

[0099] , , rr ■

[0100] U = Up = sign

[0101] Thus, perturbation Up as per (4) may be used above in eqs (2)(3) in some examples.

[0102] However, the construction based on orthogonal shifts relative to grad C is of general application, and any vector U may be used to compute its projection along grad C and take the difference of said vector U by its projection along grad C to obtain orthogonal modifier A.

[0103] In some examples still, model M* is essentially a distance function D and provides as ID results distance values D(X). In thus instances, model M* may be identified with the distance function D, and the following vector may be used instead for perturbation Up dD P,X)

[0104] U = Up = sign( dX (5) In eqs (2)-(5), instead of M*, if this is not known, the above-described approximation / re- modelling M** may be used instead.

[0105] In any one the above and below, if model M* uses transformer TF (such as embeddings), the gradient grad D° M** ° TF) « grad (D ° M* ° TF) may be used for determining A.

[0106] Although the proposed projection-based construction such as at any one of eqs (2)-(5) has been found to yield good results and affords efficient processing, it is still to be construed herein as merely one example from many, and such construction does not limit the general principles of what is disclosed herein. Specifically, geometrical vector constructions, other than projection based, are also envisaged herein in different examples. What is more, the proposed principles are not confined to geometrical constructions, as others, such as statistical, etc are also envisaged. In particular, the construction in terms of perturbation Up and pairs (X,P) is, whilst efficient, not limiting herein.

[0107] In the above, the "signum" operator sign() indicates the direction of the unit vector whilst a includes a suitably small amount of perturbation 8 that can be found by trial and error as briefly mentioned above, e.g. by conducting image perception tests on the target audience. The perturbation amount 8 may be applied vector-component-wise, as need be. Construction of the direction sign ensures that the modification A is in fact orthogonal to gradient. Modifications that are substantially orthogonal but may not necessarily be exactly orthogonal have been shown to still yield good results and are preferred herein.

[0108] The results of applying the perturbation 8 to image data may be imperceptible to the average user, and can be objectively established by conducting the said perception test using test imagery. The perturbation may be equal to or lower than the noise level in the data X acquiring detector equipment used. The noise level can be measured by experiment on phantoms for example. The noise level may be readily available from detector manufacturer's data specification sheets for the particular make and model of detector that was used to obtain the original imagery X'. The perturbation 8 may be random. It may differ from image to image, or, may be reused from image to image, as needed. Such reuse strategy may allow yet more responsive implementation which may be useful in Cloud / client-server settings, etc.

[0109] In eqs(l)-(5), the "+" operation is a vector operation, such as addition. It may be applied component-wise. As the data A may be an image, having rows and columns, a representation as a matrix may be envisaged, and the same for the shift vector A as matrx of same size as X. Thus, eqs(l)-(5) may be implemented as component-wise matrix addition, with elements X, A and treated as elements of vector space DS. Applying the modification by operations other than addition may also be envisaged in some instances so the operation "+" may indeed by a vector addition, but may also relate to other combination operation. If transformer TF is used, dimensional correspondence may need to be ensured, such as by back-transformation, padding, etc.

[0110] Reference is now made to the flow chart of Fig. 5, which illustrates steps of a computer implemented method for defending original data X' against identification attacks as envisaged herein in examples. The proposed method is not necessarily tied to the system architectures as shown above, but the said below method may be understood to constitute a teaching in its own right.

[0111] At step S405, original data X is accessed.

[0112] At an optional step S410, the original data and may be transformed into a representation more suitable for down-stream processing, as such as lower dimensional representation. Dimension reduction techniques may be used. The transformation may be implemented as an embedding into a data space (such as vector space) as points or vector. The transforming (operation) may be machine learning based. For example, an auto-encoder may be used to derive latent representations in terms of feature maps or any other. However, this transformation into a lower dimensional, or a different, otherwise more suitable, representation is optional. Thus, down-stream operation on the original data instead of on transformed data, is also envisaged herein. Preferably, such transformation TF is not used for the target model M, but may be used in the adverse model M* as embeddings for example. However, this too is optional.

[0113] At step S420 the data to be re-ID defended is received.

[0114] At step S430 the received data is modified, to yield the re-identification-attack-hardened, modified, data Xawhich is then made available at step S440 for access by any party to process, legitimate or adverse.

[0115] In more detail and in some examples, at step S450, the data Xamay be processed by a target machine learning model M, in order to obtain a regression or classification result, or any other result in an applicable domain, such as medical or other. The processing S450 may further include, in addition, or instead of, such domain processing by target ML model M, adverse processing by a (different) machine learning model M* for re-ID purposes. Thus, if the processed data X is personal data, the adverse processing by adverse model M* may wish to seek a result to confirm that the data X belongs to a certain specific individual (a person) known to the attacker.

[0116] The modification at step S430 ensures that the modified data Xaproduced has the partial invariance property as explained above, or the modification at step S430 ensures at least helps to achieve this with sufficient likelihood. Thus, the modification results in modified data Xa that preserve results obtainable by the legitimate / target domain model M, irrespective of whether original or modified data is processed, but at the same time such data modified data Xacompromises operation of the adverse model M* by causing such adverse model M* to produce different results compared to when processing the original, unmodified, data X. Thus, the legitimate user can go about processing even the modified data, as before with the original data, with difference (hence invariance) in results obtained, whilst the reidentification attacker is more likely to obtain wrong results. Thus, for example, the attacker may not be able to identify any person in the gallery using mode M*, or model M* may in fact yield a response that indicates another person as a "hit", instead of the person to whom the data X actually pertains.

[0117] In examples, the modification operation S430 is based on any one of eqs (l)-(5) or similar, or on any other data modifier approach to ensure that the modification A is not along the gradient of target model M's predictor function C. Thus, modification in a vector data space, in which the data X is presentable, is along a vector direction other than the gradient of the predictor function C of the target model M at point X. In examples, the direction is orthogonal, or is approximately (within some negligible error margin) to such gradient grad C(). A construction of modification S430 based on associative points and / or in terms of perturbation vector Up to a different point P as mentioned above is envisaged in some examples. In examples, the orthogonal construction considers the effect of the distance function D. Thus, in some examples, the construction of A is (further) based on the gradients of grad R() and / or grad D(), in particular may be based on grad D(X,P), for example as described above at eqs (4), (5).

[0118] The above method step S430 may be repeated for each data item X that needs to be hardened against re-ID attacks. The method may be practiced in prospective or retrospective fashion, as needed.

[0119] The components of system SYS, in particular or defender DF, may be implemented as one or more software modules, run on one or more general-purpose processing units PU such as a workstation associated with the imager IA, or any other computing device, such as laptop, smartphone, etc.

[0120] Alternatively, some or all components of the defender DF may be arranged in hardware such as a suitably programmed microcontroller or microprocessor, such an FPGA (field-programmable - gate-array) or as a hardwired IC chip, an application specific integrated circuitry (ASIC), integrated into the imaging system IA. In a further example still, the defender DF may be implemented in both, partly in software and partly in hardware.

[0121] The different components of the defender DF may be implemented on a single data processing unit PU. Alternatively, some or more components are implemented on different processing units PU, possibly remotely arranged in a distributed architecture and connectable in a suitable communication network such as in a cloud setting or client-server setup, etc.

[0122] One or more features described herein can be configured or implemented as or with circuitry encoded within a computer-readable medium, and / or combinations thereof. Circuitry may include discrete and / or integrated circuitry, a system-on-a-chip (SOC), and combinations thereof, a machine, a computer system, a processor and memory, a computer program.

[0123] In another example of the present invention, a computer program or a computer program element is provided that is characterized by being adapted to execute the method steps of the method according to one of the preceding examples, on an appropriate system.

[0124] The computer program element might therefore be stored on a computer unit, which might also be part of an example of the present invention. This computing unit may be adapted to perform or induce a performing of the steps of the method described above. Moreover, it may be adapted to operate the components of the above-described apparatus. The computing unit can be adapted to operate automatically and / or to execute the orders of a user. A computer program may be loaded into a working memory of a data processor. The data processor may thus be equipped to carry out the method of the invention.

[0125] This example of the invention covers both, a computer program that right from the beginning uses the invention and a computer program that by means of an up-date turns an existing program into a program that uses the invention.

[0126] Further on, the computer program element might be able to provide all necessary steps to fulfill the procedure of an example of the method as described above.

[0127] According to a further example of the present invention, a computer readable medium, such as a CD-ROM, is presented wherein the computer readable medium has a computer program element stored on it which computer program element is described by the preceding section.

[0128] A computer program may be stored and / or distributed on a suitable medium (in particular, but not necessarily, a non-transitory medium), such as an optical storage medium or a solid- state medium supplied together with or as part of other hardware, but may also be distributed in other forms, such as via the internet or other wired or wireless telecommunication systems.

[0129] However, the computer program may also be presented over a network like the World Wide Web and can be downloaded into the working memory of a data processor from such a network. According to a further example embodiment of the present invention, a medium for making a computer program element available for downloading is provided, which computer program element is arranged to perform a method according to one of the previously described examples of the invention.

[0130] It has to be noted that examples of the invention are described with reference to different subject matters. In particular, some examples are described with reference to method type claims whereas other examples are described with reference to the device type claims. However, a person skilled in the art will gather from the above and the following description that, unless otherwise notified, in addition to any combination of features belonging to one type of subject matter also any combination between features relating to different subject matters is considered to be disclosed with this application. However, all features can be combined providing synergetic effects that are more than the simple summation of the features.

[0131] While the invention has been illustrated and described in detail in the drawings and foregoing description, such illustration and description are to be considered illustrative or exemplary and not restrictive. The invention is not limited to the disclosed examples. Other variations to the disclosed examples can be understood and effected by those skilled in the art in practicing a claimed invention, from a study of the drawings, the disclosure, and the dependent claims.

[0132] In the claims, the word “comprising” does not exclude other elements or steps, and the indefinite article “a” or “an” does not exclude a plurality. A single processor or other unit may fulfill the functions of several items re-cited in the claims. The mere fact that certain measures are re-cited in mutually different dependent claims does not indicate that a combination of these measures cannot be used to advantage. Any reference signs in the claims should not be construed as limiting the scope. Such reference signs may be comprised of numbers, of letters or any of alphanumeric combination.

Claims

CLAIMS:

1. A method for de -identification of an image of a specific type, the specific type indicating a type of imaged objects, the method comprising: providing a first machine learning model to perform an object identification task and a second machine learning model to perform an image analysis task, the first machine learning model having been trained on images of the specific type, the second machine learning model having been trained on images of the specific type; using the first machine learning model for determining a first rate of change of a metric loss with respect to the image; using the second machine learning model for determining a second rate of change of an output of the second machine learning model with respect to the image; using the first rate and the second rate for determining a perturbation of the image such that a change in an output of the second machine learning model, between the image and the image perturbed with the perturbation, is mitigated; modifying the image using the perturbation.

2. The method of claim 1, wherein the image is provided in a format representing image data, the specific type further indicating the format.

3. The method of claim 1 or 2, wherein the image is provided in a format representing image data and metadata descriptive of the imaged object and / or the image, the specific type further indicating the format.

4. The method of claim 3, wherein the perturbation is used to modify the image data of the image and / or the metadata of the image.

5. The method of claim 3, wherein before the determination of the first rate and second rate, the metadata of the image is de-identified using a predefined de-identification method.

6. The method of any of the preceding claims, the first rate of change being a first vector whose elements represent the image, the second rate of change being a second vector whose elementsrepresent the image, wherein the perturbation is a vector rejection of the first vector from the second vector, the vector rejection being scaled by a scale factor.

7. The method of any of the preceding claims 1 to 5, wherein the perturbation comprises a first perturbation that increases the metric loss and a second perturbation that decreases a loss of the second machine learning model, wherein the modification comprises: using the first perturbation to modify a first part of the image and using the second perturbation to modify a second part of the image.

8. The method of any of the preceding claims, wherein the image is represented by an array, wherein the perturbation is an array of a same size as the array of the image, wherein the modification of the image comprises: scaling each element of the array of the image by a corresponding value of the perturbation array.

9. The method of any of the preceding claims, the first machine learning model being a reidentification machine learning model that is configured to perform the object identification task using as input an image of the specific type and one or more reference images.

10. The method of any of the preceding claims, the first rate of change of the metric loss being a gradient of the metric loss, the second rate of change of the output being a gradient of the output.

11. The method of any of the preceding claims, the specific type indicating a medical image type.

12. The method of any of the preceding claims, the specific type indicating an image format, wherein the image format comprises Digital Imaging and Communications in Medicine, DICOM, format.

13. A computer program product comprising a computer readable medium, the computer readable medium having computer readable code embodied therein, the computer readable code being configured such that, on execution by a suitable computer or processor, the computer or processor is caused to perform the method as claimed in any one of claims 1 to 12.

14. An apparatus comprising at least one processor; and at least one memory storing instructions that, when executed by the at least one processor, cause the apparatus at least to perform the method as claimed in any one of claims 1 to 12.

Images