Method and system for refining a trained neural network

Abstract

A method for refining a trained neural network comprises the steps of: i) generating S1 a set of inverse adversarial examples using a set of training data and a trained neural network, wherein the inverse adversarial examples are perturbed data generated from input data by applying perturbation noise N with a norm above a predetermined minimum perturbation to such input data such that the network outputs a predicted label with a confidence level above a predetermined threshold; ii) updating S2 the trained neural network using the generated set of inverse adversarial examples to generate an updated trained neural network; and iii) repeating S3 steps i) and ii) based on the updated trained neural network until a predetermined condition is met. The trained neural network is trained on a set of data comprising input data and an associated ground truth label and is configured to determine prediction logits and output a predicted label and confidence level based on the logits. Generating the set of inverse adversarial examples comprises iteratively applying perturbation noise and updating the noise by adding a sign gradient of a first objective function.

Description

TECHNICAL FIELD

[0001] The disclosure relates to training of neural networks. Specifically, the disclosure relates to methods and systems for refining a trained neural network using inverse adversarial examples. BACKGROUND

[0002] Image classification models have exhibited remarkable advances in their capabilities, in particular their ability to accurately predict image classes even when confronted with highly noisy images. Despite the impressive performance of image classification models in such challenging scenarios, instances persist in which the models manifest unwarranted overconfidence, leading to false positive predictions. Although such false positives are infrequent, there may be considerable repercussions in practical applications, for example in the scenario of face detection models used for authentication and security purposes, or the scenario of pedestrian detection models used for autonomous driving applications.

[0003] Adversarial techniques focus on the study of attacks on machine learning models and are employed to fool machine learning models by crafting adversarial examples which contain slight state perturbations that lead to very significant differences in terms of performance and predictions of a trained neural network. Examples of methods to generate adversarial examples include the Fast Gradient Sign Method (FGSM), the Iterative Fast Gradient Sign Method (I-FGSM), Momentum Iterative Fast Gradient Sign Method (MI-FGSM), Diverse Input I-FGSM (DI2FGSM), and Translation-Invariant FGSM (TI-FGSM).

[0004] However, adversarial techniques and adversarial examples do not address false positive situations involving input images with perceptible perturbation that have a prediction gap between artificial intelligence and humans. For example, the trained neural network may correctly classify a heavily perturbed image as its original class despite the heavily perturbed image being unrecognisable by a human as its original class. In other examples, the trained neural network may classify a heavily perturbed image as an indistribution class with high confidence despite the heavily perturbed image being unrecognisable by a human. SUMMARY

[0005] It is the object of the disclosure to refine and / or improve a trained neural network to enhance the specificity of such trained neural network in relation to input with perceptible perturbation leading to a prediction gap between the trained neural network and humans.

[0006] The object is achieved by the subject matter of the independent claims. Preferred embodiments are subject matter of the dependent claims.

[0007] It shall be noted that all embodiments of the present disclosure concerning a method or a series of performed steps might be carried out with the order of the steps as described, nevertheless this has not to be the only and essential order of the steps of the method. The herein presented methods or series of performed steps can be carried out with another order of the disclosed steps without departing from the respective method embodiment, unless explicitly mentioned to the contrary hereinafter.

[0008] To solve the above technical problems, the present disclosure provides a computer-implemented method for refining a trained neural network trained on a set of training data comprising input data and an associated ground truth label for each input data, the trained neural network configured to receive input data, determine prediction logits and output a predicted label and confidence level based on the determined prediction logits, wherein the method for refining the trained neural network comprises the steps of: i) generating a set of inverse adversarial examples using the set of training data and the trained neural network, wherein inverse adversarial examples are perturbed data generated from input data by applying perturbation noise N with a norm above a predetermined minimum perturbation 6 to such input data such that when an inverse adversarial example is input into the trained neural network, the trained neural network outputs a predicted label with a confidence level above a predetermined threshold; ii) updating the trained neural network using the generated set of inverse adversarial examples to generate an updated trained neural network; and iii) repeating steps i) and ii) based on the updated trained neural network until a predetermined condition is met.

[0009] The method of the present disclosure is advantageous over known methods as the specificity of the trained neural network is increased by refining or updating the trained neural network using inverse adversarial examples which may reduce in the prediction gap between the trained neural network and humans, and therefore reducing false positive rate of the trained neural network. Inverse adversarial examples are heavily perturbed data (or very noisy data) that are not recognisable by humans but are still successfully classified as the trained neural network with high or overly high confidence. Inverse adversarial examples are therefore input that would lead to the trained neural network incorrectly outputting with overly high confidence a predicted label, i.e., the trained neural network being overly confident in its prediction output. Refining the trained neural network with such inverse adversarial examples (or false positives) may potentially reduce the false positive rate of such refined trained neural network by reducing the max confidence of prediction outputs and adjusting the wrong prediction confidence or probability distribution of the trained neural network. The predetermined minimum perturbation ensures that the perturbation is perceptible, and the predetermined threshold for the confidence level ensures that the trained neural network is very sure of its prediction result (i.e., high confidence in its prediction output).

[0010] A preferred method of the present disclosure is a method as described above, wherein the input data comprises image data or audio data, and the ground truth label and the predicted label comprises a classification or regression task label.

[0011] The above-described aspect of the present disclosure has the advantage that trained neural networks are able to find patterns and make predictions based on image and audio data for both classification and regression tasks.

[0012] A preferred method of the present disclosure is a method as described above or as described above as preferred, wherein step i) of generating a set of inverse adversarial examples comprises iteratively: applying perturbation noise N to input data to generate perturbed data; and updating the perturbation noise N by adding a sign gradient of a first objective function.

[0013] The above-described aspect of the present disclosure has the advantage that using sign gradient descent is efficient and allows easier control of the step size in each iteration / optimisation step, thereby potentially generating inverse adversarial examples more efficiently or quickly. The inverse adversarial examples generated are inverse adversarial examples that the trained neural network are more likely to have an overly high confidence in its prediction output. Refining the trained neural network with such inverse adversarial example adjusts the wrong prediction confidence or probability distribution of such trained neural network by reducing the max confidence of prediction output categories and may therefore lead to a trained neural network with higher specificity which is better able to reject false positives.

[0014] A preferred method of the present disclosure is a method described above or as described above as preferred, wherein the first objective function is computed based on minimising a distance between ground truth logits, and prediction logits generated by the trained neural network with the perturbed data as input.

[0015] The above-described aspect of the present disclosure has the advantage that using a first objective function based on minimising a distance between ground truth logits and prediction logits ensures the generation of better inverse adversarial examples that the trained neural network is more likely to have an overly high confidence in its prediction output. Refining the trained neural network with the better inverse adversarial examples generated adjusts the wrong prediction confidence or probability distribution of such trained neural network by reducing the max confidence of the associated ground truth prediction output category and may therefore lead to a trained neural network with higher specificity which is better able to reject false positives.

[0016] A preferred method of the present disclosure is a method described above or as described above as preferred, wherein the predicted label comprises a classification task label and the first objective function is computed based on maximising the prediction confidence of any one class.

[0017] The above-described aspect of the present disclosure has the advantage that using a first objective function based on maximising the prediction confidence of any one class for classification task labels ensures that generation of better inverse adversarial examples that the trained neural network is more likely to have an overly high confidence in its prediction output regardless of the category or class of the prediction label / output. Refining the trained neural network with the better inverse adversarial examples generated adjusts the wrong prediction confidence or probability distribution of such trained neural network by reducing the max confidence of all potential truth prediction output categories and ensures that the trained neural network evenly distributes the prediction confidence level among all classes when an input is out of distribution, which may therefore lead to a trained neural network with higher specificity which is better able to reject false positives, particularly in situations wherein the input is out of distribution.

[0018] A preferred method of the present disclosure is a method as described above or as described above as preferred, wherein step i) of generating the set of inverse adversarial examples comprises solving objective function: argmin(^( / 0(X + N), G)), subject to ||N|| >e, N wherein f (■) represents the trained neural network, G represents the ground truth label, ^(-) represents a loss function, X represents the input data, N represents the perturbation noise, and e represents the predetermined minimum perturbation.

[0019] The above-described aspect of the present disclosure has the advantage the objective function may potentially generate better inverse adversarial examples, thereby potentially resulting in a trained neural network that has higher specificity through additional training or updating with the generated better inverse adversarial examples. The incorporation of the condition term of restricting the norm of N to above minimum perturbation e ensures that the inverse adversarial example generated is sufficiently heavily perturbed such that the trained neural network should not be able to output a predicted label with high confidence.

[0020] A preferred method of the present disclosure is a method as described above or as described above as preferred, wherein the loss function is a cross-entropy loss function.

[0021] The above-described aspect of the present disclosure has the advantage that the crossentropy loss function is computationally efficient and can be easily optimised using gradient descent. The cross-cntropy loss function also provides a measure of uncertainty or entropy that may be useful for evaluating model performance and understanding its prediction. Cross entropy loss function may also be particularly advantageous in generating inverse adversarial examples as it penalises the trained neural network more strongly for incorrect predictions with high confidence level, thereby potentially leading to the generation of inverse adversarial examples that the trained neural network classifies correctly as the ground truth label with overly high confidence.

[0022] A preferred method of the present disclosure is a method as described above or as described above as preferred, wherein the predicted label comprises a classification task label and die loss function is a negative mean maximum confidence (nMMC) loss function.

[0023] The above-described aspect of the present disclosure has the advantage that the negative mean maximum confidence (nMMC) loss function determines how well the trained neural network is able to separate the distribution of confidence level of the in-distribution inputs from the confidence level of out-of-distribution examples, which may be advantageous in generating inverse adversarial examples for classification tasks as it penalises the trained neural network for incorrect predictions with high confidence level, thereby potentially leading to the generation of inverse adversarial examples that the trained neural network classifies correctly as any other in-distribution class with overly high confidence.

[0024] A preferred method of the present disclosure is a method as described above or as described above as preferred, wherein step ii) of updating the trained neural network comprises using a second objective function, wherein the second objective function is computed based on minimising a loss function, while simultaneously optimising parameters of the trained neural network to maximise such loss function.

[0025] Tire above-described aspect of the present disclosure has the advantage that the trained neural network is updated based on the inverse adversarial examples (i.e., false positive examples generated based on true positive examples), thereby potentially increasing the specificity of the trained neural network by reducing the false positive rate of the refined trained neural network.

[0026] A preferred method of the present disclosure is a method as described above or as described above as preferred, wherein step ii) of updating the trained neural network comprises solving objective function: max^(X, G)~D [min (X + N), G)], wherein £1 represents the perturbation space, which is ||A|| e.

[0027] The above-described aspect of the present disclosure has the advantage that the max-min game of the objective function focuses on refining the trained neural network by training such trained neural network to refusc / reject false positive examples, as compared to the commonly used adversarial training that uses a min-max game that accepts false negative examples. The “min” portion of the objective function ensures that the inverse adversarial examples are heavily perturbed and can be predicted by the trained neural network with overly high confidence, while the “max” portion of the objective function ensures that the trained neural network successfully and correctly refuses / rejects the inverse adversarial examples.

[0028] A preferred method of the present disclosure is a method as described above or as described above as preferred, wherein step ii) of updating the trained neural network comprises updating parameters 0 of the trained neural network using the gradient descent: 9i = Si-i + (X + N^XG) wherein i] refers to the learning rate.

[0029] Tire above-described aspect of the present disclosure has the advantage that the gradient descent allows easy realisation using backpropagation techniques as it computes the gradient of the loss function layer and propagates the gradient to die layers of the trained neural network to update the parameters 9 iteratively.

[0030] The above-described advantageous aspects of a computer-implemented method of the present disclosure also hold for all aspects of a below-described trained neural network of the present disclosure. All below-described advantageous aspects of a trained neural network of the present disclosure also hold for all aspects of an above-described computer-implemented method of the present disclosure.

[0031] The present disclosure also relates to a trained neural network that was refined with the method of the present disclosure.

[0032] Tire above-described advantageous aspects of a computer-implemented method or trained neural network of the present disclosure also hold for all aspects of a below-described data processing system of the present disclosure. All below-described advantageous aspects of a use of a data processing system of the present disclosure also hold for all aspects of an above-described computer-implemented method or trained neural network of the present disclosure.

[0033] Tire present disclosure also relates to a data processing system comprising means for carrying out the steps of the computer-implemented method of the present disclosure.

[0034] The above-described advantageous aspects of a computer-implemented method, trained neural network, or data processing system of the present disclosure also hold for all aspects of a below-described computer program, a machine-readable storage medium, or a data carrier signal of the present disclosure. All below-described advantageous aspects of a computer program, a machine-readable storage medium, or a data carrier signal of the present disclosure also hold for all aspects of an above-described computer-implemented method, trained neural network, or data processing system of the present disclosure.

[0035] The present disclosure also relates to a computer program, a machine-readable storage medium, or a data carrier signal that comprises instructions, that upon execution on a data processing system, cause the system to perform the steps of a computer-implemented method according to the present disclosure.

[0036] In some embodiments, the present disclosure studies the out-of-distribution problem in view of inverse adversarial examples. In some situations, input images with heavy noise may be classified by a trained neural network as a particular class with counter-intuitive high confidence, which is a known problem in applications such as testing. For example, characters on a screen may contain tiny perturbations arising from transmission problems or display errors. As manually recording such cases is time and labour-consuming, the present disclosure seeks to refine / improve trained neural networks to have a higher specificity that can provide different prediction results when presented with input images with a certain degree of perturbation. The present disclosure seeks to achieve higher specificity by-discovering cases that have different predictions by humans and a trained neural network (also termed as “inverse adversarial example(s)” or “IAE(s)”) and using such cases to serve as augmentation for refining the trained neural network (also termed as “inverse adversarial training”). Inverse adversarial examples have two principles: (1) a trained neural network outputs a prediction label with high confidence when faced with an inverse adversarial example; and (2) they are heavily perturbed, and humans cannot recognize the original classes.

[0037] In some embodiments, the present disclosure uses inverse adversarial examples to find the weakness of a trained neural network that was well-trained in terms of resisting imperceptible perturbations through adversarial training but may have overconfidence when outputting a prediction label for heavily perturbed input. BRIEF DESCRIPTION OF THE DRAWINGS

[0038] These and other features, aspects, and advantages will become better understood with regard to the following description, appended claims, and accompanying drawings where:

[0039] Fig. 1 is a schematic illustration of a method of refining a trained neural network, m accordance with embodiments of the present disclosure;

[0040] Fig. 2 is an example of an algorithm that may be used to generate inverse adversarial examples for a trained neural network configured to classify input image data, also known as a classifier, in accordance with embodiments of the present disclosure;

[0041] Fig. 3 is an example of an algorithm that may be used to refine a trained neural network configured to classify input image data, also known as a classifier, in accordance with embodiments of the present disclosure;

[0042] Fig. 4 illustrates the result of a comparison experiment between inverse adversarial examples and random noise in the MNIST dataset, in accordance with embodiments of the present disclosure; and

[0043] Figs. 5 A to 5D illustrate examples of the prediction results of the classification model on sample cases of IAE and RND, in accordance with embodiments ofthe present disclosure.

[0044] In the drawings, like parts are denoted by like reference numerals.

[0045] It should be appreciated by those skilled in the art that any block diagrams herein represent conceptual views of illustrative systems embodying the principles of the present subject matter. Similarly, it will be appreciated that any flow charts, flow diagrams, state transition diagrams, pseudo code, and the like represent various processes which may be substantially represented in computer readable medium and executed by a computer or processor, whether or not such computer or processor is explicitly shown. DETAILED DESCRIPTION

[0046] In the summary above, in this description, in the claims below, and in tire accompanying drawings, reference is made to particular features (including method steps) of the disclosure. It is to be understood that the disclosure in this specification includes all possible combinations of such particular features. For example, where a particular feature is disclosed in the context of a particular aspect or embodiment of the disclosure, or a particular claim, that feature can also be used, to the extent possible, in combination with and / or in the context of other particular aspects and embodiments of the disclosure, and in the disclosure generally.

[0047] In the present document, the word “exemplary” is used herein to mean “serving as an example, instance, or illustration.” Any embodiment or implementation of the present subject matter described herein as “exemplary” is not necessarily be construed as preferred or advantageous over other embodiments.

[0048] While the disclosure is susceptible to various modifications and alternative forms, specific embodiment thereof has been shown by way of example in the drawings and will be described in detail below. It should be understood, however that it is not intended to limit the disclosure to the forms disclosed, but on the contrary, the disclosure is to cover all modifications, equivalents, and alternative falling within the scope of the disclosure.

[0049] Fig. 1 is a schematic illustration of a method 100 of refining a trained neural network, in accordance with embodiments of the present disclosure. Method 100 for refining a trained neural network may be implemented by a data processing system on any architecture and / or computing system. For example, various architectures employing, for example, multiple integrated circuit (IC) chips and / or packages, and / or various computing devices and / or consumer electronic (CE) devices, such as multi-function devices, tablets, smart phones, etc., may implement the techniques and / or arrangements described herein. Method 100 maybe stored as executable instructions that, upon execution on a data processing device and / or control unit, cause the data processing device and / or control unit to perform the steps of method 100.

[0050] According to some embodiments, method 100 may be used to refine attained neural network 108 trained on a set of training data 116. The set of training data 116 may comprise a plurality of input data 120 and an associated ground truth label 122 for each input data 120. Although the input data 120 illustrated in Fig. 1 is image data, input data 120 may be any data, including images, frames of a video, auditory data, etc. Tire ground truth label 122 may be any label associated with any task, including classification tasks and regression tasks. The example of task illustrated in Fig. 1 is that of number image classification task, and an example of a set of training data for a number image classification task is the MNIST database (Modified National Institute of Standards and Technology database) available at https: / / www.tensorflow.org / datasets / catalog / mnist, which comprises image data of handwritten digits, wherein each image is associated with a class label corresponding to the digit depicted in the image. An example of a set of training data for an image classification task is the CIFAR-10 dataset available at https: / / www.cs.toronto.edu / ~kriz / cifar.html, comprising image data in a number of predetermined classes, wherein each image is associated with a class label corresponding to the object depicted in the image. [00511 According to some embodiments, the trained neural network 108 may be configured to receive input data 120, determine prediction logits 124 and output a predicted label and a confidence level for each input data 120 based on the determined prediction logits 124. The trained neural network 108 may have any architecture suitable for the particular input data and task the trained neural network is used or trained for. For example, the trained neural network 108 may be ResNet50 if the input data comprises images. An example of the architecture of ResNet50 may be found at least in “Deep Residual Learning for Image Recognition" by He et. al. For example, ResNet50 may be trained for 200 epochs, with a learning rate of 0.001 with decay per 60 epochs, and with a cross-entropy loss function. In another example, convolutional neural networks (CNNs) may be used if the input data comprises audio signal data. Examples of CNNs that may be used for audio signal data include VGGish available at https: / / github.com / tensorflow / models / tree / master / research / audioset / vggish, YAMNet (Yet Another Multitask Network) available at https: / / www.tensorflow.org / hub / tutorials / yamnet, and CRNN (Convolutional Recurrent Neural Network). In some embodiments, the trained neural network 108 may be trained using a loss function that minimises a difference between prediction logits 124 generated with the input data 120 and ground truth logits 128 generated from the ground truth label 130 associated with the input data 120 using an encoder that converts the ground truth label 122 to an integer digit. The encoder converts categorical variables into numeric variables. An example of an encoder is the one-hot encoder which converts labels into feature vectors (e.g., a “0” label may be converted to [1,0,0,0,0,0,0,0,0,0]). Another example of an encoder is the label encoder which assigns each category an integer value.

[0052] According to some embodiments, method 100 may comprise step SI wherein a set of inverse adversarial examples 132 is generated using the set of training data 116 and the trained neural network 108. The set of inverse adversarial examples 132 may comprise a plurality of inverse adversarial examples 134 and an associated ground truth label 122 for each inverse adversarial example 134 which corresponds to the ground truth label 122 of the input data 120 that the inverse adversarial example 134 was generated from. An inverse adversarial example 132 is perturbed data generated from an input data 120 by applying perturbation noise N 138 with a norm above a predetermined minimum perturbation 6 to the input data 120 such that when an inverse adversarial example 134 is input into the trained neural network 108, the trained neural network 108 outputs a predicted label with a confidence level above a predetermined threshold. The predicted label may correspond to the ground truth label 122 associated with the input data 120 from which the inverse adversarial example 134 was generated, or may correspond to any other in-distribution label. In some embodiments, the perturbation noise N 138 may be random noise perturbations. The perturbation noise N 134 may be determined by the task that the trained neural network 108 is trained. For example, for the task of digital image reconstruction, the perturbation noise N 138 may be additive noise which affects the digital result on digital images. In some embodiments, the perturbation noise N 138 may comprise more complicated degradations such as test cut. In some embodiments, each pixel or datapoint of perturbation noise N 138 may be largerthan the predetermined minimum perturbation e. The predetermined minimum perturbation e may be any predetermined minimum perturbation level that is sufficient to render the perturbed data unrecognisable by humans. For example, the predetermined minimum perturbation 6 may be between 0.7 and 1. The predetermined threshold for the confidence level may be any predetermined threshold. For example, the predetermined threshold may be a confidence level of between 70-80% confidence to better identify perturbed data that the trained neural network 108 may correctly predict with an overly high confidence (or overconfidence).

[0053] According to some embodiments, generating an inverse adversarial example 134 comprises adding perturbation noise N 138 to input data 120 to generate perturbed data and may be expressed by the equation: X = X + N (1) wherein X represents the original input data, N represents the perturbation noise, and X represents the generated perturbed data. In general, the goal of step SI is to find a heavy perturbation N such that trained neural network 108 outputs a predicted label with overly high confidence with the generated perturbed data as input.

[0054] According to some embodiments, step SI of generating a set of inverse adversarial examples may comprise iteratively (i) applying perturbation noise N 138 to input data 120 to generate perturbed data, wherein the perturbation noise N 138 has a norm higher than a predetermined minimum perturbation e; and (ii) updating the perturbation noise N 138 by adding a sign gradient of a first objective function.

[0055] In some embodiments, the first objective function may be computed based on minimising a distance between ground truth logits 128, and prediction logits 144 generated by the trained neural network 108 with the perturbed data as input. In some embodiments, the ground truth logits 128 may be generated from the ground truth label 122 associated with the input data 120 using an encoder that converts the ground truth label 122 to an integer digit. The encoder converts categorical variables into numeric variables. An example of an encoder is the one-hot encoder which converts labels into feature vectors (e.g., a “0" label may be converted to [1,0,0,0,0,0,0,0,0,0]). Another example of an encoder is the label encoder which assigns each category an integer value.

[0056] In some embodiments, for a classification task wherein the predicted label comprises a classification task label, the first objective function may be computed based on maximising the prediction confidence or probability of any one class. In such embodiments, the first objective function may use prediction logits 144 generated by the trained neural network 108 with the perturbed data as input without necessarily using the ground truth logits 128.

[0057] According to some embodiments, generating the set of inverse adversarial examples comprises solving an objective function: argmin(^( / e(X + N),G)), subject to ||N|| >e, (2) N wherein / (-) represents the trained neural network 108, G represents the ground truth label 160, f (■) represents a loss function, X represents the input data, N represents the perturbation noise, and e represents a predetermined minimum perturbation. In general, Equation (2) may be solved by sign gradient descent. By solving Equation (2), the inverse adversarial examples (X) with the highest potential for staying in the decision boundaries of the trained neural network 108 may be identified. In some embodiments, objective function Equation (2) may seek to minimise the distance between ground truth logits 128, and prediction logits 144 generated by the trained neural network 108. An example of a loss function / (■) that may be employed in objective function Equation (2) seeking to minimise the distance between ground truth logits 128, and prediction logits 144 generated by the trained neural network 108 is a cross entropy loss function.

[0058] In some embodiments, in particular for classification tasks, objective function Equation (2) may seek to maximise a prediction confidence of any one in-distribution class. An example of a loss function ^(-) that may be employed in objective function Equation (2) seeking to maximise a prediction confidence of any one class is a negative mean maximum confidence (nMMC) loss function which may be expressed as follows: 1^7 X <3) + N),G) = 2, (max(Mx™ + N))) m=l wherein / (■) represents the trained neural network 108, max(-) extracts the maximum value in the prediction logits, Xm represents the m-th input data, M refers to the number of data, and N represents the perturbation noise.

[0059] According to some embodiments, updating perturbation noise N over a number of attack iterations T may comprise computing the sign gradient with respect to objective function Equation (2) and adding it to perturbation noise N, which may be expressed as: Nt = + asign(V-fN(^) (4) wherein a represents the step size of each iteration, sign(-) represents the signum function, V / Ntl represents the gradient of Nt_t with respect to objective function Equation (2). Updating perturbation noise N over a number of attack iterations T may comprise updating each data point (or pixel value in cases of image data) in perturbation noise N 138.

[0060] According to some embodiments, step S1 of generating a set of inverse adversarial examples 132 may comprise carrying out, in each iteration t: 1) compute the objective loss based on Equation (2) and the derivative of this loss with respect to A. By introducing the sign function and subsequently taking the negative, the gradient gradt is obtained; 2) after scaling the gradient by the step size a for each iteration, update Nt based on Equation (4); 3) enforce adjustments on each data point (or pixel for image data) to adhere to the minimum perturbation constraint 6 imposed by the || A||, ensuring the resultant perturbation is perceptually significant; 4) combine X and Nt to yield the IAE Xt; and 5) after T iterations, the final IAE X = XT is generated.

[0061] Fig. 2 is an example of an algorithm that may be used to generate inverse adversarial examples for a trained neural network 108 configured to classify input image data, also known as a classifier, in accordance with embodiments of the present disclosure.

[0062] Returning to Fig. 1, according to some embodiments, method 100 may comprise step S2 wherein the trained neural network 108 is updated using the generated set of inverse adversarial examples 132 to generate an updated trained neural network 108. In some embodiments, step S2 may comprise refining the parameters 9 of the trained neural network 108. In some embodiments, step S2 may seek to reveal the incorrect accept cases of the trained neural network 108 and enhance the specificity of the trained neural network 108 by using the capabilities of the set of inverse adversarial examples 132. Updating the trained neural network 108 is also termed as Inverse Adversarial Training (or IAT), which employs a max-min game.

[0063] According to some embodiments, step S2 may comprise using an objective function, wherein the objective function is computed based on minimising a loss function, while simultaneously optimising parameters 9 of the trained neural network 108 to maximise such loss function.

[0064] According to some embodiments, step S2 may comprise solving the objective function: max^(X,G)~2)[mn<^(X + N),G)], (5) wherein Q represents the perturbation space, which is ||A|| 3» e, f(-) represents the trained neural network, fQ) represents a loss function. G represents the ground truth label and N represents the perturbation noise. The loss function may be cross-entropy loss function or negative mean maximum confidence (nMMC) loss function. In general, the overall objective function may seek to maximise cross-entropy loss or maximise nMMC loss to update or refine the trained neural network to evenly or uniformly distribute the prediction confidence level amongst all possible outputs. Cross-entropy loss is determined using both ground truth logits 128 and prediction logits 152 generated by the trained neural network 108 w ith the inverse adversarial examples 134 as input, while nMMC loss is determined using prediction logits 152 generated by the trained neural network 108 with the inverse adversarial examples 134 as input without necessarily using ground truth logits 128.

[0065] According to some embodiments, step S2 may comprise updating parameters 9 of the trained neural network 108 in each training epoch i using the gradient descent: di = Ot-i + + Ni-O, G) (6) wherein r| refers to tire learning rate. The gradient of Equation (6) may be computed and the trained neural network fe 108 in each iteration with learning rate q. In some embodiments, any well-known optimisation algorithms such as stochastic gradient descent (SGD), stochastic gradient descent with momentum (SGDM), or Adam optimizer may be used.

[0066] According to some embodiments, step S2 of updating trained neural network 108 may comprise carrying out, in each epoch / : 1) generate IAE Xt with the trained neural network fg of a previous epoch / -1; 2) use the IAE Xt in Equation (6) to update the trained neural network fe.. 3) after I epochs, the final trained neural network fg = fgt is generated.

[0067] Fig. 3 is an example of an algorithm that may be used to refine a trained neural network 108 configured to classify input image data, also known as a classifier, in accordance with embodiments of the present disclosure.

[0068] According to some embodiments, method 100 may comprise step S3 (not illustrated) wherein steps SI and S2 are repeated based on the updated trained neural network from step S2 until a predetermined condition is met. The predetermined condition may be a number of iterations, or a specific value of an objective function.

[0069] Experiments were conducted to demonstrate the generation of inverse adversarial examples. Experiments were carried out on the MNIST dataset to show that the inverse adversarial examples generated can reveal the counter-intuitive high accuracy of highly perturbed images, especially in relation to a classification task. The accuracy of a trained ResNet50 model in relation to clean inputs and inverse adversarial examples were compared quantitatively and qualitatively.

[0070] Fig. 4 illustrates the result of a comparison experiment between inverse adversarial examples and random noise in the MNIST dataset, in accordance with embodiments of the present disclosure. Randomly generated perturbations (RND) and inverse adversarial examples (IAE) were generated from the digital images from the MNIST dataset and the prediction results were evaluated using a well-trained ResNet50 classification model that had 97% accuracy on the clear dataset. A plot figure was generated by sliding the perturbation level e from 0 to 0.9 to illustrate the capacity of inverse adversarial examples. As illustrated in Fig. 4, both IAE and RND affected the prediction accuracy of the classification model. The capacity of the classification model to classify RND appears to be generally aligned with the capacity of human beings. For example, when the perturbation is heavy, e.g., € >0.5, the image is hard to recognise and the accuracy of the classification model on RND drops to less than 30%. However, the accuracy of the classification model m relation to IAE is maintained at over 60% with perturbation e = 0.8. The results illustrate that classification models may yield different results as compared to humans when faced with input with heavy perturbation.

[0071] Figs. 5A to 5D illustrate examples of the prediction results of the classification model on sample cases of IAE and RND, in accordance with embodiments ofthe present disclosure. The examples illustrated were perturbed with a minimum perturbation level € = 0.8. As shown in Fig. 5B, the classification model correctly predicted the classification of clear input images, which indicated the recognition ability of the classification model. As illustrated in Fig. 5C, the classification model failed to correctly predict the ground truth class of most of the samples. Fig. 5D illustrates the prediction results of the classification model on IAEs. In general, the perturbation level of the IAEs are higher than the perturbation level of the RN Ds due to the search strategy used when generating IAEs. Despite the higher perturbation levels of the IAEs, the classification model was able to correctly predict the ground truth class of most of the samples, as illustrated in Fig. 5D.

[0072] Finally, the language used in the specification has been principally selected for 5 readability and instructional purposes, and it may not have been selected to delineate or circumscribe the inventive subject matter. It is therefore intended that the scope of the invention be limited not by this detailed description, but rather by any claims that issue on an application based here on. Accordingly, the embodiments of the present disclosure are intended to be illustrative, but not limiting, of the scope of the invention, which is set forth 10 in the following claims.

Claims

1. A computer-implemented method (100) for refining a trained neural network trained on a set of training data comprising input data and an associated ground truth label for each input data, the trained neural network configured to receive input data, determine prediction logits and output a predicted label and confidence level based on the determined prediction logits, wherein the method for refining the trained neural network comprises the steps of:i) generating (SI) a set of inverse adversarial examples using the set of training data and the trained neural network, wherein inverse adversarial examples are perturbed data generated from input data by applying perturbation noise N with a norm above a predetermined minimum perturbation 6 to such input data such that when an inverse adversarial example is input into the trained neural network, the trained neural network outputs a predicted label with a confidence level above a predetermined threshold;ii) updating (S2) the trained neural network using the generated set of inverse adversarial examples to generate an updated trained neural network; andiii) repeating (S3) steps i) and n) based on the updated trained neural network until a predetermined condition is met.

2. The computer-implemented method of claim 1, wherein the input data comprises image data or audio data, and the ground truth label and the predicted label comprises a classification or regression task label.

3. The computer-implemented method of any of the preceding claims, wherein step i) of generating a set of inverse adversarial examples comprises iteratively:applying perturbation noise N to input data to generate perturbed data; andupdating the perturbation noise N by adding a sign gradient of a first objective function.

4. Tire computer-implemented method of claim 3, wherein tire first objective function is computed based on minimising a distance between ground truth logits, and prediction logits generated by the trained neural network with the perturbed data as input.

5. The computer-implemented method of claim 3, wherein the predicted label comprises a classification task label and the first objective function is computed based on maximising the prediction confidence of any one class.

6. The computer-implemented method of any of the preceding claims, wherein step i) of generating the set of inverse adversarial examples comprises solving objective function:argmin(^( / 0(X + N), G)), subject to ||N|| >e, Nwherein / (•) represents the trained neural network, G represents the ground truth label, / (•) represents a loss function, X represents the input data, N represents the perturbation noise, and 6 represents the predetermined minimum perturbation.

7. The computer-implemented method of claim 6, wherein the loss function is a crossentropy loss function.

8. Tire computer-implemented method of claim 6, wherein the predicted label comprises a classification task label and the loss function is a negative mean maximum confidence (nMMC) loss function.

9. The computer-implemented method of any of the preceding claims, wherein step ii) of updating the trained neural network comprises using a second objective function, wherein the second objective function is computed based on minimising a loss function, while simultaneously optimising parameters of the trained neural network to maximise such loss function.

10. The computer-implemented method of any of the preceding claims, wherein step ii) of updating the trained neural network comprises solving objective function:max|(X, G)~D [min (X + N), G)], wherein Q represents the perturbation space, which is ||JV|| S' e.

11. The computer-implemented method of any of the preceding claims, wherein step ii) of updating the trained neural network comprises updating parameters 0 of the trained neural network using the gradient descent:= +wherein r| refers to the learning rate.

12. A trained neural network that was refined with the method of any of the preceding claims.

13. A data processing system comprising means for carrying out the steps of computer-implemented method according to any of claims 1 to 11.

14. A computer program, a machine-readable storage medium, or a data carrier signal that comprises instructions, that upon execution on a data processing system, cause the 5 system to perform the steps of a computer-implemented method according to any of claimsIto 11.

Images