Attack system, method, and program product for generating adversarial perturbations of input signals

Abstract

The present disclosure relates to attack systems, methods, and program products for generating adversarial perturbations of input signals. An attack system for generating perturbations of input signals in a recurrent neural network (RNN)-based target system using a deep reinforcement learning agent for generating perturbations is described. The attack system trains the reinforcement learning agent to determine the magnitude of the perturbations used to attack the RNN-based target system. A perturbed input sensor signal with the determined magnitude is generated and presented to the RNN-based target system so that the RNN-based target system produces an altered output in response to the perturbed input sensor signal. The system uses the altered output to identify failure modes of the RNN-based target system.

Description

[0001] CROSS-REFERENCE TO RELATED APPLICATIONS

[0002] This application is a non-provisional application of U.S. Provisional Application No. 62 / 905,106, entitled “A Deep Reinforcement Learning Based Method for Surreptitiously Generating Signals to Fool a Recurrent Neural Network”, filed in the United States on September 24, 2019, the entire contents of which are incorporated herein by reference. Technical Field

[0003] The present invention relates to an attack system for generating perturbations to an input signal of a target system having a recurrent neural network as a component, and more particularly, to an attack system for generating perturbations to an input signal of a target system using a deep reinforcement learning agent for generating perturbations. Background Art

[0004] Neural network-based systems, such as those used for static image classification, are susceptible to so-called "adversarial perturbations". Adversarial perturbations are small perturbations of the input image that are barely detectable to humans, but that fundamentally change the output of the neural network. Adversarial perturbations are specialized inputs that are created with the purpose of confusing the neural network, resulting in misclassification of a given input. The presence of such perturbations can potentially be exploited by attackers and, more generally, reveals the robustness of these models to failures caused by small amounts of noise.

[0005] Initial research has been conducted to explore perturbation-based attacks in the setting of recurrent neural networks (RNNs), which are used to process sequence-based inputs (such as video, text, audio, or time series data), and the research performs original perturbation attacks on each input at each time step. The present invention shows that these types of attacks may be a more significant challenge for RNN-based systems than previously thought. Goodfellow et al. described an original perturbation-based attack (called the fast gradient sign method (FSGM)) on a (static) image classifier in "Explaining and Harnessing Adversarial Examples" (which is incorporated herein by reference as if fully described herein (hereinafter referred to as Goodfellow et al.)) at The International Conference on Learning Representations (ICLR), 2015. This work is not directly applicable to RNNs. However, Papernot et al. gave an extension to RNN in "Crafting Adversarial Input Sequences for Recurrent Neural Networks" (which is incorporated herein by reference as if fully described herein) at Military Communications Conference, 2016.

[0006] Prior work on RNN-based time series attacks (i.e., as described above) relies on manually designed attack sizes and attack lengths, which require experts to tune through experiments. Furthermore, prior work on variable attack sizes applies iterative attacks that are not conducive to online attacks on time series RNNs (such as RNNs that process real-time sensor readings).

[0007] Therefore, there remains a need for a system that learns the timing and size of adversarial perturbations, rather than requiring expert tuning, and that can be used to attack RNNs online. Summary of the invention

[0008] The present invention relates to an attack system for generating perturbations to an input signal of a target system having a recurrent neural network as a component, and more specifically, to an attack system for generating perturbations to an input signal of a target system using a deep reinforcement learning agent for generating perturbations. The attack system includes a non-transitory computer-readable medium and one or more processors, on which executable instructions are encoded, so that when the executable instructions are executed, the one or more processors perform multiple operations. The attack system trains a reinforcement learning agent to determine the amplitude of the perturbation used to attack the RNN-based target system. A perturbed input sensor signal with the determined amplitude is generated and presented to the RNN-based target system, so that the RNN-based target system generates a modified output in response to the perturbed input sensor signal. The system uses the modified output to identify a failure mode of the RNN-based target system.

[0009] In another aspect, the reinforcement learning agent is trained to learn the timing of the perturbations.

[0010] In another aspect, the perturbed input sensor signal is generated using an attack generator.

[0011] On the other hand, at each time step of training the reinforcement learning agent, the system presents unattacked sensor data including known attack attributes to the reinforcement learning agent, wherein the reinforcement learning agent outputs a set of probability distribution parameters, and the attack parameter set is sampled by the attack generator from the set of probability distribution parameters.

[0012] In another aspect, the system determines a scalar value using the unattacked sensor data, the set of attack parameters, the perturbed input signal, and the altered output, wherein the scalar value represents a relative success rate of the attack associated with the perturbed input sensor signal.

[0013] In another aspect, the system provides the scalar value as a reward signal to the reinforcement learning agent, thereby improving an attack strategy of the reinforcement learning agent.

[0014] Finally, the present invention also includes a computer program product and a computer-implemented method. The computer program product includes computer-readable instructions stored on a non-transitory computer-readable medium, and the computer-readable instructions can be executed by a computer having one or more processors, so that when the instructions are executed, the one or more processors perform the operations listed herein. Alternatively, the computer-implemented method includes the actions of causing a computer to execute these instructions and perform the resulting operations. BRIEF DESCRIPTION OF THE DRAWINGS

[0015] Objects, features and advantages of the present invention will become apparent from the following detailed description of various aspects of the present invention, taken in conjunction with the following drawings, in which:

[0016] Figure 1 is a block diagram depicting components of an attack system according to some embodiments of the present disclosure;

[0017] Figure 2 is an illustrative diagram of a computer program product according to some embodiments of the present disclosure;

[0018] Figure 3 is a flow chart illustrating an attack system in operation according to some embodiments of the present disclosure;

[0019] Figure 4 is a flow chart illustrating a process for training components of an attack system according to some embodiments of the present disclosure;

[0020] Figure 5 is a table illustrating experimental results comparing the present invention with conventional methods according to some embodiments of the present disclosure;

[0021] Fig. 6A is a graph illustrating results of anomaly detection on an electrocardiogram (ECG) dataset using a reinforcement learning (RL) agent attack based on sensors and attack size according to some embodiments of the present disclosure;

[0022] Figure 6B is a chart illustrating the results of anomaly detection on an ECG dataset using an RL agent attack based on anomaly scores according to some embodiments of the present disclosure;

[0023] Fig. 7A is a graph illustrating results of anomaly detection on an ECG dataset based on sensor and attack size using a fixed attack size; and

[0024] Figure 7B is a graph illustrating the results of anomaly detection on an ECG dataset using a fixed attack size based on anomaly scores. DETAILED DESCRIPTION

[0025] The present invention relates to an attack system for generating perturbations to an input signal of a target system having a recurrent neural network as a component, and more specifically, to an attack system for generating perturbations to an input signal of a target system using a deep reinforcement learning agent for generating perturbations. The following description is presented to enable a person of ordinary skill in the art to make and use the present invention and incorporate it into the context of a specific application. Various modifications and multiple uses in different applications will be apparent to those skilled in the art, and the general principles defined herein can be applied to a wide range of aspects. Therefore, the present invention is not intended to be limited to the aspects presented, but is consistent with the broadest scope consistent with the principles and novel features disclosed herein.

[0026] In the following detailed description, many specific details are set forth in order to provide a more thorough understanding of the present invention. However, it will be apparent to those skilled in the art that the present invention can be practiced without being limited to these specific details. In other cases, known structures and devices are shown in block diagram form rather than in detail to avoid obscuring the present invention.

[0027] The reader's attention is drawn to all files and documents filed simultaneously with this specification and open to public inspection together with this specification, and the contents of all these files and documents are incorporated herein by reference. Unless otherwise expressly stated, all features disclosed in this specification (including any attached claims, abstracts and drawings) may be replaced by alternative features for the same, equivalent or similar purposes. Therefore, unless otherwise expressly stated, each feature disclosed is only an example in a series of equivalent or similar features.

[0028] (1) Main aspects

[0029] Various embodiments of the present invention include three "main" aspects. The first main aspect is an attack system that generates a disturbance of the input signal of the target system. The system is usually in the form of computer system operating software or in the form of a "hard-coded" instruction set. The system can be incorporated into a variety of devices that provide different functions. The second main aspect is a method that is usually in the form of software using a data processing system (computer). The third main aspect is a computer program product. A computer program product usually represents computer-readable instructions stored on a non-transitory computer-readable medium such as an optical storage device (e.g., a compact disk (CD) or a digital versatile disk (DVD)) or a magnetic storage device (such as a floppy disk or a magnetic tape). Other non-limiting examples of computer-readable media include: a hard disk, a read-only memory (ROM), and a flash memory. These aspects will be described in more detail below.

[0030] Figure 1A block diagram depicting an example of a system of the present invention (i.e., computer system 100) is provided in . The computer system 100 is configured to perform calculations, processes, operations, and / or functions associated with a program or algorithm. In one aspect, certain processes and steps discussed herein are implemented as a series of instructions (e.g., software programs) that reside in a computer-readable memory unit and are executed by one or more processors of the computer system 100. When executed, the instructions cause the computer system 100 to perform specific actions and exhibit specific behaviors, as described herein.

[0031] Computer system 100 may include an address / data bus 102 configured to transmit information. In addition, one or more data processing units (such as processor 104 (or multiple processors)) are connected to address / data bus 102. Processor 104 is configured to process information and instructions. In one aspect, processor 104 is a microprocessor. Alternatively, processor 104 can be a different type of processor, such as a parallel processor, an application specific integrated circuit (ASIC), a programmable logic array (PLA), a complex programmable logic device (CPLD) or a field programmable gate array (FPGA).

[0032] The computer system 100 is configured to utilize one or more data storage units. The computer system 100 may include a volatile memory unit 106 (e.g., random access memory (“RAM”), static RAM, dynamic RAM, etc.) coupled to the address / data bus 102, wherein the volatile memory unit 106 is configured to store information and instructions for the processor 104. The computer system 100 may also include a non-volatile memory unit 108 (e.g., read-only memory (“ROM”), programmable ROM (“PROM”), erasable programmable ROM (“EPROM”), electrically erasable programmable ROM (“EEPROM”), flash memory, etc.) coupled to the address / data bus 102, wherein the non-volatile memory unit 108 is configured to store static information and instructions for the processor 104. Alternatively, the computer system 100 may execute instructions retrieved from an online data storage unit such as in “cloud” computing. In one aspect, the computer system 100 may also include one or more interfaces, such as interface 110, coupled to the address / data bus 102. The one or more interfaces are configured to enable the computer system 100 to connect to other electronic devices and computer systems. The communication interface implemented by the one or more interfaces may include wired (e.g., serial cable, modem, network adapter, etc.) and / or wireless (e.g., wireless modem, wireless network adapter, etc.) communication technology.

[0033] In one aspect, the computer system 100 may include an input device 112 coupled to the address / data bus 102, wherein the input device 112 is configured to transmit information and command selections to the processor 104. According to one aspect, the input device 112 is an alphanumeric input device (such as a keyboard), which may include alphanumeric keys and / or function keys. Alternatively, the input device 112 may be an input device other than an alphanumeric input device. In one aspect, the computer system 100 may include a cursor control device 114 coupled to the address / data bus 102, wherein the cursor control device 114 is configured to transmit user input information and / or command selections to the processor 104. In one aspect, the cursor control device 114 is implemented using a device such as a mouse, a trackball, a trackpad, an optical tracking device, or a touch screen. Notwithstanding the foregoing, in one aspect, the cursor control device 114 is directed and / or enabled via input from the input device 112, such as in response to using special keys and key sequence commands associated with the input device 112. In an alternative aspect, cursor control device 114 is configured to be managed or directed by voice commands.

[0034] In one aspect, computer system 100 may also include one or more optional computer usable data storage devices, such as storage device 116 coupled to address / data bus 102. Storage device 116 is configured to store information and / or computer executable instructions. In one aspect, storage device 116 is a storage device such as a disk drive or optical drive (e.g., a hard disk drive ("HDD"), a floppy disk, a compact disk read only memory ("CD-ROM"), a digital versatile disk ("DVD")). According to one aspect, display device 118 is coupled to address / data bus 102, wherein display device 118 is configured to display video and / or graphics. In one aspect, display device 118 may include a cathode ray tube ("CRT"), a liquid crystal display ("LCD"), a field emission display ("FED"), a plasma display, or any other display device suitable for displaying video and / or graphic images and alphanumeric characters recognizable by a user.

[0035] The computer system 100 presented herein is an example computing environment according to one aspect. However, the non-limiting example of the computer system 100 is not strictly limited to being a computer system. For example, one aspect provides a computer system 100 representing a class of data processing analysis that can be used according to various aspects described herein. In addition, other computing systems can also be implemented. In fact, the spirit and scope of the present technology are not limited to any single data processing environment. Therefore, on the one hand, one or more operations of various aspects of the present technology are controlled or implemented using computer executable instructions (such as program modules) executed by a computer. In one implementation, such a program module includes routines, programs, objects, components and / or data structures configured to perform specific tasks or implement specific abstract data types. In addition, one aspect provides one or more aspects of the present technology implemented by utilizing one or more distributed computing environments, such as, in this computing environment, tasks are performed by remote processing devices linked by a communication network, or such as, in this computing environment, various program modules are located in local and remote computer storage media including memory-storage devices.

[0036] Figure 2 An illustrative diagram of a computer program product (i.e., storage device) embodying the present invention is depicted in . The computer program product is depicted as a floppy disk 200 or an optical disk 202 such as a CD or DVD. However, as previously mentioned, the computer program product generally represents computer-readable instructions stored on any compatible non-transitory computer-readable medium. The term "instructions" as used in connection with the present invention generally indicates a set of operations to be performed on a computer, and may represent a fragment of an entire program or a single separate software module. Non-limiting examples of "instructions" include computer program code (source or target code) and "hard-coded" electronic devices (i.e., computer operations encoded into a computer chip). "Instructions" are stored on any non-transitory computer-readable medium, such as in a computer's memory or on a floppy disk, CD-ROM, and flash drive. In any case, the instructions are encoded on a non-transitory computer-readable medium.

[0037] (2) Specific details of various implementations

[0038] It is well known that neural network-based systems, such as those used for static image classification, are vulnerable to so-called “adversarial perturbations.” An adversarial perturbation is a small perturbation of the input image that is barely detectable by a human, but which fundamentally changes the output of the neural network (e.g., a panda is now classified as a gibbon). The presence of such perturbations can potentially be exploited by an attacker and, more generally, reveals the robustness of these models to failures caused by small amounts of noise.

[0039] It is sometimes claimed that adversarial perturbations should not be a serious problem for neural network-based systems operating in real time, such as systems for processing video data. The reasoning is that if the perturbation is only on a single frame out of a large total of frames, then the misclassification due to such a perturbation is insignificant. The invention described herein is an approach that shows that there is a significant problem for real-time systems. Specifically, given a real-time neural network-based system (which is implemented as a recurrent neural network (RNN)), a system according to an embodiment of the present disclosure uses a deep reinforcement learning (RL) agent to generate small changes to the input signal in order to produce radically different network outputs. The use of reinforcement learning (RL) in the context of performing such attacks has never been explored before.

[0040] The invention described herein significantly improves existing attacks on RNN-based systems by using a deep RL-based scheduler that modifies the timing and amplitude of the attacked inputs in a way that significantly affects the output of the attacked RNN. This process extends the amount of time that the attack is effective and reduces the amount of changed data required to perform the attack, making the attack more severe and more difficult for defenders to detect. In initial proof-of-concept experiments, the effectiveness of the attack was observed to increase by 33%, the duration was observed to increase by 28%, and the number of time steps attacked was observed to decrease by 80%.

[0041] (2.1) Operation of attack system

[0042] The present invention includes Figure 3 The attack system shown and Figure 4 The process of training components of an attack system is shown. The present invention assumes the existence of a fixed (trained) target RNNf (element 300) that consumes input sensor readings (e.g., video frames, audio signals, system or human health monitoring signals, text), particularly perturbed sensor readings (element 302), and produces output predictions or estimates (element 304) that can be used to identify faults in the target system that could be exploited by a bad actor (adversary) or result in unexpected system behavior.

[0043] The invention described herein has two variants. In the first variant, the source code of f is available to the attack agent π (element 306) and the attack generator (element 312), which is called the white box case. In the second variant, the source code of f is not available to the attack agent π (element 306) and the attack generator (element 312), which is called the black box case. Typically, the black box case uses the white box case, but for another network f', in order to establish a baseline agent. Those skilled in the art will appreciate that the white box case is extended to a black box attack. The focus of the system described herein is the white box case, in which the source code is available.

[0044] In addition to the target RNNf (element 300), the attack system also includes a neural network-based reinforcement learning agent π (i.e., attack agent π (element 306)), which determines the amplitude of the perturbation used to attack the target RNNf (element 300) at each time, noting that an attack with an amplitude of 0 corresponds to no attack at all. The input of the attack agent π (element 306) at a fixed time is the current sensor reading (element 308). The output of the attack agent π (element 306) is the parameters (e.g., mean, covariance matrix) of the probability distribution over the space of attack parameters (e.g., perturbation amplitudes in various dimensions (element 310)). The implementation details of the attack agent π (element 306) are allowed to vary, and it is feasible to use various reinforcement learning methods in the implementation of the attack agent π (element 306). The output of the attack agent π (element 306) is then fed to the attack generator (element 312). There are different approaches in the literature on how to implement the attack generator (element 312); the experimental study of the present invention is based on the fast gradient symbol method (FGSM) of Goodfellow et al. and can be easily recreated by a person skilled in the art. The attack generator (element 312) attempts to generate a white-box attack to ensure that the output estimate of the target RNN (element 300) is incorrect. A white-box attack is a situation where the attacker has full access to the model being attacked. The output of the attack generator (element 312), which is the perturbed sensor reading (element 302), is then fed to the original target RNNf (element 300), which produces a prediction or estimation (element 304). Non-limiting examples of predictions / estimations include the presence of pedestrians in video frames for a video-based object detection / classification system on an autonomous vehicle, the presence of specific words in an audio sentence in a text-to-speech system, and the presence of anomalies in an anomaly detection system.

[0045] (2.2) Training of attack systems

[0046] The training of the attack agent π (element 306) will occur in discrete time steps and multiple training episodes, such as Figure 4As shown. At each time step of an epoch, training examples (unattacked sensor data) containing known attack attributes will be presented to the attack agent π (element 306) in chronological order. The attack agent π (element 306) will output the parameters of the probability distribution (element 402), from which the parameters of the attack performed at that time step will be sampled and provided to the attack generator (element 312). The attack generator will then attack the training examples (sensor data) using the parameters specified by the attack agent π (element 306) (i.e., the attacked sensor data (element 404)) and present the attacked sensor data to the target RNNf (element 300) for prediction (element 304). Any combination of the target RNNf predictions (element 304), the attacked sensor data (element 404), the non-attacked sensor data (element 400), and the attack parameters (element 402) may then be provided to an attack scoring function (element 406) to determine the relative success rate of the attack (i.e., a scalar value (element 408)), which will be provided back to the attack agent π (element 306) as a reward signal to improve its attack strategy. The attack generator and the attack scoring function (element 406) are use case specific and may be defined in any manner that captures the desired properties of the attack and assigns a scalar value (element 408) to each of them regarding the success rate of the attack. At a given training epoch, the attack agent π (element 306) may be presented with many training examples or only a single training example, and training may continue for an indeterminate number of epochs until the attack is deemed successful (e.g., when the attacked input is deemed sufficiently imperceptible by the user while also appropriately changing the target RNN's estimate).

[0047] (2.3) Experimental study

[0048] In the experimental study, the target system is an anomaly detector system that provides an anomaly score at each time step. The anomaly score is calculated using the online multi-step RNN prediction error, and the mean (μ) and covariance (Cov) of the prediction results obtained during the previous RNN training phase are formally defined as follows:

[0049] Anomaly score = (error-μ)Cov -1 (Error-μ) T

[0050] Higher anomaly scores (e.g., Figure 7B 20000) indicates a greater abnormality. These details are familiar to those skilled in the art.

[0051] refer to Figure 3In the embodiment used in the experimental study, the target RNN (element 300) is an anomaly detector system and the current sensor reading (element 308) is a time series sensor reading. For example, in the case of anomaly detection for health monitoring purposes, the current sensor reading (element 308) can be an electrocardiogram signal. The goal of this embodiment is to change the time series sensor reading by adding a small perturbation, resulting in a perturbed sensor reading (element 302) so as to prevent the anomaly from being correctly detected. In the example of health monitoring based on electrocardiogram data, the invention described herein allows its users to identify defects in their health monitoring RNN that, if deployed, may cause the patient's existing health condition to go undetected. In the experimental study, a deep reinforcement learning (RL) agent (attack agent π (element 306)) was trained via a proximal policy optimization algorithm (described by Schulman in Proximal Policy Optimization, 2017, which is incorporated herein by reference as if fully set forth herein). At each time step, the attack agent π (element 306) consumes the current sensor reading (element 308) as state and outputs the size of the attack to be performed (i.e., the attack magnitude (element 310)). The concept includes the ability to select an attack of size 0, so that within a time window, the attack agent π (element 306) can learn a strategy to control the timing and magnitude of each attack.

[0052] In this embodiment, the attack agent π (element 306) consists of two fully connected dense neural networks (actor and critic), each with two hidden layers, each with 16 nodes, and each node with a rectified linear unit (ReLU) activation function. The critic has a single linear output node that is trained to estimate the advantage of a given state, while the actor has two output nodes, namely, the mean and the standard deviation, which have a hyperbolic tangent activation function and a softplus activation function, respectively, which parameterize the normal distribution for sampled actions (e.g., attack size). The policy is restricted to the interval [-1,1] and then clipped to the interval [0,1] before being passed to the attack generator (element 312). Both the actor and the critic are optimized via the Adam optimizer with a learning rate of 1e-3 and a batch size of 64. The Adam optimizer is described in "Adam: A Method for Stochastic Optimization" by Kingman and Ba at ICLR, 2015 (which is incorporated herein by reference as if fully set forth herein). The attack agent π (element 306) maximizes the sum of future discounted rewards and receives the negative logarithm of the anomaly score as a reward so that it will effectively learn an attack strategy that minimizes the total anomaly score for a given set of time-series sensor readings (element 308). This is just one potential formulation of the reward signal; other formulations may emphasize fewer attacks, smaller attacks, or any other combination of characteristics desired by the attacker.

[0053] During training (see Figure 4 ), the attack agent π (element 306) selects a strategy (e.g., attack parameters (element 402) such as attack size) for each time step of the training example, and performs an FGSM attack (see Goodfellow et al.) of the selected size (element 404) on the data before the data is provided to the target RNN (element 300) for prediction (element 304) until the end of the training example.

[0054] After training is complete, the anomaly detector (i.e., the target system (element 300) is then queried for anomaly scores (i.e., predictions / estimates (element 304)) at each time step, and the negative logarithm of these scores is provided as a reward to the attack agent π (element 306), from which random batches of 64 time steps are sampled to compute policy improvement updates familiar to those skilled in the art. The policy rollout, anomaly score calculation, and update steps constitute a single epoch, where policy rollout and updates are the same as in standard reinforcement learning and will be familiar to anyone skilled in the art. The attack agent π (element 306) is trained for 150 such epochs to achieve Figure 5 , FIG. 6A to FIG. 6B and FIG. 7A to FIG. 7B The initial experiments were conducted with a single training example to construct a single adversarial example; however, this approach can be easily extended to incorporate many training examples to construct multiple independent adversarial examples or to train on many examples and learn a more general attack strategy that generalizes well to unseen examples.

[0055] Compared to the traditional FGSM method, the invention described in this article learns a sparser attack pattern and is more successful and effective in fooling the anomaly detection system monitoring the attacked RNN. Figure 5 is a table illustrating experimental results comparing the present invention (element 500) with conventional fixed attack size (element 502) and no attack (element 504). The results are also depicted in Figure 6B and Figure 7B As described below. Figure 5 , the present invention achieves a 61% reduction in the peak anomaly score (element 506) compared to no attack, while the traditional fixed attack size only achieves a reduction of approximately 29% compared to no attack. In addition, the method described herein also reduces the duration of detected anomalies (element 508) by 28% compared to no attack, while the traditional fixed attack size method does not improve in this regard. The present invention achieves these improvements by having an attack time step (element 510) that is 80% less than the traditional fixed attack size method (which further reduces the potential discoverability of the attack).

[0056] Fig. 6A and Figure 6B The present invention illustrates anomaly detection on an electrocardiogram (ECG) dataset using RL agent attack according to an embodiment of the present disclosure. Fig. 6A , the solid line (element 600) represents the attack size and the dashed line (element 602) represents the electrocardiogram sensor reading that monitors the heart's electrical activity. Fig. 6AThe temporal relationship between the timing and amplitude of the attack signal (solid line, element 600) generated by the attack agent and the original input signal (dashed line, element 602) is shown. Figure 6B In FIG. 6 , the non-bold dashed line (element 604 ) represents the anomaly score of the sensor data that has not been attacked, and the bold dashed line (element 606 ) represents the anomaly score of the sensor data that has been attacked. Figure 6B It is shown that in the presence of a perturbed signal generated by an attacking agent, the anomaly score is significantly reduced, meaning that if such perturbations are present in the signal, the anomaly detection system will not be able to correctly identify potential cardiovascular problems in the individual from whom the data was collected.

[0057] Fig. 7A and Figure 7B This paper demonstrates anomaly detection on ECG dataset using fixed attack size using existing techniques. Fig. 7A In FIG. 7 , the solid line (element 700) represents the attack size and the dashed line (element 702) represents the sensor reading. Figure 7B In FIG. 7 , the non-bold dashed line (element 704) represents the anomaly score of the sensor data that has not been attacked, and the bold dashed line (element 706) represents the anomaly score of the sensor data that has been attacked. Figure 6B (the present invention) and Figure 7B (Traditional method), it is obvious that the peak anomaly score (element 606) of the present invention is reduced by 61% compared to no attack (element 604), while the traditional fixed attack size (element 706) is only reduced by 29% compared to no attack (element 704).

[0058] While iterative methods for attacking RNNs exist, the idea of ​​training an RL agent to generate and schedule attacks is unique, as demonstrated by the fact that no one has attempted this approach before, despite all the recent work in machine learning and the well-known properties of basic attacks against static image classifiers. Furthermore, the present invention is unique in using a deep RL agent to learn the timing and size of adversarial perturbations. Prior work on RNN-based time series attacks relied on manually designed attack sizes and attack lengths, which required experts to tune experimentally. The present invention also provides the potential for generalization, where a learned attack for one example can be immediately generalized to examples that the RL agent has not yet seen. Prior work on variable attack sizes applied iterative attacks that are not conducive to online attacks on time series RNNs (such as RNNs processing real-time sensor readings), but the present invention is able to choose a larger single attack size in cases where there may have been iterations previously required.

[0059] The present invention is a subcomponent of a system that produces an output signal and consumes an input signal, the production of which involves the use of a recurrent neural network as a software component. Non-limiting examples of applications of the present invention include systems that perform system health monitoring on an automotive or aerospace platform based on sensor readings; computer vision systems that classify potential obstacles based on the type of video data used on an autonomous vehicle platform; systems that analyze text or speech (audio) data to perform machine translation, emotion detection, or related tasks; and systems that perform anomaly detection functions based on signals that vary over time (e.g., detecting arrhythmias from electrocardiogram data).

[0060] Finally, although the present invention has been described according to several embodiments, a person of ordinary skill in the art will readily recognize that the present invention may have other applications in other environments. It should be noted that there may be many embodiments and implementations. Furthermore, the appended claims are in no way intended to limit the scope of the present invention to the specific embodiments described above. In addition, any description of "a device for..." is intended to trigger a device-plus-function reading of the elements and claims, and any element not specifically described using "a device for..." should not be interpreted as a device-plus-function element, even if the claim otherwise includes the term "device". In addition, although specific method steps have been described in a particular order, these method steps may be performed in any desired order and fall within the scope of the present invention.

Claims

1. An attack system for generating adversarial perturbations to an input signal of a target system based on a recurrent neural network (RNN), wherein the target system based on the RNN is configured to receive an input sensor signal and generate an output, the attack system comprising: A non-transitory computer-readable medium and one or more processors having executable instructions encoded thereon, such that when the executable instructions are executed, the one or more processors perform the following operations: training a reinforcement learning agent to determine a magnitude of an adversarial perturbation for attacking the RNN-based target system; generating a perturbed input sensor signal having the determined amplitude, wherein the perturbed input sensor signal is generated by altering a set of time-series sensor readings; presenting the perturbed input sensor signal to the RNN-based target system such that the RNN-based target system produces an altered output in response to the perturbed input sensor signal; and The altered output is used to identify a failure mode of the RNN-based target system.

2. The attack system according to claim 1, wherein: The one or more processors also perform the following operations: training the reinforcement learning agent to learn the timing of the adversarial perturbation.

3. The attack system according to claim 1, wherein: The one or more processors also perform the following operations: using an attack generator to generate the perturbed input sensor signal.

4. The attack system according to claim 3, wherein: At each time step of training the reinforcement learning agent, the one or more processors also perform the following operations: presenting unattacked sensor data including known attack attributes to the reinforcement learning agent, wherein the reinforcement learning agent outputs a set of probability distribution parameters, and the attack parameter set is sampled by the attack generator from the set of probability distribution parameters.

5. The attack system according to claim 4, wherein: The one or more processors also perform the following operations: determine a scalar value using the unattacked sensor data, the set of attack parameters, the perturbed input signal, and the altered output, wherein the scalar value represents a relative success rate of the attack associated with the perturbed input sensor signal.

6. The attack system according to claim 5, wherein: The one or more processors also perform the following operations: providing the scalar value as a reward signal to the reinforcement learning agent, thereby improving an attack strategy of the reinforcement learning agent.

7. A computer-implemented method for generating an adversarial perturbation to an input signal of a target system based on a recurrent neural network (RNN), the target system being configured to receive an input sensor signal and produce an output, the method comprising the following actions: causing one or more processors to execute instructions encoded on a non-transitory computer readable medium such that, when executed, the one or more processors perform the following operations: training a reinforcement learning agent to determine a magnitude of an adversarial perturbation for attacking the RNN-based target system; generating a perturbed input sensor signal having the determined amplitude, wherein the perturbed input sensor signal is generated by altering a set of time-series sensor readings; presenting the perturbed input sensor signal to the RNN-based target system such that the RNN-based target system produces an altered output in response to the perturbed input sensor signal; and The altered output is used to identify a failure mode of the RNN-based target system.

8. The method according to claim 7, wherein: The one or more processors also perform the following operations: training the reinforcement learning agent to learn the timing of the adversarial perturbation.

9. The method according to claim 7, wherein: The one or more processors also perform the following operations: using an attack generator to generate the perturbed input sensor signal.

10. The method according to claim 9, wherein: At each time step of training the reinforcement learning agent, the one or more processors also perform the following operations: presenting unattacked sensor data including known attack attributes to the reinforcement learning agent, wherein the reinforcement learning agent outputs a set of probability distribution parameters, and the attack parameter set is sampled by the attack generator from the set of probability distribution parameters.

11. The method according to claim 10, wherein: The one or more processors also perform the following operations: determine a scalar value using the unattacked sensor data, the set of attack parameters, the perturbed input signal, and the altered output, wherein the scalar value represents a relative success rate of the attack associated with the perturbed input sensor signal.

12. The method according to claim 11, wherein: The one or more processors also perform the following operations: providing the scalar value as a reward signal to the reinforcement learning agent, thereby improving an attack strategy of the reinforcement learning agent.

13. A computer program product for generating an adversarial perturbation to an input signal of a target system based on a recurrent neural network (RNN), the computer program product comprising: Computer readable instructions stored on a non-transitory computer readable medium, the computer readable instructions being executable by a computer having one or more processors to cause the processor to: training a reinforcement learning agent to determine a magnitude of an adversarial perturbation for attacking the RNN-based target system; generating a perturbed input sensor signal having the determined amplitude, wherein the perturbed input sensor signal is generated by altering a set of time-series sensor readings; presenting the perturbed input sensor signal to the RNN-based target system such that the RNN-based target system produces an altered output in response to the perturbed input sensor signal; and The altered output is used to identify a failure mode of the RNN-based target system.

14. The computer program product of claim 13, further comprising instructions causing the one or more processors to further perform the following operations: train the reinforcement learning agent to learn the timing of the adversarial perturbation.

15. The computer program product of claim 13, further comprising instructions that cause the one or more processors to generate the perturbed input sensor signal using an attack generator.

16. The computer program product of claim 15, wherein: At each time step of training the reinforcement learning agent, the one or more processors also perform the following operations: presenting unattacked sensor data including known attack attributes to the reinforcement learning agent, wherein the reinforcement learning agent outputs a set of probability distribution parameters, and the attack parameter set is sampled by the attack generator from the set of probability distribution parameters.

17. The computer program product of claim 16, further comprising instructions for causing the one or more processors to further perform the following operations: determine a scalar value using the unattacked sensor data, the set of attack parameters, the perturbed input signal, and the altered output, wherein The scalar value represents a relative success rate of the attack associated with the perturbed input sensor signal.

18. The computer program product of claim 17, further comprising instructions causing the one or more processors to further perform the following operations: provide the scalar value as a reward signal to the reinforcement learning agent, thereby improving an attack strategy of the reinforcement learning agent.

Images