SYSTEM AND METHOD FOR DETECTING ADVERSARY ATTACKS - Patent application

Abstract

To provide a training technique of a machine learning system for detecting an adversarial attack.SOLUTION: A training process for generating a detector includes a step of classifying a first sequence as belonging to a first class indicative of a nominal sequence based on first prediction, the first sequence including an unperturbed version of sensor data. The process also includes a step of classifying a second sequence as belonging to a second class indicative of an adversarial sequence based on second prediction, the second sequence including a perturbed version of the sensor data. Combined loss data is generated for collection of sequences and is based on a first average loss with respect to incorrect classifications of the first class and a second average loss with respect to incorrect classifications of the second class.SELECTED DRAWING: Figure 7

Description

[Technical Field]

[0001] The present disclosure relates generally to machine learning systems, and more particularly to detecting adversarial data sequences. [Background technology]

[0002] In general, machine learning systems, and deep neural networks in particular, are susceptible to adversarial attacks. These adversarial attacks may include black-box attacks, which refer to attacks based on knowledge of the expected output of a machine learning system, and / or white-box attacks, which refer to attacks based on knowledge of the inner workings of a machine learning system. As an example, a machine learning system may be attacked via its input side. Such adversarial attacks are seen in perturbations of the input side, which cause changes in the output data of the machine learning system. These adversarial attacks are typically carried out by updating the perturbations of the input data based on feedback, until the machine learning system makes decisions altered by these perturbations and produces erroneous output data (e.g., misclassification of the input data). This leads to negative consequences and adverse effects. Summary of the Invention [Means for solving the problem]

[0003] overview The following is a summary of certain embodiments described in detail below. These described aspects are presented merely to provide the reader with a brief summary of these specific embodiments, and the description of these aspects is not intended to limit the scope of the present disclosure. In essence, this disclosure may encompass various aspects that may not be explicitly described below.

[0004] According to at least one aspect, a computer-implemented method relates to training a machine learning system to detect adversarial attacks. The method includes obtaining a collection of sequences. The collection of sequences includes at least a first sequence and a second sequence. The method includes classifying the first sequence as belonging to a first class indicative of a nominal sequence based on a first prediction that the first sequence includes an unperturbed version of the sensor data. The method includes classifying the second sequence as belonging to a second class indicative of an adversarial sequence based on a second prediction that the second sequence includes a perturbed version of the sensor data. The method includes generating combined loss data based on (i) a first average loss including a first-class misclassification for a first set of sequences from the collection of sequences, each sequence in the first set of sequences being a nominal sequence, and (ii) a second average loss including a second-class misclassification for a second set of sequences from the collection of sequences, each sequence in the second set of sequences being an adversarial sequence. The method includes updating parameters of the machine learning system based on the combined loss data.

[0005] According to at least one aspect, a non-transitory computer-readable medium includes computer-readable data that, when executed by a processor, causes the processor to perform a method. The method includes obtaining a collection of sequences. The collection of sequences includes at least a first sequence and a second sequence. The method includes classifying the first sequence as belonging to a first class indicative of a nominal sequence based on a first prediction that the first sequence includes an unperturbed version of the sensor data. The method includes classifying the second sequence as belonging to a second class indicative of an adversarial sequence based on a second prediction that the second sequence includes a perturbed version of the sensor data. The method includes generating combined loss data based on (i) a first average loss comprising a misclassification of the first class for a first set of sequences from the collection of sequences, wherein each sequence in the first set of sequences is a nominal sequence, and (ii) a second average loss comprising a misclassification of the second class for a second set of sequences from the collection of sequences, wherein each sequence in the second set of sequences is an adversarial sequence. The method includes updating parameters of the machine learning system based on the combined loss data.

[0006] According to at least one aspect, a computer-implemented method relates to defending against adversarial attacks. The method includes obtaining an input sequence to a first machine learning system. The method includes generating an adversarial label for classifying the input sequence as adversarial based on a statistical identification that the input sequence is a perturbed version of multiple frames of sensor data. The method includes identifying an output data sequence generated by the first machine learning system based on the input sequence. The method includes filtering the output data sequence based on the adversarial label to prevent an actuator system from being controlled based on the output data sequence.

[0007] These and other features, aspects and advantages of the present invention are discussed in the following detailed description along with the accompanying drawings, wherein like characters represent like or similar parts throughout. [Brief explanation of the drawings]

[0008] [Figure 1] 1 illustrates an example of a system including a detector and an adversarial defense system, according to one embodiment of the present disclosure. [Figure 2] 2 illustrates an example of the system of FIG. 1 in relation to mobile device technology, according to one embodiment of the present disclosure. [Figure 3A] 2 is a schematic diagram of some components of the system of FIG. 1 in a nominal mode of operation, according to one embodiment of the present disclosure. [Figure 3B] 2 is a schematic diagram of some components of the system of FIG. 1 in a defensive mode of operation, according to one embodiment of the present disclosure. [Figure 4] FIG. 1 illustrates an example of a system related to training a detector, according to one embodiment of the present disclosure. [Figure 5] 1 is a flowchart associated with training a detector, according to one embodiment of the present disclosure. [Figure 6A] 1 is a conceptual diagram of an example of an adversarial sequence generated based on a nominal sequence, according to one embodiment of the present disclosure. [Figure 6B] FIG. 10 is a conceptual diagram of another example of an adversarial sequence generated based on a nominal sequence, according to one embodiment of the present disclosure. [Figure 7] 1 is a flowchart of an example of a training process for generating a detector, according to one embodiment of the present disclosure. DETAILED DESCRIPTION OF THE INVENTION

[0009] Detailed Description The embodiments shown and described herein by way of example and many of their advantages will be understood from the foregoing description, and it will be apparent that various changes can be made in the form, construction, and arrangement of these components without departing from the disclosed subject matter or sacrificing one or more of its advantages. Rather, the described forms of these embodiments are merely illustrative. These embodiments are susceptible to various modifications and alternative forms, and the following claims are intended to encompass all such modifications and are not limited to the particular forms disclosed, but rather are intended to cover all modifications, equivalents, and alternatives falling within the spirit and scope of this disclosure.

[0010] 1 illustrates a system 100 including a sensor system 110, a control system 120, and an actuator system 130. The system 100 is configured such that the control system 120 controls the actuator system 130 based on sensor data from the sensor system 110. More specifically, the sensor system 110 includes one or more sensors and / or corresponding devices for generating sensor data. For example, the sensor system 110 may include an image sensor, a camera, a radar sensor, a light detection and ranging (LIDAR) sensor, a thermal sensor, an ultrasonic sensor, an infrared sensor, a motion sensor, a satellite-based navigation sensor (e.g., a global positioning system (GPS) sensor), a microphone, any suitable sensor, or any combination thereof. Upon obtaining sensor data of its environment, the sensor system 110 may be operable to communicate with the control system 120 via an input / output (I / O) system 140 and / or other functional modules 150, including communication technologies.

[0011] Control system 120 is configured to acquire sensor data directly or indirectly from one or more sensors of sensor system 110. In this regard, the sensor data may include sensor data from a single sensor or sensor fusion data from multiple sensors. Upon receiving an input including at least the sensor data, control system 120 may implement a software mechanism, such as a sliding window, to acquire at least one sequence from the sensor data stream. Each sequence may be of any length and include any number of elements. In one example, each sequence includes multiple elements, where each element is a frame including at least sensor data. The control system may operate to process the sensor data via processing system 160. In this regard, processing system 160 includes at least one processor. For example, processing system 160 may include an electronic processor, a central processing unit (CPU), a graphics processing unit (GPU), a microprocessor, a field programmable gate array (FPGA), an application-specific integrated circuit (ASIC), a processing circuit, any suitable processing technology, or any combination thereof. At least while processing this sensor data, processing system 160 is operable to generate output data based on communication with memory system 170. Additionally, processing system 160 is operable to provide control data to actuator system 130 based on the output data.

[0012] Memory system 170 is a computer or electronic storage system configured to store and provide access to various data to enable at least the operations and functions as disclosed herein. Memory system 170 may include a single device or multiple devices. Memory system 170 may include electrical, electronic, magnetic, optical, semiconductor, electromagnetic, any suitable memory technology, or any combination thereof. For example, memory system 170 may include random access memory (RAM), read-only memory (ROM), flash memory, disk drives, memory cards, optical storage devices, magnetic storage devices, memory modules, any suitable type of memory device, or any combination thereof. In one embodiment, memory system 170 is local, remote, or a combination thereof (e.g., partially local and partially remote) with respect to control system 120 and / or processing system 160. For example, memory system 170 may be configured to include at least a cloud-based storage system (e.g., a cloud-based database system) that is remote from other components of processing system 160 and / or control system 120.

[0013] Memory system 170 includes at least classifier 200. Classifier 200 includes machine learning system 200A. Machine learning system 200A includes at least one artificial neural network (e.g., a deep neural network) or any suitable machine learning technique. For ease of reference, machine learning system 200A may be referred to as a “first machine learning system” in this disclosure. In response to the input, processing system 160, via machine learning system 200A, may operate to generate output data based on the input. For example, upon receiving at least sensor data from sensor system 110, processing system 160, via application of machine learning system 200A, may operate to predict a class for an entity in the sensor data and provide class data as output data. As a non-limiting example, processing system 160, via machine learning system 200A, may be configured to determine that a detected entity in the sensor data most likely belongs to the pedestrian class and assign the class to the entity as a “pedestrian” from among multiple classes (e.g., traffic sign, traffic light, animal, vehicle, etc.). Concurrent with generating this output data (e.g., class data indicative of a pedestrian) via machine learning system 200A, processing system 160 is operable to generate control data for actuator system 130 based on at least this output data (e.g., "pedestrian"), whereby actuator system 130 is controlled to perform actions that take into account the detections of sensor system 110.

[0014] The memory system 170 also includes a detector 210. The detector 210 is preferably configured to distinguish between nominal and adversarial sequences. More specifically, the detector 210 includes at least one machine learning system 210A. For ease of reference, the machine learning system 210A may be referred to as a “second machine learning system” to distinguish it from the machine learning system 200A, which may be referred to as a “first machine learning system.” More specifically, the detector 210 includes at least one neural network and / or deep neural network architecture tuned to time sequences. For example, the detector 210 may include at least a recurrent neural network (RNN), a long short-term memory (LSTM) network, a gated recurrent unit (GRU), other suitable machine learning techniques, or any combination thereof.

[0015] Detector 210 is configured to receive, via processing system 160, the same inputs received and processed directly by classifier 200. More specifically, machine learning system 210A of detector 210 receives, via processing system 160, input sequences simultaneously with or within the same time frame as machine learning system 200A of classifier 200. Detector 210 may be at least partially integrated with machine learning system 200A and / or at least partially separate from machine learning system 200A. Upon receiving the input sequences, processing system 160 is configured, via detector 210, to statistically identify that the input sequences include sequences of nominal data and generate a nominal label indicating the detection of a “nominal sequence,” and to statistically identify that the input sequences include sequences of adversarial data or sequences of non-nominal data and generate an adversarial label indicating the detection of an “adversarial sequence.” By evaluating the sequences, detector 210 is configured to identify the presence or absence of an adversarial attack on machine learning system 200A. This is because adversarial attacks often involve repeated attempts at perturbing the input to induce errors in the machine learning system 200A.

[0016] Memory system 170 also includes adversarial defense system 220. Adversarial defense system 220 includes at least software technology. Additionally or alternatively, adversarial defense system 220 may include hardware technology. Adversarial defense system 220 is configured to be at least partially separate from detector 210 or integrated with detector 210. Adversarial defense system 220 is configured to receive output data from machine learning system 200A and corresponding classification data from detector 210 via processing system 160. More specifically, adversarial defense system 220 receives output data sequences from machine learning system 200A generated based on similar input sequences to classify into classification data. In this regard, processing system 160 is configured to identify output data sequences from machine learning system 200A that correspond to classification data of input sequences to machine learning system 200A via at least timestamp data or any suitable correlation data.

[0017] Adversarial defense system 220, via processing system 160, is configured to preferably ensure that at least one other system (e.g., actuator system 130) is protected from directly or indirectly receiving output data sequences from machine learning system 200A that are generated based on input sequences deemed to be adversarial sequences by detector 210. More specifically, upon receiving the adversarial labels from detector 210, adversarial defense system 220, via processing system 160, is configured to take defensive actions with respect to identified output data sequences from machine learning system 200A that correspond to adversarial sequences. For example, adversarial defense system 220 is configured to delay the output data, replace the output data with predetermined output data (e.g., default output data, an alert, etc.), reject the output data, filter the output data, discard the output data, and / or take any suitable action to protect system 100 from actions based on output data sequences generated based on detected sequences of adversarial data. Thus, the adversarial defense system 220, when working in conjunction with the detector 210 and the processing system 160, is configured to ensure that the system 100 operates at a predetermined level of accuracy by ensuring that the system 100 only operates on output data sequences that are generated based on input sequences that are considered to be nominal sequences.

[0018] 1, system 100 includes other components that contribute to the operation of control system 120 in association with sensor system 110 and actuator system 130. For example, as shown in FIG. 1, memory system 170 is also configured to store other relevant data 230 related to the operation of system 100 in association with one or more components (e.g., sensor system 110, actuator system 130, machine learning system 200A, detector 210, and adversarial defense system 220). Also shown in FIG. 1, control system 120 includes I / O system 140 that includes one or more interfaces for one or more I / O devices associated with system 100. For example, I / O system 140 provides at least one interface to sensor system 110 and at least one interface to actuator system 130. Control system 120 is also configured to provide other functional modules 150, such as any suitable hardware, software, or any combination thereof, that support and / or contribute to the functionality of system 100. For example, other functional modules 150 include operating systems and communication technologies that allow components of system 100 to communicate with each other, as described herein. Using at least the configuration discussed in the example of Figure 1, system 100 is adaptable to a variety of technologies.

[0019] FIG. 2 illustrates an example of a system 100 for mobile device technology, according to one embodiment. More specifically, in FIG. 2, the system 100 is used by a vehicle 10 in which a control system 120 controls at least one actuator system 130 of the vehicle 10 according to sensor data from a sensor system 110. Also in FIG. 2, the control system 120 includes an obstacle detection system. In this example, the control system 120 is configured to detect entities based on the sensor data and generate boundary data (e.g., a hull, outline, or frame) corresponding to the detection of the entities related to the sensor data. More specifically, upon receiving at least the sensor data and / or the boundary data of the detected entities from the sensor data, the processing system 160 is configured to predict, via a classifier 200 with its machine learning system 200A, an identifier of the entity and provide output data based on the prediction. As a non-limiting example, upon receiving sensor data and / or boundary data indicating at least one entity detected from the sensor data, processing system 160 is configured to classify the detected entity as belonging to a pedestrian class from among several classes of obstacles (e.g., traffic lights, traffic signs, vehicles, animals, road structures, etc.) via machine learning system 200A. Processing system 160 is further configured to generate class data indicating that the detected entity is a “pedestrian” via machine learning system 200A. Also in this non-limiting example, upon generation of the class data via classifier 200, control system 120 is configured to generate control data for actuator system 130 that operates at least one functional component of vehicle 10 based on the class data identifying the detected entity sensed in the environment of vehicle 10. For example, actuator system 130 may include a steering system, for which control system 120 generates control data related to a steering operation.As another example, actuator system 130 may include a braking system, where control system 120 generates control data related to braking operations. Actuator system 130 may include any actuator associated with vehicle 10, but is not limited to a steering system and / or a braking system.

[0020] Additionally, control system 120 is configured to generate classification data for each input sequence to machine learning system 200A via detector 210. The classification data classifies the input sequence as either (i) a nominal sequence containing nominal elements or (ii) an adversarial sequence containing adversarial elements. In this regard, detector 210 advantageously identifies each nominal sequence received as an input by machine learning system 200A and enables system 100 to operate, based on each corresponding output data sequence generated by machine learning system 200A, at a level of assurance that there was most likely no adversarial attack during the generation of the corresponding output data sequence. Detector 210 also advantageously identifies each adversarial sequence received as an input by machine learning system 200A and enables system 100 to take defensive action regarding a possible adversarial attack and avoid the use of each corresponding output data sequence generated from machine learning system 200A during that time frame.

[0021] Control system 120 is also configured to provide this classification data from detector 210, along with corresponding output data from machine learning system 200A, to adversarial defense system 220. In this regard, adversarial defense system 220 is configured to process output data obtained from machine learning system 200A according to the classification data obtained from detector 210. For example, upon receiving a nominal label from detector 210, adversarial defense system 220 is configured to identify output data from machine learning system 200A generated based on this input sequence that corresponds to the nominal label and provide an indication that control system 120 processes the output data in a nominal mode. In this nominal mode, control system 120 can operate to generate control data based on the corresponding output data of machine learning system 200A. Alternatively, upon receiving an adversarial label from detector 210, adversarial defense system 220 is configured to identify output data from machine learning system 200A generated based on a flagged input sequence that corresponds to the adversarial label and provide an indication that control system 120 processes the output data in a defensive mode. In this defensive mode, the adversarial defense system 220 is configured to delay the output data, replace the output data with predetermined output data (e.g., default output data), reject the output data, filter the output data, discard the output data, or take any appropriate action to defend the system 100 from the effects of output data generated based on the detected adversarial data sequences. Upon receiving communications from the adversarial defense system 220 that selectively include only output data sequences from the machine learning system 200A that correspond to the nominal sequences, the control system 120 is configured to generate control data based on these nominal sequences of output data. In response to the control data, the actuator system 130 is configured to control or assist the operation of the vehicle 10, which may be autonomously assisted, highly autonomously assisted, partially autonomously assisted, conditionally autonomously assisted, or driver-assisted.

[0022] Additionally or alternatively to the example of Figure 2, system 100 (and / or control system 120) may also operate in other applications. For example, system 100 and / or control system 120 may operate in various fields, such as computer-controlled machines, robots, home appliances, power tools, electronic personal assistants, healthcare / medical technology, mobile devices, security technology, etc. That is, system 100 and / or control system 120 are not limited to the above applications, but may be applied to any suitable application in which it is beneficial to detect adversarial attacks using iterative techniques involving perturbations across a sequence of multiple elements.

[0023] 3A and 3B show a schematic diagram of the interaction of some components of system 100, particularly machine learning system 200A, detector 210, and adversarial defense system 220. More specifically, in FIG. 3A, control system 120 operates in a nominal mode when detector 210 identifies an input sequence as nominal and / or when adversarial defense system 220 indicates the absence of an adversarial attack. In contrast, in FIG. 3B, control system 120 operates in an adversarial mode when detector 210 identifies an input sequence as adversarial and / or when adversarial defense system 220 indicates the presence of an adversarial attack. Additionally, although not shown in FIGS. 3A and 3B, processing system 160 actively interfaces with these components during both the nominal and defensive modes.

[0024] 3A illustrates an example scenario in which control system 120 operates in nominal mode. More specifically, sensor system 110 provides a sensor data stream based on its environment. The sensor data stream includes a set of nodes X={x1, x2, ... x t}, where X represents the sequence, x1 to x trepresents an element of the sequence. For example, each element of the sequence X can be referred to as a sensor data frame. Upon acquiring the sensor data, the processing system 160 is configured to generate class data for the sensor data via the machine learning system 200A. The class data output via the machine learning system 200A for the sensor data stream is represented as Y={y1, y2, ...y t}, where Y represents a sequence, y1 to y t represents an element of the sequence. For example, each element of the sequence Y can be referred to as class data generated by the machine learning system 200A for each element of the sequence X.

[0025] The detector 210 receives the same input as the machine learning system 200A (e.g., X={x1, x2, ... x t}). Upon receiving the sensor data sequence as input, detector 210 is configured to predict the sensor data sequence to be a nominal sequence and generate a nominal label, and to predict the sensor data sequence to be an adversarial sequence and generate an adversarial label. In this case, as shown in FIG. 3A, detector 210 identifies the sensor data sequence as including nominal data and generates a nominal label for the input. Adversarial defense system 220 receives the nominal labels from detector 210 and generates corresponding class data (Y={y1, y2, ... y t In this case, because detector 210 indicates that machine learning system 200A has received the nominal sequence as input, adversarial defense system 220 may be operative to indicate that control system 120 is configured to operate in a nominal mode, whereby control data for actuator system 130 is generated for the actuator system based on at least the class data from machine learning system 200A.

[0026] 3B illustrates an example scenario in which control system 120 operates in defensive mode. Unlike FIG. 3A, FIG. 3B includes an adversarial system 20 that is not part of system 100 but is generating adversarial attacks against system 100. Generally, this adversarial system 20 iteratively perturbs sensor data to machine learning system 200A, causing corruption and / or errors in machine learning system 200A. In this example, adversarial system 20 may operate to perturb the sensor data with perturbation data that may be imperceptible to the extent that machine learning system 200A generates class data for the perturbed version of the sensor data that is different from the class data that would have been generated by machine learning system 200A for the unperturbed version of the same sensor data. The adversarial system 20 often fails on its first attempt to induce an error in the machine learning system 200A, and therefore attempts several times to perturb the input to the machine learning system 200A while using the output data (e.g., class data) of the machine learning system 200A as feedback to identify perturbation data on the input that induces an error in the machine learning system 200A. In this regard, the adversarial system 20 typically relies on iterative techniques to achieve a successful adversarial attack.

[0027] 3B, the sensor system 110 generates a sensor data stream based on its environment. The sensor data stream includes a sensor data sequence, which is represented by X={x1, x2, ... x t}, where X represents a nominal sequence, x1 through x t represents an element of the sequence. For example, each element can be referred to as a sensor data frame. However, in this scenario, the adversarial system 20 generates a sequence of perturbed data and perturbs the sensor data such that the machine learning system 200A receives a perturbed version of the sensor data. For example, the sequence of perturbed data may be δ={δ, δ, ... δ t}, where δ represents the sequence of perturbation data, δ to δ t represent the various perturbation elements of this sequence, and the perturbed version of the sensor data is denoted by X'={x'1,x'2,...x' t}, where X' represents a perturbed version of the sequence X, and x'1 to x' t is a sequence of perturbed data δ={δ1,δ2,…δ t}, each representing a perturbed element of the perturbed sequence. Upon receiving the perturbed versions of the sensor data, processing system 160 is configured to generate, via machine learning system 200A, class data that classifies these perturbed versions of the sensor data. Furthermore, detector 210 also receives the same input as machine learning system 200A (i.e., X'={x'1, x'2, ... x' t}) from the machine learning system 200A. Upon receiving the perturbed version of the sensor data sequence as input, the detector 210 is configured to predict the sensor data sequence to be a nominal sequence and generate a nominal label, and to predict the sensor data sequence to be an adversarial sequence and generate an adversarial label. In this case, as shown in FIG. 3B, the detector 210 identifies the input sequence (i.e., X′) to the machine learning system 200A as an adversarial sequence and generates an adversarial label for the input sequence (i.e., X′). The adversarial defense system 220 receives the adversarial labels from the detector 210 and generates class data (Y′={y1, y2, ... y t}) as output data. In this case, because the detector 210 indicates that the input sequence to the machine learning system 200A is an adversarial sequence, the adversarial defense system 220 is configured to activate a defensive mode in which the corresponding sequence of class data generated based on the flagged input is filtered and prevented from affecting downstream systems, such as the actuator system 130. For example, in FIG. 3B , the adversarial defense system 220 receives as output data Y′={y1, y2, ... y t} does not allow the use of corresponding class data.

[0028] FIG. 4 illustrates a system 400 associated with training detector 210, according to one embodiment. In this simplified example, system 400 includes at least a memory system 410 and a processing system 420. In FIG. 4, memory system 410 is a computer or electronic storage system configured to store and provide access to various data to enable at least the operations and functions disclosed herein. This memory system 410 may include a single device or multiple devices. This memory system 410 may include electrical, electronic, magnetic, optical, semiconductor, electromagnetic, any suitable memory technology, or any combination thereof. For example, memory system 410 may include RAM, ROM, flash memory, disk drives, memory cards, optical storage devices, magnetic storage devices, memory modules, any suitable type of memory device, or any combination thereof. In one embodiment, memory system 410 may be local, remote, or a combination thereof (e.g., partially local and partially remote) with respect to processing system 420. For example, memory system 410 may be configured to include at least a cloud-based storage system (eg, a cloud-based database system) that is remote from processing system 420.

[0029] In one embodiment, as shown in FIG. 4 , memory system 410 includes detector 210, which includes machine learning system 210A. Also shown in FIG. 4 , memory system 410 includes at least training data 412 and machine learning data 414 used to generate detector 210. Furthermore, memory system 410 is configured to include other relevant data related to training and generating detector 210, as discussed herein. More specifically, training data 412 includes at least sensor data (and / or image data based on the sensor data). Machine learning data 414 includes machine learning algorithms associated with method 700 ( FIG. 7 ) for training and generating detector 210. Detector 210 includes machine learning system 210A along with various data (e.g., various layers, weights, parameter data, etc.) related to the training and / or operation of machine learning system 210A. Once trained to perform at a predetermined level of accuracy, detector 210 can be deployed and / or used by system 100 of FIG. 1 or any suitable application system.

[0030] Upon receiving the training data 412, the processing system 420 is configured to train the machine learning system 210A according to the machine learning data 414. In this regard, the processing system 420 includes at least one processor. For example, the processing system 420 may include an electronic processor, a CPU, a GPU, a microprocessor, an FPGA, an ASIC, a processing circuit, any suitable processing technology, or any combination thereof. In one embodiment, the processing system 420 is in communication with the memory system 410 to generate the detector 210 based on the training data 412 and the machine learning data 414.

[0031] 5 is a flowchart associated with a training process 500 for generating detector 210, according to one embodiment. Generally, training process 500 includes a substantial and sufficient amount of training data 412 to ensure that detector 210 functions accurately. For example, the collection of training data 412 includes at least a set of nominal sequences 412A and a set of adversarial sequences 412B. Additionally, training data 412 may include historical and / or actual adversarial attack data collected from real-world adversarial attacks on various machine learning systems.

[0032] The set of nominal sequences 412A includes at least sensor data, sensor fusion data, image data based on the sensor data, image data based on the sensor fusion data, or any combination thereof. Also, in this example, the set of adversarial sequences 412B includes at least one or more perturbed versions of the set of nominal sequences 412A. In general, the set of adversarial sequences 412B can include any sequences whose elements are perturbed by a perturbation, even if the sequence is not successful in causing an error in the machine learning system (e.g., misclassification of a sequence such as f(x′)≠f(x), where f(x′) represents the output data of a machine learning system based on a perturbed version of an element and f(x) represents the output data of a machine learning system based on an unperturbed version of that same element). After completing this training process 500 with at least this collection of training data 412, the detector 210, via at least one processor, is configured to predict a sequence to be a nominal sequence (or an unperturbed version of the sensor data) and simultaneously generate a nominal label, and predict a sequence to be an adversarial sequence (or a perturbed version of that sensor data) and simultaneously generate an adversarial label.

[0033] 6A and 6B show an example of training data 412. For example, FIG. 6A shows the training data 412 from x1 to x t6A illustrates a nominal sequence 600 including elements denoted by x, where each element is a sensor data frame. For example, each element may be an image frame taken from a video stream. Also shown in FIG. 6A, the nominal sequence 600 is continuous in time, which progresses in the direction of the arrow. FIG. 6A also illustrates an adversarial sequence 610 generated from the nominal sequence 600. In this regard, for example, upon receiving the nominal sequence 600, the processing system 420 may extract elements (e.g., x) from the nominal sequence 600. i ) and generate the adversarial sequence 610 by iteratively perturbing its elements, thereby generating multiple perturbed versions (e.g., x' i,1 =x i +δ1,…,x' i,p =x i +δ p ) is generated. In FIG. 6A, the selected element x i is perturbed "p" times, where p is the number of times the selected element is perturbed by the perturbed element x' i,p (i.e., x i +δ p ), which is used to generate f(x i +δ p )≠f(x i 6A also shows that an adversarial sequence 620 that includes the adversarial sequence 610 as a subsequence may be provided as training data 412.

[0034] FIG. 6B also illustrates an adversarial sequence 630 generated from the nominal sequence 600. More specifically, the processing system 420 is configured to generate the adversarial sequence 630 by perturbing each element of the nominal sequence. In this case, the adversarial sequence 630 includes a perturbed version of each element of the nominal sequence 600 until the machine learning system 200A makes an error. For example, the adversarial sequence 630 includes a perturbed version of the first element of the nominal sequence, a perturbed version of the second element of the nominal sequence, and so on, until the machine learning system 200A makes an error. Furthermore, FIG. 6B also illustrates an adversarial sequence 640 that includes the adversarial sequence 630 as a subsequence. In this case, the processing system 420 generates an adversarial signature (e.g., δ1, δ2, ... δ) of the perturbation. k ) to generate at least these adversarial sequences 630 and 640 by perturbing elements of the nominal sequence 600 with elements from

[0035] As discussed above, Figures 6A and 6B illustrate some examples of training data 412 that can be used to train detector 210 during training process 500. Figures 6A and 6B are advantageous in that processing system 420 can obtain at least one nominal sequence 600 and simultaneously generate these adversarial sequences 610, 620, 630, and 640. Furthermore, processing system 420 is also configured to generate other adversarial sequences 610, 620, 630, and 640 from nominal sequence 600 by attacking nominal sequence 600 with other adversarial signatures that include other perturbations. However, set of adversarial sequences 412B is not limited to the above-described adversarial sequences 610, 620, 630, and 640 (and / or adversarial sequences that induce errors in machine learning system 200A), but may include any adversarial sequence that includes a sequence of perturbative elements. In general, it is beneficial for the detector 210 to be trained with as much training data 412 as possible, to the extent that its ability to distinguish between nominal and adversarial sequences is enhanced.

[0036] 7 illustrates a flowchart of an example of the training process 500 (FIG. 5) for generating a detector 210, according to one embodiment. The training process 500 includes a method 700 for training at least one machine learning system 210A of the detector 210 to distinguish between at least one sequence of nominal data and at least one sequence of adversarial data. Preferably, the method 700 provides training data 412 including both a set of nominal sequences 412A and a set of adversarial sequences 412B, while simultaneously optimizing parameters of the machine learning system 210A of the detector 210 based on results obtained from the training data 412. Thus, following the training process 500 according to the method 700, the detector 210 becomes operable to identify sequences, predict whether the sequences are nominal / adversarial, and provide a label indicating the prediction.

[0037] In step 702, processing system 420 acquires a first set of training data via detector 210. For example, the first set of training data includes a sufficient amount of nominal data to train detector 210, thereby configuring machine learning system 210A to operate at a predetermined level of accuracy. More specifically, the first set of training data includes a set of nominal sequences 412A, each sequence including nominal data unperturbed by perturbation data. As described above, for example, the nominal data may include sensor data, sensor fusion data, image data based on sensor data, image data based on sensor fusion data, or any combination thereof. Upon acquiring set of nominal sequences 412A as training data 412, method 700 proceeds to step 706.

[0038] In step 704, the processing system 420 acquires a second set of training data via the detector 210. For example, the second set of training data includes a sufficient amount of adversarial data to train the detector 210, thereby configuring the machine learning system 210A to operate at a predetermined level of accuracy. More specifically, the second set of training data includes a set of adversarial sequences 412B, each of which includes nominal data perturbed by perturbation data. In this regard, for example, each adversarial sequence includes a plurality of perturbed sensor data, perturbed sensor fusion data, perturbed image data based on sensor data, perturbed image data based on sensor fusion data, or any combination thereof. Generally, the adversarial sequences correspond to the nominal sequences but further include element perturbations. Upon acquiring the set of adversarial sequences 412B as training data 412, the method 700 proceeds to step 708.

[0039] In step 706, processing system 420, via detector 210, classifies each sequence from a set of nominal sequences, sometimes referred to as a first set of training data. Processing system 420, via detector 210, is operable to analyze a sequence and assign one of the classes to the sequence. For example, processing system 420, via detector 210, is configured to evaluate a sequence from the set of nominal sequences and, via its machine learning model, identify the sequence as belonging to a nominal class or an adversarial class.

[0040] In step 708, processing system 420, via detector 210, classifies each sequence from the set of adversarial sequences, sometimes referred to as the second set of training data. Processing system 420, via detector 210, is operable to analyze the sequence and assign one of the classes to the sequence. For example, processing system 420, via detector 210, is configured to evaluate a sequence from the set of adversarial sequences and, via its machine learning model, identify the sequence as belonging to a nominal class or an adversarial class.

[0041] In step 710, processing system 420 generates classification data based on a first set of training data via detector 210. In this case, the first set of training data includes set 412A of nominal sequences. Detector 210 is operable to predict an input to be a nominal data sequence (or a non-adversarial data sequence) and simultaneously generate a nominal label for the input, and to predict an input to be an adversarial data sequence (or a non-nominal data sequence) and simultaneously generate an adversarial label for the input. In this regard, for example, a nominal label may be represented by one binary symbol (e.g., 0) and an adversarial label may be represented by another binary symbol (e.g., 1), or vice versa. In this case, because each input to detector 210 is a nominal sequence from the first set of training data, processing system 420 can compare the true classification data of the nominal label for a sequence in the first set with the predicted classification data (e.g., nominal label or adversarial label) for that sequence in the first set.

[0042] In step 712, processing system 420 generates classification data based on the second set of training data via detector 210. In this case, the second set of training data includes set 412B of adversarial sequences. As described above, detector 210 is operable to predict an input to be a nominal data sequence (or a non-adversarial data sequence) and simultaneously generate a nominal label for the input, and to predict an input to be an adversarial data sequence (or a non-nominal data sequence) and simultaneously generate an adversarial label for the input. In this regard, consistent with step 710, a nominal label may be represented by one binary symbol (e.g., 0), and an adversarial label may be represented by another binary symbol (e.g., 1). In this case, since each input to detector 210 is an adversarial sequence from the second set of training data, processing system 420 can compare the true classification data of the adversarial label for a sequence in the second set with the predicted classification data (e.g., nominal label or adversarial label) for this sequence in the second set.

[0043] In step 714, processing system 420 generates average loss data for detector 210 related to the difference between the predicted classification and the true classification of the first set of training data. More specifically, processing system 420 evaluates the incorrectly classified data in comparison to the correct classification data generated for the first set of training data (e.g., set of nominal sequences 412A). More specifically, with respect to this first set of training data, detector 210 (i) generates correct classification data when it receives one of these nominal sequences as an input and simultaneously predicts a nominal label via machine learning system 210A, and (ii) generates incorrect classification data when it receives one of these nominal sequences as an input and simultaneously predicts an adversarial label via machine learning system 210A. For convenience, this average loss data may be referred to as “first average loss data.”

[0044] In step 716, processing system 420 generates average loss data for detector 210 related to the difference between the predicted classification and the true classification of the second set of training data. More specifically, processing system 420 evaluates the incorrect classification data in comparison with correct classification data generated for the second set of training data (e.g., set of adversarial sequences 412B). More specifically, with respect to this second set of training data, detector 210 (i) generates correct classification data when one of the adversarial sequences is received as input and an adversarial label is predicted via machine learning system 210A, and (ii) generates incorrect classification data when one of the adversarial sequences is received as input and a nominal label is predicted via machine learning system 210A. For convenience, this average loss data is also referred to as "second average loss data."

[0045] In step 718, processing system 420 optimizes parameters of a discriminator of detector 210 based on a relative weighting function including the first average loss data and the second average loss data. More specifically, for example, processing system 420 optimizes parameters (e.g., θ) associated with a discriminator (e.g., a discriminant model or network) of machine learning system 210A, as follows:

number

[0046] In this equation, the processing system 420 determines the value of a parameter (e.g., θ) of the discriminator of the detector 210, where the coupling loss

number

number

number

[0047] As indicated by equation (1), the processing system 420 also includes at least one machine learning model that performs binary sequence classification to identify the presence or absence of an adversarial signature on an input data sequence via the detector 210. For convenience of explanation, for example, let X be the input test sequence. test , the detector 210 calculates d(X test )=0 and simultaneously assign a single binary value (e.g., a value of 0) to the test sequence as a nominal label. Furthermore, the detector 210 is configured to determine whether d(X test )=1, and simultaneously assigns another binary value (e.g., a value of 1) to the test sequence as an adversarial label. Alternatively, provided that detector 210 operates as described herein, detector 210 may be configured to assign any set of values ​​(e.g., 0 and 1) as classification data (e.g., nominal and adversarial labels).

[0048] In equation (1), λ represents a parameter that provides a relative weighting between the first average loss and the second average loss. The parameter λ enables the processing system 420 to adjust the relative weighting balance between misclassification of nominal data (e.g., nominal sequences) and misclassification of adversarial data (e.g., adversarial sequences) according to the application. In this regard, the parameter λ provides a balance factor between detection of nominal sequences (e.g., nominal sensor data) and detection of adversarial sequences (and / or adversarial attacks).

[0049] In step 720, processing system 420 completes its training with the optimized parameters, providing detector 210 ready for deployment / use. In this regard, after the parameters have been optimized and / or detector 210 has been trained with the optimized parameters, detector 210 is configured for deployment / use by system 100 or any suitable system, upon which detector 210 is evaluated to operate at a predetermined level of accuracy. Once trained, detector 210 with its machine learning system 210A is preferably capable, via at least one processor, of predicting, by statistical and / or probabilistic means, whether a sequence warrants a nominal or adversarial label, thereby providing an indication of the presence or absence of an adversarial attack.

[0050] Moreover, various modifications can be made to the above-described embodiments without departing from the spirit and scope of these embodiments. For example, in FIG. 1, instead of classifier 200 and associated machine learning system 200A, system 100 can include any software module with a machine learning system trained to suit the intended application. That is, detector 210 and adversarial defense system 220 are configured to provide the same or substantially similar benefits to any machine learning and / or software system that relies on input sequences that may be susceptible to adversarial attack.

[0051] Additionally, as another variation, in addition to or instead of synthetic-type attacks, the set of adversarial sequences 412B can include real adversarial attack data obtained from actual adversarial attacks on various machine learning systems. Furthermore, as yet another variation, the set of adversarial sequences 412B can include any perturbed version of a nominal sequence, including, but not limited to, the examples of FIGS. 6A and 6B , in which multiple perturbations occur across multiple elements (or frames) of the sequence. Additionally, as yet another variation, the process of generating adversarial sequences can be combined with a training process 500 that can generate new adversarial sequences as a training set for each iteration of training the detector 210. Furthermore, the training process 500 can be performed over multiple iterations and multiple batches. As yet another variation, during training process 500, detector 210 and adversarial system 20 may be configured to create a zero-sum game, where detector 210 may operate to minimize the combined loss, while adversarial system 20 may operate to maximize the combined loss (e.g., making sequences of adversarial data undetectable), thereby training detector 210 to become more robust by addressing a more robust adversarial system 20.

[0052] As described herein, the present embodiments include several advantageous features and benefits. For example, the present embodiments are advantageous for identifying whether an input sequence to the machine learning system 200A is a query sequence that has an adversarial goal, in the sense of understanding the limitations of a model and / or learning perturbations to the input data that cause the machine learning system 200A to induce errors. In this regard, the present embodiments are advantageous for addressing the technical problem of identifying whether a sequence input to at least one machine learning system 200A is associated with nominal data (e.g., a sensor stream from a sensor) or adversarial data (e.g., a perturbed version of the sensor stream with an adversarial query from the adversarial system 20) as a means of detecting the absence / presence of an adversarial attack. Upon detecting an adversarial sequence, the detector 210 can operate to flag the adversarial detection, thereby enabling the system 100 to respond to the adversarial attack. As an example, for example, adversarial defense system 220 may be activated to filter a corresponding sequence of output data generated by machine learning system 200A based on the adversarial sequence, such that effects resulting from the adversarial sequence are avoided and / or not realized by other systems that would otherwise receive this output data from machine learning system 200A. For example, system 100 may operate to prevent erroneous output data (e.g., erroneous class data) based on a detected sequence of adversarial data from impacting actuator system 130, thereby providing an additional level of security and safety to system 100 with respect to adversarial attacks.

[0053] That is, the above description is intended to be illustrative and not limiting, provided in the context of a particular application and its requirements. Those skilled in the art will appreciate from the foregoing description that the invention may be implemented in a variety of forms, and that various embodiments may be practiced alone or in combination. Thus, while embodiments of the invention have been described with reference to specific examples thereof, the general principles defined herein may be applied to other embodiments and applications without departing from the spirit and scope of the described embodiments, and the true scope of the embodiments and / or methods of the invention is not limited to the embodiments shown and described. This is because various modifications will become apparent to those skilled in the art upon review of the drawings, the specification, and the following claims. For example, components and functions may be separated or combined in different ways, or described using different terminology, from those in the various described embodiments. These and other variations, modifications, additions, and improvements may be included within the scope of the present disclosure, as defined in the following claims.

Claims

1. 1. A computer-implemented method for training a machine learning system to detect adversarial attacks and defending against the adversarial attacks with the trained machine learning system, comprising: obtaining a collection of sequences including at least a first sequence and a second sequence; classifying the first sequence as belonging to a first class indicative of a nominal sequence based on a first prediction that the first sequence includes an unperturbed version of sensor data; classifying the second sequence as belonging to a second class indicative of an adversarial sequence based on a second prediction that the second sequence includes a perturbed version of sensor data; generating combined loss data based on (i) a first average loss comprising a misclassification of the first class for a first set of sequences from a collection of sequences, wherein each sequence in the first set of sequences is a nominal sequence, and (ii) a second average loss comprising a misclassification of a second class for a second set of sequences from a collection of sequences, wherein each sequence in the second set of sequences is an adversarial sequence; updating parameters of the machine learning system based on the combination loss data; and for defense against adversarial attacks performed by the trained machine learning system, the computer-implemented method further comprises: obtaining an input sequence for the other machine learning system; using the trained machine learning system to generate adversarial labels for classifying the input sequence based on a prediction that the input sequence is a perturbed version of multiple frames of sensor data; identifying an output data sequence generated by the other machine learning system based on the input sequence; filtering the output data sequence from the other machine learning system based on the adversarial labels to prevent an actuator system from being controlled by control data based on the output data sequence; 11. A computer-implemented method comprising:

2. The step of updating the parameters includes: determining parameters of a discriminative model of the machine learning system that minimizes the combined loss data of a weighting function including the first average loss and the second average loss; The computer-implemented method of claim 1 .

3. The machine learning system includes a deep neural network with an architecture for processing time sequences. The computer-implemented method of claim 1 .

4. the machine learning system includes a recurrent neural network, a long short-term memory network, or a gated recurrent unit; The computer-implemented method of claim 1 .

5. the first sequence is generated such that the other machine learning system generates first class data for the first sequence, and the second sequence is a perturbed version of the first sequence such that the other machine learning system generates second class data for the second sequence, the first class data being different from the second class data; The computer-implemented method of claim 1 .

6. the first sequence is extracted from a sensor data stream, the first sequence comprising a plurality of frames of sensor data, the second sequence comprising subsequences, the subsequences comprising repeated perturbed versions of selected frames of the first sequence, one of the perturbed versions of the selected frames causing the other machine learning system to generate the second class data for the second sequence; The computer-implemented method of claim 5 .

7. the first sequence is extracted from a sensor data stream, the first sequence comprising a plurality of frames of sensor data, and each frame of the second sequence is perturbed with a respective perturbation such that the second sequence causes the other machine learning system to generate the second class data for the second sequence; The computer-implemented method of claim 5 .

8. a non-transitory computer-readable medium comprising computer-readable data that, when executed by a processor, causes the processor to perform a method for training a machine learning system to detect adversarial attacks and for defending against the adversarial attacks with the trained machine learning system; The method comprises: obtaining a collection of sequences including at least a first sequence and a second sequence; classifying the first sequence as belonging to a first class indicative of a nominal sequence based on a first prediction that the first sequence includes an unperturbed version of sensor data; classifying the second sequence as belonging to a second class indicative of an adversarial sequence based on a second prediction that the second sequence includes a perturbed version of sensor data; generating combined loss data based on (i) a first average loss comprising a misclassification of the first class for a first set of sequences from a collection of sequences, wherein each sequence in the first set of sequences is a nominal sequence, and (ii) a second average loss comprising a misclassification of a second class for a second set of sequences from a collection of sequences, wherein each sequence in the second set of sequences is an adversarial sequence; updating parameters of the machine learning system based on the combination loss data; and for defense against adversarial attacks performed by the trained machine learning system, the method further comprises: obtaining an input sequence for the other machine learning system; using the trained machine learning system to generate adversarial labels for classifying the input sequence based on a prediction that the input sequence is a perturbed version of multiple frames of sensor data; identifying an output data sequence generated by the other machine learning system based on the input sequence; filtering the output data sequence from the other machine learning system based on the adversarial labels to prevent an actuator system from being controlled by control data based on the output data sequence; Including, Non-transitory computer-readable medium.

9. updating the parameters includes determining parameters of a discriminator model of the machine learning system that minimizes the combined loss data of a weighting function including the first average loss and the second average loss; The non-transitory computer-readable medium of claim 8.

10. The machine learning system includes a deep neural network with an architecture for processing time sequences. The non-transitory computer-readable medium of claim 8.

11. the machine learning system includes a recurrent neural network, a long short-term memory network, or a gated recurrent unit; The non-transitory computer-readable medium of claim 8.

12. the first sequence is generated such that the other machine learning system generates first class data for the first sequence, and the second sequence is a perturbed version of the first sequence such that the other machine learning system generates second class data for the second sequence, the first class data being different from the second class data; The non-transitory computer-readable medium of claim 8.

13. the first sequence is extracted from a sensor data stream, the first sequence comprising a plurality of frames of sensor data, the second sequence comprising subsequences, the subsequences comprising repeated perturbed versions of selected frames of the first sequence, one of the perturbed versions of the selected frames causing the other machine learning system to generate the second class data for the second sequence; The non-transitory computer-readable medium of claim 12.

14. the first sequence is extracted from a sensor data stream, the first sequence comprising a plurality of frames of sensor data, and each frame of the second sequence is perturbed with a respective perturbation such that the second sequence causes the other machine learning system to generate the second class data for the second sequence; The non-transitory computer-readable medium of claim 12.

15. 1. A computer-implemented method for defending against adversarial attacks, comprising: obtaining an input sequence for a first machine learning system; generating adversarial labels for classifying the input sequence based on a prediction that the input sequence is a perturbed version of multiple frames of sensor data; identifying an output data sequence generated by the first machine learning system based on the input sequence; filtering the output data sequence from the first machine learning system based on the adversarial labels to prevent an actuator system from being controlled by control data based on the output data sequence; 10. A computer-implemented method comprising:

16. obtaining another input sequence to the first machine learning system, the other input sequence including another plurality of frames of sensor data; generating non-adversarial labels for the other input sequences based on another prediction that the other input sequences are not perturbed by perturbation data; obtaining another output data sequence generated by the first machine learning system based on the other input sequence; controlling the actuator system based on the other output data sequence; further comprising:

16. The computer-implemented method of claim 15.

17. and further comprising operating a sliding window to obtain the input sequence being input to the first machine learning system, wherein the plurality of frames of sensor data comprises repeated perturbed versions of selected frames of sensor data.

16. The computer-implemented method of claim 15.

18. generating the adversarial labels is performed by a second machine learning system; 16. The computer-implemented method of claim 15.

19. the second machine learning system is trained with a set of sequences, the set of sequences including a set of nominal sequences and a set of adversarial sequences, the set of adversarial sequences including perturbed versions of the set of nominal sequences such that the first machine learning system generates different class data based on the adversarial sequences compared to class data based on the nominal sequences; 20. The computer-implemented method of claim 18.

20. the second machine learning system includes a recurrent neural network, a long short-term memory network, or a gated recurrent unit; 20. The computer-implemented method of claim 18.

Images