Apparatus and method for classifying an image and assessing the robustness of the classification
By evaluating the region score map and robustness of image classifiers, and using convolutional neural networks and non-decreasing functions, the problem of image classifiers being vulnerable to adversarial patch attacks is solved, resulting in more reliable classification results and ensuring device security.
Patent Information
- Application Number
- CN202180062549.1
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Priority Date
- 2020-09-15
- Filing Date
- 2021-09-13
- Publication Date
- 2025-10-28
- Estimated Expiration
- 2041-09-13
AI Technical Summary
Existing image classifiers are vulnerable to adversarial patch attacks, leading to misclassification and affecting the security and reliability of the device.
By determining score maps for multiple region classifications, aggregating category scores, and evaluating the robustness of the score maps under adversarial patch attacks, a convolutional neural network and a non-decreasing function are used to determine the robustness of classification, and the neural network is trained to improve its resistance to adversarial patches.
It improves the robustness of image classification, reliably determines whether the classification is affected by adversarial patches, and ensures the safety and reliability of device operation.
Smart Images
Figure CN116075865B_ABST
Abstract
Description
[0001] The present invention relates to methods for classifying images, methods for operating actuators, computer programs, machine-readable storage media, classifiers, and training systems. Existing technology
[0002] Wieland Brendel and Matthias Bethge's "Approximating CNNs with Bag-of-local-Features models works surprisingly well on ImageNet" (arXiv, 20.03.2019, https: / / arxiv.org / abs / 1904.00760v1) discloses an image classifier for classifying images based on image patches.
[0003] Advantages of the present invention
[0004] Classifiers are prone to misclassifying maliciously altered input data (known as adversarial examples). In a particular form, an adversarial example can be a constrained region of a datum that has been modified to form the datum. For example, a region of an image might be overlaid with another, smaller image, thus deceiving the image classifier that would otherwise correctly classify the image.
[0005] Creating adversarial examples in this way is known as threat modeling using adversarial tiles or tile-based adversarial examples. Adversarial tiles are highly relevant to many real-world devices that employ classifiers.
[0006] For example, an image classifier can be used to identify the location of pedestrians in the environment surrounding a vehicle that is at least partially autonomous, in order to calculate a safe path for the vehicle to traverse the environment without hitting any pedestrians. In this example, adversarial patches might be applied to a portion of the environment or to at least one pedestrian themselves, causing the image classifier to miss at least one pedestrian. This could lead the vehicle to determine a path through falsely detected pedestrians, which could in turn lead to unsafe behavior by the vehicle.
[0007] Therefore, there is an expectation to protect the classifier against adversarial pieces.
[0008] The advantage of the method having the features of independent claim 1 is that, in addition to classifying the image, the method also indicates whether the classification result is provably robust against alterations to the adversarial patch. This increases classification performance because it can clearly and reliably indicate whether the classification might not be secure enough to be trusted with respect to the adversarial patch. This information, in turn, can be used, for example, to control a device using the method to classify an area surrounding the device. For instance, if a non-robust classification is detected, operation of the device can be suspended or transferred to a human operator. Summary of the Invention
[0009] In a first aspect, the present invention relates to a computer-implemented method for determining an output signal that characterizes a first classification of an input image as a category from a plurality of categories, wherein the output signal further characterizes a second classification robust to attacks with adversarial patching, wherein determining the output signal comprises the following steps:
[0010] • Determine multiple first score maps, wherein the first score maps correspond to categories from the multiple categories and include multiple region classifications, wherein each region classification characterizes whether a region belongs to a category for a region in multiple regions of the image;
[0011] • Determine multiple first category scores, wherein each first category score from the multiple first category scores corresponds to a category from the multiple categories, and is determined by aggregating the regions of the first score map corresponding to the category that represent the category;
[0012] • Determine a second category score, wherein the second category score is the minimum category score that can be obtained when adversarial patches are applied to the input image to change the region classification of the first score map corresponding to the first category with the maximum first category score;
[0013] • Determine multiple third-category scores, where the third-category score corresponds to a category different from the first category, and is the maximum category score that can be obtained when adversarial patches are applied to the input image to change the region classification of the first score map of the category;
[0014] If the second category score is greater than all third category scores, then the output signal is determined such that it characterizes the first category of the input image based on the plurality of first category scores, and it characterizes the second category of the first category as robust to the adversarial patch;
[0015] If the second category score is less than at least one third category score, then the output signal is determined such that it characterizes a first category of the input image based on the plurality of first category scores, and it characterizes the second category of the first category as not robust to the adversarial patch.
[0016] The input image can be determined, for example, from a sensor such as a camera sensor, LiDAR sensor, radar sensor, ultrasonic sensor, or thermal camera. It is also conceivable that the input image is the result of applying preprocessing steps to another image, such as cropping and / or scaling and / or rotating and / or color correction of the other image. The input image can also be a computer simulation, for example, the result of rendering a virtual environment onto an image. It is also conceivable that the input image is digitally drawn. It is further conceivable that the input image is a combination of multiple images, such as images from multiple different sensors.
[0017] This method can preferably be implemented by an image classifier configured to receive images of a predefined type and provide an output signal based on all steps of the method.
[0018] The input image can be overlaid with adversarial tiles. These adversarial tiles can be in the form of physical adversarial tiles, i.e., real-world tiles from which the input image is recorded. This can be achieved, for example, by applying an adhesive to an object. Alternatively, the input image can be overlaid with adversarial tiles after it has been recorded. This is possible, for example, if an attacker has access to the sensing device responsible for sensing the input image and can manipulate the input image after it has been sensed by the sensing device.
[0019] In the context of this invention, adversarial patches can be understood as an attempt to flip the region classification of a score map corresponding to a correctly classified category of an input image. Flipping can be understood as changing the region classification of the score map relative to what the region classification would be if the adversarial patch were not applied to the image. An attacker uses adversarial patches with the aim of flipping the region classification of the score map that is expected to predict the category for the input image, such that the aggregated category score of the score map is lower than another aggregated category score, and the image is thus misclassified.
[0020] Preferably, each score map from the plurality of first score maps is determined by a convolutional neural network. This is advantageous because convolutional neural networks are currently the highest-performing image classifiers. Since a score map can be understood as a set of categories, using a convolutional neural network to determine the score map allows for obtaining the score map with the highest accuracy.
[0021] A score map can be understood as assigning region classifications to an image, where the score map corresponds to a category, and each category either indicates that a region belongs to a category or that the region does not belong to that category. Preferably, the score map is given by a matrix with predefined height and width. Each element of the matrix can be understood as a region classification.
[0022] If the score map is determined from a convolutional neural network, then each element of the matrix obtained from the convolutional neural network (i.e., each region classification) is associated with a certain region of the input image. This region is also referred to as the receptive field of the corresponding element of the matrix. This region is preferably a small sub-part of the input image. The size of the receptive field can be controlled by the architecture of the convolutional neural network.
[0023] The score map can be aggregated into a single category score, where the category score can be understood as the unnormalized probability of the category corresponding to the score map.
[0024] The score map depends on the input image. It can be conceived as multiple classification scores for multiple regions of an adversarial patch to which adversarial patches can be applied.
[0025] It can be envisioned that an aggregated score map is implemented by determining the sum of the regions in the score map that represent the categories corresponding to the score map. Alternatively, an aggregated score map can be implemented using a neural network, wherein the neural network is configured to take the score map as input and provide category scores as output, and wherein the neural network consists only of positive weights and non-decreasing activation functions.
[0026] For both aggregation methods, the advantage lies in the fact that the mathematical function underlying the aggregation is non-decreasing. A non-decreasing function can be understood as follows: Imagine that in a score graph, the region classification for categories indicating belonging to the corresponding category in the score graph is given as 1, and the classification for categories indicating not belonging to that category is given as 0. Then, if "if M..." i，j (1) ≥M i，j (2) Then g(M) (1) )≥g(M (2) If the condition ")" holds true, then the aggregate function can be understood as non-decreasing, where It is the score chart M (1) Region classification at positions i and j It is the score chart M (2) The region is classified at position i, j, and g(.) is the function.
[0027] The authors surprisingly discovered that if the condition is met (i.e., the aggregation function is non-decreasing), the score map corresponding to the highest category score among multiple first category scores cannot be flipped such that the category score obtained from aggregating the flipped score map is lower than the category score obtained from aggregating any other flipped score map of the input image corresponding to a category score lower than the highest category score, if the following condition is met:
[0028] and
[0029] in It is for category c corresponding to the highest score t The obtained score map, where the region classification of the score map has been based on the image X with adversarial patches inserted at position p. p And flipped, M c (X p ) is the score map obtained for category c, where the region classification of the score map has been based on image X with adversarial patches inserted at position p. p And then it was flipped over.
[0030] This can be understood as determining how much the maximum first-class score can be reduced by the adversarial piece while increasing other first-class scores. If it is impossible to reduce the maximum first-class score below the maximum increase of another first-class score, then the adversarial piece cannot change the classification represented by the output signal.
[0031] This condition can be evaluated without explicitly generating a new score map for every possible location of the adversarial patch: each score map is associated with a region of the image. If the adversarial patch is placed somewhere, the worst-case scenario it can do is flip all region classifications of the score map corresponding to the highest first-class score, such that they represent classifications that do not belong to the category of the score map—if the region classifications depend on the adversarial patch. For example, if the adversarial patch extends into a region of the input image, it can flip the region classifications of the region's score map. However, the adversarial patch is only able to flip the region classifications corresponding to the region actually covered by the adversarial patch.
[0032] If the sum of the score maps is used for the aggregated score map, it can be further envisioned that in the step of determining the second category score, the second category score is determined by subtracting the maximum amount of region classification that the adversarial patch can change from the maximum first category score.
[0033] In this context, it is also conceivable that, in the step of determining the plurality of third category scores, the third category scores are determined by adding the maximum amount of region classification that the adversarial patch can change to the first category score.
[0034] The advantage of this approach is that it avoids explicitly calculating all possible locations of the adversarial patch. More precisely, the method determines the absolute amount by which the adversarial patch can alter the region classification and whether changing that amount could potentially produce a change in classification. This method requires relatively little computation because the score map does not need to be fitted, but the results can be obtained directly from the first-class score. In practice, this leads to classifying more images using the same amount of resources, i.e., energy, within a given amount of time.
[0035] If a neural network is used for the aggregated score graph, it can be further envisioned that the method for determining the output signal further includes training the neural network, wherein the training of the neural network is achieved through adversarial training.
[0036] For example, given a fixed convolutional neural network for determining a first score map from an input image, a neural network for aggregation can be trained, including the following steps:
[0037] • Identify the training images;
[0038] • Cover the training image with adversarial patches at randomly determined locations;
[0039] • Determine multiple first score maps of the input image;
[0040] • By using neural networks, the score maps are aggregated into scores for multiple categories;
[0041] • Determine the loss value representing the difference between the determined category score and multiple expected category scores;
[0042] • Determine the gradient of the loss value with respect to multiple parameters of the neural network;
[0043] • Update the parameters of the neural network based on the gradient.
[0044] Marginal loss can be used as the loss function.
[0045]
[0046] in It is the c-th category score among the determined multiple category scores, and m is the value representing the predefined margin, and It is the first category score of the expected category of the training image.
[0047] The multiple parameters of a neural network may include at least a portion of the weights of the neural network and / or at least a portion of the biases of the neural network and / or normalization parameters, such as those used in a normalization layer.
[0048] It can be further imagined that the process of determining the second category score includes the following steps:
[0049] • Determine multiple second score maps corresponding to the category with the largest first category score, wherein for each possible location of the adversarial patch in the input image, the second score map is determined, wherein for each second score map, if the adversarial patch extends into a region corresponding to a region classification, the region classification is determined to not represent the category;
[0050] • Aggregate each second score plot to determine multiple intermediate category scores;
[0051] • Provide the minimum intermediate category score from the multiple intermediate category scores as the second category score.
[0052] It can be further imagined that, in the step of determining the plurality of third-category scores, determining the third-category scores includes the following steps:
[0053] • Determine multiple second score maps corresponding to the category of the third category score, wherein for each possible location of the adversarial patch in the input image, the second score map is determined, wherein for each second score map, if the adversarial patch extends into a region corresponding to the region classification, the region classification is determined to characterize the category;
[0054] • Aggregate each second score plot to determine multiple intermediate category scores;
[0055] • Provide the largest intermediate category score from the multiple intermediate category scores as the second category score.
[0056] The advantage of this method is that it determines the actual number of regions that can be flipped. This allows for the determination of a more stringent bound on the robustness of the classification; that is, the first classification can be considered robust even if the difference between the maximum first-class score and all other first-class scores is less than twice the maximum number of regions that the adversarial patch can flip.
[0057] It can be further imagined that multiple first score maps and / or multiple second score maps are determined by a convolutional neural network.
[0058] The advantage of using convolutional neural networks (CNNs) is that they are currently the most accurate image classifiers for image classification. Since obtaining multiple first score maps and / or multiple second score maps can be understood as image classification, using CNNs for this task allows for the highest classification accuracy.
[0059] Preferably, the convolutional neural network determines the first score map by determining the output of the output layer of the convolutional neural network for the input image (x) and applying a Heaviside function to the output.
[0060] Preferably, the method further includes training a convolutional neural network, wherein training includes the following steps:
[0061] • Multiple first score maps of the training images are determined by feeding the training images into the convolutional neural network;
[0062] • Multiple first-category scores are determined by aggregating each first score map from multiple previously determined score maps;
[0063] • Determine the loss value
[0064]
[0065] in It is the c-th category score among the multiple first category scores, and m is the value representing the predefined margin. It is the first category score of the expected category of the training image (xi);
[0066] • Backpropagation is used to determine the gradients of multiple parameters of the convolutional neural network, wherein if the convolutional neural network includes a Herveside function, the gradient of the Herveside function is replaced by the gradient of the Sigmoid function;
[0067] • Adapt the parameters of the convolutional neural network according to the gradient.
[0068] The multiple parameters of a convolutional neural network may include at least a portion of the weights of the neural network and / or at least a portion of the biases of the neural network and / or normalization parameters, such as those used in a batch normalization layer.
[0069] Training a convolutional neural network in this way can be understood as the convolutional neural network learning to predict a score map that, when aggregated, has the largest possible margin between the largest and second-largest first-class scores. Therefore, it advantageously learns to become robust against attacks with adversarial patches.
[0070] The margin can be chosen based on the maximum number of regions that the adversarial patch can flip. This way, once the maximum first-class score has reached a value that makes it impossible for the adversarial patch to flip, the convolutional neural network doesn't need to increase it. This advantageously alleviates the learning problem and allows for faster training time. Given the same amount of resources, i.e., the same amount of training time, the proposed method thus increases performance because the convolutional neural network can be trained on more images.
[0071] Embodiments of the invention will be discussed in more detail with reference to the following figures. The figures illustrate:
[0072] Figure 1 Image classifier;
[0073] Figure 2 Control systems that include image classifiers that control actuators in their environment;
[0074] Figure 3 A control system for controlling at least partially autonomous robots;
[0075] Figure 4 Control system for an automated personal assistant;
[0076] Figure 5 Control system for medical analysis systems;
[0077] Figure 6 A training system for training image classifiers. Detailed Implementation
[0078] Figure 1 The image shown is an embodiment of an image classifier (60) for determining a first category (c) of an input image (x) and a second category (r) that characterizes the robustness of the first category (c).
[0079] An image classifier (60) receives an input image (x) in a score map unit (61). The score map unit (61) is configured to determine multiple score maps (M) from the input image (x). Preferably, the score map unit (61) includes a convolutional neural network that takes the input image (x) as input and provides multiple score maps (M). For this purpose, the neural network can be configured to provide tensors representing the multiple score maps (M), wherein the tensors have predefined height, width, and depth. Each matrix along the depth dimension of the tensor can represent a score map among the multiple score maps, where each dimension corresponds to a specific category. In other words, each matrix along the depth dimension of the tensor can represent a score map of a specific category. The tensor (M) includes region classifications (i.e., elements of the tensor (M), which can be 0 or 1. 0 can indicate that a region classification represents a region as not belonging to the category corresponding to the score map to which that region classification belongs. Similarly, 1 can indicate that a region classification represents a region as belonging to the category corresponding to the score map to which that region classification belongs. In other words, the tensor (M) can be a tensor of zeros and ones. In another embodiment, this dichotomous classification can also be expressed by two other values, for example, as binary variables (true or false). In yet another embodiment, the tensor (M) can also include values in the interval from 0 to 1 (and inclusive of 0 and 1), such as probability values.
[0080] Preferably, the receptive field of the output layer of the convolutional neural network providing tensors covers a small region of the input image (x). For example, the convolutional neural network can be selected such that the final layer has a receptive field of 9 by 9, 17 by 17, or 33 by 33 pixels in the input image (x).
[0081] Preferably, the convolutional neural network includes a Hervisside function as the activation function in the output layer, which provides a tensor of the score map (M). Thus, the tensor (M) consists only of zeros and ones. In another embodiment, it is conceivable to use different activations or no activations in the output layer, and then compare the results of the output layer against a predefined threshold to determine the tensor or score map (M).
[0082] The tensor (M) is then forwarded to a classification unit (63), which aggregates the score maps of the tensor (M). In this embodiment, the classification unit (63) performs aggregation by summing the elements of each matrix along the depth of the tensor, i.e., by determining the sum of each score map in the tensor (M). The result is multiple first-class scores (c). The first-class scores (c) can be understood as representing the first classification of the input image (x). In another embodiment, it is conceivable that, instead of summing the score maps, a second convolutional neural network is used, which takes the tensor (M) as input and provides multiple first-class scores (c). In these other embodiments, the second convolutional neural network is configured as a non-decreasing function. This means that the second convolutional neural network only uses non-decreasing functions as activation functions and only includes positive weights.
[0083] The tensor (M) is also forwarded to the robustness unit (62). The robustness unit (62) is configured to determine whether the classification represented by the first category score (c) can be changed if an adversarial patch is applied to the input image (x). Here, applying an adversarial patch to the input image (x) can be understood as either directly changing the input image (x) by overlaying it with an adversarial patch. Alternatively, applying an adversarial patch to the input image (x) can also be understood as applying an adversarial patch to the scene from which the input image (x) is being sensed.
[0084] The robustness unit (62) takes a tensor (M) as input and provides a robust classification (r) of multiple first category scores (c). To this end, the robustness unit determines the maximum amount of region classification that an adversarial patch can alter. In this embodiment, this is achieved by determining the amount of region classification of the score map that the adversarial patch can alter. Since each region classification depends only on a region of the input image (x), the adversarial patch can only alter the region classification if it extends into or covers that region. Therefore, the maximum amount of region classification of the score map that an adversarial patch can alter is equal to the amount of region in the input image (x) that the adversarial patch can cover or extend into. Having obtained this amount, the robustness unit (62) determines a second classification (r), wherein the second classification (r) classifies the first classification as robust if the largest first category score from the multiple first category scores (c) is greater than or equal to the sum of the second largest first category score from the multiple first category scores (c) and twice the maximum amount. If the largest first-class score from multiple first-class scores (c) is less than the sum of the second largest first-class score from multiple first-class scores (c) and twice that largest score, then the second classification (r) is determined to characterize the first classification as non-robust.
[0085] Then, multiple first category scores (c) and robust classifications (r) can be provided as the output signal (y) of the image classifier (60). In another embodiment, an optional conversion unit (64) of the image classifier (60) can process the first category scores (c) and / or robust classifications (r) and output the result as the output signal (y). For example, it is conceivable that instead of multiple category scores (c), the output signal (y) provides the category corresponding to the largest category score, along with the robust classification (r). Alternatively, if the robust classification (r) represents a robust classification, the conversion unit can provide multiple first category scores (c) or that category as the output signal (y). In this case, the robust classification (r) represents a non-robust classification, and the output signal (y) can be set to represent no clear classification result. This can be, for example, by setting the output signal (y) such that it represents a rejected classification.
[0086] Figure 2 The diagram shows an embodiment of an actuator (10) in its environment (20). The actuator (10) interacts with a control system (40). The actuator (10) and its environment (20) will be collectively referred to as the actuator system. At preferably evenly spaced time points, a sensor (30) senses the condition of the actuator system. The sensor (30) may include a plurality of sensors. Preferably, the sensor (30) is an optical sensor that acquires an image of the environment (20). The output signal (S) of the sensor (30) (or, in the case where the sensor (30) includes a plurality of sensors, the output signal (S) of each sensor) is transmitted to the control system (40), and the output signal (S) encodes the sensed condition.
[0087] Thus, the control system (40) receives a stream of sensor signals (S). It then calculates a series of actuator control commands (A) based on the stream of sensor signals (S), which are then transmitted to the actuator (10).
[0088] The control system (40) receives a stream of sensor signals (S) from the sensor (30) in an optional receiving unit (50). The receiving unit (50) transforms the sensor signals (S) into an input image (x). Alternatively, without a receiving unit (50), each sensor signal (S) can be directly taken as the input image (x). The input image (x) can be given, for example, as an excerpt from the sensor signals (S). Alternatively, the sensor signals (S) can be processed to generate the input image (x). The input image (x) includes image data corresponding to the image recorded by the sensor (30). In other words, the input image (x) is provided based on the sensor signals (S).
[0089] The input image (x) is then passed to the image classifier (60).
[0090] The image classifier (60) is parameterized by a parameter (φ), which is stored in and provided by the parameter storage device (St1).
[0091] An image classifier (60) determines an output signal (y) from an input image (x). The output signal (y) includes information that assigns one or more labels to the input image (x). The output signal (y) is transmitted to an optional conversion unit (80), which converts the output signal (y) into a control command (A). The actuator control command (A) is then transmitted to the actuator (10) for corresponding control of the actuator (10). Alternatively, the output signal (y) can be directly taken as the actuator control command (A).
[0092] The actuator (10) receives an actuator control command (A), is controlled accordingly, and performs an action corresponding to the actuator control command (A). The actuator (10) may include control logic that transforms the actuator control command (A) into another control command, which is then used to control the actuator (10).
[0093] In another embodiment, the control system (40) may include a sensor (30). In yet another embodiment, the control system (40) may alternatively or additionally include an actuator (10).
[0094] In another embodiment, it is conceivable that, instead of or attached to the actuator (10), the control system (40) controls the display (10a). The display may, for example, show the classification characterized by the output signal (y) and / or whether the classification is robust.
[0095] Furthermore, the control system (40) may include a processor (45) (or multiple processors) and at least one machine-readable storage medium (46) thereon storing instructions which, if executed, cause the control system (40) to perform a method according to one aspect of the invention.
[0096] Figure 3 One embodiment is shown, wherein the control system (40) is used to control a robot that is at least partially autonomous, such as a vehicle (100) that is at least partially autonomous.
[0097] The sensor (30) may include one or more video sensors and / or one or more radar sensors and / or one or more ultrasonic sensors and / or one or more LiDAR sensors and / or one or more location sensors (e.g., GPS). Some or all of these sensors are preferably, but not necessarily, integrated into the vehicle (100).
[0098] Alternatively or additionally, the sensor (30) may include an information system for determining the state of the actuator system. An example of such an information system is a weather information system that determines the current or future state of the weather in the environment (20).
[0099] The image classifier (60) can be configured to identify whether the vehicle is currently in an urban environment, a rural environment, or a highway in order to determine whether autonomous operation of the vehicle should be permitted. For example, it is conceivable that autonomous operation of the vehicle is permitted only if the vehicle is on a highway. It is further conceivable that the autonomous operation is enabled only if, in addition to classifying the environment as "on a highway," the output signal (y) of the image classifier (60) also indicates a robust classification (i.e., a classification that cannot be altered by adversarial patches).
[0100] The actuator (10), preferably integrated in the vehicle (100), can be given by the vehicle 100's brakes, propulsion system, engine, drivetrain, or steering mechanism. An actuator control command (A) can be determined to control the actuator (or multiple actuators) (10) so that the vehicle (100) avoids a collision with a detected object. The detected object can also be classified according to what a classifier (60) considers most likely to be (e.g., a pedestrian or a tree), and the actuator control command (A) can be determined based on this classification.
[0101] In another embodiment, the at least partially autonomous robot may be provided by another mobile robot (not shown), which may move, for example, by flying, swimming, diving, or walking. This mobile robot may, in particular, be a lawnmower that is at least partially autonomous, or a cleaning robot that is at least partially autonomous. In all the above embodiments, actuator command control (A) may be determined to control the propulsion unit and / or steering mechanism and / or brakes of the mobile robot so that the mobile robot can avoid collisions with the identified object.
[0102] Figure 4 The illustration shows an embodiment in which a control system (40) controls an automated personal assistant (250). The sensor (30) may be an optical sensor, such as a video image of a user's (249) gesture. Alternatively, the sensor (30) may also be an audio sensor, such as a voice command from the user (249).
[0103] The control system (40) then determines an actuator control command (A) for controlling the automated personal assistant (250). The actuator control command (A) is determined based on sensor signals (S) from a sensor (30). The sensor signals (S) are transmitted to the control system (40). For example, an image classifier (60) can be configured to perform gesture recognition, for example, based on at least one image of a user (249). The control system (40) can then determine the actuator control command (A) for transmission to the automated personal assistant (250). It then transmits the actuator control command (A) to the automated personal assistant (250).
[0104] For example, the actuator control command (A) can be determined based on the identified user gesture recognized by the image classifier (60). It may include information that causes the automated personal assistant (250) to retrieve information from the database and output the retrieved information in a form suitable for the user (249) to receive.
[0105] In another embodiment, it is conceivable that, instead of an automated personal assistant (250), a control system (40) controls a household appliance (not shown) based on identified user gestures. This appliance could be a washing machine, stove, oven, microwave oven, or dishwasher.
[0106] Figure 5 The illustration shows an embodiment of a medical analysis system (600) controlled by a control system (40). The medical analysis system (600) is supplied with a microarray (601), wherein the microarray includes multiple sites (602, also referred to as features) that have been exposed to a medical sample. For example, the medical sample may be, for example, a human sample or an animal sample obtained from a swab.
[0107] The microarray (601) can be a DNA microarray or a protein microarray.
[0108] The sensor (30) is configured to sense a microarray (601). The sensor (30) is preferably an optical sensor, such as a video sensor.
[0109] The image classifier (60) is configured to classify the results of the sample based on the input image (x) of the microarray supplied by the sensor (30). In particular, the image classifier (60) can be configured to determine whether the microarray (601) indicates the presence of a virus in the sample.
[0110] Then a control signal (A) can be selected so that the display (10a) shows the classification results.
[0111] Figure 6An embodiment of a training system (140) for training an image classifier (60) of a control system (40) using a training dataset (T) is shown. The training dataset (T) includes multiple input images (x, y, x) used to train the classifier (60). i The training dataset (T) is further tailored to each input image (x). i This includes the desired output signal (y). i The desired output signal (y) i ) corresponds to the input image (x) i And characterize the input image (x) i ) expected classification.
[0112] For training, the training data unit (150) accesses a computer-implemented database (St2) that provides a training dataset (T). The training data unit (150) preferably randomly selects at least one input image (x) from the training dataset (T). i ) and corresponding to the input image (x) i The desired output signal (y) i ), and input image (x i The signal is transmitted to the image classifier (60). The image classifier (60) determines the output signal based on the input image (xi). This output signal This includes at least several first-class scores obtained for the input image (xi).
[0113] Desired output signal (yi) and determined output signal It is transferred to the modification unit (180).
[0114] Based on the desired output signal (yi) and the determined output signal The modification unit (180) then determines new parameters (Φ′) for the image classifier (60). For this purpose, the modification unit (180) uses a loss function to compare the desired output signal (yi) with the determined output signal. A comparison is made. The loss function determines a first loss value, which characterizes the determined output signal. How far it deviates from the desired output signal (yi). In a given embodiment, the marginal loss
[0115]
[0116] Used as the loss function, where It is the c-th category score from multiple first category scores determined by the image classifier (60), and m is a value representing a predefined boundary. The training image (x) iThe expected category score of the first category.
[0117] In other embodiments, other loss functions can be used, especially known loss functions for multi-class classification, such as multinomial cross-entropy loss.
[0118] The modification unit (180) determines new parameters (Φ′) based on the first loss value. In a given embodiment, this is done using a gradient descent method, preferably stochastic gradient descent, Adam, or AdamW. If the image classifier (60) includes a Herveside function, the gradient of the Herveside function is replaced with the gradient of the sigmoid function.
[0119] In other preferred embodiments, the described training is iteratively repeated up to a predefined number of iterations, or iteratively repeated until a first loss value falls below a predefined threshold. Alternatively or additionally, it is also conceivable that training terminates when the average first loss value on the test or validation dataset falls below a predefined threshold. In at least one iteration, new parameters (Φ′) determined in previous iterations are used as parameters (Φ) of the classifier (60).
[0120] In this embodiment, the parameters (Φ) of the image classifier (60) to be updated are the trainable parameters of the convolutional neural network of the image classifier (60). In another embodiment, the image classifier (60) may include a second convolutional neural network, the parameters of which may also be part of the parameters to be updated of the image classifier (60), or attached to or replacing the parameters of the convolutional neural network.
[0121] Furthermore, the training system (140) may include at least one processor (145) and at least one machine-readable storage medium (146) containing instructions that, when executed by the processor (145), cause the training system (140) to perform a training method according to one aspect of the invention.
[0122] The term "computer" can be understood to encompass any device used to process predefined computational rules. These computational rules can take the form of software, hardware, or a combination of both.
[0123] Generally, a plurality can be understood as being indexed, meaning that each element in the plurality is assigned a unique index, preferably by assigning consecutive integers to the elements contained in the plurality. Preferably, if the plurality has N elements, where N is the number of elements in the plurality, then these elements are assigned integers from 1 to N. It can also be understood that the elements in the plurality can be accessed through their indices.
Claims
1. A computer implementation method for determining an output signal (y) that characterizes a first classification (c) of an input image (x) as a category from a plurality of categories, wherein the output signal (y) further characterizes a second classification (r) of the robustness of the first classification (c) against attacks with adversarial patching, wherein determining the output signal (y) comprises the following steps: • Determine multiple first score maps (M), wherein the first score maps correspond to categories from the multiple categories and include multiple region classifications, wherein each region classification characterizes whether a region belongs to a category for a region in multiple regions of the image; • Determine multiple first category scores, wherein each first category score from the multiple first category scores corresponds to a category from the multiple categories, and is determined by aggregating the regions of the first score map corresponding to the category that represent the category; • Determine a second category score, wherein the second category score is the minimum category score that can be obtained when adversarial patches are applied to the input image to change the region classification of the first score map corresponding to the first category with the maximum first category score; • Determine multiple third-category scores, where the third-category scores correspond to categories different from the first category, and are the maximum category scores that can be obtained when adversarial patches are applied to the input image (x) to change the region classification of the first score map of the category; • If the second category score is greater than or equal to all third category scores, then the output signal (y) is determined such that it characterizes the first category (c) of the input image based on the plurality of first category scores, and it characterizes the second category (r) of the first category as robust to the adversarial patch; If the second category score is less than at least one third category score, then the output signal (y) is determined such that it characterizes the first category (c) of the input image based on the plurality of first category scores, and it characterizes the second category (r) of the first category as not robust to the adversarial patch.
2. The method of claim 1, wherein the aggregated score map is achieved by determining the sum of regional classifications that characterize the category corresponding to the score map.
3. The method of claim 2, wherein the aggregated score map is implemented by a neural network, wherein the neural network is configured to accept the score map as input and provide category scores as output, and wherein the neural network comprises only positive weights and non-decreasing activation functions.
4. The method of claim 2, wherein the method further comprises training a neural network, wherein training the neural network is achieved through adversarial training.
5. The method of claim 2, wherein in the step of determining the second category score, the second category score is determined by subtracting the maximum amount of region classification that the adversarial patch can change from the maximum first category score.
6. The method of claim 2 or 5, wherein in the step of determining a plurality of third category scores, the third category scores are determined by adding the maximum amount of region classification that the adversarial patch can change to the first category score.
7. The method according to any one of claims 1 to 4, wherein in the step of determining the second category score, determining the second category score includes the following steps: • Determine multiple second score maps corresponding to the category with the largest first category score, wherein for each possible location of the adversarial patch in the input image, the second score map is determined, wherein for each second score map, if the adversarial patch extends into a region corresponding to a region classification, the region classification is determined to not represent the category; • Aggregate each second score plot to determine multiple intermediate category scores; • Provide the minimum intermediate category score from the multiple intermediate category scores as the second category score.
8. The method according to any one of claims 1 to 3, wherein in the step of determining the plurality of third category scores, determining the third category scores includes the following steps: • Determine multiple second score maps corresponding to the category of the third category score, wherein for each possible location of the adversarial patch in the input image, the second score map is determined, wherein for each second score map, if the adversarial patch extends into a region corresponding to the region classification, the region classification is determined to characterize the category; • Aggregate each second score plot to determine multiple intermediate category scores; • Provide the largest intermediate category score from the multiple intermediate category scores as the second category score.
9. The method of claim 7, wherein the plurality of first score maps and / or the plurality of second score maps are determined by a convolutional neural network.
10. The method of claim 9, wherein the convolutional neural network determines a first score map by determining the output of the output layer of the convolutional neural network for the input image (x) and applying a Herveside function to the output.
11. The method of claim 9, wherein the method further comprises training the convolutional neural network, wherein training comprises the following steps: • By training images (x i The training image (x) is provided to the convolutional neural network to determine the training image (x). i Multiple first-score plots; • Multiple first-category scores are determined by aggregating each first score map from multiple previously determined score maps; • Determine the loss value in It is the c-th category score among the multiple first category scores, and m is the value representing the predefined margin. The training image (x) i The expected category score of the first category; • Backpropagation is used to determine the gradients of multiple parameters of the convolutional neural network, wherein if the convolutional neural network includes a Herveside function, the gradient of the Herveside function is replaced by the gradient of the Sigmoid function; • Adapt the parameters of the convolutional neural network according to the gradient.
12. The method according to any one of claims 1 to 5, wherein the device is controlled according to the output signal.
13. An image classifier (60) configured to determine an output signal (y) according to any one of claims 1 to 11, wherein the image classifier comprises a convolutional neural network according to claim 9 or 10, and an aggregation unit configured to determine an aggregation of score maps according to any one of claims 2 to 4.
14. A computer program product configured such that, if the computer program product is executed by a processor, it causes a computer to perform the method and all steps thereof according to any one of claims 1 to 12.
15. A machine-readable storage medium having a computer program product according to claim 14 stored thereon.