A measure of the sensitivity of the KI module to spoofing attempts

Abstract

A method for measuring the sensitivity of a KI module to spoofing attempts, having steps: determining classification and / or regression as an undisturbed result for a pre-given open data set in an input space E, the KI module mapping the open data set onto the classification and / or regression; applying at least one interference S with dimension d < D to the open data set such that at least one disturbed data set is formed in the input space E; determining the following classification and / or regression as a disturbed result, the KI module mapping the disturbed data set onto the classification and / or regression; the deviation between the disturbed result and the undisturbed result is determined using a pre-given metric; in response to the deviation satisfying a pre-given criterion, determining that the KI module is sensitive to spoofing attempts with dimension d with respect to the open data set.

Description

Technical Field

[0001] The invention relates to the protection of a system for analyzing measurement data using a KI module against cheating attempts, in particular using so-called “adversarial examples”. Background Art

[0002] In many fields, KI modules with learning artificial intelligence, such as artificial neural networks, are used to analyze measurement data. This is particularly advantageous when the measurement data, such as image data, are very high-dimensional and a relatively low-dimensional classification and / or regression can be determined from the measurement data. For example, image data can be classified using KI modules as follows: which objects and traffic signs are contained in the road scene.

[0003] Such KI modules can be attacked using so-called "adversarial examples". This involves targeted, maliciously introduced changes to the input data of the KI module, the goal of which is to cause classification into the wrong class. Thus, for example, a traffic sign can be changed by a sticker or marking that hardly attracts the attention of the human driver, so that the traffic sign is recognized by the network as another traffic sign. Accordingly, the vehicle reacts incorrectly to the traffic sign.

[0004] DE 10 2018 200 724 A1 discloses a method for generating data signal disturbances, by means of which typical attacks of this type can be artificially simulated. These simulations can be used to study the sensitivity of classification networks and to test countermeasures. Summary of the invention

[0005] A method for measuring the sensitivity of a KI module to fraudulent attempts is developed within the scope of the invention. The KI module comprises a trainable artificial intelligence in the form of a parameterized internal processing chain. The internal processing chain can, for example, comprise an artificial neural network, KNN. The parameters of the internal processing chain can then, for example, be weights, with which the inputs applied at the neurons for activating these neurons are calculated.

[0006] The KI module to be checked by the method is designed to map an input data set from an input space E with dimension D to a classification and / or regression in an output space F with dimension G < D by means of its internal processing chain. Classification specifies, for example, for an input data set and a predefined class criterion: with which confidence the input data set belongs to each of these classes. Regression specifies, for example, for an input data set and a real-valued variable of interest: which values ​​of these variables with which confidence are consistent according to the input data set.

[0007] In most applications, G << D applies. Thus, for example, an image as large as 512x512 pixels exists in an input space E with dimension D = 262.144. The number of different objects, which images should be checked for the presence of said objects, determines the dimensionality of the output space F and is typically less than 1000.

[0008] In this method, a classification and / or regression is determined as an undisturbed result for a predefined open data set in an input space E, onto which the KI module maps the open data set.

[0009] At least one perturbation S is added to the open data set, the perturbation having a dimension d < D. In this way, at least one perturbed data set is formed in the input space E. For this purpose, it is particularly advantageous to determine an "adversarial example" for the KI module as the perturbation S. In particular for KI modules including KNN, the following algorithms are known, with which "adversarial examples" with a freely selectable dimension d can be generated.

[0010] The classification and / or regression is determined as a disturbed result, onto which the KI module maps the disturbed data set. The deviation of the disturbed result from the undisturbed result is determined using a predefined metric. The metric can be set in particular for a specific application in which the result provided by the KI module, i.e. the classification and / or regression, is to be used. The metric can then be motivated, for example, by how disturbingly the deviation affects the application.

[0011] If the KI module is used, for example, to recognize traffic signs in a scene within the scope of a driver assistance system or a system for fully or partially automated driving, then, for example, a confusion of a speed 50 sign with a speed 60 sign only results in a small disruption to the traffic. If, however, a speed 50 sign is mistakenly regarded as a stop sign, the consequence may be a sudden braking of the vehicle and a rear-end collision. If, on the other hand, a stop sign is regarded as a speed 50 sign, the potential consequences are more serious, since the vehicle may be hit laterally and essentially without braking by a vehicle having the right of way. Thus, in this application, the metric can, for example, evaluate the deviation according to the severity of the possible impact in the traffic.

[0012] If the KI module is used, for example, in an access control system to determine on the basis of measurement data whether a valid access medium (Zugangsmedium) has been presented, then the confusion of an invalid access medium with another access medium ultimately has no effect. Access is not permitted as before. If two valid access media are confused with one another, this can, for example, have the following effects: access (Zutritt) is permitted with the wrong authorization level or the wrong person is recorded in the log file about the access. If a valid access medium is mistakenly regarded as an invalid access medium, authorized persons do not gain access, which can disrupt the workflow. The worst effect is that an invalid access medium is regarded as a valid access medium and access is granted to completely unauthorized persons.

[0013] The influence of the disturbance S on the result is determined by the interaction of the sensitivity of the KI module to the disturbance S and the strength of the disturbance S. In order to measure the sensitivity of the KI module in a targeted manner, the metric can, for example, indicate the influence per unit strength of the disturbance S or be normalized to the unit strength S of the disturbance.

[0014] In response to the deviation satisfying a predetermined criterion, it is determined that the KI module is sensitive to fraud attempts with dimension d with respect to the open data set. This criterion may, for example, include exceeding or falling below a predetermined threshold value in the case of a scalar metric.

[0015] It has been recognized that those disturbances S which have a significant influence on the results of the classification and / or regression in the context of the respective application are present in a subspace C ⊂ E with a dimension c < D. If a disturbance S with a dimension smaller than c is now added to the input data set, the probability that this disturbance S is in C is low. If, for example, c=20 and only a one-dimensional disturbance is added, it is impossible for the one-dimensional straight line corresponding to the disturbance to intersect the 20-dimensional subspace C anywhere in the input space E. If, on the other hand, the disturbance S has a dimension of c or greater, the probability that this disturbance is in C increases significantly. This applies in particular when “adversarial examples” are selected as disturbances S in a targeted manner.

[0016] The perturbation S must therefore have a certain minimum dimension in order to have a noticeable effect on the result. It is recognized that this minimum dimension is a reliable measure of how robust the KI module is against deception attempts. This should be explained using the example of an image as an input data set. As explained previously, the dimension D of the input space E corresponds to the number of image pixels. The minimum dimension in this example corresponds to the number of pixels affected by the perturbation S, from which the perturbation S has a noticeable effect. For the most sensitive negative examples of the KI module, the subspace C ⊂ E is then one-dimensional. This means that a change in the image in just one pixel is sufficient to significantly change the classification and / or regression. If, on the other hand, the subspace C ⊂ E has almost the dimension D, almost all the pixels of the image must be changed in order to deceive the KI model.

[0017] The greater the minimum dimension of the disturbance S must be, the more difficult it is to add this disturbance to the input data set without being noticed. In the example of the stop sign mentioned at the beginning, for example, in the case of a sensitive KI module, an inconspicuous sticker on the sign may already be sufficient to misclassify the sign. In order to deceive a more robust KI module, however, the sign must be altered so drastically that attempts at manipulation are visually obvious and the elimination of this modification is a concern. For example, the classification result provided by each KI module, which is also so robust, can be altered with absolute safety by simply removing the stop sign and replacing it with another sign.

[0018] This means that the smallest dimension of a disturbance S which has a significant effect on the respective application according to a predefined metric is a particularly suitable measure of the sensitivity of the KI module to disturbances which can penetrate unnoticed into the input data set.

[0019] Therefore, in a particularly advantageous design, a plurality of interferences S with different dimensions d are added to the same open data set. For example, in this case, it is possible to start with a low dimension d of the interference S, for example d=1, and then gradually increase the dimension d of the interference S, for example by 1 each time. The minimum dimension c=min(d) is evaluated as a measure of the sensitivity of the KI module with respect to the open data set, for which the sensitivity of the KI module to spoofing attempts is determined according to the above-mentioned metric. For this purpose, for example, the variable s=Dc can be selected as the "intrinsic dimension" of the interference that has a strong influence. The smaller c is, that is, the greater the sensitivity is, the larger s becomes.

[0020] Furthermore, it is recognized that the sensitivity measured by c or s is not evenly distributed in the input space E in which the input data set is present. In particular, there can be such an input data set for which the KI module has a particularly high sensitivity to spoofing attempts (so-called "corner cases"). This situation is to some extent similar to a hiking trail in the mountains, which is mostly accessible even to inexperienced people, but also has some exposed places where a misstep directly leads to a fall. The method enables the identification and quantification of such "corner cases". Countermeasures can then be taken to reduce the sensitivity. For example, the KI model can be trained with additional learning data in the vicinity of the "corner cases".

[0021] In another advantageous design, the sensitivity of the KI module, measured, for example, in the form of c or s, is determined for a plurality of data sets from a pre-given set M. A summary statistic is determined from the sensitivity thus determined according to the data sets. This summary statistic can, for example, include the mean value, and / or the variance, and / or the frequency distribution, and / or the worst value of the determined sensitivity.

[0022] Thus, for example, it may be sufficient for many applications that the sensitivity remains on average within a pre-given range, while in other applications a "worst-case" estimate of the sensitivity is required, in which property or personal injury is at stake in the event of a failure.

[0023] The newly obtained quantifiability of the sensitivity of the KI module to spoofing attempts can in particular be used as feedback in order to reinforce the KI module relative to spoofing attempts.

[0024] The invention thus also relates to a method for establishing a KI module which comprises a trainable artificial intelligence in the form of a parameterized internal processing chain and is configured to map an input data set from an input space E of dimension D to a classification and / or regression in an output space F of dimension G < D by means of this internal processing chain.

[0025] In this method, the architecture of the internal processing chain is specified by so-called hyperparameters. These hyperparameters relate to the degrees of freedom existing in the design of the architecture. For example, the architecture of a KNN includes not only the order of the layers of a certain type (e.g. "10 x alternating convolutional layers and max-pooling layers, then fully connected layer") but also the specific dimensions of these layers (e.g. "512 x 512 neurons in the input layer").

[0026] A KI module is formed with an internal processing chain having an architecture according to the specified hyperparameters. The architecture then also has free parameters for training, such as weights in the case of KNN, with which the input supplied to each neuron for activating the neuron is calculated. When training the KI module, these free parameters are optimized with the aid of a set L of learning data sets and the associated learning results, so that the KI module maps the learning data sets to the learning results with a predetermined accuracy according to an error function. For example, the learning data set can include images of road scenes, and the learning results respectively associated with each image indicate which objects the KI module should recognize in the image.

[0027] After the training is completed, summary statistics of the sensitivity of the trained KI module are determined according to the previously described method using the set of validation data sets M. The set of validation data sets M should advantageously be disjoint from the set of learning data sets L, since the learning data sets are excellent insofar as the KI module is specifically optimized precisely by correct processing of these data sets. However, what is important for this application is the precise correct functioning of the KI module under unknown conditions.

[0028] The hyperparameters, ie the architecture of the KI module, are now optimized as follows: After reformulating and training the KI module, summary statistics of the sensitivities determined for this purpose indicate an overall lower sensitivity to spoofing attempts.

[0029] It has been recognized that the sensitivity of a KI module to fraud attempts is determined not only by the corresponding training of the KI module, but also by the underlying architecture. For the training, there is feedback in the form of an error function, which assesses how well a pre-given learning result is reproduced. By means of summary statistics of the correlations, feedback about the architecture of the KI module is now also available.

[0030] In a particularly advantageous embodiment, the hyperparameters are optimized in that the architecture of the internal processing chain is generated using an evolutionary algorithm. Summary statistics of the sensitivities determined after forming and training the KI module with the architecture are respectively included in the quality metric for evaluating the architecture.

[0031] Within the scope of evolutionary algorithms, the architecture of the internal processing chain can be modified, for example, using natural simulation methods of crossover and mutation, so that always those architectures with better quality measures "survive" ("Survival of the Fittest"). Evolutionary algorithms are particularly suitable for optimizing architectures, because they do not presuppose that the hyperparameters to be optimized are continuous. Thus, for example, the type of layer in KNN (e.g. convolutional layer, pooling layer or fully connected layer) is a discrete variable that is not easily understandable for the gradient descent methods frequently used in optimization tasks.

[0032] In a further particularly advantageous embodiment of the method described above, at least one open data set and / or at least one learning data set contains at least one measured value of physical measurement variables. In this way, a system for analyzing these measurement variables can be protected against a targeted, slight tampering with the measurement variables that would unnoticeably divert the analysis in the wrong direction. The measured values can in particular be determined using a sensor that records a physical effect, the manner and / or intensity of which is characterized by the physical measurement variables.

[0033] Determining the sensitivity of the KI module to spoofing attempts is generally not an end in itself, but is application-specific. It has been recognized that the above-mentioned sensitivity can be directly used as a measure of the reliability of the results determined using the KI module within the scope of such an application. Depending on the application, a trustworthy statement about the reliability can be just as important as the result itself.

[0034] The invention therefore also relates to a method for monitoring a detection area. In this method, at least one measurement data set with measurement data is detected by physically observing the detection area using at least one sensor.

[0035] The measurement data set is fed to a KI module, which comprises a trainable artificial intelligence in the form of a parameterized internal processing chain and is configured to map an input data set from an input space E with dimension D to a classification and / or regression in an output space F with dimension G < D using this internal processing chain. The input space E is here the space in which the possible measurement data can vary. The output space F and its dimension G are determined by an interrogation monitored by the monitoring, for example the type and number of objects to be classified in the measurement data.

[0036] The classification and / or regression output by the KI module is evaluated and / or output as the result of the monitoring. Additionally, the sensitivity of the KI module to spoofing attempts is measured using the method described and evaluated and / or output as a measure of the reliability of the result.

[0037] The sensitivity of the KI module to spoofing attempts determined in an automated manner in turn opens up the possibility of automatically taking countermeasures against such spoofing attempts as long as the determined sensitivity meets a predefined criterion. The criterion can, for example, in the case of a scalar sensitivity, include exceeding or falling below a predefined threshold.

[0038] As an exemplary possible countermeasure, the measurement data set may be fed to another KI module, wherein the other KI module has a different architecture than the previously utilized KI module, and / or wherein the other KI module has been trained differently than the previously utilized KI module. The more extensively the same deception attempt (e.g., "adversarial examples") matches differently trained KI modules or even differently structured KI modules, the more difficult it becomes to superimpose tampering on the measurement data so that the tampering is not otherwise detected.

[0039] As another advantageous possible countermeasure, another physical sensor can be used to detect additional measurement data. The additional measurement data can be used to check the rationality of the result. For example, another physical sensor can be activated for this purpose. The deception attempt (e.g., "adversarial sample") exists in the input space E of the KI module, but the additional measurement data exists outside this space E and is therefore not covered by the deception attempt.

[0040] As another exemplary possible countermeasure, the result may be discarded. Depending on the application, this may be a smaller disadvantage than further processing of the possibly tampered result.

[0041] As described above, in a particularly advantageous embodiment, the detection area includes at least a part of the surroundings of the vehicle. The results provided by the KI module are fed to a driver assistance system or a system for at least partially automated driving installed in the vehicle. The driver assistance system or the system for at least partially automated driving is designed to control the steering system, the transmission system and / or the braking system of the vehicle based on the results. On the one hand, this application is particularly safety-critical, and on the other hand, attempts to deceive can be easily implemented, because, for example, traffic signs are publicly available.

[0042] In the case of applications in vehicles, different specific countermeasures are meaningful individually or in combination. Thus, for example, a physical warning device that is perceptible to the driver of the vehicle can be activated. The driver of the vehicle can be asked to confirm or correct the result provided by the KI module and / or to assume control of the vehicle. In the highest upgrade level, the vehicle can be stopped on an emergency trajectory provided for a failure of an at least partially automated driving function.

[0043] As already mentioned above, access control systems are another important application. Therefore, in another particularly advantageous embodiment, the detection area includes at least a part of the following area, in which the access control system for controlling access to rooms, areas and / or information technology systems expects the presentation of an access medium. The result provided by the KI module is supplied to the access control system. The access control system is designed to determine based on the result whether a valid access medium is present in the detection area. Based on this determination, locking and / or alarm devices can be manipulated in particular to allow or prevent access. The described countermeasures can make it difficult to impersonate a valid access medium by deliberately tampering with the measurement data.

[0044] In addition, other specific countermeasures are useful, either individually or in combination, especially for access control. Thus, for example, an additional authentication, such as a PIN or a password, of the person requesting access can be required. Access can also be blocked for a predetermined time, for example, independently of the presence of a valid access medium, in order to discourage repeated fraudulent attempts. For example, an alarm can also be output to an organization responsible for the security of a room, an area or an information technology system.

[0045] In one particularly advantageous embodiment, biometric measurement data of the person requesting access are selected as measurement data. Such access media are usually difficult or completely impossible to copy, so that they represent an important alternative for an attacker to defrauding the KI module.

[0046] As described previously, the method mentioned can use additional hardware, such as additional sensors. However, this is not necessarily necessary. The method can also be implemented completely or partially in software, which brings direct customer benefits (Kundennutzen), making it possible to identify, reduce the sensitivity of the KI module to fraud attempts or weaken its results for the corresponding application. Therefore, the present invention also relates to a computer program, which contains machine-readable instructions, which when executed on a computer and / or on a control device and / or on an embedded system, prompt the computer and / or the control device and / or the embedded system to perform one of the described methods. Similarly, the present invention also relates to a machine-readable data carrier or a download product, which has a computer program. BRIEF DESCRIPTION OF THE DRAWINGS

[0047] Further measures for improving the invention are shown in more detail below together with the description of preferred exemplary embodiments of the invention with the aid of the figures.

[0048] in:

[0049] Figure 1 An embodiment of a method 100 for measuring sensitivity is shown;

[0050] Figure 2 The dimension c of the disturbance S is shown 1a 、c 1b 、c 1c , the disturbance causes a significant deviation 15 in the KI modules 1a, 1b, 1c;

[0051] Figure 3 An embodiment of a method 200 for establishing a KI module 1 is shown;

[0052] Figure 4 An embodiment of a method 300 for monitoring a detection area 2 is shown;

[0053] Figure 5 A schematic diagram showing an exemplary application of the method 300 in a vehicle 50;

[0054] Figure 6 A schematic diagram shows an exemplary application of the method 300 in the access control system 60 . DETAILED DESCRIPTION

[0055] Figure 1 An exemplary embodiment of a method 100 is shown. A predetermined open data set 11 present in an input space E with dimension E is mapped in step 110 by the KI module 1 and its internal processing chain 12 into an output space F with dimension G for classification 13 and / or regression 14 .

[0056] Additionally, in step 120, a perturbation S with dimension d<D is added to the same open dataset 11. This can be a "adversarial example" for the used KI module 1 according to block 121. According to block 122, different perturbations S with different dimensions d can also be used. The perturbed dataset 11' is mapped in step 130 onto a perturbed classification 13' or a perturbed regression 14'.

[0057] In step 140, the undisturbed results 13, 14 are compared with the disturbed results 13', 14' by means of a metric 16. In step 150, it is checked whether the determined deviation 15 meets a predetermined criterion, i.e., whether the deviation exceeds a predetermined threshold value, for example. If this is the case (true value 1), then in step 160 it is determined that the KI module 1 is sensitive to fraud attempts with a dimension d with respect to the open data set 11. In step 170, the minimum dimension c=mind(d) is evaluated as a measure 17 of the sensitivity of the KI module 1 with respect to the open data set 11, for which this is the case.

[0058] Optionally, according to block 115 , the sensitivity of the KI model 1 can be determined for a plurality of data sets 11 from a predetermined set M. According to block 175 , summary statistics 18 can be generated using the associated measures 17 , c of the sensitivity of the KI model 1 respectively determined in step 170 .

[0059] exist Figure 2 In FIG. 1 , for three exemplary KI modules 1a-1c, deviations 15 are plotted, which are produced when disturbances S of different dimensions are added to the same open data set 11 and the disturbed data set 11' is then processed by the KI modules 1a-1c, respectively. A criterion 150, which is implemented as a threshold in this example, is used to check whether the deviation 15 is significant with respect to the current application. Figure 2 The reference symbol c 1a 、c 1b or c 1c represents the minimum dimension d, which is the case for each of the three examined KI modules 1a - 1c. 1a-1c The higher , the more robust the associated KI module 1a-1c is against fraud attempts. Figure 2 In the example shown in , KI module 1b obtains the worst result and KI module 1c obtains the best result. With respect to the open dataset 11 used for this examination, the sensitivity of KI module 1a is between the sensitivities of KI modules 1b and 1c.

[0060] Figure 3 An exemplary embodiment of a method 200 for establishing a KNN module 1 is shown. In step 210, the architecture of the internal processing chain 12 of the KNN module 1 is specified by hyperparameters 12a. As described above, these hyperparameters 12a do not have to be just numerical values. Rather, the hyperparameters 12a can also include, for example, the possibility of selecting whether a layer of the KNN is a convolutional layer, a fully connected layer, or a pooling layer.

[0061] In step 220, a KI module 1 is formed with an internal processing chain 12, which has an architecture determined by the hyperparameters 12a. The architecture is an anatomy with still free parameters 12b, such as weights in KNN. These parameters 12b are trained in step 230 using a learning data set 11a from a set L and the associated learning results 13a, 14a. That is, the parameters 12b are changed until the learning data set 11a is mapped to the learning results 13a, 14a with a predetermined accuracy according to the error function.

[0062] In step 240, the sensitivity of the trained KI module 1 to fraud attempts is tested according to the previously described method 100. For this purpose, a validation data set from the set M is used as an open data set. A summary statistic 18 of the corresponding sensitivity of the KI module 1 is created from all validation data sets from the set M.

[0063] According to step 250 , the hyperparameters 12 a are now optimized as follows: after reformulation 220 and subsequent training 230 of KI module 1 , the summary statistics 18 of sensitivity indicate an overall lower sensitivity of KI module 1 to spoofing attempts. That is, the sensitivity of candidate KI modules is sequentially checked using method 100 .

[0064] exist Figure 3 In the example shown in , the hyperparameters 12a are optimized by generating a new architecture of the internal processing chain 12 of the KI module 1 to be formed using an evolutionary algorithm according to block 251. The summary statistics 18 are then respectively entered into a quality measure (fitness function) for the corresponding architecture according to block 252.

[0065] Figure 4 An exemplary embodiment of a method 300 for monitoring a detection area 2 is shown. In step 310, at least one measurement data set 11 is detected by means of a sensor 3, said measurement data set comprising measurement data 3a. In step 320, said measurement data set is supplied to a KI module 1. The KI module 1 generates a classification 13 and / or a regression 14 from the measurement data set 11 using its internal processing chain 12. In step 330, said classification 13 and / or regression 14 is evaluated and / or output as a result 13, 14 of the monitoring.

[0066] In step 340, the sensitivity 17, 18 of the KI module 1 to fraud attempts is measured using the method 100. This sensitivity 17, 18 is evaluated and / or output in step 350 as a measure of the reliability of the result 13, 14.

[0067] exist Figure 4 In addition, different countermeasures are shown, which can be taken individually or in combination in response to the determination in step 360 that the determined sensitivities 17 , 18 meet a predetermined criterion (truth value 1 ).

[0068] Thus, for example, when method 300 is used in vehicle 50, results 13, 14 are supplied to the Figure 4 A driver assistance system 52a not shown or a system 52b for at least partially automated driving,

[0069] • a physical warning device perceptible to the driver of the vehicle may be activated according to block 361,

[0070] • the driver of the vehicle 50 may be asked to confirm or correct the results 13, 14 according to block 362,

[0071] • The driver of the vehicle 50 may be requested to assume control of the vehicle 50 according to block 363, and / or

[0072] • The vehicle 50 can be stopped according to block 364 on an emergency trajectory provided for a failure of the at least partially automated driving function.

[0073] Furthermore, for example, when the method 300 is used in an access control system 60 for controlling access to a room 61 , an area 62 and / or an information technology system 63 , the access control system can check the presence of a valid access medium in the detection area 2 based on the results 13 , 14 .

[0074] • access may be prohibited for a predetermined time according to block 365 independently of the presence of valid access medium 65,

[0075] • Additional authentication of the person requesting access may be required according to block 366, and / or

[0076] • According to block 367 , an alarm can be output to the authority responsible for the security of the room 61 , the area 62 or the information technology system 63 .

[0077] In general, for example,

[0078] • according to block 370 , the measurement data set 11 is fed to another KI module 1 ′, which is trained differently and / or has another architecture, and / or

[0079] • according to block 380, using a further sensor 3' for obtaining additional measurement data 3a' in order to check the plausibility of the results 13, 14 according to block 385, and / or

[0080] • According to block 390, results 13, 14 are discarded.

[0081] If, however, the sensitivity is not particularly high (true value 0 in the test in step 360), the results 13, 14 can be used, for example, in the driver assistance system 52a or in the system 52b for at least partially automated driving according to block 395. However, the results 13, 14 can also be used, for example, in the access control system 60 according to block 396.

[0082] exist Figure 53 is briefly outlined again in the application of the method 300 in a vehicle 50. Here, the detection area 2 is part of the surroundings 51 of the vehicle 50. A driver assistance system 52a and / or a system 52b for at least partially automated driving intervenes in the driving dynamics of the vehicle based on the results 13, 14 derived from the measurement data set 11 using the method 300 by actuating a steering system 53, a transmission system 54 and / or a braking system 55 of the vehicle 50.

[0083] exist Figure 6 The use of the method 300 in an access control system 60 is briefly outlined again in FIG. The access control system 60 checks whether a valid access medium 65 for accessing a room 61, an area 62 or an IT system 63 is present in the detection area 2. Figure 6 In the example shown in , the access medium 65 is a hand with a certain biometric feature. Depending on the results 13 , 14 derived from the measurement data set 11 using the method 300 , the access control system 60 actuates the locking and / or alarm device 64 in order to allow or prevent access.

Claims

1. A method (100) for measuring the sensitivity of a KI module (1) to spoofing attempts, wherein the KI module (1) comprises a trainable artificial intelligence in the form of a parameterized internal processing chain (12) and is configured to map an input data set (11) from an input space E with dimension D to a classification (13) and / or regression (14) in an output space F with dimension G < D by means of the internal processing chain (12), where, The input data set (11) is an image data set and has the following steps: · For a pre-given open data set (11) in the input space E, the classification (13) and / or regression (14) is determined (110) as an undisturbed result (13, 14), and the KI module (1) maps the open data set (11) onto the classification (13) and / or regression (14); · At least one interference S with dimension d < D is added (120) to the open data set (11) such that at least one disturbed data set (11') is formed in the input space E; · The following classification (13') and / or regression (14') is determined (130) as a disturbed result (13', 14'), and the KI module (1) maps the disturbed data set (11') onto the classification (13') and / or regression (14'); · The deviation of the disturbed result (13', 14') from the undisturbed result (13, 14) is determined (140) using a pre-given metric (16); · In response to the deviation (15) satisfying (150) a pre-given criterion, it is determined (160) that the KI module (1) is sensitive to spoofing attempts with dimension d with respect to the open data set (11).

2. The method (100) according to claim 1, wherein an adversarial sample for the KI module (1) with dimension d is determined as the interference S, (121).

3. The method (100) according to claim 1 or 2, wherein a plurality of interferences S with different dimensions d are added to the same open data set (11), (122), and wherein the minimum dimension c = min(d) is evaluated (170) as a measure of the sensitivity of the KI module (1) with respect to the open data set (11), and the sensitivity of the KI module (1) to spoofing attempts is determined (160) for the minimum dimension.

4. The method (100) according to claim 1 or 2, wherein the sensitivity of the KI module (1) is determined with respect to a plurality of data sets (11) from a pre-given set M, (115), and wherein a summary statistic (18) is determined (175) based on the sensitivities thus determined for the data sets.

5. The method (100) according to claim 4, wherein the summary statistic (18) includes an average value, and / or a variance, and / or a frequency distribution, and / or the worst value of the determined sensitivities (17).

6. The method (100) according to claim 1 or 2, wherein at least one open data set (11) and / or at least one learning data set (11a) includes at least one measurement value of a physical measurement variable.

7. A method (200) for establishing a KI module (1), the KI module (1) comprising a trainable artificial intelligence in the form of a parameterized internal processing chain (12) and being configured to map an input data set (11) from an input space E having a dimension D to a classification (13) and / or regression (14) in an output space F having a dimension G < D by means of the internal processing chain (12), wherein, The input data set (11) is an image data set and has the following steps: · The architecture of the internal processing chain (12) is specified (210) by hyperparameters (12a), · A KI module (1) with the internal processing chain (12) is formed, · The KI module (1) is trained (230) as follows: The parameters (12b) of the internal processing chain are optimized with the aid of a set L of learning data sets (11a) and the associated learning results (13a, 14a) such that the KI module (1) maps the learning data sets (11a) to the learning results (13a, 14a) with a pre-given accuracy according to an error function; · A summary statistic (18) of the sensitivity of the trained KI module (1) is determined (240) using a set M of validation data sets (11) by the method according to any one of claims 4 to 5; · The hyperparameters (12a) are optimized (250) as follows: After reforming (220) and training (230) the KI module (1), the summary statistic (18) of the sensitivity determined therefor indicates an overall lower sensitivity to spoofing attempts.

8. The method (200) according to claim 7, wherein the hyperparameters (12a) are optimized (250) by: The architecture of the internal processing chain (12) is generated (251) using an evolutionary algorithm, wherein the summary statistic (18) of the sensitivity determined after each formation and training of the KI module (1) with the architecture is fed (252) into a quality metric for evaluating the architecture.

9. The method (100) according to claim 7 or 8, wherein at least one open data set (11), and / or at least one learning data set (11a) contains at least one measured value of a physical measurement variable.

10. A method (300) for monitoring a detection area (2), having the following steps: · Detecting (310) at least one measurement data set (11) with measurement data (3a) by physically observing the detection area (2) using at least one sensor (3); · Feeding (320) the measurement data set (11) to a KI module (1), which comprises a trainable artificial intelligence in the form of a parameterized internal processing chain (12) and is configured to map an input data set from an input space E with dimension D to a classification (13) and / or regression (14) in an output space F with dimension G < D with the aid of the internal processing chain (12); · Evaluating and / or outputting (330) the classification (13) and / or regression (14) output by the KI module (1) as the result (13, 14) of the monitoring; · The sensitivity (17, 18) of the KI module (1) to spoofing attempts is measured (340) using the method (100) according to any one of claims 1 to 5; · The determined sensitivity (17, 18) is evaluated and / or output (350) as a measure of the reliability of the result (13, 14).

11. The method (300) according to claim 10, wherein in response to the determined sensitivity (17, 18) satisfying a pre-given criterion, (360) feeding the measurement data set (11) to a further KI module (1'), (370), wherein the further KI module (1') has a different architecture than the previously used KI module (1) and / or wherein the further KI module (1') has been trained differently than the previously used KI module (1); and / or Additional measurement data (3a') are detected by means of a further physical sensor (3'), (380), and said additional measurement data (3a') are used, (385) in order to check the plausibility of said result (13, 14); and / or Discard the results (13, 14), (390).

12. A method (300) according to claim 10 or 11, wherein the detection area (2) includes at least a part of the surroundings (51) of the vehicle (50) and wherein the results (13, 14) provided by the KI module (1) are transmitted to a driver assistance system (52a) installed in the vehicle (50) or a system (52b) for at least partially automated driving, (395), wherein the driver assistance system (52a) or the system (52b) for at least partially automated driving is designed to control a steering system (53), a transmission system (54) and / or a braking system (55) of the vehicle (50) based on the results (13, 14).

13. The method (300) of claim 12, wherein in response to the determined sensitivity (17, 18) satisfying a predetermined criterion, (360), activating a physical warning device perceptible to a driver of the vehicle (50), (361) · requesting the driver of the vehicle (50) to confirm or correct the result (13, 14), (362) · requesting the driver of the vehicle (50) to assume control of the vehicle (50), (363), and / or Stopping the vehicle (50) on an emergency trajectory provided for a failure of the at least partially automated driving function, (364).

14. A method (300) according to claim 10 or 11, wherein the detection area (2) includes at least a part of the following area, in which an access control system (60) for controlling access to rooms (61), areas (62) and / or information technology systems (63) expects the presentation of an access medium (65), wherein the result (13, 14) provided by the KI module (1) is supplied to the access control system (60), (396), and wherein the access control system (60) is configured to determine based on the result (13, 14): whether a valid access medium (65) is present in the detection area (2).

15. A method (300) according to claim 14, wherein, in response to the determined sensitivity (17, 18) satisfying a predetermined criterion, (360), access is prohibited for a predetermined time, independent of the presence of a valid access medium (65), (365), and / or additional authentication of the person requesting access is required, (366), and / or an alarm is output to an agency responsible for the security of the room (61), the area (62) or the information technology system (63), (367).

16. The method (300) according to claim 14, wherein biometric measurement data of the person requesting access are selected as measurement data (3a, 3a').

17. A computer program product, comprising a computer program, wherein the computer program contains machine-readable instructions, which, when executed on a computer and / or on a control device and / or on an embedded system, cause the computer and / or the control device and / or the embedded system to perform a method (100, 200, 300) as described in any one of claims 1 to 16.

Images