Computer-implemented ID verification module and training procedure

Abstract

The invention relates to a computer-implemented ID verification module (1) and an associated training method. The ID verification module (1) comprises a feature extractor (2) and an evaluation module (3), wherein the feature extractor (2) is connectable to or connected to layers (7) of an object recognition network (4) in order to extract features from at least one layer (4) and pass them on to the evaluation module (3). The evaluation module (3) is a trained AI module and evaluates whether the extracted features indicate that the current input data (5) for object recognition, which is processed by the object recognition network (4), is covered by a distribution of the training data with which the object recognition network (4) has been trained with positive examples for object recognition. The ID verification module (1) determines and outputs (7) that an ID criterion is met if the current input data (5) is within the distribution of this training data.According to the invention, the ID verification module (1) is configured to perform an ID check for a region of the input data (5) that corresponds to a region (E1, E2) of the output data (6) of the object recognition network (4) in which the object recognition network (4) has not detected a foreign object (O1, O2). Alternatively or additionally, the object recognition network (4) is trained to also positively detect a background object (B1, B2, B3), and the ID verification module (1) is configured to perform an ID check for a region of the input data (5) that corresponds to a region of the output data (6) of the object recognition network (4) in which the object recognition network (4) has detected such a background object (B1, B2, B3).

Description

[0001] In practice, object recognition networks are commonly used to detect foreign objects. Input data is acquired via sensors and, in particular, obtained as instantaneous input from an environmental sensor. These networks typically comprise at least one neural network, especially a deep neural network, and more specifically a convolutional neural network, to perform object detection. Furthermore, object recognition networks may include additional components. The boundaries between these components can be fluid, depending on the application.

[0002] However, it is also known that neural networks, especially deep neural networks (DNNs), exhibit particularly good object recognition accuracy precisely when they are presented with input data that closely matches the training data on which the neural network was based. In practice, the phrase "currently processed input data covered by the training data distribution" has become established. The use of object recognition networks is particularly well-established in vehicles and serves to provide identified virtual foreign objects for subsequent processing in the control system, especially in driver assistance systems or autonomous vehicle control.

[0003] Object detection networks typically comprise a backbone network with multiple layers. The backbone network is fed with input data and performs various data processing operations at the interfaces between layers. These layers contain different features derived from the input data. Several of these features can be represented in a feature map, which allows for feature localization. Features can be extracted from the layers, and these extracted features can be used to characterize how the object detection network responds to the current input data.

[0004] It is also known that a trained object recognition network can react with an erroneous detection to input data that deviates significantly from the training data. This phenomenon is called overconfidence; that is, after the object recognition network outputs, a virtual foreign object is assumed to be present in the environment with a high confidence value, even though in reality no physical foreign object (or possibly a different physical foreign object) is present at the corresponding location in the environment.

[0005] It has been shown that such false-positive detection of foreign objects by an object recognition network can be uncovered by extracting and evaluating the features corresponding to the supposedly detected virtual foreign object in the layers of the object recognition network using an AI-based and trained ID verification module. The abbreviation "ID" stands for "in distribution" and refers to the check whether the input data currently being processed in the object recognition network is covered by the training data distribution with which this object recognition network was trained using positive examples of the foreign object.

[0006] An ID verification module serves to determine, based on the currently available features in the layers of the object recognition network, whether the input data processed by the object recognition network, which is supposed to correspond to a supposedly recognized virtual foreign object, sufficiently matches the training data used for learning. If such a match exists, it is determined that an ID criterion is met, i.e., that the currently processed input data falls within the distribution of the training data.

[0007] To perform this check, the ID verification module retrieves information about a virtual foreign object supposedly identified by the object recognition network, extracts features assignable to this virtual foreign object, and uses these features to check whether the ID criterion is met. If the ID criterion is not met, the virtual foreign object is either not considered for subsequent processing or is treated with a reduced trust rating. Subsequent processing can include, in particular, the activation of control systems, such as driver assistance systems in vehicles.

[0008] The object of the present invention is to demonstrate an improved ID verification technique in order to improve the reliability of object recognition using an object recognition network and / or to achieve a better result in object recognition. The invention achieves this object through the characterizing features of the independent claims.

[0009] The present invention extends ID verification, enabling additional statements about the recognition results of the object recognition network, which contribute to the uniqueness and / or reliability of the object recognition. Based on this improved uniqueness and reliability, the risk of incorrect decisions can be reduced and / or a more robust positive decision can be made in subsequent processing.

[0010] The ID verification module according to the present invention can, on the one hand, be configured and trained to perform an ID check (also) for a region of the input data that corresponds to a region of the output data of the object recognition network in which the object recognition network has not detected a foreign object. Alternatively or additionally, the object recognition network can be configured and trained to also positively detect a background object, and furthermore, the ID verification module can be configured and trained to perform an ID check for a region of the input data that corresponds to a region of the output data of the object recognition network in which the object recognition network has detected such a background object.

[0011] The ID testing module can have any type of training. One particularly promising variant involves the ID testing module being formed by, or incorporating, a multi-layer perceptron. The multi-layer perceptron itself is a software module based on AI methods and can therefore be trained separately.

[0012] A training method according to the present invention is provided for training an ID verification module and comprises the following steps. An object detection network with a neural network and a plurality of layers is provided, in which features are determined during object detection. Furthermore, an ID verification module is provided, and its feature extractor is connected to at least one of the layers (7) of the object detection network. The ID verification module is trained using an ID / OOD training dataset, which comprises, on the one hand, ID example datasets and, on the other hand, OOD example datasets. An ID example dataset comprises positive example input data for the object detection network with actually existing foreign objects and a positive basic truth about the foreign objects to be detected.Thus, the ID sample dataset can match a typical dataset used for training an object recognition network. An OOD sample dataset includes additional input data for the object recognition network in which at least one foreign object is actually absent, as well as a negative basic truth about the absence of this foreign object.

[0013] The negative fundamental truth can be expressed explicitly or implicitly. In particular, a definitive list of all foreign objects actually present in a sample dataset can, by inference, define that all other foreign objects (which the object recognition network supposedly identifies) not listed are also not actually present.

[0014] A preferred embodiment provides that the ID verification module is trained using an ID / OOD training data set, which includes a positive basic truth with a definition about the presence of at least one background object for an area of ​​the sensorially detected environment, in particular the vehicle environment, in which there are actually no relevant foreign objects, but there is free space and furthermore, in particular, interference influences may be present, especially overexposure phenomena or weather phenomena.

[0015] In this way, a positive definition is created for an area of ​​the sensorily detected environment, which creates a separate learning basis as a positive example in the training data in order to check the existence of a free space that can be driven on by a vehicle and / or to verify the absence of a virtual foreign object.

[0016] This makes it possible to distinguish between two different errors and, if necessary, to consider these errors separately in the ID check: 1. According to the output of the object recognition network, a virtual free space object has been positively detected, but in reality, no free space exists in the assigned area of ​​the input data. 2. According to the output of the object recognition network, no virtual foreign object has been positively detected (which had previously been implicitly assessed as free space), but in fact, there is no free space in the assigned area of ​​the input data.

[0017] The first error is a false-positive error for the (explicit) detection of a free space object and has not been taken into account so far.

[0018] The second error is a false-negative error indicating the presence of foreign objects.

[0019] Furthermore, the ID check significantly improves the accuracy of misclassifications of foreign objects. For example, if the virtual foreign object identified with the highest confidence (e.g., "pedestrian") is not assigned an ID criterion according to the ID check, it can be additionally checked whether the foreign object identified with the next lowest confidence ("e.g., "cyclist") might fulfill an ID criterion.

[0020] If this is the case, a limited ambiguity can be resolved in the direction of the foreign object identified by fulfilling the ID criterion.

[0021] Further advantageous embodiments are specified in the dependent claims.

[0022] The invention is illustrated in the drawings in an exemplary and schematic manner. These show: Fig. 1: An ID verification module with a first use case according to the state of the art; Fig. 2: An ID verification module according to the present invention; Fig. 3: A schematic diagram to explain the training of an ID verification module and, if applicable, an object recognition network; Fig. 4: A vehicle with an object recognition network and an ID verification module.

[0023] Fig. Figure 1 shows an object recognition network 4 with a neural network and several layers 7, in which features 5 are determined based on input data. The output data 6 of the object recognition network 4 can preferably be subdivided into regions, in particular with a reference to a coordinate system. The output data 6 contains one or more (virtual) foreign objects O1, O2, which the object recognition network 4 has supposedly identified based on the input data 5.

[0024] In Fig. Figure 1 shows an ID verification module 1*, which comprises a feature extractor 2* and an evaluation module 3*, both designed according to the state of the art. For a supposedly identified virtual foreign object O1, O2, the ID verification module 1* retrieves assignable features from a level 7 of the object recognition network 4 and checks, based on these extracted features, whether an assignable range of the input data 5 is covered by the distribution of the training data with which the object recognition network 4 was trained using positive examples for the respective foreign object O1, O2.

[0025] Fig. Figure 2 shows an ID verification module 1 with a feature extractor 2 and an evaluation module 3 according to the present invention. The ID verification module 1 is configured and trained to perform an ID check (also) for a region of the input data 5 that corresponds to a region E1, E2 of the output data 6 of the object recognition network 4 in which the object recognition network 4 has not detected a foreign object. Such regions without identification of a virtual foreign object are in Fig. 2 illustrated with circles.

[0026] Alternatively or additionally, the object recognition network 4 can also be trained to positively recognize a background object B1, B2, B3 (as a virtual object). In this case, the ID verification module 1 is preferably designed and trained to perform an ID check for a range of the input data 5 that corresponds to a range of the output data 6 of the object recognition network 4 in which the object recognition network 4 has recognized, i.e., supposedly identified, such a background object B1, B2, B3.

[0027] Fig. Section 3 describes a training method for teaching an ID verification module 1 according to the present invention. The training method comprises providing an object detection network 4 as described above. Furthermore, an ID verification module 1 is provided and its feature extractor 2 is connected to at least one of the levels 7 of the object detection network 4.

[0028] The ID test module 1 is trained using an ID / OOD training dataset LD, which is located on the left side of Fig. Figure 3 illustrates this. It comprises, on the one hand, an ID example data record 13 and, on the other hand, at least one OOD example data record 14. Preferably, several, and in particular a large number, of such ID example data records 13 and OOD example data records 14 are included.

[0029] An ID example dataset 13 comprises positive example input data 13.1 for the object recognition network 4. This positive example input data 13.1 includes actually existing foreign objects, i.e., they can be detected through a corresponding representation in the sensor data. Furthermore, an ID example dataset 13 comprises a positive basic truth 13.2 about the foreign objects to be detected.

[0030] An OOD example dataset 14 includes further input data for the object recognition network 4. In particular, this may be different or modified input data compared to the positive example input data 13.1.

[0031] In the further input data 14.1, at least one (specific or identifiable by inference) foreign object is actually not present, i.e., there is no representation in the sensor data that corresponds to this foreign object. The OOD example dataset 14 also includes a negative basic truth 14.2 about the absence of this foreign object.

[0032] The output data 6 of the object recognition network 4 are represented in a coordinate system that is in a known spatial relation to the input data 5 and / or the example data 13.1, 14.1.

[0033] The input data 5 and / or the positive example data 13.1 can preferably be obtained from at least one environmental sensor device 15 of a vehicle 16, as exemplified in Fig. Figure 4 shows that the environmental sensor device 15 can have any configuration. In particular, it can be a camera 15.2 or a lidar sensor 15.1. Accordingly, the input data 5 and / or the positive example data 13.1 can include representations of physical foreign objects in the environment of the vehicle 16 and, in particular, include image data and / or video data and / or lidar data and / or radar data.

[0034] The further example data 14.1, which is included in an OOD example dataset 14, can be generated in any way and have any content that is conducive to the differentiation of ID criteria and OOD criteria.

[0035] A preferred embodiment provides that the further example data 14.1 comprise example data that are unchanged in themselves (compared to the positive example data 13.1 or equivalent sensorially acquired raw data), but in which a false-positive detection of at least one foreign object by an object recognition network 4 has actually occurred or could occur.

[0036] Alternatively or additionally, the further example data 14.1 may include synthetic example data generated by targeted modification. The targeted modification may be carried out in any suitable manner, in particular by adding input-level disturbances, by data point-based mutation, by data point-based permutation, and / or by adding noise.

[0037] The first type of included example data (unchanged compared to sensor data) allows the ID verification module 1 to be trained using input data for which a false positive has actually been detected or could occur with a certain probability. It is suitable, for example, for handling special situations in the vehicle's environment that are expected with a certain probability. These could be situations whose occurrence is so rare that comprehensive consideration in object recognition through the classification of (new) foreign objects and positive examples would not be worthwhile, or situations that cannot be represented due to insufficient data density. An example of this is the appearance of a horse-drawn carriage in the vehicle's vicinity, which is incorrectly classified as a foreign object, either a "car" or a "truck".

[0038] Synthetically generated sample data is suitable, for example, for taking into account cases that cannot be assigned to any special state.

[0039] A particularly preferred implementation provides that the ID / OOD training dataset LD includes a positive basic truth with a definition regarding the presence of at least one background object B1, B2, B3 for an area of ​​the vehicle environment (represented in the input data / example data) in which no relevant foreign objects are present. This is particularly useful if interfering influences exist in an associated area of ​​the positive example input data 13.1 that could fundamentally negatively affect object detection. Such interfering influences can be, in particular, overexposure phenomena (glare / shadows) and / or weather phenomena (precipitation / road wetness).

[0040] Such a positive basic truth with a definition about the existence of at least one background object can be used both for training the object recognition network s 4 and for the ID verification module 1.

[0041] The ID check module 1 preferably outputs an ID rating 8, which includes an ID rating indicator for supposedly identified virtual foreign objects O1, O2.

[0042] Alternatively or additionally, the ID assessment can include an OOD assessment value for an area E1, B2 supposedly free of foreign objects and / or an ID assessment value for an area with a supposedly identified background object B1, B2, B3.

[0043] The training method according to the present disclosure is a computer-implemented method. Its steps are preferably executed by a data processing device that obtains (reads in / retrieves) an ID-OOD training data set LD and further preferably stores the result of the training as a trained ID check module 1 in a memory or on a data carrier and / or sends or receives it via a communication interface.

[0044] The invention also relates to a software product comprising instructions which, when executed on a data processing device, implement a computer-implemented ID verification module or a training method according to the present invention.

[0045] The invention further relates to a vehicle 16, in particular a motor vehicle, comprising an object recognition network 4 and an ID verification module 1 according to the present invention, and more preferably at least one environmental sensor device 15.

Claims

[1] Computer-implemented ID verification module (1) comprising a feature extractor (2) and an evaluation module (3), wherein the feature extractor (2) is connectable to or connected to layers (7) of an object recognition network (4) to extract features from at least one layer (4) and pass them to the evaluation module (3), wherein the evaluation module (3) is a trained AI module and evaluates whether the extracted features indicate that instantaneous input data for object recognition, processed by the object recognition network (4), is covered by a distribution of the training data with which the object recognition network (4) has been trained with positive examples for object recognition, and wherein the ID verification module (1) determines and outputs (7) that an ID criterion is satisfied if the instantaneous input data (5) is within the distribution of the training data, characterized by , that the ID verification module (1) is trained and configured to perform an ID check for a range of input data (5) that corresponds to a range (E1, E2) of output data (6) of the object recognition network (4) in which the object recognition network (4) has not detected a foreign object (O1, O2), AND / OR that the object monitoring network (4) is trained to also positively detect a background object (B1, B2, B3), and the ID verification module (1) is trained to perform an ID check for a range of input data (5) that corresponds to a range of output data (6) of the object detection network (4) in which the object detection network (4) has detected such a background object (B1, B2, B3). [2] ID verification module according to claim 1, wherein the ID verification module (1) is trained with an ID / OOD training data set (LD) comprising, on the one hand, ID example data sets (13) and, on the other hand, OOD example data sets (14), wherein an ID example data set (13) comprises positive example input data (13.1) for the object recognition network with actually existing objects and a positive basic truth (13.2) about the foreign objects to be recognized, and wherein an OOD example data set comprises further input data (14.1) with counterexamples for the object recognition network in which at least one object is actually not present, as well as a negative basic truth (14.2) about the absence of this object. [3] ID verification module according to one of the preceding claims, wherein the input data (5) and / or the positive example data (13.1) are obtained from at least one environment sensor device (15) of a vehicle (16), wherein the input data (5) and / or the positive example data (13.1) comprise representations of physical foreign objects in the environment of the vehicle (16) and in particular comprise image data and / or video data and / or lidar data and / or radar data. [4] ID verification module according to one of the preceding claims, wherein the ID / OOD training data set (LD) comprises a positive basic truth with a definition about the presence of at least one background object (B1, B2, B3) for an area of ​​the vehicle environment in which no relevant foreign objects but possibly interferences are present, in particular overexposure phenomena or weather phenomena. [5] ID verification module according to one of the preceding claims, wherein the output data (6) of the object recognition network (4) are represented in a coordinate system which is in a known spatial relation to the input data (5) and / or the example data (13.1, 14.1). [6] ID verification module according to one of the preceding claims, wherein the feature extractor (2) is connected to at least one level (7) of the object recognition network (4) which a. a residual fold plane with a stack normalization; AND / OR b. the features are generated in a distance-preserving manner, such that a localization of a feature in the plane (7) is in a computable relationship with a localization in the input data (5) and / or the sample data (13.1, 14.1) and / or a localization in the output data (6). [7] ID verification module according to one of the preceding claims, wherein the further example data (14.1) included in an OOD example data set (14): a. Unaltered sensor-acquired sample data in which a false positive detection of at least one foreign object actually occurred; AND / OR b. Synthetic example data generated by targeted modification, in particular by adding input-level disturbances, by data point-based mutation, by data point-based permutation, and / or by adding noise. [8] Training method for teaching an ID verification module (1) according to claim 1, comprising the following steps: a. Providing an object detection network (4) with a neural network and a plurality of layers (7) in which features are determined during the performance of object detection; b. Providing the ID verification module (1) and connecting the feature extractor (2) to at least one of the levels (7); c. Training the ID verification module (1) using an ID / OOD training dataset (LD) comprising, on the one hand, ID example datasets (13) and, on the other hand, OOD example datasets (14), wherein an ID example dataset (13) comprises positive example input data (13.1) for the object recognition network with actually present foreign objects and a positive basic truth (13.2) about the foreign objects to be recognized, and wherein an OOD example dataset (14) comprises further input data (14.1) for the object recognition network (4) in which at least one foreign object is actually not present, as well as a negative basic truth (14.2) about the absence of this foreign object. [9] Training method according to the preceding claim, further comprising: Training the ID verification module (1) using an ID / OOD training data set (LD) which includes a positive basic truth with a definition about the presence of at least one background object (B1, B2, B3) for an area of ​​a vehicle environment in which no relevant foreign objects are present, and furthermore, in particular, interference influences are present, especially overexposure phenomena or weather phenomena. [10] Software product comprising instructions which, when executed on a data processing device, implement a computer-implemented ID verification module according to claim 1 or a training method according to claim 8. [11] Vehicle, in particular motor vehicle, comprising an object recognition network (4) and an ID verification module according to claim 1.

Images