Function execution methods, methods for evaluating computer vision methods, and evaluation units

Abstract

A method for automatically executing, particularly at least partially autonomous, functions of a vehicle based on spatially resolved raw sensor data for environmental perception, the raw sensor data being generated by at least one sensor device for environmental detection of the vehicle, the method comprising: - receiving spatially resolved raw sensor data generated by at least one sensor device of the vehicle; - processing the sensor data, the sensor data being features of the spatially resolved raw sensor data, by a processing unit determining at least one region of interest (ROI) and at least one class for classifying the ROI; - processing the sensor data based on the determined ROI by an evaluation unit, thereby determining at least one local quality parameter characterizing the sensor data quality for at least one segment of the ROI; - executing the vehicle functions according to the local quality parameter.

Description

Technical Field

[0001] This invention relates to a method for automatically performing, particularly at least partially autonomous, functions of a vehicle, based on spatially resolved raw sensor data for environmental perception, generated by at least one sensor device for vehicle environmental detection. The invention also relates to a method for evaluating computer vision methods for and / or executed by a processing unit of a vehicle, wherein sensor data for environmental perception generated by at least one sensor device for vehicle environmental detection is processed and evaluated. The invention further relates to an evaluation unit for a vehicle. Background Technology

[0002] In recent years, the tremendous progress in deep neural networks (DNNs) has made detection-based vision a primary tool for the perception subsystem in highly automated driving systems.

[0003] Examples of such tasks include semantic segmentation (see, for example, Badrinalayanan et al., A Deep Convolutional Encoder-Decoder Architecture for Image Segmentation, In: arXiv: 1511.00561, 2015; Bilinski et al., Dense Decoder Shortcut Connections for Single-Pass Semantic Segmentation, In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2018; Long et al., Full Convolutional Networks for Semantic Segmentation, In: arXiv: 1411.4038, 2014; Noh et al., Learning Deconvolution Network for Semantic Segmentation, In: arXiv: 1505.04366, 2015; Ronneberger et al., U-Net: Convolutional Networks for Biomedical). ImageSegmentation, In: arXiv: 1505.04597, 2015. Detailed information regarding semantic segmentation and / or the training and architecture of (various) (artificial) neural networks is provided in these references and is incorporated herein by reference.

[0004] Another example of this type of detection task is 2D / 3D object detection (e.g., see Ross Girshick, Fast R-CNN, In: arXiv:1504.08083, 2015; Girshick et al., Rich Feature Hierarchies for Accurate Object Detection and Semantic Segmentation, In: IEEE Conference on Computer Vision and Pattern Recognition, doi 10.1109 / CVPR.2014.81, p.580-587, 2014; Ren et al., Faster R-CNN: Towards Real-Time Object Detection with RegionProposal Networks, In: Advances in Neural Information Processing Systems). 28, pp. 91-99, http: / / papers.nips.cc / paper / 5638-faster-r-cnn-towards-real-time-object-detection-with-region-proposal-networks.pdf, 2015; Redmon et al., You OnlyLook Once: Unified, Real-Time Object Detection, In: arXiv:1506.02640, 2015; He et al., SVGA-Net: Sparse Voxel-Graph Attention Network for 3D Object Detection from Point Clouds, in: arXiv: 2006.04043, 2020; Shi et al., PV-RCNN: Point-Voxel Feature SetAbstraction for 3D Object Detection, In arXiv: 1912.13192, 2019). For detailed information on 2D / 3D object detection and / or the training and architecture of (various) (artificial) neural networks, these references are incorporated herein by reference.

[0005] Another example of this detection task is image classification (see, for example, He et al., Deep Residual Learning for Image Recognition, In: arXiv: 1512.03385, 2015; Huang et al., Multi-Scale Dense Networks for Resource Efficient Image Classification, In: arXiv: 1703.09844, 2017; Krizhevsky et al., ImageNet Classification with Deep Convolutional Neural Networks, In: Advances in Neural Information Processing Systems, vl.25, https: / / proceedings.neurips.cc / paper / 2012 / file / c399862d3b9d6b76c8436e924a68c45b-Paper.pdf, 2012; Simonyan et al., Very Deep Convolutional Networks for Large-Scale Image). Recognition, In: arXiv:1409.1556, 2014; Sun et al., Automatically designing CNN architectures using genetic algorithm for image classification, In: arXiv:1808.03818, 2018; Szegedy et al., Going Deep with Convolutions, In: arXiv:1409.4842, 2014). For detailed information on image classification and / or the training and architecture of (various) (artificial) neural networks, these references are incorporated herein by reference.

[0006] For object detection and semantic segmentation, the network is trained on annotated training data. Depending on the requirements of the evaluation method, the evaluation of such networks depends on both the annotations (e.g., the so-called "mean intersection-union ratio (mIoU)") and the unannotated data (e.g., temporal consistency). However, state-of-the-art evaluation metrics do not consider the difficulty of the input data during the evaluation process. Therefore, such evaluation methods do not distinguish between hard, medium, or easy data points.

[0007] This problem involves multi-label classification, where the difficulty factors are known from existing class-level methods. Four different criteria are considered to measure the difficulty of an image. However, since this method focuses on image classification, single-object analysis is not applied.

[0008] Furthermore, according to existing technology, an advanced image dataset classification difficulty estimation method is known for predicting the accuracy of deep learning. Here, various automatic methods are applied to cluster the dataset into different difficulty levels. However, this method cannot be applied to object detection or semantic segmentation because objects vary in shape and location, making it impossible to compare different objects based on the proposed method.

[0009] Furthermore, the determination of the difficulty of visual search in images is known from existing technologies. This method uses human interpretation of difficulty as the classification difficulty, which is not sufficiently reliable compared to specific mathematical methods.

[0010] The proposed techniques all take into account image classification, rather than 2D object detection or semantic segmentation.

[0011] To date, state-of-the-art methods have only been used for image classification, which is insufficient for highly automated driving systems. In highly automated driving systems, real-time detection of individual objects in various driving scenarios is essential to ensure the safety of all participants. Therefore, it is necessary to ensure that deep neural networks can detect objects at a reasonable range of difficulty levels.

[0012] Methods and systems for detecting objects in 3D environments are known from US 10,354,406 B2. The detector is trained on a training set according to three different training schemes, each evaluated on a test set at three different difficulty levels. Three different training schemes are proposed: training on easy positive examples; training on medium positive examples (including all "easy" and all "medium" cases); and training on difficult positive examples (including all marked cars). Detection performance is then investigated. The results show that training only at a certain difficulty level does not appear to improve detection performance at that difficulty level.

[0013] US 2020 / 0249671 A1 provides a learning method for providing functional safety by using interpretable AI to warn the driver about potential hazards, the AI ​​verifying the detection process of a neural network for autonomous driving. The neural network for verification can be trained by inputting training images for verification into a neural network used for autonomous driving and an attribute extraction module. The attribute extraction module can generate a quality vector corresponding to the training images for verification, which is then input into the neural network for verification. The neural network for verification generates predicted safety information corresponding to the safety probability of performing autonomous driving by verifying the training images.

[0014] US 8,842,883 B2 describes a global classifier with local adaptation for object detection. The global classifier is proposed to be learned using all training data. The learned global classifier then divides the input space into easy and hard regions. Easy regions contain areas of data that are clearly positive or have positive values, while hard regions represent ambiguous data. To this end, the global classifier outputs a response value that classifies each image patch in a set of image patches into one of three categories: positive, indicating that the object has been considered detected in the image patch; negative, indicating that the object has not been considered detected in the image patch; or ambiguous. For each image patch classified as ambiguous, a classifier is selected based on the features of the image patch, and these features are applied to obtain the response value. Summary of the Invention

[0015] Therefore, the object of the present invention is to provide a solution that ensures that DNN or any other computer vision method used to evaluate sensor data generated by sensor devices of vehicles can detect a reasonable range of difficulty levels, while providing fast and reliable evaluation and / or detection results.

[0016] The aforementioned technical problem is solved by a method, an evaluation unit, a vehicle, and a computer-implemented method for automatically performing functions of a vehicle, particularly at least partially autonomous control, based on raw sensor data with spatial resolution for environmental perception.

[0017] A method for automatically performing, particularly at least partially autonomous, vehicle functions based on spatially resolved raw sensor data for environmental perception, generated by at least one sensor device for vehicle environmental detection, the method comprising receiving (particularly recorded) spatially resolved raw sensor data generated by at least one sensor device of the vehicle.

[0018] Furthermore, the method according to the invention includes processing sensor data via a processing unit, the sensor data being a feature of spatially resolved raw sensor data. The sensor data may be identical to the spatially resolved raw sensor data. The (spatially resolved) raw sensor data may also be preprocessed (via computer-implemented method steps) to become sensor data. Preferably, the sensor data may also be spatially resolved. In particular, the (raw) sensor data is preprocessed in a computer-implemented data preprocessing step to present the data in an input format suitable for the processing unit (particularly to present the data in an input format suitable for a neural network). For example, a camera image may be cropped to remove unwanted image regions, such as the hood of a vehicle visible in the camera image. Furthermore, the preprocessing step may include adjusting the size to a preset input size (e.g., for a neural network), particularly while maintaining a constant aspect ratio, and / or standardizing the sensor data (e.g., subtracting the average measurement value of the entire (raw) sensor data from the raw sensor data) and / or dividing by the standard deviation. Such preprocessing steps allow for faster and more stable data evaluation.

[0019] In particular, spatially resolved raw sensor data is transmitted from at least one sensor device to the processing unit. Preferably, the processing unit receives raw sensor data and / or sensor data that are characteristic of the spatially resolved raw sensor data.

[0020] The processing unit determines at least one region of interest from the (input) sensor data and at least one class for classifying the region of interest (by executing computer-implemented method steps, particularly by executing a computer vision method).

[0021] Furthermore, the method according to the invention includes processing sensor data by means of an evaluation unit based on a determined region of interest (preferably a vehicle), and specifically determining at least one local quality parameter characterizing the quality of the sensor data, relating to at least one segment of the region of interest and preferably to substantially the entire (or all) region of interest (by performing computer-implemented method steps).

[0022] Furthermore, the method according to the invention includes performing vehicle functions based on local mass parameters (particularly by the vehicle's control unit). Preferably, multiple vehicle functions (different from each other) are performed based on the local mass parameters.

[0023] Local quality parameters preferably indicate whether the (raw) sensor data regarding (essentially only) the region of interest is simple or difficult. In other words, local quality parameters preferably indicate whether the evaluation of the sensor data is (considered) difficult or simple (or moderate) in terms of extracting the content (and / or meaning and / or category) of the region of interest. Local quality parameters can be assigned a single value or a quantitative characteristic of values ​​from multiple (continuous or discretely distributed) values.

[0024] Providing local quality parameters is advantageous because it allows for the acquisition of specific information regarding the quality and / or reliability of sensor data, particularly focusing on the region of interest (only).

[0025] For example, consider an image recorded by a camera capturing a traffic environment, where the main part of the environment is in bright sunlight, and the camera is focused on an object illuminated by the sun (e.g., another vehicle). Then, a pedestrian wearing dark clothing and standing in a small shaded area of ​​the environment would be very difficult to detect using existing methods because the sensor data corresponding to the pedestrian would only create a low contrast with the background (shaded area) of the captured image.

[0026] The region of interest can be and / or may correspond to (assigned and / or detected) objects.

[0027] In particular, regions of interest (especially objects) with low contrast (intensity and / or color, etc.) to the background (at least partially) and / or small relative to the entire sensor data (or image) are considered difficult to detect.

[0028] Preferably, small regions of interest (especially objects) and / or multiple regions of interest (especially objects) with relatively low contrast to the background (region of interest) are considered difficult to classify by the processing unit for classification of regions of interest and / or difficult to detect (especially (artificial) deep neural networks) (especially objects).

[0029] The proposed method is advantageous because it is a creative approach for measuring the difficulty of a presented region of interest (especially the presented object), applicable to computer vision tasks and / or detection tasks such as 2D / 3D object detection and semantic segmentation.

[0030] Preferably, the local quality parameter is a quantity associated with the region of interest. Preferably, this means, particularly preferably, that once the region of interest is determined, only sensor data from the region of interest is used and / or processed to determine the local quality parameter. In this case, it is conceivable that only sensor data related to the region of interest will be transmitted and / or received and / or processed by the evaluation unit (method steps implemented by a computer).

[0031] It is also conceivable to use and / or process sensor data outside the region of interest to determine local quality parameters. In this case, it is preferable to use and / or process only sensor data that is relevant to the region of interest (e.g., because they are (spatially) close to and / or around the region of interest).

[0032] In particular, local quality parameters are not global quality parameters of all (raw) sensor data (received by the processing unit).

[0033] Preferably, the local quality parameter serves as a measure of sensor data quality and / or (some kind of) sensor data evaluation criterion and / or the reliability (or reasonableness) of the interpretation of (only) the determined region of interest.

[0034] Vehicle functionality can also be based on local quality parameters and involve (different and / or new) detection and / or (vehicle) computer vision tasks.

[0035] In particular, it is conceivable that the execution of a vehicle function dependent on local quality parameters could result in the vehicle function not being executed, or that the control parameters for the execution of the vehicle function could be modified before the vehicle function is executed. Preferably, the execution of a vehicle function based on local quality parameters would affect the modification, activation, deactivation, cessation, suspension, and / or adjustment of the vehicle function (e.g., by changing parameters concerning the vehicle function, particularly based on (determined) local quality parameters).

[0036] Specifically, vehicle function refers to the control functions of a vehicle, particularly those that are at least partially autonomous and preferably (fully) autonomously controlled. Vehicle function can be a vehicle function that is (particularly) automatically executed, or (at least) part of a vehicle driver assistance system.

[0037] Sensor data can be generated (and / or recorded) by the sensor device while the vehicle is in motion. Preferably, the sensor device monitors the environment surrounding the vehicle, and particularly preferably records sensor data for detecting (movable and / or moving and / or stationary) objects in the environment surrounding the vehicle (e.g., obstacles, pedestrians, vehicles, sidewalks, traffic participants, traffic lights, road markings, etc.).

[0038] Preferably, the sensor data is generated by at least one sensor device of the vehicle, selected from the group consisting of cameras, radar sensors, lidar sensors, thermal sensors, and combinations thereof. These sensor devices are currently advantageously used for environmental perception. Preferably, the sensor device is used for at least one computer vision task / method.

[0039] Preferably, the (spatially resolved) sensor data comprises multiple pixels, and more preferably each pixel contains at least one intensity information.

[0040] For example, sensor data can be (visible light) images captured by a camera. In particular, images consist of a large number (typically thousands) of pixels, which contain color and / or brightness and / or intensity information.

[0041] Preferably, the lidar sensor is used as a sensor device (as a high-precision data source). The raw sensor data of the lidar sensor consists of multiple pixels, which are specifically ordered in the lidar point cloud. The lidar points in the lidar point cloud can be mapped to voxels in a three-dimensional voxel space, and voxel clusters can be identified as objects.

[0042] The region of interest (ROI) in sensor data is preferably understood as representing a (true) subset of the sensor data. In other words, some (but not all) of the sensor data is selected to form the ROI.

[0043] In particular, the region of interest (ROI) is determined in such a way that it represents a meaningful portion of the sensor data, especially the portion that is meaningful in relation to the detection task (such as 2D / 3D object detection and / or semantic segmentation) and / or the context of the sensor data. Preferably, the ROI corresponds to (and / or includes and / or contains) an object (to be detected).

[0044] Preferably, the (determined) region of interest is classified according to the occurrence of predetermined and / or predefined and / or preset categories (within the region of interest). For example, (certain) categories may be objects. For example, predefined categories may be cars, roads, or pedestrians. These categories are predefined and / or predetermined so that (autonomous) vehicles can understand their surroundings by knowing which categories are located in specific locations around them.

[0045] Specifically, a defined subset of sensor data and / or a defined selection of sensor data (pixels of sensor data) and / or a selected portion of sensor data forming a region of interest are connected (in a topological sense) and / or correspond to a connected space.

[0046] Preferably, the region of interest (ROI) and the class used to classify the ROI are determined in such a way that the class can be assigned to at least a portion of the sensor's data within the ROI. This assignment can be done at the pixel level (e.g., semantic segmentation) or at the level of the entire object within the bounding box (e.g., in 2D / 3D object detection).

[0047] In a preferred embodiment, the determination of local quality parameters includes the determination of at least one background region of at least one segment of the region of interest and, preferably, the determination of the entire region of interest, particularly through an evaluation unit (a method step implemented by a computer). The background region is determined specifically by the background of the detected object forming the region of interest. This provides the advantage that the local quality parameters directly relate to the quality of the sensor data of the region of interest in relation to the background region.

[0048] Preferably, the background region can be the area surrounding the region of interest. In particular, the background region may not have sensor data intersecting with the region of interest. Preferably, the background region is at least partially located, and more preferably completely located, outside the region of interest.

[0049] It is conceivable that the background region is at least a segment of the region of interest, preferably a complete segment. For example, 2D / 3D object detection is used to determine the bounding box (as the region of interest). It is only known that the detected object is located somewhere within the bounding box, but the object's precise position relative to the bounding box remains unknown. Preferably, the edge region of the bounding box (but at least partially and preferably completely within the bounding box) and / or the region along the inner sidewall of the bounding box is determined as the background region. Here, it is highly likely that the determined background region represents the background region rather than a portion of the foreground (of the detected object and / or item).

[0050] Preferably, the background region is determined based on the boundary of the region of interest. Preferably, the background region follows, and more preferably covers, the boundary of the region of interest at least in segments and over its entire extension (length).

[0051] Preferably, at least a portion of the boundary of the background area and preferably the entire boundary of the background area includes curved sidewalls, and particularly (substantially) consists of curved sidewalls.

[0052] Preferably, the shape and / or trajectory of the background region boundary substantially corresponds to the shape and / or trajectory of the region of interest boundary.

[0053] In a further preferred embodiment, the background region is at least partially adjacent to the region of interest, and preferably adjacent over its entire (geometric) extension. This is advantageous because the relative arrangement of adjacent regions improves the accuracy of local quality parameters, since the foreground (object) is evaluated relative to the directly adjacent background region. Therefore, it is possible to directly assess whether the contrast of the foreground relative to the background is sufficiently high.

[0054] In a further preferred embodiment, the background region at least partially, preferably completely and / or entirely and / or substantially surrounds the region of interest along its entire boundary. This is advantageous because it allows for the evaluation of the data quality of the foreground (which substantially corresponds to the region of interest, particularly the object detected and / or identified by the processing unit) in the transition region between the foreground (object and / or region of interest) and the background. Advantageously, this transition region should contain particularly valuable information related to the evaluation of sensor data quality regarding the region of interest (particularly the object).

[0055] Preferably, the background region surrounds only (maximum) half of the boundary of the region of interest. This may be sufficient to provide reliable evaluation results on the one hand, and very fast evaluation on the other (because there is less sensor data for the background region to be evaluated).

[0056] Preferably, the ratio of the number of pixels in the sensor data of the background region used to determine the local quality parameters to the number of pixels in the sensor data of the region of interest used to determine the local quality parameters is at most 1 / 2, preferably at most 1 / 3, preferably at most 1 / 4, preferably at most 1 / 5, preferably at most 1 / 10, preferably at most 1 / 15, preferably at most 1 / 20, and particularly at most 1 / 40.

[0057] Preferably, the ratio of the number of pixels in the sensor data of the background region used to determine the local quality parameters to the number of pixels in the sensor data of the region of interest used to determine the local quality parameters is at least 1 / 100, preferably at least 1 / 40, preferably at least 1 / 20, preferably at least 1 / 15, preferably at least 1 / 10, preferably at least 1 / 5, and particularly at least 1 / 4.

[0058] In a further preferred embodiment, the determination of local quality parameters is based on at least one measurement from the sensor device, particularly a quantitative characteristic of intensity and / or brightness and / or optical values ​​(such as color, gloss, etc.). This involves sensor data from at least one segment of the region of interest, and preferably sensor data from all regions of interest. The advantage of this approach is that, when providing evaluation measures, the quantities critical to the reliability of data quality and sensor data evaluation can be considered, and these quantities of measurements (intensity, brightness, and / or color) are considered in a very flexible manner, specifically depending on the characteristics of the object (shape and / or size).

[0059] Preferably, the determination of local quality parameters is based on sensor data (especially all) of the (defined) background region.

[0060] In a further preferred embodiment, the local quality parameters are determined based on the average foreground intensity of the region of interest and / or the average background intensity.

[0061] In a further preferred embodiment, the determination of the local quality parameters depends on the quantitative characteristics, particularly the size or quantitative characteristics of at least one segment of the region of interest and preferably at least one geometric extension of the entire region of interest. This advantageously provides the possibility of taking into account the (relative) size (or dimensions) of the object in the region of interest (particularly the detected object).

[0062] Preferably, the quantity feature of at least one geometric extension of at least one segment of the region of interest, particularly the size or size quantity feature, is the quantity feature of at least one geometric extension of the detected object within the region of interest, particularly the size or size quantity feature (e.g., in the case of semantic segmentation evaluation techniques).

[0063] Preferably, the quantity feature of at least one geometric extension of at least one segment of the region of interest, particularly the size or quantity feature of the dimensions, is the quantity feature of at least one geometric extension of the bounding box, particularly the size or quantity feature of the dimensions, which is determined, for example, using 2D / 3D object detection and evaluation techniques.

[0064] Preferably, the quantity characteristics, particularly the size or dimensional quantity characteristics, of at least one geometric extension of at least one segment of the region of interest are relative quantities, which are particularly preferably determined relative to all sensor data.

[0065] Preferably, the quantity feature of at least one geometric extension of at least one segment of the region of interest, particularly the size or the quantity feature of the size, is the number of pixels in the sensor data of the segment of the region of interest (as an object) and particularly the region of interest, said number of pixels compared to the number of pixels in the entire sensor data and / or compared to the number of pixels of the object size in the sensor data and / or in the entire dataset of (a large amount) sensor data.

[0066] In a further preferred embodiment, the processing unit applies (particularly computer-implemented) 2D object detection and / or (particularly computer-implemented) 3D object detection and / or (particularly computer-implemented) semantic segmentation, etc. (method steps implemented by a computer). The advantage of using these detection and / or evaluation techniques is that object-related and / or class-related information can be extracted from sensor data.

[0067] Preferably, the 2D / 3D object detection output and / or determination of bounding boxes and classes for classifying objects within the bounding boxes are used.

[0068] In the case of semantic segmentation, each pixel of the sensor data is assigned to a predefined class (such as car, road, or pedestrian). This, in particular, allows for the extraction and / or analysis of the boundaries and / or shapes of objects and / or items corresponding to the classes assigned to the (defined) region of interest through pixel-level evaluation.

[0069] Preferably, instantaneous semantic segmentation is used. This advantageously allows for the extraction and / or analysis of the boundaries of regions of interest identified in the sensor data, with each pixel assigned to a predefined class.

[0070] In a further preferred embodiment, the determination of local quality parameters is based on a metric (determined and / or calculated), which in particular depends on the size of the region of interest and / or the contrast of the region of interest relative to its background. Specifically, the metric is used to distinguish between difficult, moderate, or easy data points, where a data point is one of a series of (input) sensor data points (generated simultaneously by the sensor device) generated sequentially by the sensor device. Specifically, the metric is used to determine and / or calculate the value, and preferably, the local quality parameter is determined based on this value.

[0071] Preferably, a specific dataset (ranking) metric is determined (a method step implemented by the evaluation unit via computer). For this purpose, preferably the region of interest size (particularly the object size) is considered and / or determined and / or taken into account based on the number of pixels and / or the relative contrast between the region of interest and its background (particularly the defined background region) (particularly the relative contrast of the object) (for determining local quality parameters).

[0072] Preferably, at least two of the three (so-called) difficulty ranking metrics are determined and / or extracted:

[0073] R joint =1–1 / 2(l2(F,B)+A),

[0074] R intensity =1–l2(F,B),

[0075] R size =1–A,

[0076] Where F is preferably the average value of the foreground pixel intensity, B is the average value of the background pixel intensity, and A is preferably the relative area of ​​the region of interest, especially the object, which is defined by the number of pixels normalized by the average object size in the entire dataset.

[0077] Here, l2(...,...) denotes the l2 norm.

[0078] Therefore, the three (difficulty) ranking metrics include:

[0079] R intensity It is the intensity difference (contrast) between the foreground and background (especially the region of interest (or object) and the defined background region).

[0080] R size It is the relative size of the region of interest, especially the object being detected, and

[0081] R joint It is a combined ranking calculated by integrating the first two criteria.

[0082] Preferably, the metric includes a difference and / or a sum of values ​​that depend on the average intensity of the foreground pixels and the average intensity of the background pixels, and on the characteristic values ​​of the relative area and / or size of the region of interest (especially the object).

[0083] R is worth considering. joint It can also be defined as R joint =1–l2(F,B)+A.

[0084] Preferably, especially when l2(...,...) and A are particularly preferred as standardized quantities, the above definitions apply to R. joint It is preferred because it is used for R joint The result value is independent of l2(F,B) and the current value of A and will not be negative.

[0085] Preferably, the local quality parameters are determined based on at least one of these measures.

[0086] The safety quantity is preferably derived and / or determined from local quality parameters and / or based on at least one (calculated and / or determined) (difficulty) level metric (particularly methodological steps implemented by the evaluation unit via computer). Preferably, the safety quantity is a characteristic of the reliability and / or quality of the detection task (performed by the processing unit).

[0087] In a further preferred embodiment, the sensor data is processed by an artificial neural network (particularly by a processing unit), providing the sensor data as an input variable to the artificial neural network, and the artificial neural network maps the input variable as a function and / or according to a parameterizable processing chain to an output variable, wherein the output variable represents at least one region of interest (ROI) of the (input) sensor data and at least one class for classifying the ROI.

[0088] Preferably, the neural network generates output data from which the vehicle's driver assistance system obtains relevant information for partial or full autonomous driving.

[0089] Preferably, the (artificial) neural network is designed as a deep artificial neural network, particularly in which a parameterizable processing chain has multiple processing layers.

[0090] Preferably, the safety measure is used for the safety argumentation of the deep neural network (particularly for evaluating vehicles with sensor data, especially during the perception DNN), where the DNN is evaluated specifically on a variety of input data points (where a data point corresponds to a sensor data record, e.g., an image), preferably using different difficulty levels.

[0091] Preferably, the (artificial) neural network is designed as a convolutional neural network (CNN) and / or a recurrent neural network (RNN).

[0092] Preferably, the artificial neural network is trained using predetermined training data, and the parameterizable processing chain is parameterized through training.

[0093] Preferably, the input layer of the artificial neural network provides a digital image captured by a camera, and the output layer of the artificial neural network provides a matrix of N-dimensional output vectors, each of which, as a probability of at least one assignment for N classes, can be interpreted in particular as a defined region of interest.

[0094] The vehicle function can be the vehicle function of a vehicle component. Preferably, the vehicle component is selected from the group consisting of: systems for driving and / or controlling the vehicle, computer vision systems, navigation systems, warning systems, chassis and / or vehicle damping systems, vehicle air conditioning control systems, entertainment systems, infotainment systems, vehicle window opening systems, comfort systems for improving passenger driving comfort, door and / or window locking systems, vehicle locking systems, roof removal mechanisms, sunroof mechanisms, vehicle safety systems (e.g., braking systems) and / or windshield wiper mechanisms, and combinations thereof.

[0095] Preferably, the spatially resolved raw sensor data is generated and / or recorded in a joint record (and thus simultaneously). It is conceivable that further records, particularly those recorded by a single sensor device, preferably recorded one after another (immediately and continuously), are used as raw sensor data.

[0096] The sensor device can be deployed on and / or inside the vehicle. The sensor device can also be deployed on the exterior of the vehicle.

[0097] Preferably, the vehicle (whose vehicle functions will be performed) includes sensor devices for detecting the vehicle's environment, particularly preferably as (especially stationary) components of the vehicle.

[0098] It can be imagined that the sensor device used for vehicle environment detection is a sensor device for the vehicle (and / or other object) that is different from the vehicle performing the vehicle's functions. For example, it may be a sensor device for another vehicle communicating with the vehicle (e.g., via "vehicle-to-vehicle", V2V communication and / or "vehicle-to-infrastructure", V2I communication and / or "vehicle-to-X", V2X communication).

[0099] The objective is further achieved by an evaluation unit of the vehicle, which evaluates spatially resolved raw sensor data for environmental perception generated by at least one sensor device of the vehicle for environmental detection. The evaluation unit is configured to receive sensor data characterized by the spatially resolved raw sensor data. The evaluation unit is configured to receive a determined region of interest (ROI) from the sensor data. The evaluation unit is characterized by being configured to process the sensor data based on the determined ROI and determine at least one local quality parameter thereon, which characterizes the quality of the sensor data with respect to at least one segment of the ROI, and preferably with respect to the entire ROI. Preferably, the ROI can be (or substantially corresponds to) the object being detected, and in particular, the detection is performed by means of a processing unit of the vehicle (e.g., the processing unit described in the description of the above method) and / or a processing unit for the vehicle (through computer-implemented method steps and / or particularly through computer vision methods).

[0100] Preferably, the evaluation unit provides at least one signal regarding the performance of vehicle functions based on local quality parameters. Preferably, this signal is transmitted to the vehicle's control unit.

[0101] Preferably, the evaluation unit is configured, adapted, and / or intended to perform the method steps described above, which have already been described in relation to the method, individually or in combination with each other, in the description of the evaluation unit. Alternatively, the method may be provided individually or in combination with each other, with all the features described in the description of the evaluation unit.

[0102] According to the invention, it is also proposed that the evaluation unit determine a local quality parameter that can serve as a measure of the quality of sensor data for a single region of interest, particularly an object. This advantageously allows for the analysis of the quality of sensor data within (or within) a region of interest in a very specific manner, depending on the specific determination of the region of interest (or object). This allows for a very fine and personalized evaluation of the quality of the (input) sensor data, which has specific significance for the detection task within the region of interest (e.g., object detection task).

[0103] The present invention also relates to a means of transportation, particularly a motor vehicle and / or designed for autonomous or partially autonomous driving, including an evaluation unit for the means of transportation according to the above embodiments.

[0104] Preferably, the vehicle includes a control unit (communicatively connected to the evaluation unit) and has at least one output for controlling at least one actuator of the vehicle, the control unit being configured to perform control functions as a function of performing vehicle functions based on local quality parameters determined by the evaluation unit, particularly by controlling at least one actuator.

[0105] The means of transport can be, in particular, a (motorized) road vehicle. The means of transport can be a motor vehicle, especially a driver-controlled motor vehicle ("driver-only"), a semi-autonomous, autonomous (e.g., Level 3, 4, or 5 of the SAE J3016 standard), or an automated driving motor vehicle. In this case, Level 5 automated driving refers to a fully automated driving vehicle. Similarly, the means of transport can be an unmanned transport system. The means of transport can be driver-controlled or automated. Furthermore, in addition to road vehicles, the means of transport can also be aircraft, airplanes, and other means of transport or other types of vehicles, such as air, water, or rail transport.

[0106] Furthermore, the present invention relates to a computer-implemented method for evaluating, particularly for verifying and / or validating, a computer-implemented computer vision method for (and / or performed by) a processing unit for a vehicle. In this computer-implemented computer vision method, sensor data for environmental perception generated by at least one sensor device for vehicle environment detection is processed and evaluated by determining at least one region of interest and a class for classifying the region of interest.

[0107] According to the present invention, for provided and / or given training data, the method includes evaluating the quality of the training data based on a local quality parameter determined based on the quality of sensor data relating to a region of interest. Specifically, the method includes determining the local quality parameter.

[0108] Preferably, the method for evaluating a computer vision method implemented by a computer includes at least one, and preferably multiple, method steps (individually and / or in combination with each other), which are performed and / or executed by the evaluation unit described in the description of the method, for automatically performing functions of a vehicle, especially a vehicle with at least partial autonomous control. Here, training data is preferably used instead of sensor data.

[0109] Preferably, the method for evaluating a computer vision method implemented by a computer includes determining local quality parameters of at least one segment of at least one region of interest, and preferably at least one background region of the complete region of interest.

[0110] Preferably, the determination of local quality parameters is based on the determination of a metric, which depends on the size of the region of interest and / or the contrast of the region of interest relative to its background. Preferably, the metric includes attributes and / or definitions as disclosed above in the description of methods for automatically performing functions of vehicles, especially vehicles with at least partial autonomous control.

[0111] It is advantageous to assess the quality of training data based on (local) data difficulty.

[0112] In a preferred embodiment, the method includes selecting training data based on local quality parameters of the training data. This training data can advantageously be used to train (artificial) neural networks, preferably based on their computer vision methods. Preferably, a sufficient amount of training data is selected, encompassing a wide range of data difficulties. This advantageously allows for very specific training of the neural network, enabling a reduction in the total amount of training data. Furthermore, the training data can be selected and generated in a highly targeted manner, as it is possible to determine which local quality parameters the (newly)selected and / or generated training data should possess. This advantageously facilitates the creation of training datasets for training neural networks.

[0113] Preferably, the computer vision method includes one or more method steps described in the description of the method steps performed by the processing unit, while training data is provided and processed and / or evaluated instead of sensor data (mentioned in the description of the method steps performed by the processing unit). In particular, the computer vision method can perform object detection tasks based on training data.

[0114] Specifically, this method includes providing training data.

[0115] In particular, computer vision methods involve processing (training) data through artificial neural networks, especially through DNNs.

[0116] Preferably, this method is used for security verification of a perceptual DNN, wherein the DNN is preferably evaluated on various input data points with different difficulty levels. In particular, the data points are a set of sensor data and / or training data.

[0117] It is preferable to use different data difficulty levels to evaluate neural networks, such as DNNs. In particular, when the local quality parameters of the regions of interest are different from each other, the two training sets are understood to have different data difficulty levels.

[0118] Preferably, the methods used for evaluation, particularly computer-implemented computer vision methods for verification and / or validation, are executed on an external server, such as a backend system.

[0119] Preferably, the quality of the training data is assessed based on the difficulty of the data, particularly based on local quality parameters of the training data. It is preferable to select a sufficient amount of training data that includes a wide range of data variations.

[0120] Therefore, the difficulty of the data points presented in the training dataset is optimized through measurement and analysis. Based on this analysis, regions of insufficient or excessive difficulty in the dataset are identified and used to select data while maintaining a reasonable balance of difficulty.

[0121] Preferably, computer vision methods, particularly neural networks (DNNs), are evaluated using different levels of difficulty in the (training) data. Specifically, the difficulty level is determined based on local quality parameters.

[0122] Therefore, it is preferable to use data difficulty metrics for the security justification of DNNs. In this way, the maximum data difficulty that a DNN can detect can be identified and compared with a specified minimum level.

[0123] It is preferable to use data difficulty measures and / or local quality parameters as computer vision methods, especially for pass / fail validation of DNNs.

[0124] In a preferred embodiment, the method includes using local quality parameters as an evaluation criterion for applying computer vision methods, particularly neural networks (e.g., DNNs), in a specific highly automated driving system.

[0125] Basically, methods for automatically performing vehicle functions and / or evaluating computer vision methods can be used for any computer vision task, particularly detection tasks within or outside the field of autonomous driving, even if both are described (only) in the context of the vehicle and / or autonomous driving fields.

[0126] For example, an external server serves as the backend for a vehicle manufacturer or service provider. The functionality of the backend or external server can be executed on an (external) server farm. The (external) server can be a distributed system. The external server and / or backend may be cloud-based.

[0127] The present invention also relates to computer programs or computer program products comprising program methods, particularly program code, which represent or encode at least one method step of each of the two methods according to the present invention, and preferably one of the preferred embodiments, and is designed to be executed by a processor device.

[0128] The present invention also relates to a data storage device on which at least one embodiment of a computer program according to the invention or a preferred embodiment of the computer program is stored. In particular, for evaluating, especially for verifying and / or validating, a computer-implemented computer vision method for (and / or executed by) a processing unit of a vehicle, the data storage device may be the aforementioned external server, which may be the backend of an OEM or service provider. Attached Figure Description

[0129] Further advantages, objectives, and features of the present invention will be described in the following description with reference to the accompanying drawings, which are by way of example only. In the drawings, similar components in different embodiments are shown with the same reference numerals.

[0130] In the attached diagram:

[0131] Figure 1 A schematic diagram of a vehicle including an evaluation unit according to an embodiment of the present invention is shown; and

[0132] Figure 2 The sample region is shown when determining local quality parameters. Detailed Implementation

[0133] Figure 1 A schematic view of vehicle 1 is shown, which includes three different sensor devices 22, 24, and 26, each of which is preferably a component of the environmental perception system 20 of vehicle 1.

[0134] Sensor data generated by sensor device 22 can be transmitted to processing device 10 and / or processing unit 14, which includes at least one processor and memory for executing executable instructions. Processing unit 14 processes the received sensor data and determines at least one region of interest (in particular an object) and at least one class for classifying the region of interest (in particular an object).

[0135] Reference numeral 12 indicates the evaluation unit (of the processing device 10) of the vehicle 1, which is configured to process sensor data based on a determined region of interest. Furthermore, the evaluation unit determines local quality parameters that characterize the quality of at least a portion of the sensor data relating to the region of interest (e.g., a detected object).

[0136] Figure 2 Example regions considered in the calculations (sensor data, such as images taken by sensor devices), particularly those used to determine local quality parameters, are shown.

[0137] Reference symbol 40 denotes a region of interest from input sensor data, such as based on pixel annotations. Region of interest 40 can be an object detected by processing unit 14 through the execution of a (computer-implemented) detection method. Here, region of interest 40 corresponds to an object that can be classified as a "pedestrian".

[0138] Reference symbol 30 represents the background region of interest 40, which completely surrounds region of interest 40. Figure 2 It can be seen that the shape of the background region 30 follows the shape of the boundary 32 of the region of interest 40.

[0139] Preferably, the pixel intensity and / or size of the presented object (region of interest 40 here) are compared with its background. In this way, small objects or objects with relatively low contrast to the background are considered difficult to detect by the evaluation technique (as a DNN).

[0140] Based on the determination of the region of interest 40 (especially the foreground region) and the background region 30, a metric can preferably be used as an evaluation standard during the verification and / or validation process of the DNN (which should be deployed in a highly automated driving system).

[0141] Preferably, two criteria, namely object size based on pixel count and the relative contrast between the object and its background (specifically, background region 30), are considered to extract at least one of the three difficulty ranking measures, as follows:

[0142] R joint =1–1 / 2(l2(F,B)+A),

[0143] R intensity =1–l2(F,B),

[0144] R size =1–A,

[0145] Where F is preferably the average pixel intensity of the foreground (corresponding to object 40), B is the average pixel intensity of the background (corresponding to background region 30), and A is preferably the relative area of ​​the object, which is defined by the number of pixels normalized by the average object size in the entire dataset.

[0146] Here, l2(...,...) denotes the l2 norm.

[0147] Therefore, the three (difficulty) ranking metrics include:

[0148] R intensity It is the difference in intensity (contrast) between the foreground and the background.

[0149] R size It is the relative size of the object, and

[0150] R joint It is a combined ranking calculated by integrating the first two criteria.

[0151] Preferably, the local quality parameters are determined based on at least one of these measures.

[0152] As mentioned above, R joint It can also be defined as R joint = 1 – l2(F,B) + A. For the reasons stated above, the above definition is preferred at least for l2(F,B) and A as standardized quantities.

[0153] The applicant reserves the right to claim all features disclosed in the application documents as essential features of the invention, provided they are novel individually or in combination with respect to the prior art. Furthermore, it should be noted that features described in the figures may be individually advantageous. Those skilled in the art will readily recognize that specific features disclosed in the figures may also be advantageous without requiring further features to be derived from those figures. Moreover, those skilled in the art will recognize that advantages may derive from combinations of different features disclosed in one or more figures.

[0154] Reference Symbol List

[0155] 1. Means of transportation

[0156] 10 Processing Equipment

[0157] 14 Processing Units

[0158] 12 Evaluation Units

[0159] 30 Background Areas

[0160] 32. Boundary of the region of interest

[0161] 40 Regions of Interest

Claims

1. A method for automatically performing functions of a vehicle (1) based on raw sensor data with spatial resolution for environmental perception, said raw sensor data being generated by at least one sensor device (22, 24, 26) for environmental detection of the vehicle (1), said method comprising: - Receive spatially resolved raw sensor data generated by at least one sensor device (22, 24, 26) of the vehicle (1); - The sensor data is processed by the processing unit (14), which is a feature of the spatially resolved raw sensor data, and the processing unit (14) determines at least one region of interest (40) and at least one class for classifying the region of interest (40); - The evaluation unit (12) processes sensor data based on the determined region of interest and thereby determines at least one local quality parameter, which characterizes the sensor data quality with respect to at least one segment of the region of interest (40), and the local quality parameter includes specific information on the sensor data quality and sensor data reliability for the region of interest; - Perform vehicle functions based on local quality parameters, the determination of which includes determining at least one segment of the region of interest (40) or at least one background region (30) of the entire region of interest (40).

2. The method according to claim 1, wherein, The background region (30) is at least partially adjacent to the region of interest (40).

3. The method according to claim 1, wherein, The determination of local quality parameters is based on the measurement quantity characteristics of at least one measurement value of sensor devices (22, 24, 26) in relation to at least one segment of the region of interest (40).

4. The method according to claim 3, wherein, The measured values ​​are intensity and / or optical values.

5. The method according to claim 1, wherein, The determination of local quality parameters is based on the average intensity of the foreground of the region of interest and / or the average intensity of the background region (30).

6. The method according to claim 1, wherein, The determination of local quality parameters depends on the quantitative characteristics of at least one geometric extension of at least one segment of the region of interest (40).

7. The method according to claim 1, wherein, The determination of local quality parameters is based on the determination of a metric, which depends on the size of the region of interest (40) and / or the contrast of the region of interest (40) relative to its background.

8. The method according to claim 1, wherein, The processing unit (14) applies 2D object detection and / or 3D object detection and / or semantic segmentation.

9. The method according to claim 1, wherein, Sensor data is processed by an artificial neural network, which provides sensor data as input variables and maps the input variables to output variables according to a parameterizable processing chain. The output variables represent at least one region of interest in the sensor data and a class for classifying the region of interest.

10. The method according to claim 1, wherein, The sensor data is generated by at least one sensor device of the vehicle (1), which is selected from the group and combination thereof including cameras, radar sensors, lidar sensors, thermal sensors.

11. An evaluation unit (12) for a vehicle (1), the evaluation unit being used to evaluate raw, spatially resolved sensor data for environmental perception generated by at least one sensor device (22, 24, 26) for environmental detection of the vehicle, wherein, An evaluation unit (12) is configured to receive sensor data, which is a feature of spatially resolved raw sensor data, wherein the evaluation unit (12) is configured to receive a determined region of interest (40) of the sensor data, characterized in that the evaluation unit (12) is configured to process the sensor data based on the determined region of interest and thereby determine at least one local quality parameter, which characterizes the quality of the sensor data relating to at least one segment of the region of interest (40), wherein the local quality parameter includes specific information for the sensor data quality and sensor data reliability of the region of interest, and the determination of the local quality parameter includes the determination of at least one background region (30) of at least one segment of the region of interest (40) or the entire region of interest (40).

12. A means of transport (1), the means of transport comprising the evaluation unit according to claim 11.

13. The vehicle according to claim 12, wherein it is a motor vehicle and / or designed for autonomous or partially autonomous driving.

14. A computer-implemented method for evaluating, verifying, and / or validating a computer-implemented computer vision method for a processing unit (14) of a vehicle (1) and / or performed by the processing unit (14) of the vehicle (1), wherein sensor data for environmental perception generated by at least one sensor device for environmental detection of a vehicle (1) is processed and evaluated by determining at least one region of interest (40) and a class for classifying the region of interest (40), characterized in that, For the provided training data, the method includes evaluating the quality of the training data based on a local quality parameter determined based on the quality of sensor data relating to the region of interest (40), wherein the local quality parameter includes specific information regarding the quality and reliability of the sensor data for the region of interest, and the determination of the local quality parameter includes determining at least one segment of the region of interest (40) or at least one background region (30) of the complete region of interest (40).

15. The method of claim 14, wherein the method comprises selecting training data based on local quality parameters of the training data respectively.

16. The method according to claim 14 or 15, characterized in that, The method includes using local quality parameters as evaluation criteria for applying computer vision methods in autonomous driving systems.

Images