Methods and computer vision systems for criticality assessment in object classification

Abstract

The present invention relates to a method for criticality assessment in a computer vision system, in which, during driving operation, an image (111) of the area in front of the vehicle is captured by a sensor. To determine a criticality limit, the image is divided into N image areas (120), a Shapley value is calculated for each of the N image areas, and the N image areas are sorted according to importance based on the Shapley value. Starting with the most important image area, a progressively less important image area is removed (130), and an object classification is performed for each remaining image. This is repeated until the object classification for an i-th remaining image yields no result (140), and the criticality limit is determined using a divisor i / N and averaging over several images.During a test drive, a standard criticality threshold is established. During a standard drive, a current criticality threshold is continuously calculated. If the current criticality threshold falls below the standard criticality threshold, a visual restriction of the sensor facing the area in front of the vehicle is detected, whereupon a warning is displayed on the driver's screen. Furthermore, a computer vision system implementing this procedure is presented.

Description

[0001] The present invention relates to a method for criticality assessment of a computer vision system, which is geared towards object classification in a vehicle approach area. Furthermore, a computer vision system implementing the method is presented.

[0002] When validating a computer vision system used in a motor vehicle to support driving operation by an Advanced Driver Assistance System (ADAS) and / or an Automated Driving System (ADS), the blockage of various image areas by, for example, sensor contamination or other influences, which lead, for example, to the non-recognition of reversing lights or rims of another vehicle, represents different safety risks.

[0003] Publication US 2021 / 0042613A1 describes the calculation of Shapley values ​​to quantify the contributions of individual input points to the outputs of a trained neural network. A probabilistic neural network, derived from the trained network, is used to calculate the distribution of the outputs. Based on this, the Shapley values ​​are estimated.

[0004] German patent application DE 11 2021 001 872 T5 discusses an object detection system that uses image and distance information to improve detection accuracy. It describes a detection processing method that integrates image and distance information to identify a target object. The document also analyzes the significance of different image areas by using Shapley values ​​to identify the regions that contribute most to correct detection.

[0005] German patent application DE 11 2021 000 596 T5 describes a technique for interpreting the results of an object recognition model, in which image areas are geometrically divided into segments in order to calculate the contribution of each segment to recognition by calculating Shapley values. These Shapley values ​​are used to evaluate the importance of individual image segments, which provides a basis for determining critical image areas.

[0006] The publication “YANG, Qing, et al. MFPP: Morphological fragmental perturbation pyramid for black-box model explanations. In: 2020 25th International conference on pattern recognition (ICPR). IEEE, 2021. pp. 1376-1383” employs a method using a morphological fragmentary perturbation pyramid to arrive at an understanding of black-box models. This involves Fig. 3. A gradual weighting of the importance of an image is shown.

[0007] The publication “PONN, Thomas; KRÖGER, Thomas; DIERMEYER, Frank. Identification and explanation of challenging conditions for camera-based object detection of automated vehicles. Sensors, 2020, Vol. 20, No. 13, p. 3699” discusses the identification and process analysis of camera-based object detection in automated vehicles under difficult conditions. Fig. 8 demonstrated the use of Shapley contributions in object detection.

[0008] German patent DE 10 2021 005 387 A1 discloses a method for issuing a warning when an object is detected in the field of view of a vehicle's surround-view camera and is blocking further travel. In calculations for cross-correlation between images from the surround-view camera, each image is divided into an area of ​​interest and a rejection area, and only the image content of the area of ​​interest is used to perform the cross-correlation.

[0009] Against this background, an object of the present invention is to propose a method for criticality assessment in a computer vision system, in which an image provided by a camera is evaluated in its image areas with regard to the recognizability of objects. Criticality here is a measure of the number of blocked or unanalyzable image areas beyond which object recognizability is no longer possible. Furthermore, a device implementing this method is presented.

[0010] To solve the aforementioned problem, a method for criticality assessment in a computer vision system is proposed, in which, during continuous operation, at least one sensor directed at a vehicle forecourt captures a respective image of the vehicle forecourt, in order to determine a criticality limit. • the respective image is divided into N image areas, • a Shapley value is calculated for each of the N image areas, • according to the Shapley value, the N image areas are sorted in a ranking according to importance, • starting with the most important image area in the ranking, step by step ◯ a respective subordinate image area is removed and ◯ a classification of objects is performed on each remaining residual image, which is repeated until the classification of objects yields no result for an i-th residual image, • a point-of-interest value is formed with a divisor i / N and stored, • The steps listed above are iterated for multiple images from at least one sensor, • an average is calculated over the point-of-interest values, and • the mean value is set as the criticality limit.

[0011] During a test drive, a standard criticality limit is established based on the criticality limit determined during that drive. Subsequently, during a standard driving operation, a current criticality limit is continuously calculated. If the current criticality limit falls below the standard criticality limit by a predefined factor, a visual impairment of the sensor facing the area in front of the vehicle is detected, and a warning indicating a malfunction of the computer vision system is displayed on a driver display.

[0012] The inventive method proposes a methodology based on Shapley values ​​regarding image area properties for determining criticality, which indicates the degree of blockage at which correct recognition is no longer possible. As already disclosed in US 2021 / 0042613 A1, Shapley values ​​indicate how strongly a specific input point, such as a pixel in an image, influences the result. Considering Shapley values ​​thus advantageously allows for the evaluation of the significance of individual pixels in the respective image of the vehicle's approach path, which is relevant for determining critical image areas that could affect the classification of objects if they are blocked or disturbed.

[0013] It is conceivable that the respective image is divided into several image regions and the division according to the invention into N image areas is carried out in each image region. The steps according to the invention for determining the criticality limit are then combined from all image regions and an average value is calculated, whereby only those image regions are used in which at least one object could be classified before the removal of subordinate image areas.

[0014] As a predefined factor by which the current criticality threshold may fall below the standard criticality threshold without indicating a limitation of the sensor's visibility, 10% is chosen, for example, meaning a warning is triggered if the current criticality threshold is only 0.9 times the standard criticality threshold.

[0015] In one embodiment of the method according to the invention, the division into N image areas is carried out by means of a checkered grid.

[0016] In a further embodiment of the method according to the invention, each image area is formed with respective adjacent image areas that overlap the checkered grid.

[0017] In a further embodiment of the method according to the invention, the at least one sensor directed towards a vehicle forecourt is formed by a sensor from the following list: camera, lidar, radar.

[0018] In a further embodiment of the method according to the invention, after prior detection of a degraded area in the respective image of the vehicle's foreground, a critical area is calculated using a limitation factor formed with the standard criticality limit. In the worst case, no classification is possible in the critical area, and it must therefore be classified as unreliable with regard to object detection.

[0019] Furthermore, a computer vision system is claimed, which is arranged in a motor vehicle and comprises a control unit, a processing unit, and at least one sensor directed at the area in front of the vehicle. The at least one sensor is designed to continuously capture an image of the area in front of the vehicle during driving operation. The processing unit is configured to determine a criticality threshold by • the respective image is divided into N image areas, • a Shapley value is calculated for each of the N image areas, • according to the Shapley value, the N image areas are sorted in a ranking according to importance, • starting with the most important image area in the ranking, step by step ◯ a respective subordinate image area is removed and ◯ a classification of objects is performed on each remaining residual image, which is repeated until the classification of objects yields no result for an i-th residual image, • a point-of-interest value is formed with a divisor i / N and stored, • The steps listed above are iterated for multiple images from at least one sensor, • an average is calculated over the point-of-interest values, and • the mean value is set as the criticality limit.

[0020] The control unit is designed to • to determine the criticality threshold during a test drive and thus establish a standard criticality threshold, • to calculate a current criticality limit during standard driving operations in continuous execution and • In the event that the current criticality limit falls below the standard criticality limit by a predetermined factor, a visual impairment of the sensor directed towards the area in front of the vehicle is detected, and a warning that there is an impairment of the computer vision system is displayed on a driver display.

[0021] In one embodiment of the computer vision system according to the invention, the division into N image areas is carried out by means of a checkered grid.

[0022] In a further embodiment of the computer vision system according to the invention, each image area is formed with respective adjacent image areas that overlap the checkered grid.

[0023] In a further embodiment of the computer vision system according to the invention, the at least one sensor is formed by a sensor from the following list: camera, lidar, radar.

[0024] In a further embodiment of the computer vision system according to the invention, the computing unit is configured to calculate a critical area after a prior detection of a degraded area in the respective image of the vehicle forecourt by means of a limitation factor formed with the standard criticality limit, whereby no safe classification is possible in the critical area.

[0025] Further advantages and embodiments of the invention will become apparent from the description and the accompanying drawing.

[0026] It is understood that the features mentioned above and those to be explained below can be used not only in the combinations specified, but also in other combinations or on their own, without leaving the scope of the present invention.

[0027] The figures are described in a coherent and comprehensive manner; identical components are assigned the same reference symbols. Fig. Figure 1 shows various steps for criticality assessment in an embodiment of the method according to the invention. Fig. Figure 2 schematically shows a determination of a critical area in a further embodiment of the method according to the invention.

[0028] In Fig. Figure 1 shows various steps for criticality assessment in an embodiment of the method according to the invention. Figure 110 shows a vehicle approach area in which an image 111 shows a vehicle 112 driving ahead (and stopped at a traffic light). The image 111 is divided into N image areas 120, a Shapley value is calculated for each image area, and they are hierarchically ordered according to their importance. Figure 130 shows important image areas designated with numerical values ​​x1 to x8, which are removed from the image 111 according to their importance until an object classification (in the example shown, a vehicle 112) yields no result. Figure 140 shows the values ​​for three analyzed images 111 or, for example,Three image regions from an entire vehicle approach area are shown with their respective graphs for comparison of the subtraction for detection, where the number of removed image regions is plotted on the x-axis to the right and the number of classified objects is plotted on the y-axis to the top. As the number of removed image regions increases, the number of classified objects decreases until, at a certain number of removed image regions, no object can be classified in the respective image 111.

[0029] In Fig. Figure 2 schematically shows a determination 200 of a critical area in a further embodiment of the method according to the invention. Following the prior detection of a degraded area 201, a mean radius r approximately describing the degraded area 201 is used. degrad212, drawn around a midpoint 211 of the degraded area 201, determined by means of a limitation factor f formed with the standard criticality limit. krit is through rkrit=(1+fcrit)∗rdegrad a critical radius r krit 213 is calculated, in which case no reliable classification is possible. A standard criticality threshold or limitation factor could be, for example, 55%, meaning that in the worst case, after 55% of the N areas have been removed, a correct classification is no longer achievable. Reference symbol list 110 Illustration of vehicle approach area 111 Image 112 Vehicle ahead 120 Split Image 130 Hierarchical Subtraction 140 Comparison of Subtraction to Detection 200 Determination of critical area 201 Degraded Area 211 Center of degraded area 212 Medium radius degraded area 213 Critical radius

Claims

[1] Method for criticality assessment in a computer vision system in which, during continuous operation, at least one sensor directed towards a vehicle forecourt captures a respective image (111) of a vehicle forecourt, in which a criticality limit is determined • the respective image (111) is divided into N image areas (120), • a Shapley value is calculated for each of the N image areas, • according to the Shapley value, the N image areas are sorted in a ranking according to importance, • starting with the most important image area in the ranking, step by step ◯ a respective subordinate image area is removed (130) and ◯ a classification of objects is carried out for each remaining residual image, which is repeated until the classification of objects yields no result for an i-th residual image (140), • a point-of-interest value is formed with a divisor i / N and stored, • the steps listed above are iterated for multiple images (111) of the at least one sensor, • an average is calculated over the point-of-interest values, and • the mean value is set as the criticality limit, whereby a standard criticality limit is formed during a test drive operation with the determined criticality limit, whereby a current criticality limit is calculated continuously during a standard drive operation, and whereby, if the standard criticality limit is undercut by the current criticality limit by a predetermined factor, a restriction of the view of the sensor directed towards the vehicle's forecourt is detected and a warning that there is an impairment of the computer vision system is displayed on a driver display. [2] Method according to claim 1, wherein the division (120) into N image areas is carried out using a checkered grid. [3] Method according to claim 2, wherein a respective image area is formed with respective adjacent image areas that overlap the checkered grid. [4] Method according to one of the preceding claims, wherein the at least one sensor is formed by a sensor from the following list: camera, lidar, radar. [5] Method according to one of the preceding claims, wherein, after prior detection of a degraded area (201) in the respective image (111) of the vehicle foreground, a critical area (200) is calculated using a limitation factor formed with the standard criticality limit, wherein no safe classification is possible in the critical area (200). [6] Computer vision system which is arranged in a motor vehicle and comprises a control unit, a computing unit and at least one sensor directed towards a vehicle foreground, wherein the at least one sensor is configured to continuously capture an image (111) of a vehicle foreground during driving operation, wherein the computing unit is configured to determine a criticality limit by • the respective image (111) is divided into N image areas (120), • a Shapley value is calculated for each of the N image areas, • according to the Shapley value, the N image areas are sorted in a ranking according to importance, • starting with the most important image area in the ranking, step by step ◯ a respective subordinate image area is removed (130) and ◯ a classification of objects is carried out for each remaining residual image, which is repeated until the classification of objects yields no result for an i-th residual image (140), • a point-of-interest value is formed with a divisor i / N and stored, • the steps listed above are iterated for multiple images (111) of the at least one sensor, • an average is calculated over the point-of-interest values, and • the mean value is set as the criticality limit, with the control unit being designed to • to determine the criticality threshold during a test drive and thus establish a standard criticality threshold, • to calculate a current criticality limit during standard driving operations in continuous execution and • In the event that the current criticality limit falls below the standard criticality limit by a predetermined factor, a visual impairment of the sensor directed towards the area in front of the vehicle is detected, and a warning that there is an impairment of the computer vision system is displayed on a driver display. [7] Computer vision system according to claim 6, wherein the division (120) into N image areas is carried out by means of a checkered grid. [8] Computer vision system according to claim 7, wherein each image area is formed with each adjacent image areas that overlap the checkered grid. [9] Computer vision system according to any one of claims 6 to 8, wherein the at least one sensor is formed by a sensor from the following list: camera, lidar, radar. [10] Computer vision system according to one of claims 6 to 9, wherein the computing unit is configured to calculate a critical area (200) after a prior detection of a degraded area (201) in the respective image (111) of the vehicle foreground by means of a limitation factor formed with the standard criticality limit, wherein no safe classification is possible in the critical area (200).

Images