Method for operating a driver assistance device for a motor vehicle, driver assistance device for a motor vehicle and computer program product

Abstract

The invention relates to a method for operating a driver assistance device for a motor vehicle (1). It is provided that image data (10) representing the surroundings of the motor vehicle (1) are acquired by means of an optical surroundings detection device (5) and examined for at least one object (2) present in the surroundings and represented by the image data (10), and a reflection (6) represented by the image data (10) is evaluated to detect an obstacle (3) present in the surroundings and concealed by the object (2). The invention further relates to a driver assistance device for a motor vehicle (1) and a computer program product.

Description

[0001] The invention relates to a method for operating a driver assistance system for a motor vehicle. It also relates to a driver assistance system for a motor vehicle and a computer program product.

[0002] For example, the prior art document US 2017 / 0287334 A1 is known. This describes a system for detecting and locating obstacles that are not on a line of sight, as well as a corresponding method. It is proposed to receive reflections, including both direct and indirect reflections, using a sensor assigned to a moving platform. Subsequently, reflections associated with static targets are to be identified in order to obtain reflections associated with moving targets. Furthermore, for moving targets, a distinction is to be made between objects that are on the line of sight and objects that are not.It is intended to locate the objects not in the line of sight relative to the platform and to signal an approach to them if the objects not in the line of sight approach the platform on a movement path that intersects the platform.

[0003] Furthermore, WO 2022 / 043983 A1 describes a system and method for detecting the movement of an object outside the line of sight of a camera. The camera captures images of a target influenced by light scattered by the object and ambient light, as well as a reference influenced by the ambient light.A processing circuit may be configured to detect the movement of the object by calculating, for each of a plurality of acquired images, a compensated light intensity value that is a target light intensity value based on pixels in the respective acquired image that represent the target and that is modified based on a reference light intensity value based on pixels in the respective acquired image that represent the reference to reduce an effect of changes in ambient light on the target light intensity value during a period between the acquisition of a given image and acquisition of the respective acquired image.

[0004] The object of the invention is to propose a method for operating a driver assistance device for a motor vehicle, which has advantages over known methods, in particular enabling a reaction to hidden obstacles.

[0005] This is achieved according to the invention with a method for operating a driver assistance device for a motor vehicle with the features of claim 1. It is provided that image data representing an environment of the motor vehicle are captured by means of an optical environment detection device and are examined for at least one object present in the environment and represented by the image data, and a reflection represented by the image data is evaluated to detect an obstacle present in the environment that is concealed by the object.

[0006] Advantageous embodiments with useful further developments of the invention are specified in the dependent claims. It should be noted that the exemplary embodiments explained in the description are not limiting; rather, any variations of the features disclosed in the description, the claims, and the figures are feasible.

[0007] The method serves to operate the driver assistance device. The driver assistance device is preferably a component of the motor vehicle, but can of course also be present separately. For example, the driver assistance device is implemented using a control unit of the motor vehicle or it comprises the control unit. The driver assistance device fundamentally serves to support a driver of the motor vehicle when performing a driving operation of the motor vehicle. During driving operation, the motor vehicle moves at a speed other than zero, particularly in its forward direction of travel.

[0008] The driver assistance system includes, for example, an adaptive cruise control system, which adjusts the speed of the motor vehicle to maintain a defined distance between the motor vehicle and the object located in front of the motor vehicle. The object is, for example, another motor vehicle, i.e., a motor vehicle traveling in front of the motor vehicle, a trailer of another motor vehicle, or the like. The driver assistance system is preferably designed and configured for automated driving at least according to Level 1, but preferably at least according to Level 2 or at least according to Level 3 according to SAE Standard J3016.

[0009] The driver assistance system has an optical environment detection device, which it uses to detect the surroundings of the motor vehicle. The optical environment detection device is implemented, for example, as a camera device with at least one camera, which captures images of the surroundings and subsequently stores them as image data. The images or image data are further processed by the driver assistance system, for example, to implement the aforementioned distance control.

[0010] In any case, the driver assistance system examines the image data for the object present in the surrounding area, which is represented by the image data. This means that the image data contains an image of the object. If the presence of the object is detected based on the image data, the vehicle's driving operation is adjusted accordingly. For example, distance control is performed based on the object, i.e., the vehicle's driving speed is adjusted so that the defined distance to the object is maintained.

[0011] Known driver assistance systems have no way of reacting to an obstacle that is obscured by the object. The obstacle is also, for example, a motor vehicle. For the purposes of this description, the motor vehicle equipped with the driver assistance system is referred to as the first motor vehicle, the object as the second motor vehicle, and the obstacle as the third motor vehicle. However, this is purely exemplary. The object and the obstacle do not necessarily have to be motor vehicles.

[0012] It is assumed that the first motor vehicle is following the second motor vehicle at a defined distance and that the third motor vehicle is approaching the third motor vehicle from behind, i.e. the distance between the first motor vehicle and the second motor vehicle is essentially constant and the distance between the second motor vehicle and the third motor vehicle is decreasing. The driver assistance system of the first motor vehicle initially has no knowledge of the third motor vehicle. Only when or after the second motor vehicle swerves out to avoid the third motor vehicle can the third motor vehicle be detected by the first motor vehicle using the optical environment detection system. Accordingly, a comparatively late reaction of the first motor vehicle to the third motor vehicle is possible.

[0013] For this reason, the image data captured by the optical environment detection device should not only be examined for the presence of the object, but should also enable detection of the obstacle concealed by the object. This is done by evaluating a reflection contained in the image data or a reflection represented by the image data in addition to the object. The image data is therefore searched not only for the object, but also for the reflection or an image of the reflection. If the presence of the reflection is detected, this is evaluated in order to detect the obstacle. It is preferably provided to determine a movement speed of the obstacle based on the reflection, in particular by evaluating images taken at a time interval which are contained in the image data.

[0014] The reflection can, for example, be reflected by the object or by infrastructure elements in the vicinity of the vehicle. Infrastructure elements are understood to be any elements in the surrounding area, such as a roadway, a warning beacon, a guardrail, a sign, a tunnel wall, a building, and / or vegetation. Particularly in situations with poor visibility, such as in the dark and / or during precipitation, the reflection can be clearly identified in the image data, allowing the presence of the obstacle to be reliably determined, especially under these conditions.

[0015] Preferably, the driver assistance system's distance control is not based solely on the distance from the object, but also on the distance of the vehicle from the obstacle and / or the speed of the obstacle. This enables extremely reliable driving of the vehicle, as the driver assistance system can react and does react very early, even to hidden obstacles. This allows all requirements of ECE Regulation 157 – also known as UNECE R157 – to be met.

[0016] A further development of the invention provides that an obstacle illuminated by ambient light and / or a self-illuminating obstacle is used as the obstacle. The obstacle therefore illuminates passively, actively, or both passively and actively. In the former case, it is illuminated by ambient light which is reflected by the obstacle. The reflected light is reflected again to generate the reflection, in particular by one of the infrastructure elements, and then detected by the environment detection device. In the case of a self-illuminating obstacle, the obstacle has a light, in particular a rear light or tail light, for example a brake light or the like. Regardless of the type of obstacle, the obstacle generates the reflection, which is detected by the environment detection device and included in the image data. The reflection is then evaluated to detect the presence of the obstacle.This achieves the advantages already mentioned.

[0017] A further development of the invention provides that the image data contains a plurality of temporally successive images captured by the environment detection device, wherein the examination for the object is carried out using at least a first of the images, and the detection of the obstacle is carried out by comparing the at least one first image with at least a second of the images. The image data comprises the plurality of images, each of which is composed of a portion of the image data. The images are recorded at temporal intervals by the environment detection device and are incorporated into the image data.

[0018] To detect the presence of an object, the at least one first image is evaluated and examined for the presence of the object. To detect an obstacle, the at least one first image representing the object is compared with the at least one second image. The comparison serves to determine whether the reflection is present in the image data. If the comparison determines that the reflection is present, it is evaluated to detect the presence of the obstacle.

[0019] Wherever this description refers to the at least one first image or the first image, the statements are always equivalent to one another. Statements relating to the at least one first image are always transferable to the first image, and conversely, statements relating to the first image are always transferable to the at least one first image. The same applies to the at least one second image and the second image. Here, too, the statements are always to be regarded as equivalent to one another.

[0020] For example, the first image is part of several first images and / or the second image is part of several second images. Thus, any number of first images can be used to examine for the presence of the object, and correspondingly any number of second images can be used to detect the presence of the reflection and, accordingly, the obstacle. The number of first images and second images is arbitrary in each case. By dividing the image data into the at least one first image and the at least one second image, which are used to detect the presence of the object and to detect the presence of the reflection and thus the obstacle, a high detection quality is achieved.

[0021] A further development of the invention provides that a set of object properties of the object is determined based on at least one of the images by means of machine learning and is taken into account when comparing the at least one first image with the at least one second image. The set of object properties of the object describes at least one property of the object. The set of object properties can thus comprise precisely one property of the object or multiple properties of the object. The term "set of object properties" is therefore used synonymously for the at least one property of the object.

[0022] The object property of the object preferably describes a state of the object, for example, whether the object's lighting is activated or deactivated, or whether the object's lighting is fully or only partially functional. This allows the comparison of the first image with the second image to take into account whether a light source contained in the image data can be assigned to the object or not. Accordingly, a precise distinction can be made between the object and the reflection. For example, the brightness of a taillight and / or license plate light or the like can be taken into account.

[0023] The object's property set is created through machine learning. Machine learning is preferably performed using a neural network, which receives the image data as input. The neural network provides the object property set, consisting of one or more object properties, as output data. The object property set is used in the comparison between the first image and the second image to improve the quality of the comparison. This further improves obstacle detection.

[0024] A further development of the invention provides that a set of motor vehicle properties is determined based on at least one operating parameter of the motor vehicle and is used when evaluating the image data to detect the obstacle. The same fundamentally applies to the set of motor vehicle properties as to the set of object properties, so reference is made to the corresponding explanations. These can be applied analogously. The set of motor vehicle properties is created based on the at least one operating parameter. The operating parameter is understood to mean, for example, a state of a motor vehicle's lighting, for example, information about whether a headlight of the motor vehicle is activated or deactivated.Using the vehicle property set, influences caused by the vehicle itself, such as reflections from the vehicle's headlights, can be eliminated when evaluating the image data. This also improves the detection quality of the obstacle.

[0025] A further development of the invention provides that at least one obstacle parameter of the obstacle is determined based on the reflection, and at least one driving parameter of the motor vehicle is adjusted using the at least one obstacle parameter. The obstacle parameter is understood to mean, for example, a movement speed and / or a movement direction of the obstacle. The obstacle parameter is determined from the image data based on the reflection, namely by evaluating one or more images contained in the image data.

[0026] For example, the image data is evaluated using machine learning. In particular, the image data, or at least the part of the image data describing the reflection, is fed to a neural network, which uses it to determine the obstacle parameter. Subsequently, the driving parameter of the motor vehicle is selected and adjusted depending on the at least one obstacle parameter. The driving parameter is understood to mean, for example, the driving speed and / or the direction of travel of the motor vehicle. These are selected in particular in such a way that a collision with the obstacle is prevented. Accordingly, safe driving of the motor vehicle is achieved.

[0027] A further development of the invention provides that an optical flow is determined based on the at least one first image and the at least one second image and is used to detect the obstacle when evaluating the image data. Optical flow is a vector field that describes the movement of individual elements, in particular pixels, between the images, i.e., from the first image to the second image. In particular, optical flow is the vector field of the speed of these elements projected into an image plane. It represents movement information of the elements and can therefore be used to detect changes between the first image and the second image.

[0028] In particular, the use of optical flow filters out influences that are contained in several of the first images. The image data of the first image therefore describes, for example, the object that is located in front of the motor vehicle and, in particular, is moving at a similar speed to it. It is assumed that the object and also the motor vehicle are initially so far away from the obstacle that no reflection can be seen in the image data. If the motor vehicle and the object approach the obstacle, the reflection occurs in the image data and can thus be identified by comparing the first image and the second image. In particular, the optical flow from the first image to the second image is determined and evaluated. This enables, in particular, the obstacle parameter of the obstacle to be easily determined.

[0029] A further development of the invention provides for the use of transfer learning and / or incremental learning in machine learning. Transfer learning and incremental learning are methods of machine learning and deep learning, respectively. They use an artificial neural network with multiple intermediate layers (hidden layers) located between an input layer and an output layer.

[0030] Transfer learning reuses existing data, such as the object property set, even if the object depicted in the image data differs from the object for which the object property set was initially created. Despite this, the training time of the neural network is typically significantly reduced. Incremental learning describes a process in which image data is continuously used to train the neural network. It also aims to prevent existing training data from the neural network from being lost over time. Transfer learning, or incremental learning, can improve the efficiency of machine learning.

[0031] The invention further relates to a driver assistance device for a motor vehicle, in particular for implementing the method as described in this description. The driver assistance device is provided and configured to capture image data representing the surroundings of the motor vehicle by means of an optical surroundings detection device and to examine the image data for at least one object present in the surroundings represented by the image data, as well as to evaluate a reflection represented by the image data to detect an obstacle present in the surroundings that is obscured by the object.

[0032] The advantages of such a design of the driver assistance system and such a procedure have already been pointed out. Both the driver assistance system and the method for its operation can be further developed according to the explanations in this description, so reference is made to these in this regard.

[0033] The invention also relates to a computer program product comprising instructions that cause the driver assistance system for the motor vehicle to execute the described method according to the details of this description. Regarding the advantages and possible advantageous developments, reference is made to the entire description.

[0034] The features and feature combinations described in the description, in particular the features and feature combinations described in the following description of the figures and / or shown in the figures, can be used not only in the respective combination specified, but also in other combinations or on their own, without departing from the scope of the invention. Thus, embodiments are also considered to be encompassed by the invention that are not explicitly shown or explained in the description and / or the figures, but which follow from or can be derived from the explained embodiments.

[0035] The invention will be explained in more detail below with reference to the exemplary embodiments shown in the drawings, without limiting the invention. In the drawings: Fig. 1 a schematic representation of a traffic situation in which several motor vehicles are moving one behind the other in one lane of a road, and Fig. 2 a schematic representation of image data and a procedure for evaluating the image data.

[0036] The Fig. 1 shows a traffic situation in which a motor vehicle 1 is moving behind an object 2, also in the form of a motor vehicle, and an obstacle 3, also in the form of a motor vehicle, on a lane of a roadway 4. Motor vehicle 1 is also referred to as the first motor vehicle, object 2 as the second motor vehicle, and obstacle 3 as the third motor vehicle.

[0037] The motor vehicle 1 has a driver assistance device with an environment detection device 5, by means of which the surroundings of the motor vehicle 1 are recorded and converted into image data representing the surroundings. The environment detection device 5 preferably has a camera by means of which light in the visible range is recorded. With the aid of the environment detection device 5, the motor vehicle 1 can detect the presence of the second motor vehicle 2. However, the third motor vehicle 3 is concealed by the second motor vehicle 2, so that it is not directly detected by the environment detection device 5.

[0038] For this reason, the driver assistance system of the motor vehicle 1 is designed to examine the image data supplied by the surroundings detection system 5 not only for the presence of the second motor vehicle 2, but also for reflections 6. If such a reflection 6 is detected, it is used to infer the presence of the third motor vehicle 3. The reflection 6 can be present, for example, on the roadway 4 itself or on an infrastructure element 7, in the exemplary embodiment shown here, on a tree. The reflection 6 is generated, for example, by a rear light 8 of the third motor vehicle 3. However, it can also be generated by light that is merely reflected by the motor vehicle 1. For example, for this purpose, the motor vehicle 1 is illuminated by ambient light or by a headlight 9 of the second motor vehicle 2.

[0039] The detection of the third motor vehicle 3 by the driver assistance system of the first motor vehicle 1 via the detour of reflection 6 has the advantage that the first motor vehicle can carry out its driving operation not only in dependence on the second motor vehicle 2, but also in dependence on the third motor vehicle 3. For example, a driving parameter of the first motor vehicle 1 is set in dependence on a driving speed of the second motor vehicle 2 and a driving speed of the third motor vehicle 3, so that a collision is reliably avoided.

[0040] The Fig.Figure 2 shows, in a top first row, a highly schematic representation of the traffic situation in front of the first motor vehicle 1 at different times t1, t2, t3, and t4. It can be seen that at times t1, t2, and t3, only the second motor vehicle 2 is present or is at a great distance from the third motor vehicle 3. At time t4, the second motor vehicle 2 has approached the third motor vehicle 3 so closely that the reflection 6 of the light from the rear light 8 of the third motor vehicle 3 can be seen on the roadway 4.

[0041] In a second row, image data 10 are shown in a highly schematic manner, representing the respective traffic situation at times t1, t2, t3, and t4. The image data 10 at times t1, t2, and t3 represent several first images 11 and, at time t4, a second image 12. In the first images 11, only the second motor vehicle 2 is recognizable, depending on the respective traffic situation. In the second image 12, the reflection 6 is also recognizable, but not the third motor vehicle 3, since the latter is obscured by the second motor vehicle 2 for the surroundings detection direction 5.

[0042] Furthermore, an analysis of the image data 10 is shown, wherein two of the images 11 and 12 are combined according to the arrows 13 to form difference images 14. Here, the images 11 from times t1 and t2, the images 11 from times t2 and t3, and the images 11 and 12 from times t3 and t4 are each combined to form one of the difference images 14. It is clear that in the first two difference images 14 (seen from the left), the third motor vehicle 3 does not appear or cannot be recognized. Only in the last difference image 14 (far right) is the reflection 6 visible, since this represents the only change compared to the first, preceding images 11.

[0043] It is provided to evaluate the reflection 6, preferably by means of machine learning, and to use it to draw conclusions about the third motor vehicle 3. In addition, it can be provided to determine at least one obstacle parameter of the third motor vehicle 3, for example a movement speed and / or a movement direction of the third motor vehicle 3, from the reflection 6, for example by evaluating the reflection 6 in the image data 10 across several images taken at different times.

[0044] By means of the described procedure, reliable detection of both the second motor vehicle 2 and the third motor vehicle 3 is ensured. Accordingly, the first motor vehicle 1 can be controlled by the driver assistance system with a high degree of reliability and / or a notification can be generated to a driver of the motor vehicle 1 with a high degree of reliability. LIST OF REFERENCE SYMBOLS: 1 motor vehicle 2 objects 3 Obstacle 4 lane 5 Environment detection device 6 Reflection 7 Infrastructure element 8 rear light 9 headlights 10 image data 11 1st picture 12 2nd picture 13 Arrow 14 Difference image QUOTES CONTAINED IN THE DESCRIPTION

[0000] This list of documents submitted by the applicant was generated automatically and is included solely for the convenience of the reader. This list is not part of the German patent or utility model application. The DPMA assumes no liability for any errors or omissions. Cited patent literature

[0000] US 2017 / 0287334 A1

[0002] WO 2022 / 043983 A1

[0003]

Claims

[1] Method for operating a driver assistance device for a motor vehicle (1), characterized by that image data (10) representing an environment of the motor vehicle (1) are recorded by means of an optical environment detection device (5) and examined for at least one object (2) present in the environment and represented by the image data (10), and a reflection (6) represented by the image data (10) is evaluated to detect an obstacle (3) present in the environment and hidden by the object (2). [2] Method according to claim 1, characterized by that an obstacle (3) illuminated by ambient light and / or a self-luminous obstacle (3) is used as the obstacle (3). [3] Method according to one of the preceding claims, characterized byin that the image data (10) contain a plurality of temporally successive images (11, 12) captured by the surroundings detection device (5), wherein the examination for the object (2) is carried out on the basis of at least a first of the images (11, 12) and the detection of the obstacle is carried out by comparing the at least one first image (11) with at least a second of the images (11, 12). [4] Method according to one of the preceding claims, characterized by that a set of object properties of the object (2) is determined on the basis of at least one of the images (11, 12) by means of machine learning and is taken into account in the comparison of the at least one first image (11) with the at least one second image (12). [5] Method according to one of the preceding claims, characterized bythat a set of motor vehicle properties is determined on the basis of at least one operating parameter of the motor vehicle (1) and is used in the evaluation of the image data (10) to detect the obstacle (3). [6] Method according to one of the preceding claims, characterized by that at least one obstacle parameter of the obstacle (3) is determined on the basis of the reflection (6) and at least one driving parameter of the motor vehicle (1) is set using the at least one obstacle parameter. [7] Method according to one of the preceding claims, characterized by that an optical flow is determined on the basis of the at least one first image (11) and the at least one second image (12) and is used in the evaluation of the image data (10) to detect the obstacle (3). [8] Method according to one of the preceding claims, characterized by that machine learning uses transfer learning and / or incremental learning. [9] Driver assistance device for a motor vehicle (1), in particular for carrying out the method according to one or more of the preceding claims, characterized by that the driver assistance device is provided and designed to capture image data (10) representing an environment of the motor vehicle (1) by means of an optical environment detection device (5) and to examine it for at least one object (2) present in the environment and represented by the image data, and to evaluate a reflection (6) represented by the image data (10) in order to detect an obstacle (3) present in the environment and concealed by the object (2). [10] Computer program product comprising instructions which cause the driver assistance device according to claim 9 to carry out the method according to one or more of claims 1 to 8.

Images