Method for representing the environment of a vehicle, driver assistance system and vehicle

Abstract

Method for displaying the environment of a vehicle (1), wherein image data depicting the environment of the vehicle (1) are acquired by means of an image acquisition system (12) of the vehicle (1) comprising at least one image acquisition device (3), wherein the image data comprise first image data in the visible spectrum and second image data in the infrared spectrum, wherein an environment display (13) is generated from the first image data and displayed on a display device (9) of the vehicle, characterized by that the second image data regarding the presence of one or several heat-emitting objects (14) in the environment are evaluated, wherein detected heat-emitting objects (14) are highlighted in the environment representation (13) by means of at least one highlighting measure.

Description

[0001] The invention relates to a method for displaying the surroundings of a vehicle, wherein image data depicting the surroundings of the vehicle are recorded by means of an image recording system of the vehicle comprising at least one image recording device. The image data comprises first image data in the visible spectrum and second image data in the infrared spectrum. A representation of the surroundings is generated from the first image data and displayed on a display device of the vehicle. The invention further relates to a driver assistance system and a vehicle.

[0002] Modern vehicles typically feature driver assistance systems that allow the driver to view areas that are difficult or obscured from the driver's seat. This can include cameras whose images are displayed on a monitor inside the vehicle. In addition to facilitating vehicle maneuvering at low speeds, such systems are increasingly being used to monitor surrounding traffic at higher speeds, thus playing a fundamentally important role in the safety of other road users.

[0003] The cameras used for this purpose typically include image sensors capable of capturing light in the visible spectrum. It is known from the prior art to use infrared cameras, which can capture light in the infrared spectrum, in addition or as an alternative.

[0004] DE 10 2010 026 223 A1 describes a vehicle with multiple image capture units for capturing an area located to the side of and behind the vehicle. The image capture units each comprise at least one infrared camera configured to capture near and / or far infrared radiation. In order to be able to display optimized image data of the vehicle's surroundings even in good visibility conditions, such as daylight, the image capture units also comprise additional optical sensors or cameras.

[0005] DE 10 2012 022 150 A1 discloses a motor vehicle with a thermal imaging camera and an image processing device designed to detect objects in a thermal image from the thermal imaging camera. At least one object property of the object is determined by evaluating the heat distribution in the thermal image. The object property is then used during operation of a vehicle system of the motor vehicle.

[0006] In order to optimise the use of image capture devices on a vehicle, it is desirable for a driver to be able to perceive the displayed image content and thus also the vehicle surroundings as intuitively as possible.

[0007] The invention is therefore based on the object of specifying an improved method for displaying the surroundings of a vehicle, which in particular increases the visual detectability of relevant objects in the vehicle environment.

[0008] To achieve this object, it is provided according to the invention in a method of the type mentioned at the outset that the second image data are evaluated with regard to the presence of one or more heat-emitting objects in the environment, wherein detected heat-emitting objects are highlighted in the environment representation by means of at least one highlighting measure.

[0009] By means of the image recording system or the one or more image recording devices of the image recording system, image data is acquired which depicts at least a partial region of the vehicle's surroundings. The image recording device or the multiple image recording devices of the image recording system, which can each be embodied as a camera, for example, are designed to generate first image data which is generated from light in the visible spectrum, and second image data which is formed from light in the infrared spectrum. The first image data and the second image data can be generated by a single image recording device or by multiple image recording devices. The second image data is generated using infrared radiation, which is also referred to herein as light in the infrared spectrum, wherein the infrared radiation can be near infrared radiation and / or far infrared radiation.

[0010] A representation of the environment is generated from the first image data and shown on the display device of the vehicle. The display device can be arranged in an interior of the vehicle so that it can be viewed by a driver or a user of the vehicle. Since the representation of the environment is generated on the basis of the first image data, it corresponds at least substantially to the human perception of light in the visible spectrum. The representation of the environment makes it possible for the driver of the vehicle to view parts of the vehicle's surroundings on the display device without, for example, having to turn their head. In addition, it is possible for the representation of the environment to include areas of the vehicle's surroundings that cannot be seen from the driver's seat of the vehicle, for example because they are too close to the vehicle body and / or are obscured by sections of the body.

[0011] The second image data are evaluated for the presence of one or more heat-emitting objects in the environment. A heat-emitting object can be detected, in particular, by the fact that it emits a particularly high level of infrared radiation or infrared light. The heat-emitting objects can be detected using machine vision, i.e., by a computer-based evaluation of the image content of at least the second image data. It is possible, for example, to differentiate between different types of objects based on their outer contour and / or other object properties recognizable in the second image data.

[0012] Heat-emitting objects detected in the second image data are highlighted in the environment representation using at least one highlighting measure. In other words, the heat-emitting objects detected based on the second image data are highlighted in the environment representation generated using the first image data.

[0013] This advantageously allows heat-emitting objects to be specifically highlighted in situations where the vehicle's surroundings are inadequately represented by the initial image data, allowing them to be more easily recognized in the surroundings display. Such situations can occur, for example, at dusk or in other poor lighting conditions, which result in particularly poor illumination of the surroundings.

[0014] Highlighting objects in the environment display has the advantage that their presence is easy and intuitive for the driver to recognize. Negative aspects, such as driver overload due to complex or hidden content, which can arise, for example, if both the first image data and the second image data are displayed side by side, can thus be avoided. The intuitive recognizability of objects also advantageously reduces the driver's reaction time if the presence of the object requires a specific driver response, such as evasive maneuvering or braking.

[0015] Compared to using an environmental view based purely on thermal or infrared images, which is used, for example, in hunting or defense applications, the method according to the invention has the advantage that a natural, true-color image of the environmental view can be generated, for example, in daylight. Furthermore, switching between different views is not necessary, since the highlighting ensures that heat-emitting objects are always recognizable, even in changing lighting conditions, for example, at dusk or similar.

[0016] Since even with high-resolution image sensors there is generally little advantage over human perception at dusk, the method according to the invention advantageously significantly increases the detectability of heat-emitting objects in the environmental display.

[0017] In a preferred embodiment of the invention, other road users, in particular vulnerable road users, and / or animals, are detected and highlighted as heat-emitting objects. Vulnerable road users, also referred to as VRUs (vulnerable road users), include, in particular, pedestrians, cyclists, scooter riders, and similar road users who are not particularly protected by the design of their vehicle.

[0018] In addition, animals can also be detected as heat-emitting objects, especially at dusk, which may be on or at least near roads. Highlighting such heat-emitting objects in the environment display advantageously increases the safety of these objects, as they can be more easily perceived by the vehicle driver.

[0019] According to the invention, it can be provided that an image recording device is used which is designed on a common recording surface both to record light in the visible spectrum and to record light in the infrared spectrum. The image recording device can, for example, comprise an image sensor which is designed both to record light in the visible spectrum and to record light in the infrared spectrum. In this case, for example, individual pixels can record light in both the visible spectrum and the infrared spectrum, wherein the first image data can be generated from the visible light and the second image data from the infrared light. This advantageously results in a pixel-precise assignment between the first image data and the second image data, which, after the heat-emitting objects in the second image data have been detected, significantly facilitates highlighting the same objects in the first image data.

[0020] In addition, an image acquisition device with a common recording surface makes it possible to maintain the full quality of the natural image produced using the visible spectrum.

[0021] According to the invention, a camera with at least one RGB-IR image sensor can be used as the image recording device. An RGB-IR image sensor refers to an image sensor with a chip in which red, green, and blue light (RGB) as well as infrared light (IR) can be detected per pixel. In this way, pixel-precisely matching first image data from the RGB component and second image data from the IR component can be easily generated. The use of an RGB-IR image sensor has the advantage that it can be easily integrated into existing system architectures of image recording devices or driver assistance systems.

[0022] In a preferred embodiment of the invention, it can be provided that an object type of a detected heat-emitting object is determined by evaluating the first image data and / or the second image data, wherein the highlighting measure is selected from a plurality of highlighting measures depending on the determined object type. The determination of an object type of the object can be carried out analogously to the recognition of the object in the second image data by evaluating the content of the image data using machine vision, for example using methods such as semantic segmentation or the like. The determination of an object type can be carried out based on the second image data, in particular together with the step of detecting the presence of the object. Additionally or alternatively, the image content of the first image data can also be evaluated in order to enable the determination of an object type of a detected object.

[0023] In a preferred embodiment of the invention, it can be provided that, as a highlighting measure, the at least one object in the environment representation is marked with a marker, in particular with a frame and / or with a warning symbol. The frame can be rectangular or designed as a box, for example. Alternatively, a frame formed by highlighting the outline or contour of the respective object is also possible. Other types of markers and / or warning symbols can also be used. The markers can surround the object or be placed next to it.In particular, the size and shape of the frame surrounding the object can be selected depending on the object geometry determined from the second image data, so that the object can be marked as a whole, even if it is not or only partially recognizable in the first image data, for example, due to poor lighting. It is possible to use different markers for different object types, or to use the same markers for all types of detected heat-emitting objects.

[0024] For example, a marker can be inserted directly into the first image data and / or the environmental view generated from it. Alternatively, the one or more markers can also be generated as an overlay, which is displayed together with the environmental view on the display device.

[0025] According to the invention, it can be provided that a highlighting measure is performed by changing the representation of the object, in particular by coloring the at least one object and / or displaying it with increased brightness. The display of the change in the object can be performed in addition to or alternatively to the use of the marking described above.

[0026] By changing the representation of the object in the surroundings view, it can be perceived more easily and quickly by the driver. An object in the surroundings view can be displayed in an incorrect color, for example red or similar, if it was detected in the second image data. A representation of such an object with increased brightness can also be provided, which enables improved perception of the object, particularly in cases of low ambient lighting. The same changes to the representation of the object can be made for all objects, or different changes, in particular different colors, can be used for different object types. In particular, the object can be highlighted true to its outline by changing the representation.The object can also be highlighted more intuitively through the changed display, as this type of highlighting can be perceived as less disturbing than, for example, an additional frame displayed around the object.

[0027] The invention further relates to a driver assistance system for a vehicle, wherein the driver assistance system comprises a control device, a display device and an image recording system with at least one image recording device, wherein the image recording system is designed to record image data of an environment of the vehicle, wherein the image data comprise first image data in the visible spectrum and second image data in the infrared spectrum, wherein the control device is designed to generate an environment representation from the first image data and to display it on the display device, wherein the control device is further designed to evaluate the second image data with regard to the presence of one or more heat-emitting objects in the environment and to highlight detected heat-emitting objects in the environment representation by means of at least one highlighting measure.

[0028] According to the invention, the image recording system can be or include a surround-view system, a front camera, a rearview camera, and / or an electronic device for rear vision. An electronic device for rear vision can, in particular, be an electronic side mirror or an electronic rearview mirror.

[0029] All advantages and embodiments described above in relation to the method according to the invention apply accordingly to the driver assistance system according to the invention and vice versa.

[0030] In particular, it can be provided according to the invention that the control device is designed to detect and highlight other road users, in particular unprotected road users, and / or animals as heat-emitting objects.

[0031] Additionally or alternatively, the invention may provide for the image recording device to comprise a common recording surface configured to record both light in the visible spectrum and light in the infrared spectrum. According to the invention, the image recording device may be a camera with at least one RGB-IR image sensor.

[0032] According to the invention, the control device can be configured to determine an object type of a detected heat-emitting object by evaluating the first images and / or the second images, wherein the highlighting measure is selected from a plurality of highlighting measures depending on the determined object type.

[0033] According to the invention, the control device can be configured to mark the at least one object in the environment representation with a marker, in particular with a frame and / or with a warning symbol, as a highlighting measure. Additionally or alternatively, the control device can be configured to change a representation of the object as a highlighting measure, in particular by coloring the at least one object and / or by displaying the object with increased brightness.

[0034] Furthermore, the invention relates to a vehicle with a driver assistance system according to the invention. All advantages and configurations of the driver assistance system according to the invention also apply analogously to the vehicle according to the invention, and vice versa. The advantages and configurations of the method according to the invention are also transferable to the vehicle according to the invention, and vice versa.

[0035] Further advantages and details of the invention will become apparent from the exemplary embodiments described below and from the drawings. These are schematic representations and show: Fig. 1 shows an embodiment of a vehicle according to the invention with an embodiment of a driver assistance system according to the invention, Fig. 2 an example of an environmental view generated from first image data and Fig. 3 an example of highlighting an object in the surrounding view to explain an embodiment of a method according to the invention.

[0036] In Fig. Figure 1 shows an embodiment of a vehicle 1. The vehicle 1 can be, for example, a motor vehicle, in particular a passenger car, a truck, or another type of commercial vehicle. It is also possible for the vehicle 1 to be a non-motorized vehicle such as a trailer or a combination consisting of a towing vehicle and a trailer. The vehicle 1 can also be a rail-bound vehicle, such as a tram or the like.

[0037] The vehicle 1 comprises an exemplary embodiment of a driver assistance system 2, which comprises an image recording system 12 with one or more image recording devices 3 and a control device 4. In the present exemplary embodiment, the image recording system 12 is designed as a surround-view system and comprises four image recording devices 3, each of which is designed as a camera. A first camera 5 is arranged as a front camera of the vehicle 1, a second camera 6 as a rear-view camera of the vehicle 1, and a third camera 7 and a fourth camera 8 are each arranged as a side camera of the vehicle 1. The control device 4 can be designed, for example, as a microcontroller, as a processor, or another type of computing device.

[0038] Alternatively, it is possible for the driver assistance system 2 to be a mirror replacement system or a computer monitor system (CMS), which comprises several electronic devices for rearward vision, in particular two electronic side mirrors and / or an electronic rearview mirror. Several of the cameras 3 can be used as a replacement for the mirrors. For example, two side cameras 7, 8 facing to the side and / or rear, which can be arranged in particular in the area of ​​the front doors of the vehicle 1, and / or the rearview camera 6 can be used.

[0039] The driver assistance system 2 further comprises a display device 9, via which graphic information can be presented to a driver or a user of the vehicle 1. The display device 9 can, for example, be one or more screens or displays arranged in an interior of the vehicle 1.

[0040] By means of the image recording devices 3, image data can be generated from a sub-area of ​​the surroundings of the vehicle 1. The image data generated by the image recording devices 3 comprise first image data in the visible spectrum and second image data in the infrared spectrum. In the present exemplary embodiment, each of the image recording devices 3 is designed to generate the first image data and the second image data of the respective environmental sub-area captured by it. For this purpose, the image recording devices 3 each comprise a recording surface 11, which is designed to record both light in the visible spectrum and light in the infrared spectrum.

[0041] The recording surface 11 can, for example, be designed as an RGB-IR image sensor which comprises a multiplicity of pixels for which visible light and infrared light can be detected and evaluated. Because the recording surface 11 can be used jointly for visible light and infrared light, each of the image recording devices 3 can simultaneously generate a first image and a second image, or first image data and second image data. The first image data and the second image data correspond, in particular, with pixel precision, i.e. the same image area is mapped onto a pixel in the first image data and the second image data, so that an assignment of the image content between the first image data and the second image data is easily possible.

[0042] The first and second image data are generated by the image recording devices 3, in particular continuously as a video stream, and transmitted to the control device 4. For this purpose, the image recording devices 3 are connected to the control device 4 via a communication connection 10, for example, a bus system of the vehicle 1 or the like.

[0043] The driver assistance system 2 is configured to perform a method for displaying the surroundings of the vehicle 1. For this purpose, the control device 4 can create a view of the surroundings from the first image data generated by the image recording devices 3 of the image recording system 12. Depending on the configuration of the image recording system 12 or the driver assistance system 2, differently oriented and / or differently sized sections of the vehicle's surroundings 1 can be displayed in the view of the surroundings. The view of the surroundings can also be configured as a panoramic view or as a top view of the vehicle 1 and the immediate surroundings of the vehicle 1.

[0044] In Fig. 2 shows an example of an environmental view 13 generated from first image data. This shows a section of the vehicle's surroundings, which is displayed, for example, when looking into an electronic side mirror on a corresponding display device 9. The environmental view 13 can be generated, for example, from the first image data of an image recording device 3 arranged on the side of the vehicle, for example one of the cameras 7 and / or 8. Fig. The environmental view 13 shown in Figure 2 represents an environmental view which was derived only from the first image data, ie from the image data generated by recording the visible spectrum.

[0045] To make it easier for a driver of vehicle 1 to detect heat-emitting objects 14 in the surroundings of vehicle 1, control device 4 searches the second image data, which are also generated by image recording devices 3 of image recording system 12, for one or more heat-emitting objects in the surroundings of vehicle 1. For this purpose, the content of the second image data can be evaluated by control device 4.

[0046] Due to the use of the common recording surface 11 in the image recording device 3, the image content of the second image data corresponds to the image content of the first image data as shown in Fig. 2, however, instead of visible light in the second image data, the image content is mapped based on infrared light. The heat-emitting object 14 can, for example, be a pedestrian 15. Furthermore, it is possible that other road users, in particular unprotected road users, and / or animals are also detected as heat-emitting objects 14.

[0047] The control device 4 is configured to detect heat-emitting objects 14, such as the pedestrian 15, in the second image data. For this purpose, the control device 4 can evaluate the content of the second image data, in particular using machine vision methods. The detection of the heat-emitting objects 14 can be achieved, for example, using semantic segmentation or other methods capable of identifying individual objects, in particular other road users, as well as the position of these objects in the image data. In particular, the detection can take into account the fact that heat-emitting objects 14 can be detected by increased infrared light emission, which is expressed by an increased brightness of the object in the second image data.

[0048] If one or more heat-emitting objects are detected in the second image data, they are highlighted in the surrounding view 13 generated from the first image data using a highlighting measure. The highlighting measure allows the user to more quickly recognize the presence of the object 14 in the surrounding view 13. This advantageously also allows the driver of the vehicle 1 to react more quickly to the presence of the object 14, should a driver response be required.

[0049] The object 14 can be marked in the environment representation 13 by means of different highlighting measures, wherein for each object 14 one highlighting measure or any combination of several of the highlighting measures described below can be used.

[0050] Fig. 3 shows an embodiment of an environmental view 13 in which the object 14 or the pedestrian 15 has been highlighted by means of several highlighting measures. As a first highlighting measure, the pedestrian 15 is surrounded by a frame 16, wherein the frame 16 has a rectangular shape or is designed as a box. The size and shape of the frame 16 can be selected depending on the object geometry determined from the second image data, so that the object 14 can be marked as a whole, even if it is not or only partially recognizable in the first image data, for example due to poor lighting. Additionally or alternatively, a frame 16 can also be generated by highlighting a contour 17 of the object 14. This is shown in Fig. 3 is schematically represented by the higher line thickness of the outline of the object 14.

[0051] Additionally or alternatively, as a second highlighting measure, a warning symbol 18 can also be displayed, for example, near the object 14 or partially on the object 14. As a further highlighting measure, the object 14 can be displayed in the surroundings view 13 in a modified manner, i.e., changed compared to the representation of the object 14 in the first image data. For this purpose, the object 14 can, for example, be colored so that it appears in a different color than usual, in particular a signal color. The coloring of the object 13 can, for example, be done by inserting a colored, partially transparent overlay in the surroundings view 13. It is also possible for the object 14 to be displayed with a changed brightness, for example, brighter than in the first images, as a highlighting measure.In this way, the recognizability of the object 14 can be improved, particularly in poor lighting conditions in the surroundings of the vehicle 1. Advantageously, sections or parts of the object 14 that cannot be distinguished from a background or surroundings of the object 14 in the first image data can also be displayed as belonging to the object 14.

[0052] The highlighting measures can be the same for all types of detected objects 14, or highlighting measures assigned to different object types can be selected from a plurality of predefined highlighting measures. The object type of an object 14 can also be determined by the control device 4 using machine vision methods.

[0053] When using type-dependent highlighting measures, object types such as pedestrians, cyclists, animals, and the like can each be provided with differently colored and / or differently shaped frames 16 or warning signs 18. Additionally or alternatively, the type of representation can also be changed differently for the different object types, so that they are displayed in the environment view 13 with a different brightness and / or a different color representation depending on the object type.

[0054] The method for generating an environmental view can be used for various types of environmental views. In addition to the image of an electric side mirror shown here as an example, which was generated using the image recording system 12 of the vehicle 1 designed as a surround view system, other vehicle environmental views can also be generated analogously. For example, the image recording system 12 can only be a front camera, only a rear view camera and / or an electric rearview mirror and / or one or more electric side mirrors. In particular, it is possible for the image recording system 2 to use fewer than the four in Fig.1, wherein, for example, only image data from the rear-view camera 5 and / or the front camera 6 are processed to form the surrounding view 13. The number of screens of the display device 9 and the arrangement of the screens in the interior of the vehicle 1 can be based on the functions provided by the image recording system 2.

[0055] In addition to the design of the image recording device 3 with the recording surface 11 used jointly for the visible light and the infrared light, image recording devices 3 can also be used that generate the first image data and the second image data separately. In such a case, by interpreting the image content of the first image data and the second image data by the control device 4, an assignment of the objects 14 to be highlighted in the surrounding view 13 can be determined even if there is no pixel-precise match between the first image data and the second image data. QUOTES CONTAINED IN THE DESCRIPTION

[0000] This list of documents submitted by the applicant was generated automatically and is included solely for the convenience of the reader. This list is not part of the German patent or utility model application. The DPMA assumes no liability for any errors or omissions. Cited patent literature

[0000] DE 10 2010 026 223 A1

[0004] DE 10 2012 022 150 A1

[0005]

Claims

[1] Method for displaying the surroundings of a vehicle (1), wherein image data depicting the surroundings of the vehicle (1) are recorded by means of an image recording system (12) of the vehicle (1) comprising at least one image recording device (3), wherein the image data comprise first image data in the visible spectrum and second image data in the infrared spectrum, wherein an environment representation (13) is generated from the first image data and displayed on a display device (9) of the vehicle, characterized by that the second image data are evaluated with regard to the presence of one or more heat-emitting objects (14) in the environment, wherein detected heat-emitting objects (14) are highlighted in the environment representation (13) by means of at least one highlighting measure. [2] Method according to claim 1, characterized bythat other road users, in particular unprotected road users, and / or animals are detected and highlighted as heat-emitting objects (14). [3] Method according to claim 1 or 2, characterized by that an image recording device (3) is used which is designed on a common recording surface (11) both for recording light in the visible spectrum and for recording light in the infrared spectrum. [4] Method according to claim 3, characterized by that a camera (4 - 7) with at least one RGB-IR image sensor is used as the image recording device (3). [5] Method according to one of the preceding claims, characterized by that an object type of a detected heat-emitting object (14) is determined by evaluating the first image data and / or the second image data, wherein the highlighting measure is selected from a plurality of highlighting measures depending on the determined object type. [6] Method according to one of the preceding claims, characterized by that as a highlighting measure, the at least one object (14) in the environmental representation (13) is marked with a marker, in particular with a frame (16) and / or with a warning symbol (18). [7] Method according to one of the preceding claims, characterized by that as a highlighting measure, a change in a representation of the object (14), in particular a coloring of the at least one object (14) and / or a representation of the at least one object (14) with an increased brightness, takes place. [8] Driver assistance system for a vehicle (1) comprising a control device (4), a display device (9) and an image recording system (12) with at least one image recording device (3), wherein the image recording system (12) is designed to record image data of an environment of the vehicle (1), wherein the image data comprise first image data in the visible spectrum and second image data in the infrared spectrum, wherein the control device (4) is designed to generate an environment representation (13) from the first image data and to display it on the display device (9), wherein the control device (4) is further designed to evaluate the second image data with regard to the presence of one or more heat-emitting objects (14) in the environment and to highlight detected heat-emitting objects (14) in the environment representation (13) by means of at least one highlighting measure. [9] Driver assistance system according to claim 8, characterized by that the image recording system (12) is or comprises a surround view system, a front camera, a rear view camera and / or an electronic device for rear vision, in particular an electronic rear view mirror and / or an electronic side mirror. [10] Vehicle comprising a driver assistance system (2) according to claim 8 or 9.

Images