Method for operating a driver assistance device for a motor vehicle, driver assistance device for a motor vehicle and computer program product

Abstract

The invention relates to a method for operating a driver assistance device (2) for a motor vehicle (1), wherein a first image is recorded by means of a first camera (4) using a first color filter matrix (8), and a second image is recorded by means of a second camera (5) using a second color filter matrix (9) having a different color filter arrangement than the first color filter matrix (8). It is provided that color information of the first image is adapted using color information of the second image, and that image content is created from the color-adapted first image and displayed by means of a display device (16). The invention further relates to a driver assistance device (2) for a motor vehicle (1) and to a computer program product.

Description

[0001] The invention relates to a method for operating a driver assistance device for a motor vehicle, wherein a first image is recorded by a first camera using a first color filter matrix, and a second image is recorded by a second camera using a second color filter matrix having a different color filter arrangement than the first color filter matrix. The invention further relates to a driver assistance device for a motor vehicle and a computer program product.

[0002] For example, the prior art document DE 10 2013 102 207 A1 is known. This document describes a camera system for a vehicle for image capture for one or more driver assistance systems, comprising a first camera module and a second camera module. The first camera module and the second camera module each comprise at least one imaging system and one image recording element, and the first camera module and the second camera module serve to implement at least two different image capture functions.

[0003] Furthermore, US 2019 / 0191127 A1 discloses a vehicle visualization system with a plurality of cameras, including a forward-view camera and several color cameras, as well as with a display device that can display video images derived from the image data captured by the color cameras. A processing unit comprises a first processing chip having an image processor for machine processing the captured image data, and a second processing unit that receives vehicle data and image data captured by the color cameras. The first processing chip processes the image data captured by the cameras for object detection and object classification. The first processing chip controls the operating parameters of the color cameras to improve object detection based on the machine processing of the image data captured by the color cameras by the first processing chip.The second processing chip controls the operating parameters of the color cameras for displaying video images on the display device.

[0004] Document DE 11 2017 007 579 T5 discloses a method and a system for generating a first color image from a first night vision image. First, predicted chrominance values ​​are determined using a prediction function based on luminance values ​​of the first night vision image. Then, the first color image is generated by combining the luminance values ​​of the first night vision image with the predicted chrominance values.

[0005] Furthermore, the prior art documents DE 10 2014 213 397 A1 and DE 10 2022 120 236 B3 are known.

[0006] It is an object of the invention to propose a method for operating a driver assistance device for a motor vehicle, which has advantages over known methods, in particular provides a more comprehensive visualization of an environment of the motor vehicle for a driver of the motor vehicle.

[0007] This is achieved according to the invention with a method for operating a driver assistance device for a motor vehicle with the features of claim 1. In this case, it is provided that color information of the first image is adapted using color information of the second image and an image content is created from the color-adapted first image and displayed by means of a display device, wherein when the image content is created, at least part of the second image is superimposed into the first image and / or the first image is supplemented by at least part of the second image such that the first image is enlarged by means of the second image, wherein a camera which has a different orientation than the second camera is used as the first camera, wherein a field of view of the first camera lies at least partially in a field of view of the second camera.

[0008] Advantageous embodiments with useful further developments of the invention are specified in the dependent claims. It should be noted that the exemplary embodiments explained in the description are not limiting; rather, any variations of the features disclosed in the description, the claims, and the figures are feasible.

[0009] The method serves to operate the driver assistance device. The driver assistance device is preferably a component of the motor vehicle, but can of course also be present separately. For example, the driver assistance device is implemented by means of a control unit of the motor vehicle or it comprises the control unit. The driver assistance device fundamentally serves to support a driver of the motor vehicle when carrying out a driving operation of the motor vehicle. During driving operation, the motor vehicle moves at least temporarily at a speed other than zero, in particular in its forward or reverse direction.

[0010] The driver assistance system includes, for example, an adaptive cruise control system, which adjusts the speed of the motor vehicle to maintain a defined distance between the motor vehicle and an object located in front of the motor vehicle. The object is, for example, another motor vehicle, i.e., a motor vehicle traveling in front of the motor vehicle, a trailer of another motor vehicle, or the like. The driver assistance system is preferably designed and configured for assisted driving at least according to Level 1 or at least according to Level 2, or for automated or autonomous driving at least according to Level 3 according to SAE Standard J3016.

[0011] The driver assistance device has an optical environment detection device with which it detects the surroundings of the motor vehicle. The optical environment detection device comprises at least the first camera and the second camera, by means of which images of the surroundings are recorded. The images are therefore images of the surroundings or contain such images. They are at least partially further processed by the driver assistance device, for example to implement the aforementioned distance control. In particular, the image from the first camera is used for this purpose. This camera is preferably present as the front camera of the motor vehicle and records the surroundings of the motor vehicle viewed forward in the direction of a longitudinal axis of the motor vehicle. The second camera, on the other hand, is used, for example, to visualize the surroundings of the motor vehicle for its driver.In this case, it may be provided that the second image is displayed to the driver at least temporarily by means of the display device.

[0012] The first camera can of course be a component of multiple first cameras; analogously, the second camera can be a component of multiple second cameras. Wherever reference is made to the first camera or the second camera within the scope of this description, the statements are always to be understood as referring to at least one first camera or at least one second camera, respectively. The statements for the first camera are preferably applicable to each of the multiple first cameras, if present. Accordingly, the statements for the second camera are applicable to each of the multiple second cameras, likewise only if present.

[0013] The first camera can basically be any camera. For example, it is designed for visible light, i.e. it detects electromagnetic radiation with a wavelength of 380 nm to 750 nm. Additionally or alternatively, the first camera also detects light in the infrared range, i.e. with a wavelength of more than 750 nm, in particular of at least 780 nm on the one hand and at most 1500 nm, at most 1200 nm or at most 900 nm on the other. The first camera can also be a night vision camera. For this purpose, it has, for example, a residual light amplifier or is designed as such. Alternatively, it is a thermal imaging camera. The second camera, on the other hand, is preferably designed for visible light, in particular exclusively. The first camera and the second camera can differ in this respect with regard to the wavelength ranges they detect. For example, the wavelength ranges overlap but are offset from one another.They can also be directly adjacent to each other or spaced apart from each other.

[0014] The first camera captures the first image using the first color filter matrix, and the second camera captures the second image using the second color filter matrix. Each of the color filter matrices corresponds to an arrangement of color filters, in particular with a regularly repeated color filter pattern. The color filter pattern describes, for example, a 2x2 arrangement of color filters, a 3x3 arrangement of color filters, or a 4x4 arrangement of color filters, which is regularly repeated to generate the respective color filter matrix. Each of the color filter matrices is therefore composed of several repetitions of the color filter pattern.

[0015] One such color filter matrix is ​​a Bayer filter matrix, for example, which consists of one red, two green, and one blue color filter. The color filters within each color filter matrix or in each color filter pattern are preferably the same size, i.e., they have identical areas or surface areas. One of the color filter matrices can be selected such that only transparent color filters are present, so that the respective camera captures the corresponding image as if no color filter were present.

[0016] What is important, however, is that the first color filter matrix of the first camera is different from the second color filter matrix of the second camera and differs from the latter in terms of the color filter arrangement. For example, the first color filter matrix and the second color filter matrix are composed of different color filter patterns. Thus, the first color filter matrix has a first color filter pattern or is composed of multiple repetitions of this first color filter pattern, and the second color filter matrix has the second color filter pattern or is composed of multiple repetitions of this second color filter pattern.

[0017] Accordingly, the images from the cameras differ in terms of their color impression, even if the field of view is (only assumed to be) identical. In particular, the first image does not reproduce the colors present in the recorded scene correctly, or only partially, whereas the color reproduction of the second image recorded by the second camera is more accurate, i.e., has a higher color fidelity. The second image is therefore better suited for reproduction via the display device than the first image, as the driver could be irritated by the distorted color reproduction of the first image. For example, the first image is distinguished from the second image by a better level of detail, in particular because the first camera records the first image with a higher resolution than the second camera records the second image. This means that details in the surroundings can be better recognized using the first image.

[0018] For this reason, the first image is adapted to the second image in terms of its color reproduction and only then is it displayed to the driver with the aid of the display device, namely in the form of the image content created from the first image after its color adjustment. Ultimately, this means that the color information of the second image is used to adapt or correct the color information of the first image. In this case, a structure of the first image is retained; in this respect, the first image is merely colorized, while edges and similar structures contained in the first image, for example, are retained unchanged. In particular, the adaptation of the first image is preferably carried out before demosaicing, in which sensor data from the cameras is interpolated.

[0019] This adjusts the color reproduction of the first image to that of the second image, so that both images produce the same color impression for the driver. Preferably, the color information of the first image is adjusted in such a way that the driver or a human observer can no longer distinguish between the two images, namely the first image and the second image, based on their color reproduction.

[0020] This makes it possible, for example, to display images recorded with different color filter matrices in the same context to the driver, in particular using the display device. This means, in particular, that the two images are displayed side by side or at least in such a way that the driver can visually perceive them simultaneously. Without color adjustment of the first image, the driver would notice the different color reproduction between the images. Under certain circumstances, this could even lead to the driver being unable to identify images of the same object contained in the two images. This could reduce the level of support provided by the driver assistance device for the driver and also the driver's acceptance of the driver assistance device. Such an effect is avoided using the procedure described.

[0021] A further development of the invention provides that the first camera is used with a camera that has a different focal length and / or a different image resolution and / or a different wavelength range than the second camera, wherein a field of view of the first camera lies at least partially within a field of view of the second camera. The two cameras therefore differ not only with regard to the color filter matrix, but also with regard to at least one other camera parameter, in particular with regard to their focal length and / or their orientation and / or their image resolution or sensor resolution.

[0022] For example, the focal length of the first camera is greater than the focal length of the second camera. In particular, one of the cameras has a focal length of no more than 24 mm, no more than 16 mm, no more than 10 mm, or no more than 6 mm. The other camera, on the other hand, has a focal length of no less than 40 mm, no less than 80 mm, or no less than 120 mm. Due to the different focal lengths of the cameras, areas of the environment captured by both cameras are sometimes reproduced with different levels of sharpness in the first image and the second image. For this reason, it is particularly useful to use the first image to create the image content, but to supplement it with the color information from the second image.

[0023] Preferably, additionally or alternatively, the image resolution of the first camera is higher than the image resolution of the second camera, in particular by a factor of at least 1.5, at least 2.0, or at least 2.5. The first image provided by the first camera thus has a higher image resolution and thus greater sharpness than the second image provided by the second camera. Therefore, only the color information from the second camera is used to adjust or correct the colors of the first image.

[0024] The two cameras can have the same orientation, for example, be arranged with parallel optical axes. According to the invention, however, their orientations are different from one another, in particular arranged with optical axes that are angled or skewed towards one another. In any case, however, the fields of view of the cameras overlap; the field of view of the first camera therefore lies at least partially in the field of view of the second camera and vice versa. This ensures that the images at least partially depict the same scene or object, so that the color information can be transferred between them. The procedure described results in a particularly high information content of the images, since they reproduce different, yet overlapping fields of view.

[0025] A further development of the invention provides that a camera used for automatic driver assistance, in particular for the longitudinal and / or lateral guidance of the motor vehicle, is used as the first camera and / or a camera of a visualization system is used as the second camera. The first camera is used to carry out the automatic driver assistance, in which the images recorded by the first camera are evaluated using artificial intelligence, for example to carry out autonomous or semi-autonomous driving of the motor vehicle, wherein the longitudinal or lateral guidance of the motor vehicle is carried out on the basis of the first image recorded by the first camera. For example, the first image from the first camera is fed to an artificial neural network in order to carry out object recognition, for example recognition of a road user, a traffic sign, a traffic light or the like.Automatic driver assistance is used to carry out assisted or automated driving of the motor vehicle.

[0026] Implementing longitudinal guidance of the motor vehicle includes, in particular, setting the driving speed of the motor vehicle, in particular by accelerating and / or decelerating the motor vehicle. Lateral guidance includes steering the motor vehicle. Longitudinal guidance occurs, for example, within the framework of the aforementioned distance control using adaptive cruise control. Lateral guidance is preferably performed within the framework of a lane change assistant. In any case, the first image is used for longitudinal and / or lateral guidance, or the first image is evaluated for this purpose.

[0027] Additionally or alternatively, the second camera is a component of the visualization system, for example a top-view camera system. This is understood to be a camera system by means of which an area surrounding the motor vehicle is recorded, preferably in close-up, and displayed to the driver. In particular, the camera system comprises a plurality of second cameras which together record an area surrounding the motor vehicle, preferably continuously and uninterruptedly in at least one imaginary plane. The second images recorded by the plurality of second cameras of the camera system are combined with one another and displayed to the driver of the motor vehicle, at least temporarily. The procedure described results in particularly high flexibility and comprehensive support for the driver.

[0028] The invention provides that, when creating the image content, at least a portion of the second image is superimposed into the first image and / or the first image is supplemented by at least a portion of the second image. For example, it is provided that the portion of the second image or even the entire second image replaces a portion of the first image, i.e., the second image is at least partially superimposed into the first image. However, it can also be provided that the first image is supplemented using at least a portion of the second image. In this case, the first image is enlarged using the second image.

[0029] For example, it is provided to convert both the first image and the second image into a bird's-eye view of the motor vehicle and to overlay the second image onto the first image at this time or subsequently, or to combine the image content from the images converted in this way. Thus, the resulting image content, which is subsequently displayed by the display device, is composed, for example, of the second image in the near field of the motor vehicle and of the first image in the far field of the motor vehicle. Alternatively, the image content can of course only comprise the first image, which is color-adjusted using the color information of the second image. In any case, particularly comprehensive information for the driver is ensured.

[0030] A further development of the invention provides that a false-color filter matrix is ​​used as the first color filter matrix, which has at least one clear filter or is present as an RCCC filter matrix or RYYCy filter matrix. The first color filter matrix is ​​therefore not designed to reproduce the surroundings of the motor vehicle in true colors, but rather distorts the colors or does not capture them at all. In order to achieve particularly reliable machine evaluation of the first image recorded by the first camera, the first color filter matrix has at least one clear filter. This means that the color filter pattern has the clear filter, which in this respect can be found, for example, multiple times in the first color filter matrix.

[0031] One such color filter matrix is ​​the RCCC filter matrix, which features a 2x2 color filter pattern with a red filter and three clear filters arranged in a repeated pattern. The RYYCy filter matrix can also be used. This has a 2x2 color filter pattern with a red filter, two yellow filters, and a cyan filter. This type of color filter matrix allows for good evaluation of the first image, making it particularly suitable for automated driver assistance, particularly for longitudinal and / or lateral guidance of the vehicle.

[0032] A further development of the invention provides that a true-color filter matrix, in particular a Bayer filter matrix, is used as the second color filter matrix. The second color filter matrix is ​​thus designed to reproduce the colors present in the surroundings of the motor vehicle at least approximately correctly, i.e., to create an image of the environment that is as perceptually true as possible in terms of color impression. The true-color filter matrix preferably has at least one red filter, one green filter, and one blue filter. For example, the Bayer filter matrix is ​​used, which consists of one red filter, two green filters, and one blue filter, arranged in the aforementioned 2x2 color filter pattern.

[0033] Other filter matrices can, of course, also be used, as long as they ensure at least approximately true color reproduction. Examples of such filter matrices include an RGBC filter matrix consisting of a red filter, a green filter, a blue filter, and a clear filter, and an RGBCy filter consisting of a red filter, a green filter, a blue filter, and a cyan filter. The described procedure ensures high color fidelity of the second image and thus enables high-quality color adjustment of the first image.

[0034] A further development of the invention provides that, taking into account the field of view of the first camera and the field of view of the second camera, at least one first pixel of the first image is assigned to at least one second pixel of the second image, and the color information of the at least one first pixel is adapted using the color information of the at least one second pixel. The first image is composed of a plurality of first pixels, and the second image is composed of a plurality of second pixels. If, in the context of this description, reference is made to the first pixel or to the at least one first pixel, the statements are always equivalent to one another. The statements for the first pixel or the at least one first pixel are also preferably transferable to each of the plurality of first pixels of the first image.The same applies to the second pixel or the at least one second pixel and the plurality of second pixels contained in the second image.

[0035] Preferably, each of the first pixels of the first image is assigned to one of the second pixels of the second image and / or vice versa, namely taking into account the fields of view of the cameras, which result in particular from the focal lengths and / or the orientations of the cameras. Preferably, the image resolution of the cameras is also taken into account. For example, the assignment of the first pixel to the second pixel, or of the first pixels to the second pixels, is performed during a calibration of the cameras and stored in the driver assistance device. The calibration can be performed either as part of a static calibration or as a dynamic calibration.

[0036] Based on the assignment, the color information of the second pixel or pixels is transferred to the first pixel or pixels. For example, the color intensity of the first pixel and / or brightness information of the first pixel is retained or at least taken into account. This avoids a loss of quality or a loss of sharpness of the first image due to the color adjustment using the second image. The described procedure provides a simple procedure for achieving high-quality images that can be carried out with little computing power. This includes, in particular, the projection of color information from the lower-resolution second image into the higher-resolution first image, which is optimized in particular for artificial intelligence methods.

[0037] A further development of the invention provides that the color information of the first image is adjusted using a neural network to which the second image is fed as a context data set. Methods for colorizing a false-color image, for example, a black-and-white image, using artificial intelligence are generally known, so they will not be discussed in detail. In any case, the color adjustment of the first image is performed using the neural network, taking the second image into account.

[0038] The second image is preferably fed to the neural network as a context data set in order to achieve particularly high color fidelity when adjusting the color of the first image. For this purpose, the neural network assigns a context window, the contents of which it uses to adjust the color information of the first image. The second image is fed to this context window so that it is subsequently available as context for the neural network when adjusting the color information of the first image. The neural network therefore bases the color adjustment of the first image on the second image or its color information. This ensures that the first image is adjusted with particularly high color fidelity.

[0039] The invention further relates to a driver assistance device for a motor vehicle, in particular for carrying out the method according to the explanations in the context of this description, wherein the driver assistance device is provided and designed to record a first image by means of a first camera using a first color filter matrix and to record a second image by means of a second camera using a second color filter matrix having a different color filter arrangement than the first color filter matrix.The driver assistance device is further provided and designed to adapt color information of the first image using color information of the second image and to create an image content from the color-adapted first image and to display it by means of a display device, wherein when creating the image content at least a part of the second image is superimposed into the first image and / or the first image is supplemented by at least a part of the second image such that the first image is enlarged by means of the second image, wherein a camera which has a different orientation than the second camera is used as the first camera, wherein a field of view of the first camera lies at least partially in a field of view of the second camera.

[0040] The advantages of such a design of the driver assistance system and such a procedure have already been pointed out. Both the driver assistance system and the method for its operation can be further developed according to the explanations in this description, so reference is made to these in this regard.

[0041] The invention also relates to a computer program product comprising instructions that cause the driver assistance system for the motor vehicle to execute the described method according to the details of this description. Regarding the advantages and possible advantageous developments, reference is made to the entire description.

[0042] The features and feature combinations described in the description, in particular the features and feature combinations described in the following description of the figures and / or shown in the figures, can be used not only in the respective combination specified, but also in other combinations or on their own, without departing from the scope of the invention. Thus, embodiments are also considered to be encompassed by the invention that are not explicitly shown or explained in the description and / or the figures, but which follow from or can be derived from the explained embodiments.

[0043] The invention will be explained in more detail below with reference to the exemplary embodiments shown in the drawings, without limiting the invention. Fig. 1 a schematic representation of a motor vehicle with a driver assistance device which detects the surroundings of the motor vehicle by means of an environment detection device.

[0044] The Fig. Figure 1 shows a schematic representation of a motor vehicle 1 with a driver assistance system 2 having an environment detection system 3. A first camera 4 and a second camera 5 are shown purely by way of example in the environment detection system 3, each of which captures an environment 6 of the motor vehicle. The two cameras 4 and 5 are arranged and configured such that they have overlapping fields of view. Accordingly, they each capture an object 7 present in the environment 6. Purely by way of example, object 7 is shown here as a vegetation object, more precisely, as a tree.

[0045] Indicated purely schematically is the fact that the first camera 4 captures the surroundings 6 using a first color filter matrix 8 and the second camera 5 captures it using a second color filter matrix 9. It can be seen that each of the color filter matrices 8 and 9 is composed of several color filters 10, 11, 12, and 13, which are arranged in a respective color filter pattern 14 and 15, respectively, which is repeated throughout to form the respective color filter matrix 8 and 9, respectively.

[0046] The first color filter matrix 8 is configured as an RYYCy filter matrix, and its color filter pattern 14 is composed of a red filter, two yellow filters, and a cyan filter. Color filter 10 is configured as a red filter, color filters 11 and 12 as yellow filters, and color filter 13 as a cyan filter. The second color filter matrix 9 is configured as a Bayer filter matrix and thus has a red filter, two green filters, and a blue filter, which comprise the color filter pattern 15. Color filter 10 is configured as a red filter, color filters 11 and 12 as green filters, and color filter 13 as a blue filter.

[0047] A first image is captured by the first camera 4, and a second image is captured by the second camera 5. The color information of the first image is adapted using color information of the second image. From this color-adapted first image, an image content is created, which is then displayed by a display device 16 of the motor vehicle 1. For example, the image content corresponds directly to the color-adapted first image. However, the perspective of the color-adapted first image can also be adapted, for example, by converting it into a bird's-eye view of the motor vehicle 1.

[0048] The described procedure achieves a true-color reproduction of the first image, or at least of the image content created based on the first image. Particularly preferably, the image content is composed of the first image and the second image, so that the driver is provided with a maximum amount of information visually. This increases the acceptance of the driver assistance system 2 by the driver of the motor vehicle 1. LIST OF REFERENCE SYMBOLS: 1 motor vehicle 2 Driver assistance system 3 Environment detection device 4 1. Camera 5 2. Camera 6 Environment 7 Object 8 1. Color filter matrix 9 2. Color filter matrix 10 color filters 11 color filters 12 color filters 13 color filters 14 color filter patterns 15 color filter patterns 16 Display device

Claims

[1] Method for operating a driver assistance device (2) for a motor vehicle (1), wherein a first image is recorded by means of a first camera (4) using a first color filter matrix (8) and a second image is recorded by means of a second camera (5) using a second color filter matrix (9) having a different color filter arrangement than the first color filter matrix (8), wherein color information of the first image is adapted using color information of the second image and an image content is created from the color-adjusted first image and displayed by means of a display device (16), wherein, when creating the image content, at least a part of the second image is superimposed into the first image and / or the first image is supplemented by at least a part of the second image such that the first image is enlarged by means of the second image, wherein a camera is used as the first camera (4),which has a different orientation than the second camera (5), wherein a field of view of the first camera (4) lies at least partially in a field of view of the second camera (5). [2] Method according to claim 1, characterized by that a camera is used as the first camera (4) which has a different focal length and / or a different image resolution and / or a different wavelength range than the second camera (5), wherein a field of view of the first camera (4) lies at least partially in a field of view of the second camera (5). [3] Method according to one of the preceding claims, characterized by that a camera used for automatic driver assistance is used as the first camera (4) and / or a camera of a visualization system is used as the second camera (5). [4] Method according to one of the preceding claims, characterized bythat a false color filter matrix is ​​used as the first color filter matrix (8), which has at least one clear filter or is in the form of an RCCC filter matrix or RYYCy filter matrix. [5] Method according to one of the preceding claims, characterized by that a true color filter matrix is ​​used as the second color filter matrix (9). [6] Method according to one of the preceding claims, characterized by that, taking into account the field of view of the first camera (4) and the field of view of the second camera (5), at least a first pixel of the first image is assigned to at least one second pixel of the second image and the color information of the at least one first pixel is adapted using the colors of at least one second pixel. [7] Method according to one of the preceding claims, characterized bythat the color information of the first image is adjusted using a neural network to which the second image is fed as a context data set. [8] Driver assistance device (2) for a motor vehicle (1), in particular for carrying out the method according to one or more of the preceding claims, wherein the driver assistance device (2) is provided and designed to record a first image using a first color filter matrix by means of a first camera (4) and a second image using a second color filter matrix (9) having a different color filter arrangement than the first color filter matrix by means of a second camera (5), wherein the driver assistance device (2) is further provided and designed to adapt color information of the first image using color information of the second image and to create an image content from the color-adapted first image and to display it by means of a display device (16),wherein, when creating the image content, at least a part of the second image is superimposed into the first image and / or the first image is supplemented by at least a part of the second image such that the first image is enlarged by means of the second image, wherein the first camera (4) is a camera which has a different orientation than the second camera (5), wherein a field of view of the first camera (4) lies at least partially in a field of view of the second camera (5). [9] Computer program product comprising instructions which cause the driver assistance device (2) according to claim 8 to carry out the method according to one or more of claims 1 to 7.

Images