Method for generating an image representation, control unit, driver assistance system, vehicle, computer program and data carrier

Abstract

The invention relates to a method for generating an image representation from at least one camera image (12) of at least one camera (3), wherein the at least one camera image (12) is projected onto a virtual projection surface (13) and wherein the image representation is generated from a back-projection image (15) generated with the aid of the projection surface (13), wherein the back-projection image (15) to be generated is processed by means of at least one image processing function comprising a filter core (16), wherein the filter core (16) for processing at least one pixel in the back-projection image (15) is set depending on at least one pixel density assigned to the at least one pixel, wherein the pixel density describes the number of pixels from the camera image (12) apportioned to the at least one pixel of the back-projection image (15).

Description

[0001] Description

[0002] Method for generating an image representation, control unit, driver assistance system, vehicle, computer program and data carrier

[0003] The invention relates to a method for generating an image representation from at least one camera image from at least one camera, wherein the at least one camera image is projected onto a virtual projection surface, wherein the image representation is generated from a rear-projection image produced using the projection surface, and wherein the rear-projection image to be generated is processed with at least one image processing function comprising at least one filter core. The invention further relates to a control unit, a driver assistance system, a vehicle, a computer program, and a data carrier.

[0004] Modern vehicles use visual displays generated from one or more camera images to depict, for example, a portion of the vehicle's surroundings. These camera images, typically captured by cameras mounted on the vehicle, can be projected onto a virtual projection surface to create views of the environment from perspectives different from those of the cameras. This allows, for instance, the creation of top-down views of the vehicle's surroundings or perspective views from a vantage point located next to the vehicle.

[0005] To capture as much of the vehicle's surroundings as possible with a camera, cameras with a wide-angle lens, such as a fisheye lens, can be used. These offer a large field of view, for example, 180° or more, so that, for instance, four cameras positioned on different sides of the vehicle can capture the entire area around the vehicle. However, such lenses can exhibit distortions, which can increase, especially towards the edges of the field of view. When the camera images are projected onto a virtual projection surface, this distortion can result in image areas with different resolutions, which can negatively affect the image quality of any resulting visual representation. Various image processing techniques are known from the prior art to improve the image quality of visual representations.

[0006] German patent DE 102020211 896 A1 describes a method for generating an image of the vehicle environment of an ego-vehicle. The ego-vehicle comprises several surround-view cameras for capturing the vehicle environment, and the images provided by the surround-view cameras are used to generate an image of the vehicle environment.

[0007] Surround-view cameras each feature a fisheye lens. The generated image of the vehicle's surroundings is divided into several sub-images, which are then combined using different filter kernels, each with configurable filter kernel values.

[0008] DE 102016209391 A1 describes an adaptive image filtering method. First, a camera captures an image. Then, a filter kernel dependent on the viewing angle is calculated for viewing angles between a first and a second viewing angle. This filter kernel is then used to filter an image area that lies between the first and second viewing angles.

[0009] From EP 3380357 B1, a method and a device for image data processing of an environmental image of a vehicle's surroundings are known. In this method, an environmental image composed of several camera images from the vehicle's cameras is processed with an adaptive filter, whereby an image area belonging to an object contained within the composite environmental image is filtered. The image area belonging to the object is determined by an environmental data model generated using the vehicle's environmental sensors. The invention is based on the objective of providing an improved method for generating an image representation that, in particular, reduces undesirable, filter-based image distortions.

[0010] This problem is solved according to the invention in a method of the type mentioned at the outset by adjusting the filter core for processing at least one pixel in the rear projection image as a function of at least one pixel density assigned to the at least one pixel, wherein the pixel density describes the number of pixels from the camera image that fall on the at least one pixel of the rear projection image.

[0011] The camera can, for example, be a vehicle camera, whereby the image displayed is, in particular, a view of the surroundings depicting at least a partial area of ​​the vehicle's environment. However, the method according to the invention is not limited to applications in vehicles or vehicle cameras. The method according to the invention can also be used with other types of cameras that generate an image display using a virtual projection surface.

[0012] The camera features, in particular, an optic with a distortion that affects at least part of the camera image. According to the invention, at least one camera can be a camera with a wide-angle lens, in particular a fisheye lens. This allows the camera to have a detection range of at least 180° in at least one spatial direction.

[0013] At least one camera image is generated using at least one camera. The camera image captured by the camera, which can also be referred to as the source image, is projected onto a virtual projection surface. The projection surface can, in particular, have a curvature and, for example, the shape of a cylindrical segment or a bowl surrounding the vehicle in a virtual space. Other shapes of the projection surface are also possible. The shape of the projection surface can be fixed or adaptively adjustable. When using a camera mounted on the vehicle, adaptive adjustment of the projection surface can occur, for example, depending on the position and / or geometry of objects in the vehicle's environment detected by the vehicle's environmental sensors.

[0014] By projecting the camera image onto the virtual projection surface, it is possible to derive a rear-projection image whose viewpoint differs from the actual viewpoint of the cameras. In other words, a rear-projection image can be generated from a virtual camera, whereby the position and orientation of the virtual camera in the virtual space can be set essentially arbitrarily and can, in particular, differ from the actual position and orientation of the camera with which the camera image was captured.

[0015] Projecting the camera image onto the projection surface also corrects the image's distortion. For example, a stored mapping rule can define which pixel of the source image is projected to which position on the projection surface. With camera images exhibiting significant distortion, this can result in one or more image areas with higher resolution and one or more other image areas with lower resolution.

[0016] The rear-projection image generated using the projection surface is processed with at least one image processing function that includes a filter kernel. The filter kernel can also be referred to as a kernel or filter core. The filter kernel can be described, for example, as a matrix that is shifted pixel by pixel over the source image, assigning values ​​to individual pixels. These values ​​are calculated from the values ​​of the pixels encompassed by the matrix. The influence of each pixel can be weighted by corresponding coefficients within the filter kernel. Depending on the size of the filter kernel and / or the values ​​of its coefficients, different image processing functions or effects can be generated.

[0017] According to the invention, the filter kernel for processing at least one pixel in the rear projection image is adjusted as a function of at least one pixel density assigned to that at least one pixel. The pixel density describes the number of pixels from the camera image that correspond to the at least one pixel of the rear projection image. In other words, the pixel density of an image or image section can be expressed as the quotient of the number of pixels in the camera image (or the source image) and the number of pixels in the rear projection image.

[0018] As previously described, the number of pixels in the camera image varies across the projection surface due to distortion caused by the camera optics. The number of pixels in the rear-projection image that correspond to a given image section also depends, for example, on the geometry of the projection surface and the position of the virtual camera. Therefore, depending on the display situation, the pixel density can differ for specific sections of the rear-projection image. The pixel density can be specified for each individual pixel of the rear-projection image and can also change, for example, continuously, for neighboring pixels. The pixel density can therefore also be referred to as local pixel density.

[0019] The pixel-density-dependent processing of the rear projection image according to the invention has the advantage that, regardless of the display situation, areas with high pixel density and areas with low pixel density can be processed differently. With processing that depends solely on the properties of the camera lens, the use of the projection surface can lead to suboptimal processing with respect to the generated image display. This can result, for example, in an undesirable, blurry display in areas with low image information density or frame rate, such as in the edge regions. Because the entire rear projection image to be generated is processed, it is unnecessary to split the image and process the resulting sub-images separately. This advantageously prevents the occurrence of undesirable tiling or other artifacts.Block artifacts in the final image rendering are avoided.

[0020] The method according to the invention can in particular

[0021] be a computer-implemented method, which can be implemented, for example, in a control device connected to the at least one camera for receiving the at least one camera image.

[0022] According to the invention, the processing of the at least one pixel with the filter kernel can take place during the projection of the camera image and / or during the generation of the rear projection image. The pixel density of the at least one pixel, and in particular the pixel density assigned to all pixels of the rear projection image, can be determined if the extrinsic and intrinsic parameters of the at least one camera, the geometry of the projection surface, and the position and orientation of the virtual camera are defined. In other words, the pixel density can be determined before the camera image is projected and / or the rear projection image is generated, or before the corresponding calculations are performed.Accordingly, it is advantageously possible to perform image processing directly during the generation of the rear projection image, resulting in efficient and fast image processing, especially for applications that essentially run in real time.

[0023] In a preferred embodiment of the invention, the size of the filter kernel and / or the values ​​of one or more coefficients of the filter kernel can be adjusted depending on the pixel density. Depending on the pixel density of the at least one pixel to be processed, the filter kernel can be implemented, for example, as a square matrix or an nxn matrix of different sizes, e.g., as a 3x3 matrix, 5x5 matrix, 7x7 matrix, 9x9 matrix, etc. It is also possible to implement the filter kernel as matrices of other dimensions. Additionally or alternatively, the values ​​of one or more of the coefficients of the filter kernel can also be adjusted depending on the pixel density. For example, the magnitudes and / or signs of one or more of the coefficients can be changed depending on the pixel density assigned to the at least one pixel to be processed.

[0024] According to the invention, it can be provided that, at a pixel density greater than 1, the filter core is set as a Dirac filter core or as an adaptive low-pass filter core. The size and / or the coefficients of the low-pass filter core can, for example, depend on the magnitude of the respective pixel density.

[0025] Additionally or alternatively, the invention may provide that the filter core is set for adaptive image sharpening when the pixel density is less than 1. This advantageously allows for better detection of edges in areas of the rear projection image where the pixel density is low due to distortion correction, while, for example, a correct reproduction of the camera image (possibly downscaled) is achieved in an image area of ​​the rear projection image corresponding to the center of the camera image.

[0026] In a preferred embodiment of the invention, the filter core can be adjusted depending on the brightness of the at least one pixel, wherein, at a brightness above a threshold value, a larger value is set for at least some of the coefficients of the filter core, and / or wherein, at a brightness below the threshold value or below a further threshold value, a smaller value is set for at least some of the coefficients of the filter core. The terms "smaller" and "larger" refer to the respective values ​​of the coefficients and are to be understood here relative to each other and relative to the filter cores used for areas that do not meet the brightness criterion defined by the threshold value and / or the further threshold value.According to the invention, it can be provided that at least one camera is a camera of a vehicle that captures at least a partial area of ​​a vehicle's surroundings and / or that the generated image display is reproduced on a display device of a vehicle.

[0027] For a control unit according to the invention, it is provided that it is configured to execute a method according to the invention when at least one camera image is provided by at least one camera.

[0028] A driver assistance system according to the invention is provided to include one or more cameras and a control unit according to the invention.

[0029] A vehicle according to the invention is provided that it has a driver assistance system according to the invention.

[0030] A computer program according to the invention comprises instructions which, upon receiving at least one camera image from at least one camera, cause a computing device to execute a method according to the invention. The computing device can be, for example, the control unit of a driver assistance system or another type of computing device.

[0031] A data carrier according to the invention is provided to include a computer program according to the invention.

[0032] All advantages and embodiments described above in relation to the method according to the invention apply accordingly to the control unit, the driver assistance system, the vehicle, the computer program, and the data carrier according to the invention, and vice versa. The advantages and embodiments described above in relation to the other inventions listed above are also transferable to the other inventions. Further advantages and details of the invention will become apparent from the exemplary embodiments described below and from the drawings. These are schematic representations and show:

[0033] Fig. 1 shows an embodiment of a vehicle according to the invention.

[0034] Fig. 2 shows a schematic diagram of an embodiment of the method according to the invention.

[0035] Fig. 3 shows an example of a camera image taken using a camera with a fisheye lens and

[0036] Fig. 4 shows an image representation generated from the camera image in Fig. 3 using the embodiment of the method according to the invention.

[0037] Figure 1 shows an embodiment of a vehicle 1. The vehicle 1 can be, for example, a motor vehicle, in particular a passenger car, a truck, or another type of commercial vehicle. It is also possible that the vehicle 1 is an unmotorized vehicle, for example, a trailer, or that it is a combination of a towing vehicle and a trailer. The vehicle 1 can also be a rail-bound vehicle, such as a tram or the like, or a robot.

[0038] The vehicle 1 comprises an embodiment of a camera system 2, which includes one or more cameras 3 and a control unit 4. In the present embodiment, the vehicle 1 comprises four cameras 3, which form a camera arrangement configured as a surround-view system. A first camera 5 is arranged as a front camera of the vehicle 1, a second camera 6 as a rear-view camera of the vehicle 1, and a third camera 7 and a fourth camera 8 each as a side camera of the vehicle 1. The number and arrangement of the cameras 3 are exemplary; the camera system 2 can also comprise a different number and / or cameras 3 arranged at different locations on the vehicle 1.

[0039] The control unit 4 can be configured, for example, as a microcontroller, a processor, or another type of computing device. The control unit 4 includes a storage device 9 in which data can be stored and / or in which one or more calculation instructions or...

[0040] Algorithms can be implemented. The control unit 4 is configured to implement one or more driver assistance functions for the vehicle 1 by evaluating the camera images from the cameras 3. Additionally or alternatively, the one or more driver assistance functions can also be implemented by at least one other control unit of the vehicle 1.

[0041] The vehicle 1 further includes a display device 10, which can present graphical information to a driver or user of the vehicle 1. The display device 10 can, for example, be one or more screens or displays arranged in the interior of the vehicle 1, or the like.

[0042] Using the cameras 3, image data can each be generated from a sub-area of ​​the vehicle 1's surroundings. The cameras 3 are connected to the control unit 4 via a communication link 11. The communication link 11 can, for example, transmit multiple signals.

[0043] The communication link 11 may include point-to-point connections or a bus connection such as a CAN bus or the like. Image information or image data is transmitted from the cameras 3 to the control unit 4 via the communication link 11, in particular continuously and / or as a video stream. The control unit 4 can process this image data and use the image data or the processed image data to provide at least one driver assistance function. Figure 2 shows a schematic diagram of an embodiment of a method for generating an image from at least one camera image 12 of at least one of the cameras 3 of the vehicle 1. The image depicts at least a section of the vehicle 1's surroundings and can, in particular, be displayed on the vehicle 1's display unit 10.The procedure can, for example, be implemented as a computer-implemented procedure and executed by the control unit 4 of the vehicle 1.

[0044] The image display can, for example, reproduce the view from one of the vehicle's cameras 3 and be generated accordingly from one or more camera images 12 from camera 3. For example, the image display can depict an area in front of, beside, or behind the vehicle 1.

[0045] Alternatively, the image display can also be generated from several camera images from several of the cameras 3 and, for example, be a perspective view from a viewpoint next to and / or above the vehicle 1, for example a bird's-eye view or the like.

[0046] To generate the image display, at least one camera image 12 is projected onto a virtual projection surface 13. The image display is determined from a rear projection image 15 generated using the projection surface 13. The rear projection image 15 can be determined from the viewpoint of a virtual camera 14, whereby the position and / or orientation of the virtual camera 14 relative to the projection surface 13 can correspond to or differ from the actual position and orientation of the camera 3 on the vehicle 1.

[0047] The rear-projection image 15 to be generated is processed with at least one image processing function comprising at least one filter core. The filter core is adjusted to process at least one pixel in the rear-projection image 15 as a function of at least one pixel density assigned to that pixel. The pixel density describes the number of pixels from the camera image that correspond to at least one pixel in the rear-projection image 15. Figure 3 shows an example of a camera image 12, which was captured by the reversing camera 3, 6 of the vehicle 1. In the present embodiment, the reversing camera 3, 6 includes a fisheye lens, which results in a barrel distortion of the surroundings in the camera image 12, particularly in the edge regions of the camera image 12.By projecting the camera image 12 onto the virtual projection surface 13, which in this case has, for example, the shape of a cylindrical shell segment, the camera image 12 can be rectified. The projection of the camera image 12 is carried out by an assignment rule that maps the pixels 17 of the camera image 12 onto the projection surface 13.

[0048] By deriving the rear-projection image 15 from the virtual camera 14, different pixel densities can result for individual image areas of the rear-projection image 15 to be generated, depending on the position and orientation of the virtual camera 14. In other words, the areas of the rear-projection image 15 can have different densities of image information. In particular, areas where a section of the vehicle's surroundings depicted at the edge of the camera image 12 is shown may contain less image information than areas of the surroundings shown in the center of the camera image 12. This can lead to a poorer rendering of the vehicle's surroundings, especially at the edges, for example, a lower contrast rendering or similar issues.

[0049] To address this, the rear projection image 15 to be generated is processed using the image processing function comprising the filter core 16, whereby the filter core 16 is set to process at least one pixel of the rear projection image 15 depending on the pixel density assigned to that pixel. The processing of the rear projection image 15 to be generated can take place during the projection of the camera image 12 onto the projection surface 13, so that no subsequent processing step is required after deriving the rear projection image 15. Alternatively, a preliminary rear projection image 15 can first be determined from the projected camera image 12, and the image processing can then take place after the creation of the preliminary rear projection image, thereby generating the rear projection image 15 to be used as the image display.

[0050] When adjusting the filter kernel 16 depending on the local pixel density in the rear projection image 15 to be generated, the size of the filter kernel 16 and / or the values ​​of one or more coefficients of the filter kernel 16 are set depending on the pixel density. The filter kernel 16 can, in particular, be continuously adjusted according to the respective local pixel density or the pixel density assigned to each pixel to be processed.

[0051] Depending on the pixel density of the at least one pixel to be processed, the filter kernel can be configured, for example, as a square matrix or an nxn matrix of varying sizes, such as a 3x3, 5x5, 7x7, 9x9, etc. matrix. Figure 2 shows the filter kernel 16 as an example of a 3x3 matrix, which assigns a value to the middle of the nine pixels. This value depends not only on an initial value of the middle pixel but also on the values ​​of one or more of the surrounding eight pixels. The pixel value can be, for example, the value in one or more color channels of the pixel and / or a luminance value of the pixel.

[0052] Depending on the pixel density, the values ​​of one or more of the coefficients of filter kernel 16 can also be adjusted. For example, the magnitudes and / or signs of one or more of the coefficients can be changed depending on the pixel density assigned to the at least one pixel to be processed.

[0053] For example, at a pixel density greater than 1, the filter core 16 is configured as a Dirac filter core or an adaptive low-pass filter core. At a pixel density less than 1, the filter core is configured for adaptive image generation. Furthermore, the size and / or coefficients of the filter core 16 can also be adjusted depending on the pixel density value. For a first pixel density, the filter core 16 can, for example, have the values ​​[0, 1, 0; 1, -2, 1; 0, 1, 0], and for a second pixel density, the values ​​[0, 2, 0; 2, -4, 2; 0, 2, 0]. The coefficient values ​​for the second pixel density are each larger.In particular, it is possible that the coefficients of the filter kernel 16 are continuously adjusted, in whole or in part, for pixel densities between the first pixel density and the second pixel density, so that, for example, for pixel densities that change from the first pixel density to the second pixel density over several pixels, a continuous adjustment of the coefficients of the filter kernel 16 results.

[0054] Furthermore, the filter kernel 16 can be adjusted depending on the brightness of at least one pixel. For brightness levels above a certain threshold, a larger value can be set for at least some of the coefficients of the filter kernel 16 than for pixels with a brightness below the threshold or below a different, additional threshold.

[0055] Figure 5 shows a rear-projection image in which a filter kernel for adaptive image sharpening was used in areas with low pixel density. Such an area is marked by a frame 18. The rear-projection image was captured from a camera position that corresponds at least substantially to the position of cameras 3, 6 on the vehicle 1. A filter kernel for image sharpening was used as the filter kernel 16. Therefore, due to the pixel-density-dependent image processing, the image display exhibits improved edge recognition in the image areas processed by the image processing function, corresponding to the edge region of the original camera image. There is no difference compared to the display of surrounding areas originally contained in the center of the camera image 12, thus advantageously enabling the most faithful possible reproduction of the vehicle's surroundings.

[0056] Processing the rear-projection image to be generated using the pixel-density-dependent filter kernel 16 allows for targeted processing of image areas containing comparatively little image information or image data with a low frequency. By taking pixel density into account, it is advantageously achieved that the image processing can be adapted to the actual display situation; that is, depending on the position of the virtual camera 14 and / or the shape of the projection surface 13, sharpening, for example, can always be performed in areas with low pixel density.This enables improved image quality, therefore also in image representations which, for example, through the appropriate choice of position and orientation of the virtual camera 14, show an environment representation from a different perspective, for example a bird's-eye view of the vehicle 1 or the like.

[0057] Furthermore, targeted image processing of image areas with little image information is advantageously achieved when the image display is generated using a projection surface 13 with an adaptive projection surface geometry. With an adaptive projection surface geometry, the shape of the projection surface 13 can, for example, be adapted to objects in the vehicle's environment detected by the vehicle's environmental sensors in order to achieve improved display. When the projection surface 13, for example a bowl-shaped one, surrounding the vehicle 1 is adapted, the pixel density also changes, so that efficient image processing is advantageously possible even in such cases using the method according to the invention.

Claims

Patent claims 1. Method for generating an image representation from at least one camera image (12) of at least one camera (3), wherein the at least one camera image (12) is projected onto a virtual projection surface (13), wherein the image representation is generated from a rear projection image (15) generated using the projection surface (13), and wherein the rear projection image (15) to be generated is processed with at least one image processing function comprising at least one filter kernel (16). characterized by that the filter kernel (16) is set to process at least one pixel in the rear projection image (15) depending on at least one pixel density assigned to the at least one pixel, wherein the pixel density describes the number of pixels from the camera image (12) that correspond to the at least one pixel of the rear projection image (15).

2. Method according to claim 1, characterized by that the processing of at least one pixel with the filter kernel (16) takes place during the projection of the camera image (12) and / or during the generation of the rear projection image (15).

3. Method according to claim 1 or 2, characterized by that a size of the filter kernel (16) and / or the values ​​of one or more coefficients of the filter kernel (16) are set depending on the pixel density.

4. Method according to any of the preceding claims, characterized by that at a pixel density greater than 1 the filter kernel (16) acts as a It is set as a Dirac filter core or as an adaptive low-pass filter core.

5. Method according to one of the preceding claims, characterized in that that when the pixel density is less than 1, the filter core (16) is set for adaptive image sharpening.

6. Method according to any of the preceding claims, characterized by that the filter core (16) is adjusted depending on the brightness of the at least one pixel, wherein, for a brightness above a limit value, a value with a larger amount is set for at least some of the coefficients of the filter core (16), and / or wherein, for a brightness below the limit value or below a further limit value, a value with a smaller amount is set for at least some of the coefficients of the filter core (16).

7. Method according to any of the preceding claims, characterized by that at least one camera (3) is a camera with a wide-angle lens, in particular a fisheye lens.

8. Method according to any of the preceding claims, characterized by that at least one camera (3) is used which captures at least a part of the vehicle's surroundings and / or that the generated image display is reproduced on a display device (10) of a vehicle (1).

9. Control unit which is configured to execute a method according to one of the preceding claims when at least one camera image (12) is provided to at least one camera (3).

10. Driver assistance system comprising one or more cameras (3) and a control unit according to claim 9.

11. Vehicle comprising a driver assistance system according to claim 10.

12. Computer program comprising instructions which cause a computing device, upon provision of at least one camera image (12) from at least one camera (3), to execute a method according to one of claims 1 to 8.

13. Data carrier comprising a computer program according to claim 12.

Images