Method for reducing color shift of image pixels in images of automobiles taken by a camera

Abstract

The present invention relates to a method for reducing color shifts of image pixels (8) of an image (7) of a motor vehicle (1) taken by a camera, a computer program product and a control device (2) for a motor vehicle (1). The method comprises the steps of: for each image pixel (8) of the image (7), measuring (S1) color information (9) describing the color of the image pixel (8), checking (S2) whether the measured color information (9) is greater than a minimum color information (10) and less than a maximum color information (11), and only if said check is successful, calculating (S3) corrected color information (13) for the image pixel (8) taking into account the measured color information (9) and a correction factor (12), and providing (S4) the calculated corrected color information (13) instead of the measured color information (9), wherein the minimum color information (10) and the maximum color information (11) define a color information gamut within which a reference image pixel describing a reference object (42) fluctuates, and wherein the corrected color information (13) has a reduced color shift compared to the measured color information (9).

Description

[Technical Field]

[0001] The present invention relates to a method for reducing color shifts of image pixels in images of a vehicle captured by a camera, and further to a computer program product and a control device for a vehicle. [Background technology]

[0002] In many cases, an optical imaging system for a vehicle equipped with multiple cameras can provide multiple views of the vehicle's environment. Each view can be synthesized from multiple camera images captured by the imaging system. First, the individual camera images, each depicting a specific sub-area of ​​the vehicle's environment, can be newly mapped onto the overall view, for example by selecting and combining the individual camera images for this purpose. In this way, a mosaic image can be created as a new overall image representing a top view of the vehicle's environment with the vehicle positioned in the center. Such a top view can alternatively be referred to as a top view.

[0003] The camera images and / or overview images taken by the vehicle's cameras are often modified by at least one image processing algorithm before being displayed to the vehicle user on a display device inside the vehicle.

[0004] For example, US2018 / 0204310A1 shows a visualization system for automobiles, in which a lens shading algorithm is applied to improve the noise level of the image data of the visualization system.

[0005] The reason for processing the camera image and / or the overall image is that, for example, changes or differences in lighting conditions and / or behavior of the camera lens of the camera employed may result in, for example, color shifts in one or more image pixels of the camera image or overall image that need to be corrected. Summary of the Invention

[0006] The object of the present invention is to provide a solution that allows reducing the color shift of image pixels.

[0007] This problem is solved by the subject matter of the independent claims.

[0008] A first aspect of the present invention relates to a method for reducing color shifts of image pixels in an image of a vehicle captured by a camera. It is assumed that at least one image is captured by, for example, a camera on the vehicle. The image depicts, for example, a sub-area of ​​the vehicle's environment. That is, the image at least partially depicts, for example, at least a portion of the road surface of the road on which the vehicle is located, at least a portion of infrastructure adjacent to the road surface, e.g., a building or factory, and / or at least a portion of other road users, e.g., vehicles and / or humans. The image may alternatively be referred to as a camera image. The image may also be a complete image synthesized from multiple camera images. The image includes many image pixels, which together form the image. The camera capturing at least a portion of the image may be, for example, a front camera, a rear camera, and / or a side camera on the vehicle. Furthermore, the captured image may not be captured by the vehicle itself, but may be provided to the vehicle from an external device, for example, another vehicle, a computing device, and / or an infrastructure device such as a traffic camera, via, for example, a communication link.

[0009] The present invention is based on the insight that images often exhibit a color shift in the hue of image pixels in at least one subarea of ​​the image. The color shift may alternatively be referred to as a hue shift. The color shift may be due to camera lens-related non-uniformities in the color representation of the respective camera lenses. Depending on the state of the camera lens, for example, the image hue may have a blue-green or blue cast in one subarea of ​​the image, while having a reddish-purple or red cast in another subarea, resulting in local fluctuations in the hue of, for example, a monochromatic area in the image. For example, this may cause individual subareas of the image to appear bluish to the viewer (a blue cast), while other subareas of the image appear reddish (a red cast). This non-uniformity in hue often occurs in edge regions of the image that are located away from the image center. Particularly noticeable is the color shift characteristic of large monochromatic objects in the image, such as the gray surface of a road. Therefore, the color values ​​of individual image pixels of an image should be corrected to enable obtaining an overall view of the image that is harmonious in terms of color representation.

[0010] It may be assumed that non-uniformities in color representation have already been mitigated by the camera's image signal processor, for example, by applying the image signal processor's lens shading algorithm to the image. However, this may not be sufficient, and further measures may be required to reduce the color shift of individual image pixels of the image. This may be the case, for example, when the color fading between individual camera lenses is so severe that the color fading profile cannot be modeled in a simple manner, for example, using a polynomial curve fit. In this case, it may be useful to additionally correct the color values ​​of at least the individual image pixels.

[0011] This means that the image should be provided with a further processing step suitable for reducing the color shift of the image pixels. Since the observed color shift may depend on the incident light, e.g. depending on the position of the sun the color shifted image pixels will change. For this reason the processing step should provide a correction for each image pixel individually that always takes into account the current color shift to which the respective image pixel is affected.

[0012] The method according to the invention comprises a number of method steps that are performed for each image pixel of an image captured by a camera. Preferably, each individual image pixel of the captured image or at least one sub-area of ​​the captured image is considered as part of the method. A method step envisages measuring color information describing the color of the image pixel. The image information may indicate, for example, whether the image pixel is a red, orange, yellow, green, blue, purple, or magenta hue. Alternatively, the color information may be referred to as color data. The color data describes the color of the image pixel, for example, by color values. The color values ​​may, for example, be numerical values ​​or coordinates in the color space of a color model.

[0013] In a further method step, a check is made as to whether the measured color information is greater than the minimum color information and less than the maximum color information. The minimum color information and the maximum color information define a color information gamut, in particular a typical gamut, in which the reference image pixel depicting the reference object fluctuates. The reference object can be, for example, a road surface, i.e., a light gray, dark gray, or black object. The minimum color information and the maximum color information define, for example, a color space region of the color space of the color model, according to which the measured color information is described. In the color information gamut, there is present the color information of many, in particular all, image pixels of the image that depict the reference object or an object similar to the reference object at least in terms of color. The minimum color information and the maximum color information can be determined, for example, by a calibration method that is performed, for example, before or at the start of applying the above-mentioned method. In other words, it is checked whether the color information of the image pixel is within the color information gamut defined by the minimum color information and the maximum color information. Preferably, the minimum color information and the maximum color information define a predetermined subarea of ​​the overall color space. The sub-area simply lies within a typical color information gamut of a reference object, such as a gray road surface. The defined color information gamut contains color information for color casts or hue shifts, such as toward blue-green or reddish-purple. That is, when checking, for example, each image pixel is checked to see if its color information suggests that it describes a reference object, such as a gray road surface, because the measured color information (measured color information) is greater than the minimum color information and less than the maximum color information. If the reference object is a gray road surface, for example, an image pixel depicting a blue sky or green grass in the car's environment in this example will not be recognized as the color information of the image pixel, even though it lies within the color information gamut defined by the minimum and maximum color information.

[0014] Only if the measured color information is greater than the minimum color information and less than the maximum color information is corrected color information calculated for the image pixel. The calculation is performed taking into account the measured color information and a correction coefficient. The correction coefficient is predetermined, for example, depending on the camera lens of the camera that captured the image. The calculated corrected color information is then provided instead of the measured color information. The corrected color information has a reduced color shift compared to the measured color information. This is achieved by recalculating the color information of the affected image pixel and shifting it to the corrected color information according to the correction coefficient. The corrected color information differs from the calculated color information. In other words, for all image pixels where the check is successful, the measured color information is replaced with the corrected color information. If the measured color information is not greater than the minimum color information or less than the maximum color information, it is assumed that no corrected color information is calculated. Therefore, the measured image information is provided for these image pixels. In other words, only the color information assigned to the image pixels where the check is successful is modified.

[0015] The method can be terminated by providing the image with all its image pixels. For all image pixels for which the check was successful, correction color information is provided in place of the measured color information. That is, only image pixels whose color information is outside the color information gamut defined by the minimum and maximum color information are corrected. All other image pixels remain unchanged in terms of their color information. This ultimately achieves a reduction in color shift at selected image pixels, as correction color information having a reduced color shift compared to the measured color information is provided in place of the measured color information, if necessary.

[0016] In an advantageous embodiment, the reference object is assumed to be a road surface or a light box. The road surface is specifically a gray road surface, for example, a light gray or dark gray road surface. The road surface is, for example, the surface of an asphalt road. The light box is, for example, an object that emits light of a predetermined wavelength, typically a white light range. The color and / or brightness information of the light emitted by the light box is preferably known by being predetermined. As a result, the color of the reference object is preferably within a color information range from black / dark gray to light gray / white. Since automobiles are often located on gray road surfaces, objects having color information within this color information range are often found in the automobile's environment. Therefore, a top view of the environment directly adjacent to the automobile typically includes at least one gray subarea. This subarea often occupies a large subarea of ​​the top view compared to the rest of the top view. Depending on the selected brightness, a gray area in the color space of the color model may correspond to a white area in the color space. Thus, by referring to the light box as a reference object, the color shift can be recognized and transferred to a gray object, e.g., the road surface. The choice of reference object therefore specializes the method to recognize and correct color shifts in color intervals specific to the automotive environment, leading to a wide range of application of the method in automotive.

[0017] In a further embodiment, it is assumed that each measured color information is described by a U color value and / or a V color value in the YUV color model. That is, a common color model suitable for describing the color information of an image pixel can be relied upon. The YUV color model can describe the brightness of an image pixel by a Y value and the color by U and V values, i.e., the U and V color values. The U and V values ​​cover a color space. This color space can be referred to as a UV color space, in which all colors for each Y value can be represented. As a result, when only the U and V color values ​​are considered, a two-dimensional color space is mapped from the U axis to the V axis, each of which ranges from, for example, 0 to 255. Therefore, the color information includes two individual color values, i.e., the U color value and the V color value. By indicating the two color values ​​together, the color information can be accurately represented, and thus the color of the image pixel can be described.

[0018] In particular, it is assumed that the minimum color information is described by the minimum U and / or minimum V color value, and that the maximum color information is described by the maximum U and / or maximum V color value, so that a total of four color extremes are known, which ultimately define a sub-area of ​​the UV color space of the YUV color model.

[0019] In a further embodiment, the minimum and maximum U color values ​​and the minimum and maximum V color values ​​are assumed to define a color space region arranged around the center of the UV color space of the YUV color model. The UV color space at a U color value of 128 and a V color value of 128 includes a center. Around the center, colors ranging from black / dark gray to light gray / white are arranged according to the Y value. By locating the color-defined color information gamut at this position, only objects in the image having colors ranging from black / dark gray to light gray / white can be corrected in terms of their color information. Meanwhile, objects in the image having different colors, for example, blue, green, red, or yellow objects, remain unaffected in terms of their color information. This ensures that color shift reduction is performed only in subareas of the image where it makes sense to reduce color shift, for example, because there are large areas of gray road surfaces where color shift is particularly noticeable.

[0020] In a further embodiment, it is assumed that the further the measured color information is from the center of the UV color space, the greater the distance between the measured color information and the corrected color information in the UV color space. That is, it is assumed that the degree of correction of color information varies depending on the individual pixel. For example, image pixels whose color information is farther away from the center of the UV color space due to a large color cast will be corrected more strongly than image pixels whose color information is located closer to the center of the UV color space and therefore has a small color cast. Therefore, for image pixels with a large color cast, the difference between the measured U and V color values ​​and the corrected U and V color values ​​will be larger than for image pixels with a small color cast. That is, not all image pixels that require correction in the checking step will be corrected to the same extent.

[0021] The calculation of the corrected U color value from the measured U color value can be performed, for example, by first subtracting the value 128 from the measured U color value of the image pixel to obtain a difference value from the center of the UV color space. The corrected U color value is calculated as the difference. The measured U color value is the minuend, and the product of the difference value and the correction factor is the subtrahend. The calculation of the corrected V color value is performed in a similar manner. The correction factor is, for example, 0.75. Alternatively, the correction factor can be between 0.5 and 1, between 0.6 and 0.9, and in particular between 0.7 and 0.8. This achieves that the degree of correction applied varies depending on the distance of the measured U or V color value, respectively, from the center of the UV color space and, therefore, on the individual image pixel. Finally, in this way, color information closer to the edge regions of the defined color information gamut is corrected more strongly toward the center, thereby achieving, overall, as uniform a color representation as possible of, for example, a gray road surface in the provided image. By taking into account the measured color information and correction coefficients in this way, rather than all image pixels being corrected to the same extent, a gradual correction can be made, or gradients can be taken into account during the correction.

[0022] In an embodiment, it may be assumed that the number of image pixels whose measured color information is greater than the minimum color information and less than the maximum color information is measured. Then, it is checked whether the measured number (measured number) is greater than the minimum number. If the measured number is equal to or less than the minimum number, alternative minimum color information and alternative maximum color information are determined by evaluating the image pixels. The color information gamut defined by the alternative minimum color information and alternative maximum color information is located particularly far from the center of the UV color space of the YUV color model. Then, for example, the above-described method may be implemented taking into account the alternative minimum color information and alternative maximum color information. Thus, for example, if a vehicle is positioned on a grassy surface rather than a gray road surface, the minimum color information and maximum color information may first be adjusted for this environment of the vehicle. This is because, for example, despite the method described above, all or at least some of the image pixels in the captured image of the grassy surface do not depict gray or white objects. Even in this case, it may be assumed that color shifts of the image pixels can be reduced by calculating alternative minimum color information and maximum color information and taking them into account as follows, thereby reducing possible deviations in the green gamut. First, it can be envisaged to carry out a new calibration so that alternative minimum and maximum colour information can be calculated using reference objects with different colours, for example green in particular.

[0023] Alternatively or additionally, it may be envisaged that correction color information for the corresponding image pixels is calculated only if the number of measurements is greater than a minimum number. That is, for example, the following control step may be provided: In this control step, if, for example, there are few gray image pixels in the image (because the number is less than the minimum number), no further method steps are performed, and it is simply determined that the captured image is not suitable for the method because there are not enough image pixels in the relevant color space region. That is, first, it is checked, for example, whether the captured image contains gray and / or white regions in which correction can be performed, and therefore is suitable for processing by the method. This avoids unnecessary calculations that would occur if the method were to be performed continuously without taking into account the number of affected image pixels.

[0024] The minimum number may be expressed as a percentage of the total number of image pixels of the image, for example, 5 percent, 10 percent, 20 percent, 30 percent, 40 percent, 50 percent, 60 percent, 70 percent, 80 percent, 90 percent, or particularly a value greater than 90 percent and less than 100 percent.

[0025] Furthermore, in embodiments, it is envisioned that a harmonic function is applied to the image before checking the measured color information of each image pixel. The harmonic function performs brightness and / or color harmonization on the image. Therefore, it may be envisioned that a brightness and color harmonization step of the image is first performed before checking whether the measured color information is greater than the minimum color information and less than the maximum color information. Applying a harmonic function is particularly meaningful when the image is an overall view synthesized from individual camera images taken under different lighting conditions, since the images show various regions of the automotive environment. Then, brightness and color corrections can be performed by applying the harmonic function. In particular, brightness and color adjustments are performed between the two camera images in the transition regions between the two camera images. However, it is still possible that color shifts of individual image pixels may be observed in the harmonized image due to, for example, differences in the condition and quality of camera lenses for a particular camera type. However, such minor color shifts are typically not corrected by applying a harmonic function, and an inventive method for calculating corrected color information is needed. Thus, by applying the harmonic functions early, the colors in the image are pre-corrected, so that the starting image provides a reasonable base for fine-tuning corrections that can be achieved by calculating the corrective color information, which can ultimately result in an optimal reduction of color shifts.

[0026] Furthermore, in embodiments, it may be envisioned that the image is a partial image of a whole image synthesized from at least two partial images. The whole image may be synthesized from, for example, four partial images: one captured by a front camera, one captured by a rear camera, and two captured by side cameras located on the vehicle's side mirrors. The synthesized image is obtained by combining or concatenating the individual partial images. Each partial image thus depicts a sub-area of ​​the vehicle's environment. That is, a partial image depicting the environment in the area in front of the vehicle, a partial image depicting the environment in the area behind the vehicle, a partial image depicting the environment on the left side of the vehicle's longitudinal direction, and a partial image depicting the environment on the right side of the vehicle's longitudinal direction are provided. These four partial images may, for example, be combined to form a 360-degree image of the environment or a top view of the vehicle. For example, it may be envisioned that a step of synthesizing the individual partial images into a whole image is performed before applying the harmonic function. The harmonic function may in particular harmonize the boundary areas between the individual images, i.e., adjust their brightness and / or color values ​​relative to one another. In other words, images captured by multiple cameras on the vehicle can be optimally utilized.

[0027] In another embodiment, it is envisioned that the overall image is synthesized after checking the measured color information for each image pixel of at least two images. Therefore, first, the measured color information for each image pixel of at least two different images is checked and possibly corrected. Then, the overall image is synthesized from the at least two images. As a result, the overall image includes at least two partial images. Therefore, the individual images of the overall image have already been checked for color information of their image pixels according to the above-described method, and corresponding image pixels are either corrected by providing corrected color information or not corrected by maintaining the measured color information. A harmonic function is then applied to the overall image. In the overall region, at least two images depict different subareas of the vehicle's environment, specifically the front, rear, and / or side regions of the vehicle. The harmonic function performs brightness and / or color harmonization on the overall image, as described above. That is, the above-described method envisioned for checking the measured color information and, if possible, calculating and providing corrected color information to corresponding image pixels is applied to the partial images, which may simply be combined and adjusted to each other by the harmonic function.

[0028] In an embodiment, each partial image is a viewport of a camera image captured by a camera, and the overall image is assumed to show a top view of the vehicle's environment. Therefore, it can be assumed that the entire image captured by the camera is not displayed and further considered, but rather that a viewport is selected from the entire image. A viewport is a term for a display window. To display a top view of the vehicle's environment, for example, a region of the environment that is located on top of the vehicle's height and included in the camera image, such as part of the sky, is not required. Therefore, only the viewport is considered as the partial image from which the overall image is synthesized. This is a particularly suitable display for vehicles and can be useful for the vehicle driver, for example, in parking maneuver assistance.

[0029] In a further embodiment, it is envisioned that at least one object in an image is recognized by applying an object recognition algorithm to the image. Only if the recognized object is a road surface, respective correction color information for each image pixel of the image is calculated. Therefore, whether a captured image, a partial image, or an entire image is suitable for the described method is checked by first performing an object check based on the object recognition algorithm and then checking whether the recognized object is a road surface. The object recognition algorithm is, for example, a computer program and includes at least one rule or instruction that enables recognition of at least one predetermined object in the image. The predetermined object may be a road surface. Furthermore, the object recognition algorithm can distinguish between several types of objects, i.e., it can recognize, for example, multiple classes of objects. This avoids calculating correction color information for image pixels of an image that is completely unsuitable for application of the described method, for example, because it does not display a road surface.

[0030] In a further embodiment, it may be assumed that the correction coefficients are predetermined depending on the minimum color information and the maximum color information. For example, a large correction coefficient may be assumed when the distance between the minimum color information and the maximum color information is large, compared to when a smaller correction coefficient may be selected when the distance between the minimum color information and the maximum color information is small. That is, the smaller the color information space defined by the minimum color information and the maximum color information, the smaller the correction coefficient may be selected. Vice versa. This allows the correction coefficients to be adjusted to the behavior of the camera lens, thereby achieving a reasonable reduction in color shift at all times.

[0031] Furthermore, embodiments are envisaged in which the reference image pixels are captured by multiple cameras, each equipped with a substantially identical camera lens, when performing the calibration method. The multiple cameras may be assigned to a common camera type. The common camera type may be associated with a common camera lens. That is, the multiple cameras each have a substantially identical camera lens. As a result, the multiple cameras have, for example, a common type of camera lens, and thus the same camera lens. Substantially identical means that there are maximum differences between the individual camera lenses that are unrelated to the color information gamut in which the color information of the image pixels can vary. Preferably, the camera used to capture the image also has the camera lens used to perform the calibration method. Therefore, the camera used to capture the image is also assigned to a common camera type. In other words, the calibration method is performed for a specific camera lens.

[0032] Preferably, a reference image having at least 10 reference pixels is captured of a reference object, from which minimum and maximum color information are calculated. For example, a road surface area and / or a light box area are depicted in the reference image and selected from them. Subsequently, it is calculated or measured in which color information range, i.e., in which color space area, image pixels depicting the road surface or light box fluctuate, even though they should actually represent the same color information. That is, it is determined whether the reference object in the reference image is depicted by at least one reference pixel with, for example, a reddish or bluish color shift. The color information that is most or least shifted relative to the center of the color space is taken as the maximum or minimum color information, respectively. In this regard, it is appropriate to first perform multiple calibration measurements for a common camera lens before the minimum and maximum color information can be provided as predetermined values.

[0033] A further aspect of the present invention relates to a computer program product. The computer program product may be, for example, a computer program. The computer program product may be stored on a computer, such as a control device of a motor vehicle. The computer program product comprises at least one instruction, preferably a plurality of instructions, that causes the computer to implement the above-described method when the program is executed by the computer, i.e., for example, by a control device of a motor vehicle. The computer may be, for example, a computing device. Furthermore, a second computer program product may be provided, in which a camera captures environmental and / or reference images of the motor vehicle. Furthermore, a third computer program product may be provided, in which the described calibration method may be implemented.

[0034] A further aspect of the present invention relates to a control device for a motor vehicle. The control device is configured to implement the above-mentioned method. The control device comprises a processor device capable of implementing the described method. The processor device for this purpose may comprise at least one microprocessor, and / or at least one microcontroller, and / or at least one FPGA (Field Programmable Gate Array), and / or at least one DSP (Digital Signal Processor). Furthermore, the processor device may include a computer program product. The control device preferably implements the described method.

[0035] The embodiments described in connection with the method according to the invention equally apply, insofar as applicable and insofar as they are not mutually exclusive, to the computer program product according to the invention as well as to the control device according to the invention, individually or in combination with each other.

[0036] A further aspect of the invention may be a motor vehicle equipped with a control device according to the invention. The motor vehicle may be, for example, a car, a truck, a bus and / or a motorcycle. [Brief explanation of the drawings]

[0037] [Figure 1] 1 is a schematic diagram of a vehicle with a control device and multiple cameras. [Figure 2] 1 is a signal flow graph schematic of a method for reducing color shift of image pixels in an image of a car captured by a camera. [Figure 3] Schematic representation of the UV color space in the YUV color model. [Figure 4] Schematic diagram of the correction of a car camera image. [Figure 5] 3 is a signal flow graph schematic diagram of a further method step of the method of FIG. 2; [Figure 6] 3 is a signal flow graph schematic of the method steps of the calibration method for the method of FIG. 2; [Figure 7] Schematic of a flow graph for applying an object recognition algorithm to an image. DETAILED DESCRIPTION OF THE INVENTION

[0038] 1 depicts a motor vehicle 1 equipped with a control unit 2. The control unit 2 is a computing device that may comprise, for example, at least one microprocessor and / or microcontroller. In other words, the control unit 2 of the motor vehicle 1 is a computer that is capable of executing a computer program product, for example a computer program.

[0039] The automobile 1 is equipped with multiple cameras, namely a front camera 3, a rear camera 4, and two side cameras 5, each located on the side mirrors of the automobile 1. In combining the camera images captured by these four cameras of the automobile 1, for example, a 360° environment of the automobile 1 and / or a planar view of the automobile 1 can be displayed. The displayable environment is defined by at least the capture area of ​​each camera, namely the front camera 3, the rear camera 4, or the side camera 5.

[0040] A display device 6, which may be, for example, a display, in particular a touchscreen display, is arranged in the automobile 1. Camera images captured by, for example, the front camera 3, the rear camera 4, and / or the respective side cameras 5 may be displayed on the display device 6. For this purpose, the camera images may first be manipulated or processed in the control device 2. The individual cameras, i.e., the front camera 3, the rear camera 4, and / or the side cameras 5, may, for example, be equipped with fisheye lenses capable of capturing the environment at 180°. A top view of the environment of the automobile 1 may be provided, for example, by selecting a corresponding section from each camera image. This means that, for example, the display device 6 may display the current environment of the automobile 1 to the user of the automobile 1 at least partially, preferably in its entirety, in order to assist the user in parking.

[0041] 2 illustrates a method for reducing color shifts in image pixels 8 of an image 7 of a vehicle captured by a camera, specifically a front camera 3, a rear camera 4, and / or a side camera 5. The image 7 may be, for example, a camera image captured by the respective camera.

[0042] In a first step S0, an image 7 of the surroundings of the motor vehicle 1 can be captured by the front camera 3, the rear camera 4 and / or the side camera 5. The image 7 is made up of individual image pixels 8, the number of which depends on the respective camera. The further method steps S1 to S5 described below are preferably performed for each image pixel 8 of the image 7 captured by each camera, i.e. the front camera 3, the rear camera 4 and / or the side camera 5.

[0043] In a first method step S1, color information 9 describing the color of image pixels 8 is measured. The color information 9 preferably describes U and / or V color values ​​of the YUV color model. Thus, the color information 9 preferably includes two values ​​for each image pixel 8, namely a U color value and a V color value.

[0044] In method step S2, a check is made as to whether the measured color information (measured color information) 9 is greater than minimum color information 10 and less than maximum color information 11. Minimum color information 10 may alternatively be referred to as minimum color information. Also, maximum color information 11 may alternatively be referred to as maximum color information. Minimum color information 10 and maximum color information 11 define a color information gamut within which reference image pixels depicting the reference object fluctuate. For this purpose, minimum color information 10 may include a minimum U color value 14 and / or a minimum V color value 15. Also, maximum color information 11 may include a maximum U color value 16 and / or a maximum V color value 17.

[0045] Only if the measured color information 9 is greater than the minimum color information 10 and less than the maximum color information 11 is the correction color information 13 calculated for the image pixel 8 in method step S3. The calculation is performed taking into account the measured color information 9 and a correction coefficient 12. The correction coefficient 12 can be, for example, 0.75. In method step S4, the calculated correction color information 13 is provided instead of the measured color information 9. The correction color information 13 has a reduced color shift compared to the measured color information 9. This means that the individual color information 9 of the image pixel 8 for which check step S2 was successful has been corrected so as to achieve a uniform color distribution in the image 7. Furthermore, if check step S2 is unsuccessful, i.e., if the measured color information 9 is either greater than the minimum color information 10 or less than the maximum color information 11, the correction color information 13 is not calculated, and therefore the already measured color information 9 is provided for the respective image pixel 8 in method step S5. Overall, the image 7 may eventually contain image pixels 8 that contain either measured color information 9 or corrected color information 13, depending on which result the check step S2 produced.

[0046] FIG. 3 shows a schematic representation of a UV color space 18 of the YUV color model. A minimum U color value 14, a maximum U color value 16, a minimum V color value 15, and a maximum V color value 17 are depicted for illustrative purposes. These four extreme values ​​are arranged around a center 19 of the UV color space 18. The center 19 of the UV color space 18 has a U value of 128 and a V value of 128, where the U and V axes of the UV color space 18 each extend between 0 and 255, as is typical in the YUV color model. The four extreme values ​​define a color space region 20 surrounding the center 19 of the UV color space 18. The color space region 20 is the range of color information within which a reference pixel depicting a reference object fluctuates.

[0047] The UV color space 18 has areas of different colors, which are indicated by different hatching depending on the color. The colors range from magenta 50 to red 51, yellow 52, ​​green 53, and blue 54. Areas of mixed colors between the above colors are indicated by overlapping hatching. Around the central point 19, the gray tones are also differentiated from one another, ranging from reddish gray 55 to pure gray 56 to bluish gray 57.

[0048] 3, two exemplary color information 9 are depicted for each image pixel 8: a first color information 9a and a second color information 9b. Both of these are located in the depicted color space region 20. That is, when the check is performed in method step S2, each color information 9a, 9b is evaluated as a color information 9 that is greater than the minimum color information 10 and less than the maximum color information 11. Therefore, in method step S3, corrected color information 13, i.e., a first corrected color information 13a and a second corrected color information 13b, is calculated for each of these two color information 9a, 9b. It is clear that in the UV color space, the distance 21 between the first measured color information 9a and the first corrected color information 13a is greater than the distance 21 between the second measured color information 9b and the second corrected color information 3b. This is because the further the measured color information 9 is from the center 19 of the UV color space 18, the greater the distance between the measured color information 9 and the corrected color information 13. Here, the first measured color information 9a is farther from the center than the second measured color information 9b.

[0049] In the example shown in FIG. 3, for example, the minimum U color value 14 could be at 124. The maximum U color value 16 could be at 130. The minimum V color value could be at 123. The maximum V color value could be at 137.

[0050] If check step S2 is successful, then to first calculate the corrected U color value from the measured U color value, a difference value from the center of the UV color space can be determined by subtracting the number 128 from the measured U color value of image pixel 8. The corrected U color value is then calculated as the difference. The measured U color value is the minuend, and the product of the previously calculated difference value and the correction factor 12 is the subtrahend. The calculation for the corrected V color value is performed in a similar manner. Thus, in method step S4, the corresponding calculated values ​​are provided for image 7 as corrected color information 13, which includes the corrected U color value and the corrected V color value.

[0051] FIG. 4 shows, for example, how four images 7 from four cameras of a vehicle 1 can be predefined. That is, for method step S0, an image 7 from the front camera 4 is displayed at the top left, an image 7 from the rear camera 4 at the top right, an image 7 from the left side camera 5 in the longitudinal direction of the vehicle 1 at the bottom left, and an image 7 from the right side camera 5 in the longitudinal direction of the vehicle 1 at the bottom right. Each image 7 depicts a viewing area 22, alternatively or additionally referred to as a sub-image 23. Each image 7 shows the environment of the vehicle 1. In the image 7, a road surface 24 on which the vehicle 1 is located is depicted over a wide area. The road surface 24 is gray, specifically, a gray ranging from black to dark gray and light gray to white. Furthermore, each image 7 depicts parking space markings 25, the shadow 26 of the vehicle 1, several background objects 27, such as trees, the sky 28, and grass 29. Purely by way of example, two areas in image 7 are highlighted in their color by corresponding hatching. These areas exhibit a reddish gray 55. Alternatively, image 7 may include at least one area of ​​bluish gray 57 and / or further areas of reddish gray 55. Furthermore, other or alternative color casts may be envisaged.

[0052] In method step S6, the four view areas 22 can be combined into respective partial images 23 to form the overall image 30. In other words, the individual images 7, specifically the view areas 22, can be respective partial images 23 of the overall image 30 combined from multiple partial images 23. Each partial image 23 depicts a sub-area of ​​the environment of the automobile 1, specifically the front, rear, and / or side areas of the automobile 1. Therefore, the image 7 can alternatively be considered as the overall image 30 combined from multiple partial images 23. The automobile 1 is located in a central area of ​​the overall image 30 that is not captured by the automobile 1's camera. By considering the four view areas 22, the overall image 30 exemplarily shows a top view of the environment of the automobile 1. In the combined overall image 30, image areas where a red shift occurs are highlighted, for example. For this purpose, the same shading as in FIG. 3 is used.

[0053] In method step S7, a harmonic function 31 can be applied to image 7, here to overall image 30. Harmonic function 31 performs brightness harmonization 32 and / or color harmonization 33 on image 7 or overall image 30, respectively. An example image of harmonic image 34 is shown here. It is clear from this image that, for example, the transition areas between the individual viewing areas 22 have been adjusted to each other, achieving a continuous transition in terms of color and brightness between the individual partial images 23. Furthermore, several objects from the individual camera images 7, i.e., for example, parking space markings 25 or shadows 26, are also visible in harmonic image 34. Harmonic image 34 is therefore the result of combining images 7, here viewing areas 22, in step S6 and the subsequent application of harmonic function 31 to overall image 30.

[0054] The method according to method steps S1 to S3 can then be applied to the harmonic image 34 as image 7. The harmonic image 34 thus corresponds to the image 7 whose individual image pixels 8 have been considered in method steps S1 to S5. A final image 35 is depicted which is provided by the image pixels 8 provided in method steps S4 and S5, respectively. This corrects particularly red-shifted areas of the image 7, i.e. areas of red-tinted gray 55. The color shifts present there have thus been reduced, providing a correspondingly improved image 7 in the form of the final image 35. The final image 35 preferably shows only the road surface 24 in pure gray 56 without any noticeable color cast.

[0055] As an alternative to the sequential arrangement of the method steps according to Figure 4, method steps S1 to S5 can already be performed immediately after method step S0, and only then can the overall image 30 be constructed from the individual partial images 23 (method step S6). Then, in method step S7, a harmonic function is applied to the composite image, thereby providing the final image 35.

[0056] 5 shows further possible method steps S8 to S10 in a schematic manner. In method step S8, the number 36 of image pixels 8 whose measured color information 9 is greater than the minimum color information 10 and less than the maximum color information 11 is measured. In method step S9, it is checked whether the measured number 36 is greater than the minimum number 37. If the measured number 36 is less than or equal to the minimum number 37, then in method step 10, an alternative minimum color information 10′ and an alternative maximum color information 11′ are determined. The color space region 20 defined by the alternative minimum color information 10′ and the alternative maximum color information 11′ is arranged in particular away from the center 19 of the UV color space 18 of the YUV color model. For example, it can be assumed that if the vehicle 1 is not within the area of ​​the center 19 of the UV color interval 18 in terms of color, for example because it is located on a green grassland that does not correspond to the gray hue of the road surface 24, then alternative minimum color information 10' and alternative maximum color information 11' are determined and then method steps S1 to S5 are performed, for example taking into account the alternative minimum color information 10' and alternative maximum color information 11'.

[0057] Furthermore, it can be assumed that the correction color information 13 for the corresponding image pixel 8 is calculated only if the number of measurements 36 is greater than the minimum number 37, i.e. method step S3 is performed, i.e. it can be assumed that it is first checked whether enough image pixels 8 in the corresponding color region to be corrected are provided.

[0058] The calibration method is illustrated schematically in Figure 6. In the calibration method, an image 7 containing reference pixels may first be provided by taking images of, for example, a gray road surface 24 or a light box 39 with the front camera 3, the rear camera 4, the side camera 5 and / or another reference camera 38, for example located outside the vehicle. Preferably, at least 10 reference images are taken as images 7 in method step S11 so that the calibration method can be reliably provided.

[0059] In method step S12, a calibration algorithm 40 comprising at least one rule for calibrating the camera is applied to the photographed reference pixels and their respective photographic information 9. The reference pixels represent reference objects, i.e., road surface 24 and / or light box 39. For this reason, the measured color information 9 is always positioned around the center 19 of UV color space 18. By applying the calibration algorithm 40, minimum color information 10 and maximum color information 11 can be determined.

[0060] The cameras employed in method step S11 all have substantially identical camera lenses, i.e., they can be assigned to a common camera type. The employed camera lenses are therefore comparable with one another, at least in terms of the color shifts observed in the images 7 captured by the respective camera lenses. It can therefore be concluded that the color space region 20 specific to a camera lens is defined by certain extreme values. Depending on the camera lenses of the front camera 3, rear camera 4, and / or side camera 5 of the automobile 1, the corresponding minimum and maximum color information 11 determined in method step S12 can be provided to the automobile 1. Furthermore, method steps S11 and S12 can be performed in the automobile 1, for example, based on at least 10 images 7 captured by each camera of the automobile 1.

[0061] An embodiment is illustrated schematically in Figure 7. In this embodiment, in method step S13, an object recognition algorithm 41 is applied to the image 7 to recognize at least one object 42 in the image 7. If in method step S14 it is determined that the recognized object 42 is a road surface 24, then method steps S2 to S5 are performed, since it has been determined that the car 1 is located on the grey road surface 24, and therefore the above-described method relating to the centre 19 of the UV colour space 18 in the grey region can be performed.

[0062] If the check performed in method step S14 is not successful, then method step S5 can be performed, for example, i.e., for each image pixel 8, already measured color information 9 can be provided and no corrected color information 13 can be calculated or provided. Alternatively or additionally, method step S10 can be performed, i.e., alternative minimum color information 10' and alternative maximum color information 11' can be determined. Alternatively, in method step S15, the method can be terminated, i.e., method steps S1 to S5 can not be performed at all, since the road surface 24 is not displayed in the image 7 and therefore the image 7 is not recognized as suitable for the method.

[0063] Overall, these examples illustrate an algorithm for mitigating color distribution non-uniformities in a camera lens. It is suggested that the described method be applied after applying a harmonic function 31 to a composite overall image 30, which is a top-view view of the environment of the automobile 1. This individually corrects individual image pixels 8 distorted in a particular direction in the UV color space 18. Each image pixel 8 is considered individually by the method, which checks how far the measured color information 9 of the image pixel 8 is from the center 19 of the UV color space 18, which contains the coordinates (128, 128), relative to the UV coordinates of the UV color space 18. The road surface 24 is the most homogeneous surface seen by the camera of the automobile 1 in top view and therefore likely to occupy a large area in the overall image 30 synthesized from the individual viewing areas 22. Furthermore, the harmonic function 31 corrects each sub-image 23 toward the center 19 of the UV color space 18. In method steps S1-S5, the distance 21 of each color information 9 from the center 19 of the UV color space 18 is checked with respect to its U and V color values. The distance 21 is shifted by shifting the U and V values ​​towards the centre 19 by multiplication with the correction factor 12, as described above. The respective colour information 9 of each image pixel 8 is shifted by a gradient to arrive at the corrected colour information 13. This gradient determines the strength of the non-uniform camera lens performance, which should be measurable by the variance in colour values ​​of a uniform surface of a reference object, here given by the road surface 24 or a light box 39. This allows the correction to be performed only in certain colour regions in the image, while for example unaffected image pixels 8 of a different colour, e.g. depicting a parking space marking 25 or a different background object 27, can remain unmodified.

Claims

1. 1. A method for reducing color shifts of image pixels (8) of an image (7) of a car (1) taken by a camera, comprising: The method comprises the steps of: for each image pixel (8) of the image (7) captured by the camera: - measuring (S1) color information (9) describing the color of said image pixel (8), - checking (S2) whether said measured color information (9) is greater than a minimum color information (10) and less than a maximum color information (11), said minimum color information (10) and said maximum color information (11) defining a color information range within which a reference image pixel describing a reference object (42) fluctuates; - calculating (S3) corrected color information (13) for the image pixel (8) taking into account the measured color information (9) and correction coefficients (12) only if the measured color information (9) is greater than the minimum color information (10) and less than the maximum color information (11), and providing (S4) the calculated corrected color information (13) instead of the measured color information (9), wherein the corrected color information (13) has a reduced color shift compared to the measured color information (9).

2. 2. The method of claim 1, wherein the reference object (42) is a road surface (24), in particular a grey road surface (24), or a light box (39).

3. 2. The method of claim 1, wherein each of the color information (9, 13) is described by a U and / or a V color value of the YUV color model, in particular the minimum color information (10) is described by a minimum U color value (14) and / or a minimum V color value (15), and the maximum color information (11) is described by a maximum U color value (16) and / or a maximum V color value (17).

4. 4. The method of claim 3, wherein the minimum U color value (14) and the maximum U color value (16), and the minimum V color value (15) and the maximum V color value (17), define a color space region (20) arranged around a center (19) of a UV color space (18) of the YUV color model.

5. 5. The method of claim 4, wherein the further the measured color information (9) is from the center (19) of the UV color space (18), the greater the distance (21) between the measured color information (9) and the corrected color information (13) in the UV color space (18).

6. Measure the number (36) of image pixels (8) whose measured color information (9) is greater than the minimum color information (10) and less than the maximum color information (11) (S8), and check whether the measured number (36) is greater than the minimum number (37) (S9); - if the measured number (36) is less than or equal to the minimum number (37), determining (S10) an alternative minimum color information (10') and an alternative maximum color information (11') by evaluating the image pixels (8), the color space region (20) defined by the alternative minimum color information (10') and the alternative maximum color information (11') being located particularly far from the center (19) of the UV color space (18) of the YUV color model, and / or - calculating (S3) the correction color information (13) for the corresponding image pixel (8) only if the measured number (36) is greater than the minimum number (37); The method of claim 4.

7. 2. The method of claim 1, further comprising applying (S7) a harmonic function (31) to the image (7) before checking the measured color information (9) of each image pixel (8), the harmonic function (31) performing luminance harmonization (32) and / or color harmonization (33) on the image (7).

8. 8. The method according to claim 7, wherein the image (7) is a partial image (23) of an overall image (30) synthesized (S6) from at least two partial images (23), each partial image (23) depicting a sub-area of ​​the environment of the vehicle (1), in particular a front area, a rear area and / or a side area of ​​the vehicle (1).

9. 2. The method according to claim 1, further comprising: after checking the color information (9) measured for each image pixel (8) of at least two images (7), synthesizing an overall image (30) comprising each of the at least two images (7) as a partial image (23), and applying a harmonic function (31) to the overall image (30), wherein the at least two images (7) depict different sub-areas of the environment of the motor vehicle (1), in particular the front, rear and / or side regions of the motor vehicle (1), and wherein the harmonic function (31) performs brightness harmonization (32) and / or color harmonization (33) on the overall image (30).

10. 9. The method of claim 8, wherein each partial image (23) is a view area (22) of a camera image taken by the camera, and the overall image (30) shows a top view of the environment of the vehicle (1).

11. 2. The method of claim 1, further comprising: applying an object recognition algorithm to the image to recognize (S13) at least one object in the image; and calculating (S14) the corrected color information for each image pixel of the image only if the recognized object is a road surface.

12. 2. The method of claim 1, wherein the correction coefficients (12) are predetermined depending on the minimum color information (10) and the maximum color information (11).

13. 2. The method of claim 1, wherein the reference image pixels (8) are captured by a plurality of cameras each equipped with a substantially identical camera lens (42) when performing a calibration method, and in particular the cameras used when capturing the images (7) also have the camera lens.

14. A computer program comprising at least one instruction that, when said computer program is run by a computer, causes said computer to carry out the method of any one of claims 1 to 13.

15. A control device (2) for a motor vehicle (1), said control device (2) being configured (7) to carry out the method according to any one of claims 1 to 13.

Images