Method for reducing color shift of image pixels in images of automobiles taken by a camera

Abstract

The present invention relates to a method for reducing color shifts of image pixels (8) of an image (7) of a motor vehicle (1) taken by a camera, a computer program product and a control device (2) for a motor vehicle (1). The method comprises the steps of: for each image pixel (8) of the image (7), measuring (S1) color information (9) describing the color of the image pixel (8), checking (S2) whether the measured color information (9) is greater than a minimum color information (10) and less than a maximum color information (11), and only if said check is successful, calculating (S3) corrected color information (13) for the image pixel (8) taking into account the measured color information (9) and a correction factor (12), and providing (S4) the calculated corrected color information (13) instead of the measured color information (9), wherein the minimum color information (10) and the maximum color information (11) define a color information gamut within which a reference image pixel describing a reference object (42) fluctuates, and wherein the corrected color information (13) has a reduced color shift compared to the measured color information (9).

Description

[Technical field]

[0001] The present invention relates to a method for reducing color shifts of image pixels in images of a vehicle captured by a camera.Furthermore, the present invention relates to a computer program product and a control device for a vehicle. [Background technology]

[0002] An optical imaging system for a motor vehicle, often with several cameras, allows several views of the vehicle's environment to be obtained. Each of the views can be synthesized from several camera images taken by the imaging system. Firstly, the individual camera images, each depicting a certain sub-area of ​​the vehicle's environment, can be newly mapped in the overall image, for example by selecting and combining the individual camera images with one another for this purpose. In this way, a mosaic image can be created as a new overall image, which represents a top view of the vehicle's environment with the vehicle in the center. Such a top view can alternatively be called a top view.

[0003] The camera images and / or overall image views captured by the vehicle's cameras are often modified by at least one image processing algorithm before being displayed to a vehicle user on a display device inside the vehicle.

[0004] For example, US2018 / 0204310A1 shows a visualization system for automobiles, in which a lens shading algorithm is applied to improve the noise level of image data of the visualization system.

[0005] The reason for processing the camera image and / or the overall image is that, for example, changes or differences in lighting conditions and / or the behavior of the camera lens of the employed camera may result in, for example, color shifts in one or more image pixels of the camera image or overall image, respectively, which must be corrected. Summary of the Invention

[0006] The object of the present invention is to provide a solution which allows reducing the color shift of image pixels.

[0007] This problem is solved by the subject matter of the independent claims.

[0008] A first aspect of the invention relates to a method for reducing color shifts of image pixels of an image of a motor vehicle taken by a camera. It is envisaged that at least one image is taken by, for example, a camera of the motor vehicle. The image depicts, for example, a sub-area of ​​the environment of the motor vehicle. That is to say, the image at least partially depicts at least a part of the surface of, for example, a road on which the motor vehicle is located, at least a part of infrastructure adjacent to the surface, for example a building or a factory, and / or at least a part of other road users, for example motor vehicles and / or humans. The image may alternatively be referred to as a camera image. The image may furthermore be a total image synthesized from a number of camera images. The image comprises many image pixels, which together form the image. The camera which takes at least a part of the image is, for example, a front camera, a rear camera and / or a side camera of the motor vehicle. Furthermore, the taken image may not be taken by the motor vehicle itself, but may be provided to the vehicle by an external device, for example via a communication link, for example from another motor vehicle, a computing device and / or an infrastructure device such as a traffic camera.

[0009] The invention is based on the insight that images often have a color shift in the hue of the image pixels present therein, at least in one sub-area of ​​the image. The color shift can alternatively be referred to as a hue shift. The color shift may be due to a camera lens-related non-uniformity in the color representation of the camera lens of the respective camera. Depending on the state of the camera lens, for example, the hue of an image may have a blue-green or blue-ish color cast in one sub-area of ​​the image, and a magenta or red-ish color cast in another sub-area, so that the hue of, for example, a monochromatic area in the image may fluctuate locally. For example, this may cause individual sub-areas of the image to appear bluish to the viewer of the image (blue cast), while other sub-areas of the image appear reddish (red cast). This non-uniformity in terms of hue often occurs in image edge regions located away from the image center. Particularly noticeable is the color shift characteristic of large monochromatic objects in the image, for example the gray surface of a road. Therefore, the color values ​​of the individual image pixels of the image should be corrected in such a way that it is possible to obtain an overall picture of the image that is harmonious in terms of color representation.

[0010] It may be assumed that non-uniformities in color representation have already been mitigated by the image signal processor of the camera, for example by applying a lens shading algorithm in the image signal processor to the image. However, this may not be sufficient and further measures may be required to reduce the color shift of individual image pixels of the image. This may be the case, for example, when the color fading between the individual camera lenses is so severe that the color fading profile cannot be modeled using, for example, a polynomial curve fit. In this case, it may be useful to additionally correct at least the color values ​​of the individual image pixels.

[0011] That is to say, a further processing step should be provided for the image that is suitable for reducing the color shift of the image pixels. Since the observed color shift may depend on the incident light, e.g. depending on the position of the sun the color shifted image pixels change. For this reason, the processing step should provide a correction for each image pixel individually that always takes into account the current color shift to which the respective image pixel is affected.

[0012] The method according to the invention comprises a number of method steps which are carried out for each image pixel of the image captured by the camera. Preferably, each individual image pixel of the captured image or at least one sub-area of ​​the captured image is considered as part of the method. A method step envisages measuring colour information describing the colour of the image pixel. The image information indicates for example whether the image pixel is a shade of red, orange, yellow, green, blue, purple or magenta. Alternatively, the colour information may be referred to as colour data. The colour data describes the colour of the image pixel, for example by means of a colour value. The colour value may for example be a numerical value or may be a coordinate in the colour space of a colour model.

[0013] In a further method step, a check is made as to whether the measured color information is greater than the minimum color information and less than the maximum color information. The minimum and maximum color information define a color information region in which the reference image pixel describing the reference object fluctuates, in particular typically fluctuates. The reference object can be, for example, a road surface, i.e. a light gray, dark gray or black object. The minimum and maximum color information define, for example, a color space region of the color space of the color model, according to which the measured color information is described. In the color information region, there is color information of many, in particular all, image pixels of the image, which image pixels describe the reference object or an object similar to the reference object at least in terms of color. The minimum and maximum color information can be determined, for example, by a calibration method carried out, for example, before or at the start of applying the above-mentioned method. That is, in other words, it is checked whether the color information of the image pixel is within the color information region defined by the minimum and maximum color information. Preferably, the minimum and maximum color information define a predetermined sub-area of ​​the overall color space. The sub-area simply lies in a typical color information gamut of a reference object, e.g. a gray road surface. The defined color information gamut contains color information for color casts or hue shifts, e.g. towards blue-green or red-purple. That is, when checking, it is checked whether, for example, each image pixel suggests that its color information describes a reference object, e.g. a gray road surface, since the measured color information (measured color information) is greater than the minimum color information and less than the maximum color information. If the reference object is a gray road surface, for example, an image pixel depicting a blue sky or green grass in the environment of the car in this example will not be recognized as the color information of the image pixel, even though it lies in the color information gamut defined by the minimum and maximum color information.

[0014] A calculation of correction color information for an image pixel is performed only if the measured color information is greater than the minimum color information and less than the maximum color information. The calculation is performed taking into account the measured color information and a correction factor. The correction factor is, for example, predetermined depending on the camera lens of the camera that captured the image. Then, a step is performed to provide the calculated correction color information instead of the measured color information. The correction color information has a reduced color shift compared to the measured color information. This is achieved by recalculating the color information of the affected image pixel and shifting it to the correction color information depending on the correction factor. The correction color information is different from the calculated color information. In other words, the measured color information is replaced by the correction color information for all image pixels for which the check is successful. If the measured color information is not greater than the minimum color information or less than the maximum color information, it is assumed that no correction color information is calculated. Thus, the measured image information is provided for these image pixels. That is to say, only the color information assigned to the image pixels for which the check is successful is modified.

[0015] The method may be terminated by providing the image with all its image pixels. For all image pixels where the check has been successful, correction color information is provided instead of the measured color information, i.e. only image pixels whose color information is outside the color information gamut defined by the minimum and maximum color information are corrected. All other image pixels are kept unchanged with respect to their color information. Thus, a reduction in color shift is finally achieved in selected image pixels, since, if necessary, correction color information is provided instead of the measured color information, which has a reduced color shift compared to the measured color information.

[0016] In an advantageous embodiment, it is assumed that the reference object is a road surface or a light box. The road surface is in particular a grey road surface, for example a light grey or dark grey road surface. The road surface is, for example, the surface of an asphalt road. The light box is an object that emits light, for example in a range of wavelengths, typically white light. The color and / or brightness information of the light emitted by the light box is known, preferably by being predetermined. As a result, the color of the reference object is preferably within a color information range of black / dark grey to light grey / white. Since automobiles are often located on grey road surfaces, objects with color information in this color information range are often found in the environment of the automobile. Therefore, it is customary for the top view of the environment directly adjacent to the automobile to include at least one grey sub-area. This sub-area often occupies a large sub-area of ​​the top view compared to the rest of the top view. Depending on the selected brightness, the grey area in the color space of the color model may correspond to a white area in the color space. Thus, by referring to the light box as a reference object, the color shift can be recognized and transferred to a gray object, e.g. the road surface. The choice of the reference object thus makes the method specialized for recognizing and correcting color shifts in color intervals specific to the automotive environment, leading to a wide range of application of the method in automotive.

[0017] In an additional embodiment, it is assumed that each measured color information is described by a U color value and / or a V color value of the YUV color model. That is, one can resort to a common color model suitable for describing the color information of an image pixel. The YUV color model allows for the brightness to be described by a Y value and the color by U and V values, i.e., the U and V color values, for an image pixel. The U and V values ​​cover a color space. This color space can be referred to as a UV color space in which all colors for each Y value can be represented. As a result, when considering only the U and V color values, a two-dimensional color space is mapped from the U axis to the V axis, each of which, for example, ranges from value 0 to value 255. The color information therefore includes two separate color values, i.e., the U and V color values. By showing the two color values ​​together, the color information can be accurately shown. That is, the color of the image pixel can be described in turn.

[0018] In particular, it is assumed that the minimum color information is described by the minimum U and / or minimum V color values, and that said maximum color information is described by the maximum U and / or maximum V color values. Thus, in total, four color extremes are known, which ultimately define a sub-area of ​​the UV color space of the YUV color model.

[0019] In a further embodiment, it is assumed that the minimum and maximum U color values, as well as the minimum and maximum V color values, define a color space region arranged around the center of the UV color space of the YUV color model. The UV color space at U color value 128 and V color value 128 includes the center. Around the center, colors are arranged according to the Y value, ranging from black / dark gray to light gray / white. By arranging the color information area defined for the color at this position, only objects in the image having a color lying between black / dark gray and light gray / white can be corrected with respect to their color information. On the other hand, objects in the image having a different color, i.e., blue, green, red or yellow objects, remain unaffected with respect to their color information. This ensures that the reduction of the color shift is only performed in sub-areas of the image where it is reasonable to reduce the color shift, for example because there is a large surface representation of a gray road surface where the color shift is particularly noticeable.

[0020] In an additional embodiment, it is assumed that the further the measured color information is from the center of the UV color space, the greater the distance between the measured color information and the corrected color information in the UV color space. That is, it is assumed that the degree of correction of the color information varies depending on the individual pixel. This means that, for example, image pixels whose color information is far from the center of the UV color space due to a large color cast are modified more strongly than image pixels that are located closer to the center of the UV color space and therefore have a small color cast in comparison. Thus, for color information of image pixels with a large color cast, the difference between the measured U and V color values ​​and the corrected U and V color values ​​is larger than for image pixels with a small color cast. That is, not all image pixels that require correction in the check step are corrected to the same extent.

[0021] The calculation of the corrected U color value from the measured U color value can be performed, for example, by first subtracting the value 128 from the measured U color value of the image pixel to obtain a difference value from the center of the UV color space. The corrected U color value is calculated as the difference. The measured U color value is the minus and the product of the difference value and the correction factor is the subtrahend. The calculation of the corrected V color value is performed in a similar manner. The correction factor is, for example, 0.75. Alternatively, the correction factor can be 0.5 to 1, 0.6 to 0.9, in particular 0.7 to 0.8. This achieves that the degree of correction applied varies depending on the distance of the measured U color value or the measured V color value, respectively, from the center of the UV color space and thus the individual image pixel. Finally, in this way, color information close to the edge regions of the defined color information gamut is corrected more strongly towards the center, so that, overall, in the provided image, a color representation that is as homogeneous as possible, for example of a gray road surface, can be obtained. By taking into account measured color information and correction coefficients in this way, rather than all image pixels being corrected to the same extent, a gradual correction is possible, or gradients can be taken into account during the correction.

[0022] In an embodiment, it may be envisaged to measure the number of image pixels whose measured color information is greater than the minimum color information and less than the maximum color information. Then, it is checked whether the measured number (measured number) is greater than the minimum number. If the measured number is less than or equal to the minimum number, the alternative minimum color information and the alternative maximum color information are determined by evaluating the image pixels. The color information gamut defined by the alternative minimum color information and the alternative maximum color information is located particularly away from the center of the UV color space of the YUV color model. Then, for example, the above-mentioned method may be carried out taking into account the alternative minimum color information and the alternative maximum color information. Thus, for example, if a car is not placed on a grey road surface, but for example on a grass surface, first an adjustment of the minimum color information and the maximum color information is carried out for this environment of the car. This is because, for example, the method described so far is set, but all or at least some of the image pixels of the captured image of the grass surface do not depict a grey or white object. It may be envisaged that in this case too, it is possible to reduce the color shift of the image pixels, since the alternative minimum and maximum color information can be calculated and taken into account as follows to reduce possible deviations in the green gamut. Firstly, it may be envisaged to carry out a new calibration so that alternative minimum and maximum colour information can be calculated using reference objects with different colours, in particular the colour green, for example.

[0023] Alternatively or additionally, it may be envisaged to calculate the correction color information for the corresponding image pixels only if the number of measurements is greater than the minimum number. That is, for example, the following control step may be provided: in this control step, if, for example, there are few gray image pixels in the image (because the number is less than the minimum number), no further method steps are carried out at all, but it is simply determined that the captured image is not suitable for the method since there are not enough image pixels in the relevant color space region. That is, first it is checked, for example, whether the captured image contains gray and / or white regions in which a correction can be carried out, and therefore is suitable for processing by the method. This avoids unnecessary calculation operations that may occur if the method is carried out continuously without taking into account the number of affected image pixels.

[0024] The minimum number may be expressed as a percentage of the total number of image pixels of the image, for example 5 percent, 10 percent, 20 percent, 30 percent, 40 percent, 50 percent, 60 percent, 70 percent, 80 percent, 90 percent, or especially a value greater than 90 percent and less than 100 percent.

[0025] Furthermore, in the embodiment, it is envisaged to apply a harmonic function to the image before checking the measured colour information of each image pixel. The harmonic function performs brightness and / or colour harmonization on the image. It can therefore be envisaged to perform a brightness and colour harmonization step on the image before checking if the measured colour information is greater than the minimum colour information and less than the maximum colour information. The harmonic function application is particularly meaningful if the image is an overall representation, for example, composed of individual camera images taken under different lighting conditions, as these images show different areas of the automotive environment. Then, by applying the harmonic function, brightness and colour corrections can be performed. In particular, in the transition areas between the two camera images, brightness and colour adjustments are made between the two camera images. However, it is also possible that colour shifts of individual image pixels can be observed in the harmonic image, for example due to different conditions and quality of the camera lenses of a certain camera type. However, such minor colour shifts are typically not corrected by applying the harmonic function, and an inventive method for calculating the corrected colour information is needed. Thus, the early application of the harmonic functions pre-corrects the colors in the image, so that the starting image provides a reasonable base for fine-tuning corrections that can be achieved by calculating the correction color information, which may ultimately result in an optimal reduction of color shifts.

[0026] Furthermore, in an embodiment, it can be envisaged that the image is a partial image of a total image, which is synthesized from at least two partial images. The total image can be synthesized, for example, from four partial images, one of which is taken by a front camera, one by a rear camera and two by respective side cameras arranged in the side mirrors of the vehicle. The synthetic image is obtained by combining or concatenating the individual partial images. Each partial image thus depicts a sub-area of ​​the environment of the vehicle. That is to say, a partial image is given which depicts the environment in the front area of ​​the vehicle, a partial image which depicts the environment in the rear area of ​​the vehicle, a partial image which depicts the environment of the vehicle on the left side in the longitudinal direction of the vehicle and a partial image which depicts the environment of the vehicle on the right side in the longitudinal direction of the vehicle. These four partial images can, for example, be combined into a 360° image of the environment or a top view of the vehicle. For example, it can be envisaged to carry out a step of synthesizing the individual partial images into a total image before applying the harmonic function. The harmonic function allows in particular the border areas between the individual images to be harmonized, i.e. to be adjusted to one another in terms of their brightness and / or colour values. This means that images captured by multiple cameras on the vehicle can be utilized optimally.

[0027] In another embodiment, it is envisaged to synthesize the overall image after checking the measured color information for each image pixel of at least two images. Thus, first the measured color information for each image pixel of at least two different images is checked and possibly corrected. Then the overall image is synthesized from the at least two images. As a result, the overall image comprises at least two partial images. Thus, the individual images of the overall image have already been checked for color information of their image pixels according to the above-mentioned method, and the corresponding image pixels have been corrected by providing corrective color information or have not been corrected by maintaining the measured color information. Now, a harmonic function is applied to the overall image. In the overall area, at least two images depict different sub-areas of the environment of the vehicle, in particular the front area, the rear area and / or the side area of ​​the vehicle. The harmonic function performs luminance and / or color harmony on the overall image as described above. That is to say, the above-mentioned method envisaged for checking the measured color information and possibly calculating corrective color information and providing it to the corresponding image pixels is applied to the partial images, which can only be synthesized and adjusted to each other by the harmonic function.

[0028] In an embodiment, it is assumed that each partial image is a viewing area of ​​a camera image taken by a camera, and that the overall image shows a top view of the environment of the vehicle. It can therefore be assumed that the entire image taken by the camera is not displayed and further considered, but that the viewing area is selected from the entire image. The viewing area is a designation for a viewing window. To display a top view of the environment of the vehicle, for example, a region of the environment that is located at the top in the height direction of the vehicle and is included in the camera image, such as a part of the sky, is not necessary. For this reason, only the viewing area is considered as a partial image with which the overall image is synthesized. This is a display that is particularly suitable for vehicles and can be useful for the driver of the vehicle, for example in parking maneuver assistance.

[0029] In a further embodiment, it is envisaged to recognize at least one object in the image by applying an object recognition algorithm to the image. Only if the recognized object is a road surface, the respective correction color information for the respective image pixel of the image is calculated. Thus, it is checked whether the captured image, partial image or entire image is suitable for the described method by first performing an object check based on the object recognition algorithm and then performing a check as to whether the recognized object is a road surface. The object recognition algorithm is, for example, a computer program and comprises at least one rule or instruction that allows recognition of at least one predefined object in the image. The predefined object may be a road surface. Furthermore, the object recognition algorithm is able to distinguish between several types of objects, i.e. to recognize, for example, several classes of objects. This avoids calculating correction color information for image pixels of an image that is not at all suitable for application of the described method, for example because it does not display a road surface.

[0030] In a further embodiment, it can be assumed that the correction coefficient is predetermined depending on the minimum color information and the maximum color information. For example, a large correction coefficient can be assumed when the distance between the minimum color information and the maximum color information is large compared to when the distance between the minimum color information and the maximum color information is small, and a smaller correction coefficient can be selected. That is, the smaller the color information space defined by the minimum color information and the maximum color information, the smaller the correction coefficient can be selected. And vice versa. This can be achieved to always reduce the color shift reasonably, since the correction coefficient is adjusted to the behavior of the camera lens.

[0031] Furthermore, embodiments are envisaged in which the reference image pixels are taken by a number of cameras, each of which is equipped with a substantially identical camera lens, when performing the calibration method. The cameras can be assigned to a common camera type. The common camera type can be related to a common camera lens, i.e. the cameras each have a substantially identical camera lens. As a result, the cameras have, for example, a common type of camera lens and thus the same camera lens. Substantially identical means that there are maximal differences between the individual camera lenses that are independent of the color information gamut in which the color information of the image pixels can vary. Preferably, the camera used when taking the image also has the camera lens used when performing the calibration method. Thus, the camera used when taking the image is also assigned to a common camera type. In other words, the calibration method is performed for a specific camera lens.

[0032] Preferably, a reference image with at least 10 reference pixels is captured for the reference object, from which the minimum and maximum color information are calculated. In the reference image, for example, an area of ​​the road surface and / or an area of ​​the light box are shown and selected from them. Then, it is calculated or measured in which color information range, i.e. in which color space area, the image pixels depicting the road surface or the light box fluctuate, although they should actually represent the same color information. That is, it is determined whether the reference object in the reference image is depicted by at least one reference pixel with, for example, a reddish or bluish color shift. The color information that is maximally or minimally shifted with respect to the center of the color space is taken as the maximum or minimum color information, respectively. In this regard, it is appropriate that firstly, for a common camera lens, several calibration measurements are performed before the minimum and maximum color information can be provided as predetermined values.

[0033] A further aspect of the invention relates to a computer program product. The computer program product may be, for example, a computer program. The computer program product may be stored in a computer, such as, for example, a control device of a motor vehicle. The computer program product comprises at least one instruction, preferably a plurality of instructions, which causes the computer to carry out the above-mentioned method when the program is executed by the computer, i.e., for example, by a control device of a motor vehicle. The computer is, for example, a computing device. Furthermore, a second computer program product may be provided. According to the second computer program product, a camera takes environmental and / or reference images of the motor vehicle. Furthermore, a third computer program product may be provided. According to the third computer program product, the described calibration method may be carried out.

[0034] A further aspect of the invention relates to a control device for a motor vehicle. The control device is configured to implement the above-mentioned method. The control device comprises a processor device capable of implementing the described method. A processor device for this purpose may comprise at least one microprocessor, and / or at least one microcontroller, and / or at least one FPGA (Field Programmable Gate Array), and / or at least one DSP (Digital Signal Processor). Furthermore, the processor device may include a computer program product. The control device preferably implements the described method.

[0035] The embodiments described in connection with the method according to the invention likewise apply to the computer program product according to the invention as well as to the control device according to the invention, individually or in combination with each other, insofar as applicable and insofar as they are not mutually exclusive.

[0036] A further aspect of the invention may be a motor vehicle equipped with a control device according to the invention. The motor vehicle may be, for example, a car, a truck, a bus and / or a motorcycle. [Brief description of the drawings]

[0037] [Figure 1] 1 is a schematic diagram of a vehicle equipped with a control device and multiple cameras. [Diagram 2] 1 is a schematic signal flow graph of a method for reducing color shift of image pixels in an image of a car captured by a camera; [Diagram 3] Schematic diagram of the UV color space in the YUV color model. [Figure 4] Schematic diagram of the correction of a camera image of an automobile. [Diagram 5] 3 is a schematic signal flow graph of a further method step of the method of FIG. 2; [Figure 6] 3 is a schematic signal flow graph of the method steps of a calibration method for the method of FIG. 2; [Figure 7] Schematic of a signal flow graph for applying an object recognition algorithm to an image. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0038] In Fig. 1, a motor vehicle 1 is depicted which comprises a control device 2. The control device 2 is a computing device which may comprise, for example, at least one microprocessor and / or microcontroller, in other words, the control device 2 of the motor vehicle 1 is a computer which is capable of executing a computer program product, for example a computer program.

[0039] The car 1 is equipped with several cameras, namely a front camera 3, a rear camera 4 and two side cameras 5, each arranged on the side mirrors of the car 1. In a composition of camera images taken by these four cameras of the car 1, for example a 360° environment of the car 1 and / or a planar view of the car 1 can be displayed. The displayable environment is defined by at least the respective capture area of ​​each camera, i.e. the front camera 3, the rear camera 4 or the side cameras 5.

[0040] A display device 6, which is for example a display, in particular a touch-sensitive display, is arranged in the automobile 1. Camera images captured by, for example, the front camera 3, the rear camera 4 and / or the respective side camera 5 can be displayed on the display device 6. For this purpose, the camera images can first be manipulated or processed in the control device 2. The individual cameras, i.e. the front camera 3, the rear camera 4 and / or the side camera 5, can for example be equipped with a fish-eye lens, which allows capturing the environment at 180°. A top view of the environment of the automobile 1 can for example be provided by selecting a corresponding section from each camera image. This means that, for example, the display device 6 can for example display to a user of the automobile 1 the current environment of the automobile 1 at least partially, preferably in its entirety, in order to assist in a parking maneuver.

[0041] 2 illustrates a method for reducing color shifts of image pixels 8 of an image 7 of a car captured by a camera, in particular a front camera 3, a rear camera 4, and / or a side camera 5. The image 7 can be, for example, a camera image captured by the respective camera.

[0042] In a first step S0, an image 7 of the surroundings of the motor vehicle 1 can be captured by the front camera 3, the rear camera 4 and / or the lateral camera 5. The image 7 is composed of individual image pixels 8, the number of which depends on the respective camera. The further method steps S1 to S5 described below are preferably performed for each image pixel 8 of the image 7 captured by the respective camera, i.e. the front camera 3, the rear camera 4 and / or the lateral camera 5.

[0043] In a first method step S1, a measurement of color information 9 describing the color of the image pixels 8 is performed. The color information 9 preferably describes a U color value and / or a V color value of the YUV color model. Thus, the color information 9 preferably comprises two values ​​for each image pixel 8, namely a U color value and a V color value.

[0044] In method step S2, a check is made whether the measured color information (measured color information) 9 is greater than the minimum color information 10 and less than the maximum color information 11. The minimum color information 10 may alternatively be referred to as minimum color information. And the maximum color information 11 may alternatively be referred to as maximum color information. The minimum color information 10 and the maximum color information 11 define a color information gamut within which the reference image pixels describing the reference object fluctuate. For this purpose, the minimum color information 10 may include a minimum U color value 14 and / or a minimum V color value 15. And the maximum color information 11 may include a maximum U color value 16 and / or a maximum V color value 17.

[0045] Only if the measured color information 9 is greater than the minimum color information 10 and less than the maximum color information 11, is a calculation of the correction color information 13 for the image pixel 8 performed in method step S3. The calculation is performed taking into account the measured color information 9 and a correction factor 12, which can be, for example, 0.75. In method step S4, the calculated correction color information 13 is provided instead of the measured color information 9. The correction color information 13 has a reduced color shift compared to the measured color information 9. This means that a correction of the individual color information 9 of the image pixel 8 for which the check step S2 was successful has been achieved in such a way that a homogeneous color distribution in the image 7 is achieved. Furthermore, this result is that if the check step S2 is unsuccessful, i.e. if the measured color information 9 is either greater than the minimum color information 10 or less than the maximum color information 11, the correction color information 13 is not calculated, so that in method step S5 the already measured color information 9 is provided for the respective image pixel 8. Overall, image 7 may ultimately contain image pixels 8 that contain either measured color information 9 or corrected color information 13, depending on which result checking step S2 produced.

[0046] FIG. 3 shows a schematic representation of the UV color space 18 of the YUV color model. Here, a minimum U color value 14, a maximum U color value 16, a minimum V color value 15, and a maximum V color value 17 are exemplarily depicted. These four extreme values ​​are arranged around a center 19 of the UV color space 18. The center 19 of the UV color space 18 has a U value 128 and a V value 128 if the U and V axes of the UV color space 18 each extend between 0 and 255 as is typical in the YUV color model. The four extreme values ​​define a color space region 20 surrounding the center 19 of the UV color space 18. The color space region 20 is the color information range within which a reference pixel describing a reference object fluctuates.

[0047] The UV color space 18 has areas of different colors, which are indicated by different hatching depending on the color. The colors range from magenta 50 to red 51, yellow 52, ​​green 53, and blue 54. Areas of mixed colors between the above-mentioned colors are indicated by overlapping hatching. Around the central point 19, the gray tones are also differentiated from one another, ranging from reddish gray 55 to pure gray 56 to bluish gray 57.

[0048] In Fig. 3, for each image pixel 8, two exemplary color information 9 are depicted, namely a first color information 9a and a second color information 9b, both of which are in the depicted color space region 20. That is to say, when the check is carried out in method step S2, the respective color information 9a, 9b is evaluated as a color information 9 that is greater than the minimum color information 10 and less than the maximum color information 11. For this reason, in method step S3, a corrected color information 13 is calculated for each of these two color information 9a, 9b, namely a first corrected color information 13a and a second corrected color information 13b. It is clear that in the UV color space, the distance 21 between the first measured color information 9a and the first corrected color information 13a is greater than the distance 21 between the second measured color information 9b and the second corrected color information 3b. This is because the further the measured color information 9 is from the center 19 of the UV color space 18, the greater the distance between the measured color information 9 and the corrected color information 13. Here, the first measured color information 9a is farther away from the center than the second measured color information 9b.

[0049] 3, for example, the minimum U color value 14 could be at 124; the maximum U color value 16 could be at 130; the minimum V color value could be at 123; and the maximum V color value could be at 137.

[0050] If the check step S2 is successful, then in order to first calculate the corrected U color value from the measured U color value, a difference value from the center of the UV color space can first be determined by subtracting the numerical value 128 from the measured U color value of the image pixel 8. The corrected U color value is then calculated as the difference. The measured U color value is the minuend and the product of the previously calculated difference value and the correction factor 12 is the subtrahend. The calculation for the corrected V color value is carried out in a similar manner. Thus, in method step S4, the corresponding calculated values ​​are provided for the image 7 as corrected color information 13 which comprises the corrected U and corrected V color values.

[0051] FIG. 4 shows, for example, in a schematic way, that first four images 7 of the four cameras of the car 1 can be predefined. That is, first for a method step S0, an image 7 of the front camera 7 is displayed at the top left, an image 7 of the rear camera 4 at the top right, an image 7 of the left side camera 5 in the longitudinal direction of the car 1 at the bottom left, and an image 7 of the right side camera 5 in the longitudinal direction of the car 1 at the bottom right. In each of the images 7, a viewing area 22, alternatively or additionally called a partial image 23, is depicted. The individual images 7 show the environment of the car 1. In the images 7, the road surface 24 on which the car 1 is located is depicted over a large area. The road surface 24 is grey, in particular a grey between black and dark grey and light grey and white. Furthermore, in each image 7, a parking space mark 25, a shadow 26 of the car 1, a number of background objects 27, for example trees, the sky 28, as well as grass 29 are depicted. Purely by way of example, two areas in image 7 are highlighted in their colour by corresponding hatching. These areas exhibit a reddish grey colour 55. Alternatively, image 7 may contain at least one area of ​​blue-grey colour 57 and / or further areas of reddish grey colour 55. Furthermore, other or alternative colour casts are conceivable.

[0052] In method step S6, the four viewing areas 22 can be combined as respective partial images 23 into the overall image 30. In other words, the individual images 7, in particular the viewing areas 22, can be respective partial images 23 of an overall image 30 combined from a number of partial images 23. Each partial image 23 depicts a sub-area of ​​the environment of the motor vehicle 1, in particular a front area, a rear area and / or a side area of ​​the motor vehicle 1. The image 7 can therefore alternatively be considered as an overall image 30 combined from a number of partial images 23. In the central area of ​​the overall image 30, the motor vehicle 1 is located, which is not captured by the camera of the motor vehicle 1. By considering the four viewing areas 22, the overall image 30 exemplarily shows here a top view of the environment of the motor vehicle 1. In the combined overall image 30, a number of image areas, in which a deviation occurs, for example in red, are highlighted. For this purpose, the same shading as in FIG. 3 is used.

[0053] In method step 7, a harmonic function 31 can be applied to the image 7, here to the overall image 30. The harmonic function 31 performs a luminance harmonization 32 and / or a color harmonization 33 on the image 7 or the overall image 30, respectively. Here, an example image of a harmonic image 34 is shown. It is clear from this image that, for example, the transition areas between the individual viewing areas 22 have been adjusted to one another to achieve a continuous transition in terms of color and luminance between the individual partial images 23. Furthermore, several objects from the individual images 7 of the camera are also visible in the harmonic image 34, i.e., for example, the markings 25 or the shadows 26 of the parking spaces. The harmonic image 34 is thus the result of the combination of the images 7, here the viewing areas 22, in step S6 and the subsequent application of the harmonic function 31 to the overall image 30.

[0054] The method according to the method steps S1 to S3 can then be applied to the harmonic image 34 as image 7. The harmonic image 34 thus corresponds to the image 7 whose individual image pixels 8 have been taken into account in the method steps S1 to S5. A final image 35 is depicted which is provided by the image pixels 8 provided in the method steps S4 and S5, respectively. This results in correction of particularly red-shifted areas of the image 7, i.e. areas of red-tinted grey 55. A correspondingly improved image 7 in the form of the final image 35 is thus provided, with the colour shifts present being reduced. The final image 35 preferably shows only the road surface 24 in pure grey 56 without any noticeable colour cast.

[0055] As an alternative to the sequential arrangement of the method steps according to figure 4, the method steps S1 to S5 can already be carried out immediately after method step S0, and only then is the overall image 30 constructed from the individual partial images 23 (method step S6). Then, in method step S7, i.e. a harmonic function is applied to the composite image, so that the final image 35 can be provided.

[0056] Further possible method steps S8 to S10 are shown diagrammatically in Fig. 5. In method step 8, the number 36 of image pixels 8 whose measured color information 9 is greater than the minimum color information 10 and less than the maximum color information 11 is measured. In method step S9, it is checked whether the measured number 36 is greater than the minimum number 37. If the measured number 36 is less than or equal to the minimum number 37, then in method step 10, a determination of an alternative minimum color information 10' and an alternative maximum color information 11' takes place. The color space region 20 defined by the alternative minimum color information 10' and the alternative maximum color information 11' is arranged in particular at a distance from the center 19 of the UV color space 18 of the YUV color model. For example, it can be envisaged that if the vehicle 1 is not within the area of ​​the centre 19 of the UV colour interval 18 in terms of colour, for example because it is located on a green grassland which does not correspond to the grey hue of the road surface 24, then alternative minimum colour information 10' and alternative maximum colour information 11' are determined and then method steps S1 to S5 are carried out, for example taking into account the alternative minimum colour information 10' and the alternative maximum colour information 11'.

[0057] Furthermore, it can be assumed that the correction color information 13 of the corresponding image pixel 8 is calculated only if the measurement number 36 is greater than the minimum number 37, i.e. the method step S3 is performed, i.e. it can be assumed that it is first checked whether sufficient image pixels 8 in the corresponding color region to be corrected are provided.

[0058] The calibration method is illustrated diagrammatically in Fig. 6. In the calibration method, an image 7 with reference pixels may first be provided by photographing the road surface 24, which is in particular grey, or a light box 39 with the front camera 3, the rear camera 4, the side camera 5 and / or another reference camera 38, for example arranged outside the vehicle. Preferably, at least 10 reference images are taken as image 7 in method step S11, so that the calibration method can be reliably provided.

[0059] In method step S12, a calibration algorithm 40, which comprises at least one rule for calibrating the camera, is applied to the photographed reference pixels and their respective photographic information 9. The reference pixels represent reference objects, i.e. the road surface 24 and / or the light box 39. To this end, the measured color information 9 is always positioned around the center 19 of the UV color space 18. By applying the calibration algorithm 40, the minimum color information 10 and the maximum color information 11 can be determined.

[0060] The cameras employed in method step S11 are all equipped with substantially identical camera lenses, i.e. they can be assigned to a common camera type. The employed camera lenses are therefore comparable with one another at least in terms of the color shifts observed in the images 7 taken with the respective camera lenses. For this reason, it can be concluded that the camera lens-specific color space region 20 is defined by certain extreme values. Depending on the camera lenses of the front camera 3, the rear camera 4 and / or the side camera 5 of the motor vehicle 1, the corresponding minimum and maximum color information 11 determined in method step S12 can be provided to the motor vehicle 1. Furthermore, method steps S11 and S12 can be implemented in the motor vehicle 1, for example on the basis of at least 10 images 7 taken by the respective cameras of the motor vehicle 1.

[0061] An embodiment is illustrated diagrammatically in Figure 7. In this embodiment, in method step S13, an object recognition algorithm 41 is applied to the image 7 in order to recognize at least one object 42 in the image 7. If in method step S14 it is determined that the recognized object 42 is a road surface 24, then method steps S2 to S5 are performed, since it has been determined that the car 1 is located on the grey road surface 24, and thus the above-described method relating to the centre 19 of the UV colour space 18 in the grey region can be performed.

[0062] If the check carried out in method step S14 is not successful, then for example method step S5 can be carried out, i.e. for each image pixel 8 already measured color information 9 can be provided and no corrected color information 13 is calculated or provided. Alternatively or additionally, method step S10 can be carried out, i.e. alternative minimum color information 10' and alternative maximum color information 11' can be determined. Alternatively, in method step S15 the method can be terminated, i.e. method steps S1 to S5 can not be carried out at all since the image 7 is not recognized as suitable for the method since the road surface 24 is not displayed in the image 7.

[0063] Overall, these examples show an algorithm for mitigating the unevenness of the color distribution of the camera lens. It is suggested to apply the described method after applying the harmonic function 31 to the composite overall image 30, which is a top view of the environment of the car 1. This causes the individual image pixels 8, which are distorted in a certain direction in the UV color space 18, to be individually corrected. Each image pixel 8 is considered individually by the method, which checks how far the measured color information 9 of the image pixel 8 is from the center 19 of the UV color space 18, which contains the coordinates (128, 128), with respect to the UV coordinates of the UV color space 18. The road surface 24 is the most homogeneous surface seen in top view by the camera of the car 1 and is therefore the surface that may occupy a large area in the overall image 30, which is synthesized from the individual viewing areas 22. Furthermore, the harmonic function 31 corrects the individual partial images 23 towards the center 19 of the UV color space 18. In the method steps S1 to S5, it is checked how far the respective color information 9 is from the center 19 of the UV color space 18 with respect to its U and V color values. The distance 21 is shifted by shifting the U and V values ​​towards the centre 19 by multiplication with the correction factor 12 as described above. The respective colour information 9 of each image pixel 8 is shifted by a gradient to arrive at the corrected colour information 13. This gradient determines the strength of the non-uniform camera lens performance, which should be measurable by the variance in colour values ​​of a uniform surface of a reference object, given here by the road surface 24 or the light box 39. This allows the correction to be performed only in certain colour regions in the image, while for example unaffected image pixels 8 of different colours, e.g. depicting a parking space marking 25 or a different background object 27, remain unmodified.

Claims

1. A method for reducing color shifts in image pixels (8) of an image (7) of a car (1) taken by a camera, comprising the steps of: The method comprises the steps of: for each image pixel (8) of the image (7) captured by the camera: - measuring (S1) color information (9) describing the color of said image pixels (8), - checking (S2) whether said measured color information (9) is greater than a minimum color information (10) and less than a maximum color information (11), said minimum color information (10) and said maximum color information (11) defining a color information gamut within which a reference image pixel describing a reference object (42) fluctuates, - calculating (S3) corrected color information (13) for the image pixel (8) taking into account the measured color information (9) and a correction coefficient (12) only if the measured color information (9) is greater than the minimum color information (10) and less than the maximum color information (11), and providing (S4) the calculated corrected color information (13) instead of the measured color information (9), the corrected color information (13) having a reduced color shift compared to the measured color information (9).

2. 2. The method according to claim 1, wherein the reference object (42) is a road surface (24), in particular a grey road surface (24), or a light box (39).

3. 3. The method according to claim 1 or 2, wherein each said color information (9, 13) is described by U and / or V color values ​​of the YUV color model, in particular said minimum color information (10) is described by a minimum U color value (14) and / or a minimum V color value (15) and said maximum color information (11) is described by a maximum U color value (16) and / or a maximum V color value (17).

4. 4. The method of claim 2 or 3, wherein the minimum (14) and maximum (16) U color values, and the minimum (15) and maximum (17) V color values ​​define a color space region (20) arranged around a center (19) of a UV color space (18) of the YUV color model.

5. 5. The method of claim 4, wherein the further the measured color information (9) is from the center (19) of the UV color space (18), the greater the distance (21) between the measured color information (9) and the corrected color information (13) in the UV color space (18).

6. Measure (S8) the number (36) of image pixels (8) whose measured color information (9) is greater than the minimum color information (10) and less than the maximum color information (11), and check (S9) whether the measured number (36) is greater than a minimum number (37); - if the measured number (36) is less than or equal to the minimum number (37), determining (S10) an alternative minimum color information (10') and an alternative maximum color information (11') by evaluating the image pixels (8), the alternative minimum color information (10') and the alternative maximum color information (11') being positioned in particular away from the center (19) of the UV color space (18) of the YUV color model, and / or - calculating (S3) the correction color information (13) for the corresponding image pixel (8) only if the measured number (36) is greater than the minimum number (37); The method according to claim 4 or 5.

7. The method according to any one of claims 1 to 6, further comprising applying (S7) a harmonic function (31) to the image (7) before checking the measured color information (9) of each image pixel (8), said harmonic function (31) performing a luminance harmonization (32) and / or a color harmonization (33) on the image (7).

8. 8. The method according to claim 7, wherein the image (7) is a partial image (23) of an overall image (30) synthesized (S6) from at least two partial images (23), each partial image (23) depicting a sub-area of ​​the environment of the vehicle (1), in particular the front area, the rear area and / or the side area of ​​the vehicle (1).

9. 7. The method according to claim 1, further comprising: after checking the measured color information (9) for each image pixel (8) of at least two images (7), synthesizing an overall image (30) comprising each of the at least two images (7) as partial images (23) and applying a harmonic function (31) to the overall image (30), wherein the at least two images (7) depict different sub-areas of the environment of the motor vehicle (1), in particular the front, rear and / or lateral regions of the motor vehicle (1), and wherein the harmonic function (31) performs a luminance harmonization (32) and / or a color harmonization (33) on the overall image (30).

10. 10. The method according to claim 8 or 9, wherein each partial image (23) is a view area (22) of a camera image taken by the camera and the overall image (30) shows a top view of the environment of the vehicle (1).

11. The method according to any one of claims 1 to 10, further comprising applying an object recognition algorithm (41) to the image (7) to recognize (S13) at least one object (42) in the image (7) and calculating (S14) the respective corrected colour information (13) for each image pixel (8) of the image (7) only if the recognized object (42) is a road surface (24).

12. The method according to any one of the preceding claims, wherein the correction factor (12) is predetermined depending on the minimum color information (10) and the maximum color information (11).

13. The method according to any one of claims 1 to 12, wherein the reference image pixels (8) are taken with a plurality of cameras each equipped with a substantially identical camera lens (42) during the implementation of the calibration method, in particular the cameras used during the taking of the images (7) also equipped with said camera lens.

14. A computer program product comprising at least one instruction causing a computer to carry out the method according to any one of claims 1 to 13, when said program is executed by a computer.

15. A control device (2) for a motor vehicle (1), said control device (2) being configured (7) to carry out a method according to any one of claims 1 to 13.

Images