Image harmonization method, image processing device, camera system and motor vehicle

Abstract

Method for image harmonization of at least two images (FV, RV, ML, MR), wherein a respective image (FV, RV, ML, MR) was captured by a respective camera (3, 4, 5, 6) of a camera system (2), the method comprising the steps: - for each of the at least two images (FV, RV, ML, MR) performing a colour analysis of the image (FV, RV, ML, MR); - determining at least one color correction value (CC) according to a first determination rule (R1) depending on a result of the color analysis; - applying the at least one color correction value (CC) to at least a part of the respective image (FV, RV, ML, MR); characterized by the steps: - depending on the result of the color analysis, determining whether at least one predetermined condition indicating artificial lighting conditions is met; and - in the event that the at least one predetermined condition is met, determining the at least one color correction value (CC) according to a second determination rule (R2) instead of according to the first determination rule (R1).

Description

[0001] The invention relates to a method for image harmonization of at least two images, wherein a respective image was captured by a respective camera of a camera system. For each of the at least two images, a color analysis of the image is performed, and at least one color correction value is determined according to a determination rule depending on a result of the color analysis. The at least one color correction value is then applied to at least part of the respective image. The invention also relates to an image processing device, a camera system, and a motor vehicle.

[0002] In existing multi-camera automotive vision systems, various types of views can be generated using multiple camera images. In such a visualization involving multiple cameras and an electronic processing unit, the raw camera images are first mapped onto a target viewport and merged, thus creating a mosaic image representing the view from specific three-dimensional points around the vehicle. In the simplest case, the virtual camera is positioned above the vehicle, looking down from the highest central position, and the projective surface represents a flat two-dimensional plane. This view is commonly referred to as a top view.More general cases include, for example, a spherical view, where the projective two-dimensional surface has the shape of a sphere, and the virtual camera or virtual viewpoint is located at an arbitrary position in three-dimensional space.

[0003] In plan views and any other views that require the fusion of more than one camera input image, a harmonization algorithm is typically applied to harmonize the fused camera images in terms of brightness or luminance and chrominance. Such harmonization can at least partially correct any differences in luminance and chrominance between the different images.

[0004] For example, US 2015 / 0071534 A1 describes a method for harmonizing a combined image, comprising receiving a respective image or image frame from a respective one of two or more cameras, wherein the images from two of the cameras represent the same area in an overlap region. Furthermore, pixel statistics of at least some of the pixels in the overlap region are measured for the image from each of the two cameras, a difference in the pixel statistics of the images from a respective one of the two cameras in the overlap region is determined, and a correction factor is calculated that is predicted to produce a reduction in the difference in the pixel statistics when applied to the image from either of the two cameras.

[0005] Furthermore, EP 2 701 120 A2 describes an improved alpha blending of images from a camera system of a motor vehicle. Here, alpha blending is applied to two images in an overlapping area by determining an alpha matrix with alpha values ​​for pixels in the overlapping area that define transparency values ​​for one image and / or opacity values ​​for the other image. Respective brightness values ​​of the images, at least in the overlapping area, are determined, and the alpha matrix is ​​determined depending on the brightness values.

[0006] Using these harmonization methods, harmonization is achieved by averaging the luminance and chrominance across different images. However, such averaging may not always be advantageous. In particular, a motor vehicle performs many different maneuvers, such as parking, exiting a parking space, driving around a parking space, entering a tunnel, and so on. Thus, the multi-camera surround-view system is exposed to an enormous spectrum of environments, dynamic brightness ranges, and colored objects in all of these scenarios. Such environments and objects that the surround-view system must adequately display on the display unit include grass areas, lighting conditions when entering a garage and a tunnel, three-dimensional objects in the views, such as other motor vehicles, road cones, and so on. Therefore, a motor vehicle can be surrounded by various scene environments.Therefore, the harmonization should be able to distinguish cases where the vehicle is surrounded by a colored texture, such as green grass, or by a colored object, such as a red vehicle, from the actual color and brightness discrepancy of the environment, since it is obviously not desirable for colored textures and objects to be corrected for the purpose of harmonization.

[0007] Therefore, it is an object of the present invention to provide a method for image harmonization, an image processing device, a camera system and a motor vehicle which enable image harmonization which is better adapted to different environmental conditions and situations.

[0008] This object is achieved by a method, an image processing device, a camera system, and a motor vehicle having the features specified in the respective independent claims. Advantageous embodiments of the invention are the subject of the dependent claims, the description, and the figures.

[0009] A method according to the invention serves for image harmonisation of at least two images, wherein each image was captured by a respective camera of a camera system. A colour analysis of the image is carried out for each of the at least two images. Furthermore, at least one colour correction value is determined according to a first determination rule depending on a result of the colour analysis. Furthermore, the at least one colour correction value is applied to at least part of the respective image. Furthermore, depending on the result of the colour analysis, it is determined whether at least one predetermined condition indicating artificial lighting conditions is met, and if the at least one predetermined condition is met, the at least one colour correction value is determined according to a second determination rule instead of the first determination rule.

[0010] By determining the at least one color correction value according to the first or second determination rule depending on whether a predetermined condition is met or not, in particular one that indicates artificial lighting conditions, much better adaptations of the harmonization method to different ambient conditions, in particular those caused by different lighting conditions, can be provided. The invention is based on several insights. First, as already described at the beginning, it would be desirable for strong colors that are actually part of the scene not to be corrected and instead to be displayed as they are.However, harmonization that depends only on the presence or absence of strong colors in images would fail in the following situation: in the case of a vehicle entering a garage with artificial lighting inside, or a tunnel that also has artificial lighting, this is considered a particularly difficult scene to harmonize because the interior lighting may be amber and close to some color casts introduced by the camera. In a digital representation, the chrominance values ​​from each camera for the scene would be different due to the cameras' positions within such complex, highly dynamic environmental conditions. The difficulty with the tunnel case is that the front and rear cameras can partially see the outside world as well as the lighting inside the tunnel.Therefore, the color casts from the front and rear cameras may be less pronounced than the color casts in images captured by, for example, a left and right mirror camera. This is because the left and right mirror cameras only see the lighting inside the tunnel. Therefore, their color casts are quite sensitive to the tunnel lighting conditions. A color harmonization algorithm that would only depend on the presence or absence of strong colors within an image to decide whether correction is necessary would, in such a case, decide to correct the images from the front and rear cameras, since their color casts are not as strong and would therefore be classified as a color cast that should be corrected.However, the images from the left and right mirror cameras would not be corrected, as the color harmonization algorithm would then interpret the strong color within the images as a colored object or texture that should be displayed as is and not corrected. As a result, the color harmonization algorithm would decide to correct the images from the front and rear cameras, but not those from the left and right mirror cameras. However, this is undesirable, as the result would be even more visually disturbing than if none of the image textures were corrected for color cast.

[0011] However, this problem can be solved by introducing a predetermined condition indicating artificial lighting conditions. Consequently, the manner in which an image is corrected using the at least one color correction value can be made dependent on whether artificial lighting conditions prevail or not. Thus, the harmonization algorithm can advantageously automatically detect or recognize that the vehicle is surrounded by artificial interior lighting or any lighting with a color temperature other than daylight at midday, and adjust the determination of the at least one color correction value according to the second determination rule instead of the first determination rule.

[0012] According to one embodiment of the invention, when the at least one color correction value is determined according to the first determination rule, the at least one color correction value is calculated as a composition of a relative color correction value, which is determined in particular as a function of a relative color tone difference between two images of the at least two images, and of an absolute color correction value, in particular which is determined for a respective image independently of the other images of the at least two images. Thus, under normal conditions, as defined later, a color correction can be provided which advantageously takes into account a relative correction as well as an absolute correction. By means of the relative correction, as provided by the relative color correction value, different color tones of different images can be adjusted relative to one another, in particular on a neighboring camera basis.In this way, different color casts can be adjusted to one another, for example, like averaging. This can reduce differences in color tones between different images, but a global color cast of the overall image cannot be reliably avoided in every situation. Therefore, it is very advantageous to also provide an absolute color correction value that allows absolute correction of the chrominance, and in particular, also of the luminance. Such an absolute correction can, for example, be based on some kind of defined reference, similar to the way white balance is performed. In particular, a gray area around the center of the UV color plane of the YUV color model is a suitable reference for absolute color correction.

[0013] Advantageously, the determination of the at least one color correction value differs when artificial lighting conditions are detected. In particular, there are many advantageous options for determining the at least one color correction value in the case of artificial lighting conditions, all of which have in common that no absolute color correction is performed. These options are explained below.

[0014] According to a first possibility, when the at least one color correction value is determined according to the second determination rule, a previous color correction value, which was determined in particular under predetermined normal lighting conditions, is set as the at least one current correction value. In other words, if the predetermined condition indicating artificial lighting conditions is met, the color correction values ​​are no longer updated; rather, the color correction calculation is frozen, and the color correction is applied using only the previous color correction values ​​calculated in an environment deemed reliable for its calculation. This is considered to be an outdoor environment with normal lighting conditions. Thus, the environment surrounding the vehicle can be adequately represented, which is considered to be the case when freezing the color correction estimate is performed.

[0015] According to an advantageous embodiment of the invention, when the at least one color correction value is determined according to the second determination rule, exclusively a relative correction value is determined as the at least one color correction value, wherein the relative color correction value is determined as a function of the relative hue difference between two of the at least two images. In other words, the relative color correction value can be determined as previously described. This allows at least a relative correction of the images with respect to one another.

[0016] Using only previous values ​​for at least one color correction value is still better than an algorithmic approach, where a final color correction value is calculated as a combination of the absolute and relative corrections for harmonization. On the other hand, performing only relative color correction has the advantage of at least partially compensating for color casts caused by artificial lighting conditions. However, using only relative corrections may not be entirely reliable, as it aims to estimate color corrections so that relatively matched color tones are obtained, especially across neighboring cameras, which can lead to unstable execution in some cases.

[0017] Therefore, a very advantageous embodiment of the invention is that, when the at least one color correction value is determined according to the second determination rule, the at least one color correction value is determined as a weighted average of a previous color correction value, which was determined in particular under predetermined normal lighting conditions, and a relative color correction value, which is determined in particular as a function of a relative hue difference between two of the at least two images. This advantageously combines the advantages of the previously described options for determining the color correction value.

[0018] Furthermore, a parameter defining the weighting of the weighted average can be adjusted depending on at least one defined situation parameter. This provides even more flexibility for adapting to different situations.

[0019] The following describes how to determine whether artificial lighting conditions prevail or not. In particular, if at least one condition is not met, the lighting conditions can then be classified as normal lighting conditions.

[0020] According to an advantageous embodiment of the invention, determining whether the at least one predetermined condition indicating artificial lighting conditions is met involves determining whether the image has at least one predetermined, highly colored region. This is particularly advantageous because highly colored regions in the image are an indicator of artificial lighting conditions. Furthermore, if a colored object is present in the image, a highly colored region in the image would also be determined. Thus, advantageously, adaptation options can be provided in an appropriate manner not only with regard to artificial lighting conditions, but also with regard to a highly colored ambient scene.This is because when the predetermined condition indicating artificial lighting is met, no absolute color correction is performed on the image. This means that even if strong colored objects are present in the image, these strong colors are not corrected or compensated for by absolute correction. This allows both situations where artificial lighting prevails and situations where colored objects are present in the image to be handled appropriately.

[0021] To check whether the image has at least one predetermined strongly colored region, an average chrominance value can be determined and whether the chrominance value lies within a predetermined chrominance range or not. If the determined chrominance value lies outside the predetermined chrominance range, it is determined that the image has at least one predetermined strongly colored region. For example, the chrominance value can be determined by two independent chrominance components U and V according to the YUV color model.

[0022] In other words, a predetermined range can be determined in the UV color plane, particularly around the center of the U and V axes, which define a gray area. Therefore, chrominance values ​​within this range indicate that the respective image does not contain any strong colors. In particular, this gray area around the center of the UV color plane can also serve as a suitable reference value for absolute color correction as described above.

[0023] However, although the YUV color model provides a very simple way to determine whether an image contains strong colors or not, other color models, for example the RGB color space, can also be used in a similar manner, where one or more corresponding regions that do not include strong colors can also be defined. Furthermore, determining whether the at least one predetermined condition indicating artificial lighting conditions is met is preferably performed separately for each of the at least two images. In other words, a decision can be made separately for each image whether color correction is performed according to the first or second determination rule. This allows for particularly good adaptations to respective situations.

[0024] According to a further advantageous embodiment of the invention, in order to determine whether the at least one predetermined condition indicating artificial lighting conditions is met, it is checked for each of the at least two images whether the respective image has at least one predetermined strongly colored area, and it is determined for each respective image that the condition is met if at least one of the two following criteria is met: the first criterion is that the respective image has at least one predefined strongly colored area, and the second criterion is that more than one of all of the at least two images has at least one predetermined strongly colored area.

[0025] Thus, for example, if it is determined for a first image that the image has a predetermined strongly colored region, a correction value is calculated according to the second determination rule as described above. If the first image does not have any strongly colored regions, but at least two of the other images captured by other cameras in the camera system have such strongly colored regions, then a color correction value is also determined for this first image according to the second determination rule. In all other cases, a correction value is determined for the first image according to the first determination rule. This is done in the same way for all images from the different cameras in the same frame.This decision logic is very advantageous because it does not affect the implementation of harmonization in a non-indoor environment, which is also important, so there is no need to introduce a separate set of algorithm parameters. For example, in the case of grass next to the vehicle, only the image from the specific camera in question is corrected according to the second determination rule, but all other cameras are color-corrected according to the first determination rule, also applying the absolute correction.On the other hand, if the vehicle is traveling through a tunnel and it is determined, for example, that the image from the front camera does not contain any strongly colored areas, the image from the front camera is nevertheless corrected using the color correction value determined according to the second determination rule, since in this case it would be determined that the respective images from the left and right mirror cameras contain strongly colored areas. It is very important to note that this decision logic is very advantageous because it still allows for these two cases to be handled differently—namely, when strongly colored objects are actually present in the scene and when strong colors result from artificial lighting conditions—thus allowing for very good situational adaptation.

[0026] Ultimately, an overall image, especially a top-down view, can be created as a composite image based on the at least two images. Due to the harmonization of the individual images described above, the overall image will have a consistent and highly consistent appearance, even when environmental conditions and driving situations vary greatly.

[0027] The invention also relates to an image processing device for a camera system of a vehicle, in particular a motor vehicle, wherein the image processing device is designed to carry out a method according to the invention or one of its embodiments.

[0028] Furthermore, the invention also relates to a camera system for a vehicle comprising at least two cameras, preferably four cameras, namely a front camera, a rear camera, a left mirror camera and a right mirror camera, in order to capture a respective image, and an image processing device according to the invention.

[0029] Furthermore, the invention also relates to a vehicle, in particular a motor vehicle, which has a camera system according to the invention.

[0030] The advantages described with reference to the method according to the invention and its embodiments apply equally to the image processing device according to the invention, the camera system according to the invention and the vehicle according to the invention.

[0031] Further features of the invention emerge from the claims, the figures and the description of the figures. The features and combinations of features mentioned above in the description as well as the features and combinations of features mentioned below in the description of the figures and / or shown alone in the figures can be used not only in the respectively specified combination, but also in other combinations without departing from the scope of the invention. Thus, embodiments are to be regarded as encompassed and disclosed by the invention that are not explicitly shown and explained in the figures, but which emerge and can be produced by separate combinations of features from the explained embodiments. Embodiments and combinations of features are also to be regarded as disclosed that therefore do not have all the features of an originally formulated independent claim.Furthermore, embodiments and combinations of features are to be regarded as disclosed, in particular by the embodiments set out above, which go beyond or deviate from the combinations of features set out in the reliances of the claims.

[0032] Showing: Fig. 1 is a schematic representation of a motor vehicle with a camera system according to an embodiment of the invention; Fig. 2 shows a schematic representation of image regions of an image captured by respective cameras of the camera system, on the basis of which an overall image is generated, according to an embodiment of the invention; Fig. 3 is a schematic representation of the UV color plane of the YUV color model and a predetermined chrominance range used to check whether an image has a strongly colored area or not; and Fig. 4 is a flowchart illustrating a method for image harmonization according to an embodiment of the invention.

[0033] Fig. 1 shows a schematic representation of a vehicle 1 with a camera system 2 according to an exemplary embodiment of the invention. The camera system 2 is designed as a surround-view camera system and, in this example, has a front camera 3, a rear camera 4, a left mirror camera 5, and a right mirror camera 6. Each of the cameras 3, 4, 5, 6 is designed as a wide-angle camera and can have a fisheye lens or similar optics. Therefore, a very large field of view can be captured by each of the cameras 3, 4, 5, 6. In this example, the front camera 3 has a corresponding field of view FOV1, which is delimited in the horizontal plane by the illustrated boundary lines 3a. Analogously, the rear camera 4 has a corresponding field of view FOV2, which is delimited in the horizontal plane by the corresponding boundary lines 4a.The left mirror camera 5 has a corresponding field of view FOV3, which is delimited in the horizontal plane by the illustrated boundary lines 5a, and the right mirror camera 6 has a corresponding field of view FOV4, which is delimited in the horizontal plane by the illustrated boundary lines 6a. Therefore, each of the cameras 3, 4, 5, 6 is designed to capture a corresponding part 7a, 7b, 7c, 7d of the surroundings 7 of the vehicle 1. Furthermore, the fields of view FOV1, FOV2, FOV3, FOV4 overlap in pairs. The camera system 2 also has an image processing module 8, which harmonizes the images captured by the respective cameras 3, 4, 5, 6.Before harmonization, the image processing module 8 maps the acquired images onto a target surface, such as a sphere or a flat two-dimensional plane, and subsequently fuses and renders the images onto the viewport of an arbitrarily selected virtual camera, namely the virtual viewpoint. Corresponding image regions of each image acquired by the respective cameras 3, 4, 5, 6 can be determined depending on the determined virtual viewpoint. This has the advantage that only parts of the acquired images need to be harmonized, analyzed, and modified, and computing power can be saved because parts of the images do not need to be analyzed or modified since they are not visible from the selected perspective anyway. These image regions are in . Fig. 2 shown.

[0034] Fig. Figure 2 shows a schematic representation of image regions of images captured by the various cameras 3, 4, 5, and 6. The image region captured by the front camera 3 is designated FV, the image region captured by the rear camera 4 is designated RV, the image region captured by the left mirror camera 5 is designated ML, and the image region captured by the right mirror camera 6 is designated MR. Furthermore, the respective image regions have overlapping portions. In particular, parts 9a, 9b of the image areas FV, ​​ML of the front camera 3 and the left mirror camera 5, parts 10a, 10b of the image areas FV, ​​MR of the front camera 3 and the right mirror camera 6, parts 11a, 11b of the image areas MR, RV of the right mirror camera 6 and the rear camera 4, and parts 12a, 12b of the image areas RV, ML of the rear camera 4 and the left mirror camera 5 overlap.These overlapping parts 9a, 9b, 10a, 10b, 11a, 11b, 12a, 12b therefore represent the most important areas, since by comparing these respective overlapping parts 9a, 9b, 10a, 10b, 11a, 11b, 12a, 12b, differences in the image characteristics, in particular with regard to brightness and colour, can be derived, which are to be minimised in the context of harmonisation.

[0035] To achieve this, all relevant image areas FV, ​​RV, ML, MR from each camera 3, 4, 5, 6 are first analyzed with respect to chrominance. The UV components of each area FV, RV, ML, MR of each input image are analyzed and mapped to the UV color plane, as shown in Fig. 3 shown, illustrated.

[0036] In particular, Fig. 3 shows a schematic representation of the UV color plane 13 of the YUV color model. Within this UV color plane 13, a chrominance range 14 is defined around the intersection of the U and V axes, which essentially comprises grayscale colors, while chrominance values ​​outside this range 14 can be considered strong colors. If the vehicle 1 travels through an artificially lit tunnel or any other artificially lit environment, this will influence the colors in the captured images, particularly those from the right and left mirror cameras 5, 6. Therefore, artificially lit environments can be detected by checking whether the chrominance values ​​of the respective image areas FV, ​​RV, MR, ML lie within or outside the defined chrominance range 14.

[0037] In this Fig. In the example shown in Figure 3, the chrominance values ​​of the respective image areas FV, ​​MR, RV, ML are each marked by an X in the UV color plane 13. In this example, the chrominance values ​​F ML , F MR of the image areas MR, ML of the left and right mirror cameras 5, 6 outside the defined chrominance range 14, while the chrominance values ​​F FV , F RVthe image areas FV, ​​RV of the front and rear cameras 3, 4 lie within the defined chrominance range 14. This is typical for a tunnel driving scenario, since in a tunnel the front camera 3 and the rear camera 4 usually see at least partially the outside world as well as the lighting inside the tunnel. The left and right mirror cameras 5, 6, however, only see the lighting inside the tunnel and therefore exhibit much stronger color casts. In the same way, colored textures in an image, such as green grass or brightly colored objects, can also be advantageously detected, since image areas that exhibit such strong colors due to color textures or objects would also have chrominance values ​​that would lie outside the defined chrominance range 14.Therefore, the color analysis of the respective image areas FV, ​​RV, ML, MR with reference to the UV color plane 13 is particularly advantageous in order to be able to detect such situations and adapt the harmonization accordingly, which will now be described in more detail with reference to . Fig. 4 is explained.

[0038] Fig. 4 shows a flowchart illustrating a method for image harmonization according to an embodiment of the invention. The method begins in step S1, in which respective images are captured by the respective cameras 3, 4, 5, 6 of the vehicle 1, and at least one color analysis of the captured images is performed. Preferably, a brightness analysis is also performed, since a harmonization of the respective images or at least their relevant regions FV, ​​RV, ML, MR with respect to brightness is also performed. This can, however, be done according to known standard brightness harmonization methods.

[0039] Based on the color analysis of the respective image areas, it can be determined whether the respective images contain any strongly colored areas, as with reference to Fig. 3. For this purpose, for each of the images or each of the image areas FV, ​​RV, MR, ML, the respective chrominance values ​​F ML , F MR , F FV , F RVare determined, and it can be checked whether these lie within the defined chrominance range 14 or not. If they lie outside, it is determined that the respective images or image areas FV, ​​RV, MR, ML have strong colors; otherwise, they are not. Thus, in step S2, it is checked whether at least two of the respective image areas FV, ​​RV, MR, ML have strong colors. If this is the case, at least one color correction value CC is determined for each image or image area FV, RV, MR, ML according to a second determination rule R2, which can be described by the following formula: CCi=α⋅RCi+(1−α)⋅PCi, where CC denotes the current color correction value for the respective camera 3, 4, 5, 6, RC represents a relative color correction value for the respective camera 3, 4, 5, 6, PC represents a previous color correction value of the respective camera 3, 4, 5, 6, α represents a parameter that can be selected from the interval [0; 1], and i represents an index indicating the respective camera 3, 4, 5, 6.

[0040] α can be fixed, for example, α can be equal to 0.4 or 0.2. Generally, it is preferred that α be between 0 and 0.5. Alternatively, α can be dynamically adjusted depending on a defined situational parameter, providing even more flexibility for adaptation to different situations.

[0041] According to this second determination rule R2, the current color correction value CC can advantageously be calculated by a weighted average of the relative correction value RC and a previous color correction value PC. This avoids an absolute correction, which is very advantageous in situations where images exhibit strong colors due to strongly colored objects in the image or colored textures, as well as in situations where the vehicle 1 travels through an artificially lit tunnel or other artificially lit environment.

[0042] Furthermore, if it is determined in step S2 that at least two images or image regions do not have strong colors, a check is carried out in step S4 to determine whether a respective image or a respective image region FV, RV, MR, ML has a strongly colored region as previously described. If the respective image or image region FV, RV, MR, ML has a strong color, then the current color correction value CC is also determined for this image or image region FV, RV, MR, ML according to the second determination rule R2 in step S5, and then the resulting color correction value CC is applied to the respective image. Otherwise, the current color correction value CC is determined according to a first determination rule R1 in step S6 and applied to the respective image.When the current color correction value CC is determined according to the first determination rule R1, the color correction value CC is calculated as a combination of the relative correction value RC and an absolute correction value AC. This color correction is particularly advantageous for non-artificially lit environments.

[0043] To implement this logic, each camera 3, 4, 5, 6 can be assigned a flag: flagML, flagMR, flagFV, flagRV.

[0044] If any UV color component of the cameras lies outside the correction limit, namely the defined chrominance range 14 on the UV color space map 13, the corresponding camera flag is set to 1.0. For example, in the tunnel flow, the resulting flags could be: flagML = 1.0; flagMR = 1.0; flagFV = 0.0; flagRV = 0.0.

[0045] In addition, a flag can be assigned to the sum of all camera flags as “TotalFlag” in the following way: TotalFlag=flagML+flagMR+flagFV+flagRV.

[0046] Then the current camera flag status and the status of the TotalFlag are checked according to the following logic: If (current camera flag=1.0) or (TotalFlag>1.0)…

[0047] Therefore, it is first checked whether the current flags of cameras 3, 4, 5, 6 have been set to 1.0 or whether at least two of the other cameras 3, 4, 5, 6 are outside the boundary 14. This allows detection of whether or not vehicle 1 is surrounded by artificial lighting.

[0048] If either of these two cases applies, the color correction value for each camera is calculated according to one of the following ways: Either only the previously estimated color corrections PC are used as the current correction values ​​CC, or a relative color correction estimation is performed, resulting in the relative correction value RC, and excluding absolute camera color correction as performed in the normal case scenario. Relative color correction refers to calculating a color correction for each camera 3, 4, 5, 6 based on the relative hue difference between neighboring cameras. A third possibility would be the combination of the previous possibilities as a weighted sum, where the weights can be set by a parameter α as described previously. For example, α can be set within a certain dynamic range, for example, between 0 and 0.5, by a sequence evaluation.For example, if α is set equal to 0.2, a reduced relative correction and previous correction values ​​are used.

[0049] This method is particularly advantageous because it does not degrade the harmonization performance in a non-indoor environment, which is also important, eliminating the need to introduce a separate set of algorithm parameters. If it is determined that the vehicle is in an artificially lit scenario, the chrominance disparity can be mitigated by calculating the corrections using one of the methods described above. This allows for automatic adaptation of the harmonization to different lighting situations.

Claims

[1] Method for image harmonization of at least two images (FV, RV, ML, MR), wherein a respective image (FV, RV, ML, MR) was captured by a respective camera (3, 4, 5, 6) of a camera system (2), the method comprising the steps: - for each of the at least two images (FV, RV, ML, MR) performing a colour analysis of the image (FV, RV, ML, MR); - determining at least one color correction value (CC) according to a first determination rule (R1) depending on a result of the color analysis; - applying the at least one color correction value (CC) to at least a part of the respective image (FV, RV, ML, MR); characterized by the steps: - depending on the result of the color analysis, determining whether at least one predetermined condition indicating artificial lighting conditions is met; and - in the event that the at least one predetermined condition is met, determining the at least one color correction value (CC) according to a second determination rule (R2) instead of according to the first determination rule (R1). [2] Method according to claim 1, characterized by that, when the at least one color correction value (CC) is determined according to the first determination rule (R1), the at least one color correction value (CC) is calculated as a composition of a relative color correction value (RC), which is determined in particular as a function of a relative color tone difference between two images (FV, RV, ML, MR) of the at least two images (FV, RV, ML, MR), and of an absolute color correction value (AC), in particular which is determined for a respective image (FV, RV, ML, MR) independently of the other images (FV, RV, ML, MR) of the at least two images (FV, RV, ML, MR). [3] Method according to one of the preceding claims, characterized bythat, when the at least one color correction value is determined according to the second determination rule (R2), a previous color correction value (PC), which was determined in particular in predetermined normal lighting conditions, is set as the at least one current correction value (CC). [4] Method according to one of claims 1 or 2, characterized by that, when the at least one color correction value is determined according to the second determination rule (R2), exclusively a relative correction value (RC,) is determined as the at least one color correction value (CC), wherein the relative color correction value (RC) is determined as a function of the relative color tone distance between two images (FV, RV, ML, MR) of the at least two images (FV, RV, ML, MR). [5] Method according to one of claims 1 or 2, characterized bythat, when the at least one color correction value (CC) is determined according to the second determination rule (R2), the at least one color correction value (CC) is determined as a weighted average of a previous color correction value (PC), which was determined in particular in predetermined normal lighting conditions, and a relative color correction value (RC), which is determined in particular as a function of a relative color tone distance between two images (FV, RV, ML, MR) of the at least two images (FV, RV, ML, MR). [6] Method according to claim 5, characterized by that a parameter (α) which defines the weighting of the weighted averaging is adjustable depending on at least one defined situation parameter. [7] Method according to one of the preceding claims, characterized bythat the determination of whether the at least one predetermined condition indicating artificial lighting conditions is met includes determining whether the image (FV, RV, ML, MR) has at least one predetermined strongly colored area. [8] Method according to claim 7, characterized by in that, in order to check whether the image (FV, RV, ML, MR) has at least one predetermined strongly colored area, an average chrominance value is determined and it is checked whether the chrominance value lies within a predetermined chrominance range or not, and in the event that the determined chrominance value lies outside the predetermined chrominance range, it is determined that the image (FV, RV, ML, MR) has at least one predetermined strongly colored area. [9] Method according to claim 8, characterized by that the chrominance value is determined by two independent chrominance components U and V according to the YUV color model. [10] Method according to one of the preceding claims, characterized by that the determination of whether the at least one predetermined condition indicating artificial lighting conditions is met is carried out separately for each of the at least two images (FV, RV, ML, MR). [11] Method according to one of the preceding claims, characterized by in that, to determine whether the at least one predetermined condition indicating artificial lighting conditions is met, it is checked for each of the at least two images (FV, RV, ML, MR) whether the respective image (FV, RV, ML, MR) has at least one predetermined strongly colored area, and it is determined for each of the respective images (FV, RV, ML, MR) that the condition is met if at least one of the following criteria is met: a) the respective image (FV, RV, ML, MR) has at least one predetermined strongly coloured area; b) more than one of the at least two images (FV, RV, ML, MR) has at least one predetermined strongly colored area. [12] Method according to one of the preceding claims, characterized by that an overall image, in particular a top view image, is generated as a composite image (FV, RV, ML, MR) based on the at least two images (FV, RV, ML, MR). [13] Image processing device (8) for a camera system (2) of a motor vehicle (1), which is designed to carry out a method according to one of the preceding claims. [14] Camera system (2) for a motor vehicle (1) with at least two cameras (3, 4, 5, 6) for capturing a respective image (FV, RV, ML, MR) and an image processing device (8) according to claim 13. [15] Motor vehicle (1) with a camera system (2) according to claim 14.

Images