Method for providing color correction for particular camera sensor

Abstract

The invention relates to a method for providing color correction for a specific camera sensor, comprising the steps of: providing a reference image, which is generated from the detection of the specific camera sensor, performing color interpolation of the reference image to provide an interpolated image, determining at least one region of red, green and blue to be inspected in the interpolated image, and correcting the color correction for the specific camera sensor. An average value of the color values is formed in each region to be inspected to obtain a respective generated average color, a respective reference color is assigned to each generated average color, where the reference colors comprise at least red, green and blue, a color correction matrix is generated based on the color values of the reference image, intermediate variables for red, green and blue are calculated based on the color correction matrix, respectively, for which respective differences between color values of the reference image and the intermediate variables are minimized in order to provide color correction for a particular camera sensor. The invention also relates to a computer program, a device and a storage medium therefor.

Description

Technical Field

[0001] The present invention relates to a method for providing color correction for a specific camera sensor. The present invention also relates to a computer program, a device, and a storage medium for this purpose. Background Art

[0002] A color correction matrix (English: "Color Correction Matrix", CCM) is a tool in digital image processing that is used to correct and adapt colors in a digital image. The main purpose of the color correction matrix is to optimize the color reproduction of an image sensor by correcting the color deviation that can be formed by the lens and sensor of a recording device through the color correction matrix.

[0003] The color correction matrix includes a matrix of values applied to the color channels of an image to correct the colors. Typically, the color correction matrix includes a 3×3 matrix, where each row of the matrix represents the red, green, and blue channels of the image. By multiplying the color values of a pixel by the matrix, color deviation can be corrected, color saturation can be adapted, and the overall color balance can be improved.

[0004] For example, in the prior art, weighted sums are used to define the color correction matrix, and look-up tables are mostly also used in practice to reconstruct the image. Usually, further information about the current camera sensor or special existing hardware is also required, and this information must be obtained, for example, based on measurements of the wavelengths of the recorded images parallel to the camera sensor.

[0005] For example, document US20090268044A1 describes a method for adjusting the pixel colors of an image. Here, after performing white balance on the original image data of a digital image, the white-balanced image data is forwarded as a color vector in a color space to a color correction module for color adaptation by means of a color correction matrix.

[0006] Document US20110019913A1 discloses a method for generating a color correction matrix (CCM) for an image sensor. Here, the quantum efficiency spectrum (QE) of the pixels of an image sensor irradiated by a physical light source is measured. Then, the color values of the image sensor and the color values in a predetermined color space are determined based on the QE spectrum and predetermined reference data important for deriving the color values. Finally, a color correction matrix for the image sensor is generated by applying an adaptation algorithm to the color values of the image sensor and the color values in the preset color space. Summary of the Invention

[0007] The subject matter of the present invention is a method having the features of claim 1, a computer program having the features of claim 8, a device having the features of claim 9, and a computer-readable storage medium having the features of claim 10. Further features and details of the present invention result from the respective dependent claims, the description, and the drawings. Herein, the features and details described in connection with the method according to the present invention are of course also applicable in connection with the computer program according to the present invention, the device according to the present invention, and the computer-readable storage medium according to the present invention, and vice versa, such that with respect to the disclosure of the various aspects of the present invention, reference is always made to or can always be made to each other.

[0008] The subject matter of the present invention is in particular a method for providing color correction for a specific camera sensor, comprising the following steps, wherein the steps can be performed repeatedly and / or sequentially. The camera sensor preferably detects images in the visible range for humans. By means of the expression "specific" in the context of the camera sensor, for example, it means a determined type or a determined model.

[0009] In a first step, preferably a reference image is provided, wherein the reference image is generated from the detection of a specific camera sensor. It is also conceivable to provide at least two reference images. The reference image can be detected by means of an FPGA-based frame grabber. The data, i.e., in particular at least one reference image, can be provided in real time, for example, using RAW16 CSI encoding at 30 FPS, in order to advantageously implement the method according to the present invention for real-time applications or the color correction provided by the application. Furthermore, within the scope of the present invention, applications based on the unified computing device architecture (CUDA) can apply an image signal processor (ISP) and image reconstruction (IMR).

[0010] In another step, preferably color interpolation of the reference image is performed in order to provide an interpolated image. Color interpolation can also be denoted by the English terms Demosaicing or Debayering. Color interpolation, or demosaicing or de-bayering, is a method in which a complete color image can be generated from the Bayer samples of a reference image that contains information only about luminance, in particular. Herein, in particular, a full-color image can be created from the original, color-filtered data. Herein, by interpolating the information from adjacent pixels, the color of each pixel can be calculated. For example, for a red pixel, the green and blue values are estimated from the surrounding pixels in order to obtain a complete RGB color value.

[0011] In another step, at least one corresponding area to be examined of the colors red, green, and blue is preferably determined in the interpolated image. In particular, red is a color with a wavelength of approximately 620 - 750 nanometers, green is a color with a wavelength of approximately 495 - 570 nanometers, and blue is a color with a wavelength of approximately 450 - 495 nanometers. This determination can be made, for example, manually by the user or automatically, for example, based on object or pattern recognition.

[0012] In another step, the average value of the color values is preferably formed in each determined area to be examined in order to obtain the corresponding resulting average color. Thereby, an uneven color distribution in the reference image can be advantageously compensated, for example, by shadows or unevenness on the surface.

[0013] In another step, a corresponding reference color is preferably assigned to each resulting average color, where the reference color at least includes the colors red, green, and blue. Simply put, for example, the reference color red is assigned to the resulting average color that basically corresponds to or should correspond to the color red. The same operation can be performed for the colors green and blue.

[0014] In another step, a color correction matrix is preferably generated based on the color values of the reference image. The color values of the reference image can be the measured color channel values of the reference image. The color correction matrix (English "Color Correction Matrix", CCM) is particularly generated in the form of nine variables, where three variables are provided for the respective colors, namely, in particular, red, green, and blue. Thus, the color correction matrix preferably includes a 3×3 matrix, where each row of the matrix represents the red, green, and blue channels of the image.

[0015] In another step, intermediate variables for the colors red, green, and blue are preferably calculated respectively based on the generated color correction matrix, in particular, also based on the color values of the reference image. For example, this can be performed in the sense of the following equations.

[0016] R' = R * CCM[0][0] + G * CCM[0][1] + B * CCM[0][2]

[0017] G' = G * CCM[1][0] + G * CCM[1][1] + B * CCM[1][2]

[0018] B' = B * CCM[2][0] + G * CCM[2][1] + B * CCM[2][2]

[0019] Here, R'G'B' are particularly the intermediate variables and R, G, B are the measured color channel values of the reference image. Here, R, G, B represent red, green, and blue respectively.

[0020] In another step, preferably, the corresponding differences between the color values of the reference image and the respectively calculated intermediate variables are minimized for the colors red, green, and blue in order to provide color correction for a specific camera sensor. This is exemplarily shown according to the following pseudocode, where the variables R_real, G_real, and B_real represent the corresponding color values of red, green, and blue of the reference image respectively:

[0021] Minimize(abs(R' - R_real))

[0022] Minimize(abs(G' - G_real))

[0023] Minimize(abs(B' - B_real))

[0024] By means of the method according to the invention, image processing can be advantageously improved by the provided color correction, and thus, for example, the detection of objects in the image can be improved. This can be particularly advantageous in the context of applications in vehicles, especially in applications in at least partially automated vehicles. The image to which the provided color correction has been applied can advantageously have colors that are perceived by humans in a strengthened manner, or in other words, appear more natural visually to humans.

[0025] It can be proposed that the method further comprises the following steps:

[0026] - Render the interpolated image in order to determine (103) at least one corresponding area to be examined based on the rendered interpolated image.

[0027] By rendering the interpolated image, especially if it is determined manually by the user, the determination of the area to be examined in the interpolated image can be simplified.

[0028] Furthermore, it can be considered that the assignment (105) comprises the following steps:

[0029] - Create a table, where the table includes the corresponding assignment of reference colors to the resulting average colors.

[0030] By means of this table, a database for other steps of the method can be advantageously provided, and the database can be modified according to the application situation.

[0031] In another example, a non-integer equation solver is used to perform the generation of the color correction matrix, where three variables are generated for each color. Using a non-integer equation solver can lead to a faster and more efficient generation of the color correction matrix because fewer iterations are required.

[0032] The non-integer equation solver can be, for example, an advanced process optimizer or an interior point optimizer.

[0033] The Advanced Process Optimizer (APOPT) is an optimization algorithm that can be used to solve mixed-integer nonlinear programming problems (MINLP). The way APOPT works can be summarized as follows: For example, APOPT is suitable for problems that include continuous and discrete decision variables. In particular, APOPT uses a branch-and-bound algorithm. In particular, this is a method for solving optimization problems with integer constraints. The algorithm works especially by systematically dividing the solution space into smaller sub-regions (branches) and examining said sub-regions individually. By setting an upper bound (boundary) on the optimal value, hopeless regions in the solution space can be excluded to accelerate the search. For the continuous variables of the problem, APOPT preferably uses nonlinear programming methods to find the optimal or near-optimal solution. For example, this includes using techniques such as the gradient method or the interior point method. For discrete variables, APOPT specifically takes into account the integer constraints to ensure that the final solution corresponds to the requirements of the problem. For example, the algorithm tests different combinations of integer values and evaluates their impact on the overall optimum. One aspect of using APOPT is the difference between the global optimum and the local optimum. In particular, APOPT aims to find the global optimum depending on the complexity of the problem and the specific parameter settings, but can also provide the local optimum.

[0034] In particular, the Interior Point Optimizer (IPOPT) is a numerical optimization algorithm for solving large-scale nonlinear programming problems (NLP). IPOPT is especially good at solving nonlinear programming problems. For example, the problem includes an objective function that should be minimized or maximized while observing nonlinear equations and inequalities as auxiliary conditions. IPOPT preferably uses an interior point scheme. For example, the scheme differs from boundary point methods (such as the simplex method in linear problems) in that: the scheme stays inside the allowed region during the solution process instead of navigating along the boundary. This usually enables a more efficient traversal of the solution space. The core of IPOPT is especially the iterative strategy. In each iteration step, an approximate solution can be generated based on the current estimate of the optimal value. Then the approximate solution can be used to calculate the next estimate. To determine the direction and magnitude of the next step, IPOPT preferably solves a system of nonlinear equations. This can be performed with the help of techniques such as the Newton method and linear programming. The algorithm preferably iterates until a solution within a preset tolerance range is found. This means that the changes between successive iterations are especially small enough to conclude that the solution is close to the optimal value.

[0035] It is also conceivable that the non-integer equation solver is configured such that it performs a breadth search in a mode for nonlinear programming. Nonlinear programming relates in particular to the solution of problems in which at least one objective function and / or at least one constraint is a nonlinear function. In particular, a breadth search refers to a special way of searching the solution space, for example in optimization problems involving discrete decisions (e.g. integer variables). In particular, a breadth search ("Breadthfirst" in English) is in particular a strategy in which all branches on one level of a decision tree are first checked before transitioning to the next level. This is in contrast, for example, to a depth search ("Depth first" in English) strategy in which one penetrates as deeply as possible into one branch of the tree before transitioning to other branches.

[0036] In another example, the method further includes the following steps:

[0037] - Determining color values of the reference image based on an analysis of the reference image.

[0038] This can be performed, for example, by pixel-by-pixel analysis, histogram analysis or automatic color recognition (eg with the aid of machine learning).

[0039] It is possible to use the method according to the invention in a vehicle, in particular a specific camera sensor for at least one camera of the vehicle. The vehicle can be designed, for example, as a motor vehicle and / or a passenger car and / or as an at least partially automated vehicle. The vehicle can have vehicle devices, for example, to provide autonomous driving functions and / or driver assistance systems. The vehicle devices can be designed to at least partially automatically control and / or accelerate and / or brake and / or steer the vehicle.

[0040] The subject of the invention is also a computer program, in particular a computer program product, comprising instructions which, when the computer program is executed by a computer, cause the computer to carry out the method according to the invention. The computer program according to the invention therefore entails the same advantages as described in detail with reference to the method according to the invention.

[0041] The subject matter of the invention is also a device for data processing, which is designed to carry out the method according to the invention. For example, a computer that executes the computer program according to the invention can be implemented as the device. The computer can have at least one processor for executing the computer program. A non-volatile data memory can also be provided, in which the computer program is stored and from which the computer program can be read by the processor for execution.

[0042] The subject matter of the present invention can equally be a computer-readable storage medium having a computer program according to the present invention and / or comprising instructions which, when executed by a computer, cause the computer to carry out a method according to the present invention. The storage medium is for example configured as a data memory such as a hard disk and / or a non-volatile memory and / or a memory card. The storage medium can for example be integrated into the computer.

[0043] Furthermore, the method according to the present invention can also be configured as a computer-implemented method. BRIEF DESCRIPTION OF THE DRAWINGS

[0044] Other advantages, features and details of the present invention result from the following description in which embodiments of the present invention are described in detail with reference to the drawings. Here, the features mentioned in the claims and the description can be important for the present invention individually or in any combination. Among them:

[0045] Figure 1 Schematic visualizations of a method, a device, a storage medium and a computer program according to an embodiment of the present invention are shown. DETAILED DESCRIPTION OF THE EMBODIMENTS

[0046] Figure 1 A method 100, a camera sensor 1, a device 10, a storage medium 15 and a computer program 20 according to an embodiment of the present invention are schematically shown.

[0047] Figure 1 In particular, an embodiment of a method 100 for providing color correction for a specific camera sensor 1 is shown. In a first step 101, a reference image is provided, wherein the reference image is generated from the detection of a specific camera sensor 1. In a second step 102, color interpolation of the reference image is performed to provide an interpolated image. In a third step 103, at least one respective area to be checked for the colors red, green and blue is determined in the interpolated image. In a fourth step 104, an average value of the color values is formed in each of the determined areas to be checked to obtain a respective resulting average color. In a fifth step 105, a respective reference color is assigned to each resulting average color, wherein the reference color at least includes the colors red, green and blue. In a sixth step 106, a color correction matrix is generated based on the color values of the reference image. In a seventh step 107, intermediate variables for the colors red, green and blue are respectively calculated based on the generated color correction matrix. In an eighth step 108, the respective differences between the color values of the reference image and the respectively calculated intermediate variables are minimized for the colors red, green and blue in order to provide color correction for the specific camera sensor 1.

[0048] After performing the demosaicing step in an image signal processor, the image can still include the raw values of the RGB diodes. These values represent the response of the photodiodes to specific wavelengths and are particularly unsuitable for being interpreted as true colors when rendering an image.

[0049] Thus, according to an embodiment of the present invention, the following system of equations with nine variables is solved. A color correction matrix (English "Color Correction Matrix", CCM) can advantageously combine the outputs of each color in the raw RGB image, since each photodiode responds more or less to the entire visible spectrum.

[0050] R' = R * CCM[0][0] + G * CCM[0][1] + B * CCM[0][2]

[0051] G' = G * CCM[1][0] + G * CCM[1][1] + B * CCM[1][2]

[0052] B' = B * CCM[2][0] + G * CCM[2][1] + B * CCM[2][2]

[0053] When performed correctly, the output image, i.e., the image to which color correction provided according to an embodiment of the present invention has been applied, can advantageously resemble the colors perceived by humans.

[0054] A reference image can be detected by means of an FPGA-based frame grabber. Data, in particular at least one reference image, can be provided in real time at 30 FPS using RAW16 CSI encoding. In addition, an application based on the unified computing device architecture (CUDA) can apply an image signal processor (ISP) and image reconstruction (IMR). Image reconstruction can be performed immediately after the demosaicing step. In this way, it is easier to determine or select the regions of interest (ROI) to be inspected in the reference image, especially in the case of, for example, manual selection by the user.

[0055] According to an embodiment of the present invention, the following steps can be performed to obtain and apply a color correction matrix. In the first step, a reference image can be provided, which can be generated from the detection of a specific camera sensor 1. In the second step, a decompression and / or interpolation step (the "debayering step") can be applied to the detected reference image. In the third step, the output of the previous step can be rendered so that the detected reference image can be visualized. In the fourth step, at least one region of interest (ROI) to be examined can be selected for each color of the detected reference image. In the fifth step, the area in each region to be examined can be averaged, and the average color obtained therefrom can be determined for the corresponding region to be examined. In the sixth step, a table can be created with the measured colors (i.e., the average colors respectively generated by the regions to be examined) and the corresponding reference colors. Thus, the corresponding reference colors can be assigned to each region to be examined. In the seventh step, nine variables of the color correction matrix can be generated using an advanced process optimizer (APOPT) or an interior point optimizer (IPOPT) solver, which is configured to perform a breadth first branching in the mode of non-linear programming (NLP). For each color, three intermediate variables can also be introduced into the solver. The use of the intermediate variables can advantageously contribute to the convergence here. The equations obtained from the above seventh step are shown as follows:

[0056] R' = R * CCM[0][0] + G * CCM[0][1] + B * CCM[0][2]

[0057] G' = G * CCM[1][0] + G * CCM[1][1] + B * CCM[1][2]

[0058] B' = B * CCM[2][0] + G * CCM[2][1] + B * CCM[2][2]

[0059] Here, R'G'B' are the intermediate variables in the solver and R, G, B are the measured color channel values. Here, R, G, B represent red, green, and blue respectively.

[0060] In the eighth step, lenses can be used instead of equations. Three lenses can be preset for each reference color. The goal is particularly to minimize the absolute value of the difference between the measured color channels and the adapted color channels, i.e., the intermediate variables determined in particular. This is exemplarily shown according to the following pseudocode:

[0061] Minimize(abs(R' - R_real))

[0062] Minimize(abs(G' - G_real))

[0063] Minimize(abs(B' - B_real))

[0064] In the ninth step, a solver can be executed and the output can be visualized.

[0065] The above explanations of the embodiments describe the present invention only within the scope of examples. Of course, as long as it is technically reasonable, the various features of the embodiments can be freely combined with each other without departing from the scope of the present invention.

Claims

1. A method (100) for providing color correction for a specific camera sensor (1), comprising the following steps: - providing (101) a reference image, wherein the reference image is generated from the detection of the specific camera sensor (1), - performing (102) color interpolation on the reference image to provide an interpolated image, - determining (103) at least one corresponding area to be inspected for each of the colors red, green, and blue in the interpolated image, - forming (104) an average value of the color values in each determined area to be inspected to obtain a corresponding generated average color, - assigning (105) a corresponding reference color to each generated average color, wherein the reference color at least includes the colors red, green, and blue, - generating (106) a color correction matrix based on the color values of the reference image, - respectively calculating (107) intermediate variables for the colors red, green, and blue based on the generated color correction matrix, - minimizing (108) the corresponding differences between the color values of the reference image and the respectively calculated intermediate variables for the colors red, green, and blue so as to provide the color correction for the specific camera sensor (1).

2. The method (100) according to claim 1, wherein, the method (100) further comprises the following steps: - rendering the interpolated image so as to determine (103) at least one corresponding area to be inspected based on the rendered interpolated image.

3. The method (100) according to any one of the preceding claims, wherein, the assignment (105) comprises the following steps: - creating a table, wherein the table includes the corresponding assignment of the reference color to the generated average color.

4. The method (100) according to any one of the preceding claims, wherein, a non-integer equation solver is used to perform the generation (106) of the color correction matrix, wherein three variables are generated for each color.

5. The method (100) according to claim 4, wherein, the non-integer equation solver is an advanced process optimizer or an interior point optimizer.

6. The method (100) according to any one of claims 4 or 5, wherein, the non-integer equation solver is configured such that the non-integer equation solver performs a breadth search in a mode for non-linear programming.

7. The method (100) according to any one of the preceding claims, wherein, the method (100) further comprises the following steps: - determining the color values of the reference image based on the analysis of the reference image.

8. A computer program (20), comprising instructions which, when the computer program (20) is executed by a computer (10), cause the computer (10) to execute the method (100) according to any one of the preceding claims.

9. A device (10) for data processing, the device being designed to: execute the method (100) according to any one of claims 1 to 7.

10. A computer-readable storage medium (15) comprising instructions that, when executed by a computer (10), cause the computer to perform the steps of the method (100) according to any one of claims 1 to 7.

Images