Method for providing color correction for particular camera sensor

Abstract

To provide a method for providing color correction for a particular camera sensor.SOLUTION: A method (100) includes the steps of providing (101) a reference image from a particular camera sensor (1), performing color interpolation of the reference image (102), determining (103) an area to be inspected in the interpolated image with respect to red, green, and blue colors, obtaining (104) an average color of the color values of the inspection area, and associating (105) a reference color including at least red, green, and blue colors with the average color, generating (106) a color correction matrix on the basis of the color values of the reference image, calculating (107) one intermediate variable each for red, green, and blue colors on the basis of the generated color correction matrix, and minimizing (108) differences in red, green, and blue colors between the color values of the reference image and the intermediate variables to provide color correction for the particular camera sensor (1).SELECTED DRAWING: Figure 1

Description

Technical Field

[0001] The present invention relates to a method for providing color correction for a specific camera sensor. The present invention further relates to a computer program, an apparatus, and a memory medium for this purpose.

Background Art

[0002] Prior Art A color correction matrix (English: “Color Correction Matrix” (CCM)) is a tool in digital image processing used for color correction and adjustment in digital images. The main purpose of the color correction matrix is to optimize the color reproduction of an image sensor by correcting possible color deviations caused by the sensor and the objective lens of a photographing device.

[0003] The color correction matrix includes a matrix of values applied to the color channels of an image to correct the color. Typically, the color correction matrix includes a 3×3 matrix, and each row of the matrix represents the red channel, the green channel, and the blue channel of the image. By multiplying the color value of a pixel by this matrix, color bleeding can be corrected, color saturation can be adjusted, and general color balance can be improved.

[0004] In the prior art, for example, a weighted sum is used to define the color correction matrix, and in practice, a look-up table is often further used to reconstruct the image. Also, for existing camera sensors or for special existing hardware, additional information is often required, and this information has to be specified, for example, based on measuring the wavelength of an image taken parallel to the camera sensor.

[0005] U.S. Patent Application Publication No. 20090268044 describes, for example, a method for setting the pixel color of an image. In this method, after the raw image data of a digital image is white balance processed, this white balance processed image data is transferred to a color correction module as a color vector in a color space for color adjustment using a color correction matrix.

[0006] U.S. Patent Application Publication No. 20110019913 discloses a method for generating a color correction matrix (CCM) for an image sensor. In this method, the quantum efficiency (QE) spectrum of the pixels of the image sensor illuminated by a physical light source is measured. Subsequently, the color values of the image sensor and the color values in a predetermined color space are determined according to the QE spectrum and predetermined reference data important for deriving the color values. Finally, a color correction matrix for the image sensor is generated by applying an adjustment algorithm to the color values of the image sensor and the color values in a predetermined color space.

Prior Art Documents

Patent Documents

[0007]

Patent Document 1

Patent Document 2

Summary of the Invention

Means for Solving the Problems

[0008] Disclosure of the Invention The subject matter of the present invention is a method having the features according to claim 1, a computer program having the features according to claim 8, an apparatus having the features according to claim 9, and a computer-readable memory medium having the features according to claim 10. Further features and details of the present invention will become apparent from the respective dependent claims, the specification and the drawings. In this case, the features and details described in connection with the method according to the present invention naturally also apply in connection with the computer program according to the present invention, the apparatus according to the present invention and the computer-readable memory medium according to the present invention, and vice versa, and thus, with respect to the disclosure of the individual aspects of the present invention, cross-references can always be made or are possible.

[0009] The subject matter of the present invention is, in particular, a method for providing color correction for a specific camera sensor, the method comprising the following steps, which can be carried out repeatedly and / or sequentially in succession. The camera sensor preferably detects an image in the region visible to humans. The expression "specific" represents, with respect to the camera sensor, for example, a given type or a given model.

[0010] In a first step, preferably a reference image resulting from the detection of a specific camera sensor is provided. It is also conceivable to provide at least two reference images. The reference image can be detected using an FPGA-based frame grabber. The data, i.e., in particular, at least one reference image, can be provided in real time at 30 FPS, for example, using RAW16CSI coding, which advantageously enables the method according to the present invention or the real-time application of the provided color correction. Further, within the framework of the present invention, an application based on the Compute Unified Device Architecture (CUDA) can apply an Image Signal Processor (ISP) and Image Reconstruction (IMR).

[0011] In a further step, preferably, color interpolation of the reference image is performed to provide an interpolated image. Color interpolation is sometimes also referred to by the English terms "Demosaicing" or "Debayering". Color interpolation or demosaicing or debayering is also a method by which a complete color image can be generated from a Bayer pattern of a reference image that contains only information regarding luminance. In particular, in this case, a full-color image is created from the raw data that has been color-filtered. In this case, by interpolating information from adjacent pixels, one color can be calculated for each pixel. For example, in the case of a red pixel, the green and blue values from surrounding pixels are estimated in order to obtain the complete RGB color value.

[0012] In a further step, preferably, at least one respective region to be inspected is determined for red, green, and blue in the interpolated image. In particular, red is a color having a wavelength of approximately 620 to 750 nanometers, green is a color having a wavelength of approximately 495 to 570 nanometers, and blue is a color having a wavelength of approximately 450 to 495 nanometers. This determination may be carried out manually by the user, for example, or may be carried out automatically based on object recognition or pattern recognition, for example.

[0013] In a further step, preferably, the average value of the color values in each determined region to be inspected is formed to obtain the resulting average color for each. Thereby, advantageously, for example, non-uniform color distribution in the reference image due to shadows or undulations on the surface can be compensated for.

[0014] In a further step, preferably, each resulting average color is associated with a respective reference color, the reference color including at least red, green, and blue. Put simply, for example, a resulting average color that substantially corresponds to or should correspond to red is associated with the reference color of red. The same is done for green and blue.

[0015] In a further step, preferably, a color correction matrix is generated based on the color values of the reference image. The color values of the reference image may be the measured color channel values of the reference image. The color correction matrix (English: “Color Correction Matrix” (CCM)) is generated particularly in the form of nine variables, with three variables provided for each color, that is, particularly for red, green, and blue. Thus, the color correction matrix preferably includes a 3×3 matrix, and each row of the matrix represents the red channel, green channel, and blue channel of the image.

[0016] In a further step, preferably, based on the generated color correction matrix and particularly also based on the color values of the reference image, one intermediate variable each for red, green, and blue is calculated. This can be implemented, for example, in the sense of the following equations. R’ = R * CCM[0][0] + G * CCM[0][1] + B * CCM[0][2] G’ = G * CCM[1][0] + G * CCM[1][1] + B * CCM[1][2] B’ = B * CCM[2][0] + G * CCM[2][1] + B * CCM[2][2]

[0017] In this case, R’G’B’ are particularly intermediate variables, and R, G, B are the measured color channel values of the reference image. In this case, R, G, and B represent red, green, and blue, respectively.

[0018] In a further step, preferably, for each of red, green, and blue, the respective differences between the color values of the reference image and the respective calculated intermediate variables are minimized, providing color correction for a particular camera sensor. This is shown, by way of example, based on the following pseudo-code, where the variables R_real, G_real, and B_real represent the corresponding color values for red, green, and blue of the reference image, respectively. Minimize(absolute value(R’ - R_real)) Minimize(absolute value(G’ - G_real)) Minimize(absolute value(B’ - B_real))

[0019] According to the method according to the invention, thanks to the provided color correction, advantageously, image processing, and thus, for example, the detection of objects in an image can be improved. This can be particularly advantageous within the framework of applications in vehicles, especially in at least partially automated vehicles. The image to which the provided color correction is applied can advantageously show the colors perceived by humans in an emphasized state, or, in other words, can act more naturally visually for humans.

[0020] · Performing step (103) of rendering the interpolated image and determining at least one respective area to be inspected, based on the rendered interpolated image can further be assumed to be included. By rendering the interpolated image, the step of determining the areas to be inspected in the interpolated image can be performed more easily, especially when it is assumed that it is determined manually by the user.

[0021] Furthermore, the associating step (105) · includes a step of creating a table, the table including the respective associations of the reference colors to the resulting average colors It is assumed to include. By means of a table, advantageously, a database for further steps of the method can be provided, and this database can be modified according to each application case.

[0022] In yet another example, the step of generating the color correction matrix is carried out using a non-integer equation solver, and three variables are generated for each color. By using a non-integer equation solver, it becomes possible to carry out the step of generating the color correction matrix more quickly and efficiently. This is because the required iterations can be reduced.

[0023] The non-integer equation solver may be, for example, Advanced Process Optimizer or Interior Point Optimizer.

[0024] The Advanced Process Optimizer (APOPT) is an optimization algorithm that can be used, in particular, to solve mixed-integer non-linear programming problems (MINLPs). The functioning mode of APOPT can be summarized as follows. That is, APOPT is suitable, for example, for problems that include both continuous and discrete decision variables. APOPT uses, in particular, the Branch and Bound algorithm. This is a technique, in particular, for solving optimization problems using integer constraints. This algorithm operates, in particular, by systematically subdividing the solution space into smaller sub-regions (Branches) and examining those sub-regions individually. By setting upper bounds (Bounds) for the optimal value, it is possible to exclude unpromising regions of the solution space and accelerate the search. In the case of the continuous variables of the problem, APOPT preferably utilizes non-linear programming techniques to find an optimal or near-optimal solution. This includes applying techniques such as, for example, the gradient method or the interior point method. In the case of discrete variables, APOPT takes into account, in particular, the integer constraints to ensure that the final solution meets the requirements of the problem. This algorithm tests, for example, various different combinations of integer values and evaluates the impact that those combinations have on the global optimum value. One aspect of using APOPT is to distinguish between the global optimum and the local optimum. APOPT aims, in particular, to find the global optimum, but depending on the complexity of the problem and specific parameter settings, it can also supply a local optimum.

[0025] The Interior Point Optimizer (IPOPT) is a numerical optimization algorithm, especially for solving large-scale non-linear programming problems (NLP). IPOPT is specialized in solving non-linear programming problems in particular. Such problems include, for example, an objective function to be minimized or maximized while maintaining non-linear equalities and non-linear inequalities as constraints. IPOPT preferably uses an interior point method approach. This approach is different from the boundary point method (such as the simplex method in linear problems), for example, in that it stays inside the feasible region during the solution process rather than moving along the boundary. This often enables a more efficient traversal of the solution space. The core of IPOPT is especially an iterative strategy. In each iterative step, an approximate solution can be generated based on the actual estimate of the optimal value. This approximate solution can then be used to calculate the next estimate. To determine the direction and magnitude of the next step, IPOPT preferably solves a system of non-linear equations. This can be implemented using techniques such as Newton's method and linear programming methods. This algorithm is preferably iterated until a solution within a predetermined tolerance is found. This means that the change between successive iterations is especially small enough to infer that the solution is close to the optimal value.

[0026] Furthermore, it is conceivable that the non-integer equation solver is configured to perform a breadth search in a mode for non-linear programming. Non-linear programming is especially concerned with solving problems where at least one objective function and / or at least one constraint is a non-linear function. The breadth search is especially related to a special way of exploring the solution space in an optimization problem that includes, for example, discrete decisions (such as integer variables). The breadth search (English: “Breadth first”) is especially a strategy where all branches at one level of the decision tree are explored first and then the search moves to the next level. This is in contrast to the depth first search strategy (English: “Depth first”), for example, where one delves as deep as possible into one branch of the tree and then moves to other branches.

[0027] In a further example, the method · determining the color values of the reference image based on the analysis of the reference image is further included. This can be implemented, for example, by per-pixel analysis, histogram analysis, or automatic color identification, for example, using machine learning.

[0028] The method according to the invention can be used in a vehicle, in particular for a specific camera sensor of at least one camera of the vehicle. The vehicle may be configured, for example, as a motor vehicle and / or a passenger vehicle and / or at least partially automated vehicle. The vehicle may have, for example, a vehicle device for providing an autonomous driving function and / or a driver assistance system. The vehicle device may be configured to control and / or accelerate and / or brake and / or steer the vehicle at least partially automatically.

[0029] Similarly, the subject matter of the invention is a computer program, in particular a computer program product, comprising instructions for causing a computer to perform the method according to the invention when executed by the computer. Thus, the computer program according to the invention provides the same advantages as described in detail in connection with the method according to the invention.

[0030] Similarly, the subject matter of the invention is a device for data processing configured to perform the method according to the invention. As this device, for example, a computer executing the computer program according to the invention can be provided. The computer may have at least one processor for executing the computer program. A non-volatile data memory can also be provided, in which the computer program can be stored, and from which the computer program can be read by the processor for execution.

[0031] Similarly, the subject of the present invention may be a computer-readable memory medium having a computer program according to the present invention and / or containing instructions for causing a computer to implement the method according to the present invention when executed by the computer. The memory medium is configured as a data memory such as, for example, a hard disk and / or a non-volatile memory and / or a memory card. The memory medium can be incorporated, for example, into a computer.

[0032] Furthermore, the method according to the present invention can also be implemented as a computer-implemented method.

[0033] Further advantages, features and details of the present invention will become apparent from the following description, in which embodiments of the present invention are described in detail with reference to the drawings. In this case, the features described in the claims and the specification can each, or in any combination, constitute the essence of the present invention.

Brief Description of the Drawings

[0034]

Figure 1

Embodiments for Carrying Out the Invention

[0035] FIG. 1 schematically shows a method 100, a camera sensor 1, an apparatus 10, a memory medium 15 and a computer program 20 according to an embodiment of the present invention.

[0036] FIG. 1 shows an embodiment of a method 100 for providing color correction for a particular camera sensor 1. In a first step 101, a reference image resulting from the detection of the particular camera sensor 1 is provided. In a second step 102, color interpolation of the reference image is performed to provide an interpolated image. In a third step 103, at least one respective area to be inspected is determined for red, green, and blue in the interpolated image. In a fourth step 104, an average value of the color values in each determined area to be inspected is formed to obtain each resulting average color. In a fifth step 105, each resulting average color is associated with a respective reference color, which includes at least red, green, and blue. In a sixth step 106, a color correction matrix is generated based on the color values of the reference image. In a seventh step 107, one intermediate variable each for red, green, and blue is calculated based on the generated color correction matrix. In an eighth step 108, the respective differences for red, green, and blue between the color values of the reference image and each calculated intermediate variable are minimized to provide color correction for the particular camera sensor 1.

[0037] After the debayering processing step in the image signal processor, the image may further include raw values for the RGB diodes. These values represent the response of the photodiodes to a given wavelength and are not particularly suitable for being interpreted as the correct color when rendering the image.

[0038] Therefore, according to an embodiment of the present invention, the following system of equations having nine variables is solved. The color correction matrix (Color Correction Matrix (CCM)) can advantageously combine the output of each color in the raw RGB image. This is because each photodiode responds more or less to the entire visible spectrum. R’=R*CCM[0][0]+G*CCM[0][1]+B*CCM[0][2] G’ = G * CCM[1][0] + G * CCM[1][1] + B * CCM[1][2] B’ = B * CCM[2][0] + G * CCM[2][1] + B * CCM[2][2]

[0039] When correctly implemented, the output image, i.e., the image to which color correction provided according to the embodiments of the present invention is applied, can advantageously approximate the colors perceived by humans.

[0040] The reference image can be detected using an FPGA-based frame grabber. Data, i.e., in particular, at least one reference image, can be provided in real time at 30 FPS, for example, using RAW16 CSI coding. Furthermore, applications based on the Compute Unified Device Architecture (CUDA) can apply an Image Signal Processor (ISP) and Image Reconstruction (IMR). The image reconstruction may be performed immediately after the de-bayer processing step. In this way, the region to be inspected in the reference image, i.e., the "region of interest" (ROI), can be more easily determined or selected, especially when manually selected by the user, for example.

[0041] According to an embodiment of the present invention, the following steps can be implemented to obtain and apply a color correction matrix. In the first step, a reference image can be provided that can be obtained as a result of the detection of a specific camera sensor 1. In the second step, a debayering step and / or an interpolation step (English: “debayering step”) can be applied to the detected reference image. In the third step, the output of the previous step can be rendered, thereby visualizing the detected reference image. In the fourth step, for each color of the detected reference image, at least one region of interest (ROI) to be inspected can be selected. In the fifth step, the area in each region to be inspected can be averaged, and the resulting average color can be specified for each region to be inspected. In the sixth step, a table can be created having the measured colors, i.e., the resulting average color of each region to be inspected, and the corresponding reference colors. In this way, one corresponding reference color can be associated with each region to be inspected. In the seventh step, nine variables of the color correction matrix can be generated using an Advanced Process OPTimizer (APOPT) solver or an Interior Point OPTimizer (IPOPT) solver configured to perform breadth first branching in a mode for non-linear programming (NLP). For each color, three additional intermediate variables can be introduced into the solver. In this case, using the intermediate variables can advantageously contribute to convergence. The equations obtained from the seventh step above are shown below. R’ = R * CCM[0][0] + G * CCM[0][1] + B * CCM[0][2] G’ = G * CCM[1][0] + G * CCM[1][1] + B * CCM[1][2] B’ = B * CCM[2][0] + G * CCM[2][1] + B * CCM[2][2]

[0042] In this case, R’G’B’ are intermediate variables in the solver, and R, G, B are measured color channel values. In this case, R, G, and B represent red, green, and blue, respectively.

[0043] In the eighth step, the objective function can be used instead of the equation. For each reference color, three objective functions can be defined. The goal is, in particular, to minimize the absolute value of the difference between the measured color channel and the adjusted color channel, i.e., in particular, the determined intermediate variable. This is shown, by way of example, based on the following pseudo-code. Minimize (absolute value (R’ - R_real)) Minimize (absolute value (G’ - G_real)) Minimize (absolute value (B’ - B_real))

[0044] In the ninth step, the solver can be executed and the output can be visualized.

[0045] The above description of the embodiments is only illustrative of the present invention within the framework of examples. Of course, the individual features of the embodiments can be freely combined with each other without departing from the scope of the present invention as long as they are technically meaningful.

Claims

1. A method (100) for providing color correction for a specific camera sensor (1), wherein the method (100) comprises: - providing a reference image (101) resulting from the detection of the specific camera sensor (1); - performing color interpolation on the reference image to provide an interpolated image (102); - determining, in the interpolated image, at least one respective area to be inspected for red, green, and blue (103); - forming an average value of color values in each respective determined area to be inspected to obtain a respective resulting average color (104); - associating each respective resulting average color with a respective reference color (105), the reference color including at least red, green, and blue; - generating a color correction matrix based on the color values of the reference image (106); - calculating one intermediate variable for each of the red, green, and blue based on the generated color correction matrix (107); - minimizing the respective differences for the red, green, and blue between the color values of the reference image and the respective calculated intermediate variables to provide color correction for the specific camera sensor (1) (108). The method (100) as described above.

2. The method (100) further comprises: - rendering the interpolated image and performing the step (103) of determining at least one respective area to be inspected based on the rendered interpolated image. The method (100) according to claim 1.

3. The associating step (105) comprises: - creating a table that includes the respective associations of the reference colors to the resulting average colors. The method (100) according to claim 1 or 2.

4. The step (106) of generating the color correction matrix is performed using a non-integer equation solver, and three variables are generated for each color. The method (100) according to any one of claims 1 to 3.

5. The non-integer equation solver is Advanced Process Optimizer or Interior Point Optimizer. The method (100) according to claim 4.

6. The non-integer equation solver is configured to perform a breadth search in a mode for non-linear programming. The method (100) according to claim 4 or 5.

7. The method (100) - a step of determining the color value of the reference image based on an analysis of the reference image The method (100) according to any one of claims 1 to 6, further comprising.

8. A computer program (20) comprising instructions for causing the computer (10) to perform the method (100) according to any one of claims 1 to 7 when executed by the computer (10).

9. An apparatus (10) for data processing, configured to perform the method (100) according to any one of claims 1 to 7.

10. A computer-readable memory medium (15) comprising instructions for causing the computer (10) to perform the steps of the method (100) according to any one of claims 1 to 7 when executed by the computer (10).

Images