Method for providing a color image on the basis of color data from a sensor having a color filter array

Abstract

The invention relates to a method for providing a color image on the basis of color data from a sensor having a color filter array. The color filter array is, for example, a Bayer color filter array (CFA). The color filter array can have, for example, the colors red, green and blue, i.e. an RGGB CFA. It is also conceivable for the color filter array to have transparent, red and green color filter elements, also referred to as an RCCG CFA. The invention also relates to a computer program, to a device, and to a storage medium for this purpose.

Description

[0001] R.413384 - 1 -Description Title Method for Providing a Color Image Based on Color Data from a Sensor with a Color Filter Array The invention relates to a method for providing a color image based on color data from a sensor with a color filter array. The invention further relates to a computer program, a device, and a storage medium for this purpose. State of the Art For the reproduction of color information in a digital image, at least three color filters, the so-called color channels, are required at each pixel location. Red (R), green (G), and blue (B) are frequently used for this purpose. Digital color cameras are usually based on a single area sensor whose pixels measure the amount of incident light over a broad wavelength range. For the image sensor signal to include all three color filters, the camera would have to be equipped with three separate sensors.Since three separate sensors incur considerable costs, most cameras are based on a single area sensor whose pixels measure the amount of incident light across a broad wavelength range that varies regularly from pixel to pixel. Technically, this is achieved using a so-called Color Filter Array (CFA), a regular arrangement of color filters on the individual pixels. Most often, a so-called "Bayer" color filter array (Bayer CFA) is used. The "Bayer" color filter includes, in particular, green, red, and blue color filter elements and can also be referred to as an RGGB CFA. Because only one of the three color values ​​R, G, or B is measured per pixel, the color information for the other two color values ​​is missing for each pixel. The task of a demosaicing method is then to estimate the two missing color values ​​from the surrounding measurements of a pixel. The result is R.413384 -. 2 -This is a three-channel color image. One way to reconstruct the missing color values ​​at a pixel is by linearly interpolating these values ​​from the immediate vicinity. The linear interpolation of the color values ​​is a low-pass filter with a moving average. Since such a demosaicing algorithm is agnostic with respect to textures in the image, artifacts, for example in the form of zippering and false colors, occur, especially at edges or other high-frequency image structures. Disclosure of the invention: The invention relates to a method with the features of claim 1, a computer program with the features of claim 8, a device with the features of claim 9, and a computer-readable storage medium with the features of claim 10. Further features and details of the invention will become apparent from the respective dependent claims, the description, and the drawings.Features and details described in connection with the inventive method naturally also apply to the inventive computer program, the inventive device, and the inventive computer-readable storage medium, and vice versa, so that mutual reference is always possible with regard to the disclosure of the invention. The invention relates in particular to a method for providing a color image based on color data from a sensor with a color filter array, comprising the following steps, wherein the steps can be performed repeatedly and / or sequentially, or in a specific order. The color filter array is, for example, a Bayer color filter array (CFA). The color filter array can, for example, have the colors red, green, and blue, a so-called RGGB CFA.In this case, the first color can correspond to "green", the second to "red", and the third to "blue". It is also conceivable that the color filter array has transparent (English: "Clear"), red, and green color filter elements, wherein, for the following description of the method, the respective red color filter elements correspond to the pixel positions of the second color, and the respective transparent color filter elements correspond to the pixel positions of the first color. R.413384 -. 3 -The color and the respective green color filter elements are each assigned to the pixel positions of the third color. Within the scope of the present invention, this color filter array can also be referred to as RCCG CFA. For example, it can be provided with 50% transparent and 25% red and green color filter elements each. The method according to the invention can also be applied to other color filter arrays, i.e., color filter arrays with different color combinations. With such an RCCG CFA, the method according to the invention can be particularly advantageous, since a regular application of the Hamilton & Adams "Adaptive Color Plane Interpolation" method with linear interpolation based on differences can lead to significant color artifacts. In a first step of the method, the color data is preferably provided, wherein the color data results from a sensor acquisition with the color filter array. The sensor is, in particular, an image sensor.In a further step, a missing color value of a first color at a respective pixel position of a second color is preferably determined based on horizontal and vertical gradients at the respective pixel position of the second color. Within the scope of the present invention, a gradient is to be understood in particular as a brightness profile. The gradient represents, in particular, a respective horizontal and vertical brightness profile at surrounding pixel positions of the first and second colors, for example, green and red pixel positions in the case of an RGGB CFA. In a further step, a missing color value of the first color at a respective pixel position of a third color is preferably determined based on horizontal and vertical gradients at the respective pixel position of the third color. The gradients are calculated, for example, similarly to the above in the case of the pixel position of the second color.In a further step, a missing color value of the third color at a respective pixel position of the first color is preferably determined based on a linear interpolation of a quotient of a respective color value of the third color and a respective color value of the first color of adjacent pixel positions. In other words, the quotient is a result that occurs when the respective color value of the R.413384 -. 4 -The third color is divided by the color value of the first color, which also applies analogously to the following quotients. Using the quotient instead of the difference can advantageously lead to more homogeneous colors and reduce false colors. In a further step, a missing color value of the third color at the respective pixel position of the second color is preferably determined based on a linear interpolation of a quotient of a respective color value of the third color and a respective color value of the first color of adjacent pixel positions. In a further step, a missing color value of the second color at the respective pixel position of the first color is preferably determined based on a linear interpolation of a quotient of a respective color value of the second color and a respective color value of the first color of adjacent pixel positions.In a further step, a missing color value of the second color is preferably determined at the respective pixel position of the third color based on a linear interpolation of a quotient of a respective color value of the second color and a respective color value of the first color of adjacent pixel positions. In a further step, the color image is preferably provided based on the color data and the determined missing color values ​​of the first, second, and third colors. A respective color value, for example, a green value, red value, or blue value, is in particular a signal received by the sensor during detection, above which a corresponding color filter element, for example, a green, red, or blue color filter element, is applied, wherein the color data represents the received signals.A given color value of the first color is, for example, a signal received by the sensor during detection, over which a corresponding color filter element of the first color is placed. A given color value of the second color is, for example, a signal received by the sensor during detection, over which a corresponding color filter element of the second color is placed. A given color value of the third color is, for example, a signal received by the sensor during detection, over which a corresponding color filter element of the third color is placed. In the case of an RCCG CFA, R.413384 applies. 5 -In particular, a respective color value of the first color is a signal received by the sensor during detection, over which a corresponding transparent color filter element is placed. A respective color value of the second color in an RCCG CFA is, in particular, a signal received by the sensor during detection, over which a corresponding red color filter element is placed. A respective color value of the third color in an RCCG CFA is, in particular, a signal received by the sensor during detection, over which a corresponding green color filter element is placed. Furthermore, it can be advantageous within the scope of the invention that the method further comprises the following step: - Limiting a result of the respective linear interpolation to a minimum and / or a maximum of the adjacent pixel positions used for the respective linear interpolation. Preferably, the limit is set to both a minimum and a maximum.This process step advantageously ensures that the color values ​​remain within the measurable range of the sensor. This limitation effectively prevents artificial overshoots or undershoots. It is conceivable that the method further comprises the following steps: - Calculating a quotient of a channel of the second color and a channel of the first color, and / or a quotient of a channel of the third color and a channel of the first color, where each channel encompasses all color values ​​of the respective color; - Filtering the calculated quotient(s); - Calculating back to the channel of the second color and / or the channel of the third color using the filtered quotients. The filtering is, in particular, a low-pass filter and can be performed with a Gaussian filter and / or a median filter. Filtering can advantageously reduce the occurrence of false colors.Also part of the invention is a computer program, in particular a computer program product, comprising instructions which, when the computer program is executed by a computer, cause it to execute the method according to the invention. Thus, the invention R.413384 brings about... 6 -The computer program offers the same advantages as those described in detail with reference to a method according to the invention. Also part of the invention is a data processing device configured to execute the method according to the invention. The device can, for example, be a computer that executes the computer program according to the invention. The computer can have at least one processor for executing the computer program. A non-volatile data storage device can also be provided in which the computer program is stored and from which the computer program can be read by the processor for execution. Furthermore, it is conceivable that the data processing device is designed as an image signal processor (ISP), so that the method according to the invention can be carried out by hardware of the image signal processor.The invention can also relate to a computer-readable storage medium which contains the computer program according to the invention and / or includes instructions that, when executed by a computer, cause it to execute the method according to the invention. The storage medium is, for example, designed as a data storage device such as a hard drive and / or non-volatile memory and / or a memory card. The storage medium can, for example, be integrated into the computer. Furthermore, the method according to the invention can also be implemented as a computer-implemented method. Alternatively or additionally, at least one of the disclosed method steps can be computer-implemented and / or carried out automatically. Further advantages, features, and details of the invention will become apparent from the following description, in which exemplary embodiments of the invention are described in detail with reference to the drawings.The features mentioned in the claims and the description can each be essential to the invention individually or in any combination. See R.413384. 7 -Fig. 1 is a schematic visualization of a method, a sensor, a device, a storage medium, and a computer program according to exemplary embodiments of the invention. Fig. 2 is a schematic representation of a color filter array with various color filter elements. Fig. 3 is a schematic representation of a color filter array with various color filter elements. Fig. 4 is a schematic representation of a color filter array with various color filter elements. Fig. 5 is a schematic representation of a color filter array with various color filter elements. In Fig. 1, a method 100, a sensor 1, a device 10, a storage medium 15, and a computer program 20 according to exemplary embodiments of the invention are schematically represented. Figures 2 to 5 each show a color filter array 2 with various color filter elements 3.Figure 1 shows in particular an embodiment of a method 100 for providing a color image based on color data from a sensor 1 with a color filter array 2. In a first step 101, the color data is provided, the color data resulting from an acquisition by the sensor 1 with the color filter array 2. In a second step 102, a missing color value of a first color at a respective pixel position of a second color is determined based on horizontal and vertical gradients at the respective pixel position of the second color. In a third step 103, a missing color value of the first color at a respective pixel position of a third color is determined based on horizontal and vertical gradients at the respective pixel position of the third color.In a fourth step 104, a color value of the third color is generated at a respective pixel position of the first color based on a linear interpolation of a quotient of a respective color value of the R.413384 -. 8 -In a fifth step, a missing color value of the third color at the respective pixel position of the second color is determined based on a linear interpolation of a quotient of a respective color value of the third color and a respective color value of the first color at adjacent pixel positions. In a sixth step, a missing color value of the second color at the respective pixel position of the first color is determined based on a linear interpolation of a quotient of a respective color value of the second color and a respective color value of the first color at adjacent pixel positions.In a seventh step (107), a missing color value of the second color at the respective pixel position of the third color is determined based on a linear interpolation of the quotient of a respective color value of the second color and a respective color value of the first color at adjacent pixel positions. In an eighth step (108), the color image is provided based on the color data and the determined missing color values ​​of the first, second, and third colors. To avoid or reduce the problems of the prior art, edge-directed interpolation can analyze the area around each pixel to determine whether a preferred interpolation direction exists. This avoids interpolation across an edge and the resulting artifacts. One implementation can be achieved, for example, using the "Adaptive ColorPlane Interpolation" method by Hamilton & Adams.This method is explained below using an example with reference to Figures 2 to 5 for an RGGB CFA. The numbers used below for pixel positions 1 to 9 can also be replaced by the letters a to i and serve only for referencing in Figures 2 to 5. In a first step, a missing green value at a respective red pixel position is determined. To estimate the missing green value at a red pixel position 5 (G5) (see Figure 2), horizontal and vertical gradients at red pixel position 5 (R5) are preferably used. The horizontal gradient can be calculated via the red pixels (R3, R5, and R7) and the green pixels (G4 and G6). The vertical gradient can be calculated via the red pixels (R1, R5, and R9) and the green pixels (G2 and G8). This is illustrated below using a pseudo-code: R.413384 -. 9 -In a second step, a missing green value at a blue pixel position is preferably determined. To estimate the missing green value at blue pixel position 5 (G5) (see Fig. 3), horizontal and vertical gradients at blue pixel position 5 (B5) are preferably used. The gradients are calculated, for example, similarly to the red pixel position described above. This is illustrated below using pseudo-code: In a third step, a missing blue value at a green pixel position is preferably determined (see Fig. 4). A linear interpolation of a difference B – G is preferably determined, particularly under the assumption that B – G varies only weakly in space: R.413384 - 10 -In a fourth step, a missing blue value at a red pixel position is determined (see Fig. 4). Preferably, a linear interpolation of a difference B – G is determined, particularly under the assumption that B – G varies only slightly in space. In a fifth step, a missing red value at a green pixel position is determined (see Fig. 5). Preferably, a linear interpolation of a difference R – G is determined, particularly under the assumption that R – G varies only slightly in space. In a sixth step, a missing red value at a blue pixel position is determined (see Fig. 5). Preferably, a linear interpolation of a difference R – G is determined, particularly under the assumption that R – G varies only slightly in space. As an alternative to the widely used RGGB CFA, sensors, especially cameras, with CFAs that measure transparent (clear), red, and green can also be used. 11 -The color filter elements comprise, in particular, 50% transparent (clear) and 25% each of red and green color filter elements, hereinafter referred to as RCCG CFA. For comparison with the Hamilton & Adams method described above, the nomenclature RGGB, in the sense of the "Bayer" color filter, is used again below, i.e., "R = R", "G = C", and "B = G". Due to the widespread use of the RGGB CFA, publications on demosaicing are mostly based on an RGGB CFA, including the "Adaptive Color Plane Interpolation" method by Hamilton & Adams described above. The assumptions made therein, such as the absence of high-frequency components in the difference channels RG and BG, do not apply to the RCCG CFAs. Therefore, the application of this method in the RCCG case can lead to significant color artifacts.For this reason, the "Adaptive Color Plane Interpolation" method by Hamilton & Adams was adapted and extended according to embodiments of the present invention. According to embodiments of the present invention, a color-preserving chrominance calculation is provided for estimating blue and red values ​​at a green pixel position, where, within the scope of the present invention, chrominance is understood to mean, in particular, the ratios R / G and B / G. Furthermore, according to embodiments of the present invention, a limiting of a value range for the linear interpolation for estimating blue and red values ​​at the green pixel position is provided. Additionally, according to embodiments, a chrominance low-pass filtering of the three color channels is provided. The three aspects mentioned can be used independently of one another. The method according to embodiments can be used for an RCCG-CFA, but also for any three-channel CFA.The color-preserving chrominance calculation of the blue and red values ​​at a green pixel position according to embodiments of the invention can advantageously lead to more homogeneous colors and reduce false colors. The color-preserving chrominance calculation of the blue and red values ​​at a green pixel position is based in particular on the interpolation of the surrounding pixels. The method according to R.413384-. 12 -In exemplary embodiments, the value range of the interpolation result is preferably limited to a minimum and / or maximum of the surrounding pixels used for interpolation. This allows estimated color values ​​to advantageously lie only within the measurable value range of the sensor. Color values ​​that would not be within the measurable value range of the sensor can, for example, appear as black pixels. The chrominance low-pass filtering of the three color channels according to exemplary embodiments of the invention can mitigate the occurrence of false colors. A method according to exemplary embodiments of the invention is described below in comparison with the method according to Hamilton & Adams. Regarding the determination of the missing green value at the red pixel position, there is preferably no change compared to the method according to Hamilton & Adams.Regarding the determination of the missing green value at the blue pixel position, there is preferably no change compared to the method according to Hamilton & Adams. Regarding the determination of the missing blue value at the green pixel position, the method according to embodiments of the invention differs. The blue channel is preferably calculated taking into account the already interpolated green channel. According to embodiments of the invention, the linear interpolation is not performed on the difference B - G, but on the quotient B / G. By calculating the quotient, color ratios can be advantageously preserved, and more homogeneous color gradients can be achieved. R.413384 - 13 - The method according to exemplary embodiments of the invention also differs with regard to determining the missing blue value at the red pixel position. The linear interpolation is preferably not performed on the difference B - G but on the quotient B / G. By forming the quotient, color ratios can be advantageously preserved, and more homogeneous color gradients can be achieved. The method according to exemplary embodiments of the invention also differs with regard to determining the missing red value at the green pixel position. The red value is preferably calculated taking into account the already interpolated green value. In particular, the linear interpolation is not performed on the difference R - G, but on the quotient R / G. By forming the quotient, color ratios can be advantageously preserved, and more homogeneous color gradients can be achieved. The method according to exemplary embodiments of the invention also differs with regard to determining the missing red value at the blue pixel position. In particular, the linear interpolation is not performed on the difference R - G, but on the quotient R / G. By forming the quotient, color ratios can be advantageously preserved, and more homogeneous color gradients can be achieved.Furthermore, according to exemplary embodiments, a limitation of the value range of the results from the preceding steps can be performed. The results from the interpolation are preferably limited to the minimum and / or maximum of the adjacent pixels of the same color. This limitation advantageously avoids artificial overshoots or undershoots. In addition, according to exemplary embodiments, a chrominance low-pass filtering can be performed. For this purpose, the chrominance can first be calculated as follows: Cr = R-channel / G-channel Cb = B-channel / G-channel R.413384 -. 14 -Subsequently, Cr and Cb can be low-pass filtered using a Gaussian filter and / or a median filter. A Gaussian filter can be used instead of a mean filter to prevent aliasing effects. After filtering, the calculation can be reversed as follows: R-channel = Cr_filtered x G-channel; B-channel = Cb_filtered x G-channel. The preceding explanation of the embodiments describes the present invention solely by way of examples. Of course, individual features of the embodiments can be freely combined with one another, provided this is technically feasible, without departing from the scope of the present invention.

Claims

R.413384 - 15 -Claims 1. A method (100) for providing a color image based on color data from a sensor (1) with a color filter array (2), comprising the following steps: - providing (101) the color data, wherein the color data results from a detection by the sensor (1) with the color filter array (2), - determining (102) a missing color value of a first color at a respective pixel position of a second color based on horizontal and vertical gradients at the respective pixel position of the second color, - determining (103) a missing color value of the first color at a respective pixel position of a third color based on horizontal and vertical gradients at the respective pixel position of the third color.- Determining (104) a missing color value of the third color at a respective pixel position of the first color based on a linear interpolation of a quotient of a respective color value of the third color and a respective color value of the first color of adjacent pixel positions, - Determining (105) a missing color value of the third color at the respective pixel position of the second color based on a linear interpolation of a quotient of a respective color value of the third color and a respective color value of the first color of adjacent pixel positions, - Determining (106) a missing color value of the second color at the respective pixel position of the first color based on a linear interpolation of a quotient of a respective color value of the second color and a respective color value of the first color of adjacent pixel positions,-Determining (107) a missing color value of the second color at the respective pixel position of the third color based on a linear, R.413384 - 16 -1. Interpolation of a quotient of a respective color value of the second color and a respective color value of the first color of adjacent pixel positions, -provision (108) of the color image based on the color data and the determined missing color values ​​of the first, second and third colors.

2. Method (100) according to claim 1, characterized in that a respective color value is a signal received by the sensor (1) during detection, over which a corresponding color filter element (3) is placed, wherein the color data represent the received signals. 3.Method (100) according to one of the preceding claims, characterized in that the color filter array (2) comprises transparent, red, and green color filter elements (3), wherein respective red color filter elements (3) are assigned to the pixel positions of the second color, respective transparent color filter elements (3) to the pixel positions of the first color, and respective green color filter elements (3) to the pixel positions of the third color.

4. Method (100) according to one of the preceding claims, characterized in that the method (100) further comprises the following step: - Limiting a result of the respective linear interpolation to a minimum and / or a maximum of the adjacent pixel positions used for the respective linear interpolation. R.413384 - 17 -5. Method (100) according to any one of the preceding claims, characterized in that the method (100) further comprises the following steps: - Calculating a quotient of a channel of the second color and a channel of the first color and / or a quotient of a channel of the third color and a channel of the first color, wherein each of the channels comprises all color values ​​of the respective color, - Filtering the calculated quotient, - Calculating back to the channel of the second color and / or the channel of the third color using the filtered quotients.

6. Method (100) according to claim 5, characterized in that the filtering is a low-pass filter.

7. Method (100) according to claim 5 or 6, characterized in that the filtering is performed with a Gaussian filter and / or a median filter. 8.

9. Computer program (20), comprising instructions which, when executed by a computer (10), cause the computer to execute the method (100) according to any one of the preceding claims.

10. Device (10) for data processing, which is configured to execute the method (100) according to any one of claims 1 to 7.

11. Computer-readable storage medium (15), comprising instructions which, when executed by a computer (10), cause the computer to execute the steps of the method (100) according to any one of claims 1 to 7.

Images