Method and image processor unit for processing data provided by an image sensor

Abstract

The present invention discloses a method for processing image data provided by an image sensor, the image data comprising a frame formed by an array of pixels, the pixel array being overlaid with a color filter array such that the pixels represent color information according to a specific color pattern defined by the color filter array. The method includes multiplying each pixel of the frame by a respective gain factor predefined in a gain matrix stored for a pixel position corresponding to the pixel in the frame when the gain matrix is positioned over the frame, the gain matrix being smaller than the frame and being arranged at a plurality of different positions over the frame to multiply each pixel of the frame by the respective gain factor of the gain matrix.

Description

Technical Field

[0001] The present invention relates to a method for processing image data provided by an image sensor, the image data comprising a frame formed by an array of pixels, the pixel array being overlaid by a color filter array such that the pixels represent color information according to a specific color pattern defined by the color filter array.

[0002] The present invention further relates to an image processor unit for processing image data provided by an image sensor, the image sensor comprising a sensor pixel array that provides a frame formed by an array of pixels overlaid by a color filter array such that the pixels represent color information according to a specific color pattern defined by the color filter array.

[0003] The present invention further relates to a computer program comprising instructions for performing the steps of the aforementioned method.

Background Art

[0004] The pixel array of a standard digital image sensor is designed to capture light without favoring any particular wavelength. The resulting image is a monochromatic image without any color information. To generate color information, the pixel array is overlaid by a color filter array, which is a grid of optical filters for different wavelengths. The most common color filter array pattern is the Bayer color filter consisting of 2×2 basic cells containing one red (R), two green (G), and one blue (B) pixel cell points. The basic cells are repeated over the entire pixel array of the image sensor to generate an image with R, G, and B pixels. Due to this process, the color information is incomplete. The process of reconstructing the missing color information, e.g., reconstructing R and G and B for all pixels, is known as demosaicking.

[0005] There exist other types of color filter arrays, such as Quad Bayer, HexaDeca, or 6×6 color filter arrays. Thus, advanced image sensors use complex color filter array patterns and employ not only 2×2 Bayer, but also larger N×M RGB or RGBW color filter arrays (W = white).

[0006] The problem of pixel crosstalk occurs especially as the effect increases for higher-order color filter array patterns. The crosstalk problem occurs particularly for RGBW where white and colored interact with each other. The effect on white pixels depends on nearby colored pixels. As a result, an artificial periodic lattice structure of N×M pixels on the white pixels is visible.

[0007] US 10,681,290 B1 discloses a method, an image sensor, and an image processing device for reducing crosstalk noise. A raw image generated by a sensor array is obtained, which comprises a plurality of image pixels and a plurality of phase detection pixels corresponding to phase detection sensing elements. The exposure and system gain of the image sensor, the pixel coordinates of the phase detection pixels, and the sharpness information of the raw image are considered to determine whether to compensate the image data of the current image pixel.

[0008] US 8,947,563 B2 discloses a method for reducing video crosstalk in a display camera system by capturing a first image of a local site while projecting an image of a remote site having a first gain, and capturing a second image of the local site while projecting an image having a second gain different from the first gain. A mixed image of the local site including the first image combined with the projected image having the first gain is captured, and a second mixed image of the local site including the second image combined with the projected image having the second gain is captured. Crosstalk reduction is performed on the mixed images to create a reconstructed image of the local site by determining whether the variation in pixel values between the mixed images is affected by movement in the first and second images of the local site.

[0009] US 10,580,384 B1 discloses a method of configuring a display panel, wherein a crosstalk gain is calculated for a given value of a color component using a generated non-linear model and an input value of a pixel. The crosstalk gain is applied to the color components of the pixel to create a crosstalk-compensated component value. Crosstalk gain calculation uses measured values of the panel to create several electrical to optical transfer functions, which are then compared to each other to create a gain, which is applied to the output of the panel crosstalk model with a maximum multiplication to create an adjusted crosstalk-corrected signal used to generate the overall gain that the system applies to linear RGB inputs.

[0010] US 2021 / 0227185 A1 discloses a calibration circuit calibrated to receive a digital image signal generated based on pixel signals output from a pixel array of an image sensor. The color gain of the digital image signal is calculated based on a set of coefficients calculated based on a reference image signal generated by a reference image sensor under a first light source having a first color temperature. The color gain is applied to the digital image signal to generate a calibrated image signal.

Summary of the Invention

[0011] The object of the present invention is to provide an improved method and an image processor unit for processing image data provided by an image sensor.

[0012] This object is achieved by a method comprising the steps according to claim 1, an image processor unit according to claim 13, and a computer program according to claim 15. Preferred embodiments are described in the dependent claims.

[0013] To process image data comprising a frame formed by an array of pixels and to reduce the visual effect of pixel crosstalk caused by a color filter array pattern, a grid-based gain is applied to the pixels of the frame. Each pixel value is multiplied by one of various gain coefficients of a gain matrix providing the gain grid.

[0014] Reducing the crosstalk effect can be achieved by multiplying each pixel of the frame by a respective gain coefficient predefined in a gain matrix (i.e., gain grid) stored at a pixel position corresponding to the pixel in the frame when the gain matrix is positioned on the frame, the gain matrix being smaller than the frame and being functionally (e.g., virtually or computationally / mathematically) arranged at a plurality of different positions on the frame to multiply each pixel of the frame by a respective gain coefficient of the gain matrix.

[0015] Positioning a gain matrix on the pixel matrix of a frame provided by the image data of an image sensor is understood as a functional positioning for assigning a pixel position in the frame to each pixel position in the gain matrix. There is no need to physically overlay or move a grid matrix on the pixel matrix of the frame. A virtual or computational assignment is sufficient and functions in the same way as if the gain matrix were overlaid on the frame pixel matrix. This can be achieved by storing the gain matrix in data storage and calculating the multiplication of the selected pixel values at the pixel positions of the frame having the preselected stored gain coefficients of the gain matrix. The gain matrix can then be virtually repeated to cover all the pixels of the frame by accessing the same gain coefficients in the gain matrix for a plurality of pixels in the pixel matrix of the frame, whereby an effect can be created as if the gain matrix were positioned over several sections of the frame to cover the whole frame.

[0016] By using different gain values for regions of the frame pixel array of the size of the gain matrix, crosstalk can be significantly reduced. Particularly in the case of white pixels, the gain coefficients can be different between the white pixel and the surrounding colored pixels. This reduces the crosstalk effect between the white pixel value and the nearby colored pixel values.

[0017] The gain coefficients assigned to the pixel positions of the gain matrix can be calibrated on a flat field reference image.

[0018] The size of the gain matrix can preferably be the size of the color filter array pattern, which is repeated to form a color filter array of the size of the frame. From this, the size of the gain matrix is determined by the size of the basic pattern of the color filter array. This addresses the artificial periodic lattice-like structure on the white pixels that occurs at the size of the basic color filter array matrix due to the pixel crosstalk effect.

[0019] The N×M gain grid is periodic according to the N×M cells of the basic color filter array pattern, and the gain matrix is repeated over the size of the frame according to the color filter array pattern.

[0020] The gain matrix can be repeatedly positioned on the frame in a periodic form such that the color pattern of the frame matches the color pattern assignment of the gain coefficients in the gain matrix. Each pixel can be multiplied on the frame by each gain coefficient of the periodically repeated gain matrix. From this, the gain matrix is globally distributed over the frame and provides a gain coefficient pattern that matches the color filter array pattern.

[0021] However, the crosstalk pattern can vary over the frame independently of the color filter array pattern. Preferably, a set of different gain matrices is provided, and each gain matrix is assigned to each position of the gain matrix on the frame to multiply each pixel of the frame by each gain coefficient at the pixel position in the associated stacked gain matrix.

[0022] From this, different gain grids are used according to the position in the frame to define the N×M gain grid not only globally but also locally.

[0023] A selected gain matrix from the set of gain matrices can be assigned to the associated support points in the frame. The support points that define the support pixel positions in the gain matrix assign the gain coefficients at the pixel positions of the selected gain matrix to each pixel of the frame and match the pixel positions in the frame at the associated support points defined for the frame to multiply the pixel values by the associated gain coefficients.

[0024] From this, several support points are defined in the frame, for example, a set of U×V support points in the frame is defined. These "macroscopic" support points are spread around throughout the frame. Preferably, they are spread evenly throughout the frame. The set of gain matrices comprises angular points, i.e., support points of the gain lattice, which are assigned to each respective support point in the associated gain matrix when functionally arranging the selected gain matrix across the frame. For each of the U×V support points, one selected N×M gain lattice is assigned to arrange the gain lattice selected across the frame at the position of the associated support point.

[0025] The use of a plurality of gain matrices and a plurality of support points, each having an individual set of gain coefficients for each pixel position in the gain matrix, enables reducing different crosstalk patterns across the frame.

[0026] One particular gain matrix can be provided for each support point in the frame.

[0027] Preferably, the gain matrix can be interpolated between adjacent gain matrices located at each respective adjacent support point. From this, the local gain lattice is interpolated between support points. As a result, it is not necessary to arrange the gain matrix over all the pixels of the frame to obtain the gain coefficients for the entire pixel matrix of the frame. The gain coefficients for pixel positions not covered by different gain matrices at the support points are determined by interpolation with the gain coefficients of the neighboring gain matrices.

[0028] A set of different gain matrices can be provided, where each gain matrix is assigned to a respective color. This enables the color adaptation gain lattice to reduce pattern artifacts that reappear on objects having saturated colors, for example. The reason for such color pattern artifacts is, for example, the white - colored interaction according to the color of the incident light.

[0029] The solid color can be determined for a matrix of pixels in a frame. The gain factor is determined for the pixel positions of a matrix of pixels in a frame having a different set of gain matrices, based on the solid color determined for such a matrix of pixels in the frame.

[0030] From this, instead of having a single set of gain matrices with associated gain factors, a plurality of K sets of gain matrices are provided, where each set of gain matrices is calibrated for one particular color.

[0031] The solid color can be determined by calculating the average color value of the primary color pixels in the sliding-window local neighbourhood of the matrix of pixels in the location of the associated gain matrix. The size of the matrix of pixels of the sliding window can be different from the size of the gain matrix. However, it is preferred that the size of the matrix of pixels of the sliding window is exactly the same as the N×M size of the gain matrix so that the measurement data does not already contain exactly that pattern which the algorithm should compensate for. The average of the red, green, and blue primary color pixel colors in the sliding-window local neighbourhood of the N×M pixels of the color matrix position on the frame can be calculated to determine the solid color.

[0032] The gain factor can be interpolated between the sets of gain matrices based on the determined solid color. This allows for having a reduced number of sets of gain matrices for some particular colors. Solid colors that do not match a particular color of a predefined gain matrix are addressed by interpolating the gain factor between at least two gain matrices of the set assigned to a particular color similar to the determined solid color. However, interpolation can also be performed based on more than two gain matrices of neighbouring colors.

[0033] Preferably, the weight for the gain factor in a set of gain matrices is calculated based on the respective solid color average value of the colored pixel values in the associated matrix of pixels in the frame.

[0034] The present invention will be described in more detail by way of illustrative embodiments with the accompanying drawings. The drawings are as follows.

Brief Description of the Drawings

[0035]

Figure 1

Figure 2

Figure 3

Figure 4

Modes for Carrying Out the Invention

[0036] FIG. 1 presents an illustrative schematic block diagram of an image processor unit 1 comprising a camera 2 and an image processor unit 3 for processing raw image data IMG provided by an image sensor 4 of the camera 2. RAW

[0037] The image sensor 4 comprises an array of pixels P RAW such that the raw image IMG x, y is a data set in the raw matrix of pixels for each image. A color filter array CFA is provided in the optical path in front of the image sensor 4 to capture the colors in the image. The camera 2 comprises an opto-mechanical lens system 5, for example, a fixed uncontrolled lens.

[0038] The image processor unit 1 can be incorporated into a handheld device such as a smartphone, tablet, wearable, picture or video camera, etc.

[0039] The image processor unit 3 receives image data IMG from the image sensor 4 that captures an imageRAW configured to process, and a frame is also considered an image in the sense of the present invention.

[0040] Advanced image sensors use complex color filter array patterns. The simplest color filter array pattern is the 2x2 Bayer color filter array. However, larger N×M RGB or RGBW color filter arrays (R = red, G = green, B = blue, W = white) exist. For example, 4×4 Quad Bayer, 6×6 Nonacell, 2×2 RGBW Bayer color filter array, 4×4 RGBW #1 Kodak color filter array, etc.

[0041] There is a problem of pixel crosstalk between adjacent pixels, which increases for larger color filter array patterns, especially those including white pixels where white and colors interact with each other. The effect on white pixels depends on nearby colored pixels. Pixel crosstalk is the result of an artificial periodic lattice structure on white pixels, especially having the size of N×M pixels of the color filter array pattern.

[0042] To reduce the pixel crosstalk effect, a lattice-based gain W i is applied to the pixel P RAW of the raw image IMG x, y

[0043] FIG. 2 is an example of a frame pixel array having an N×M lattice matrix periodically positioned across a frame.

[0044] The image of the frame comprises an array of pixels at x,y positions, where each pixel is assigned to a respective color of the color filter array CFA. For ease of understanding, this example is based on a 2×2 Bayer color filter array comprising a 2×2 matrix including a green - red - blue - green pattern.

[0045] ​Each N×M = 2×2 gain matrix with weights W1, W2, W3, and W4 for each pixel position in the gain matrix can be provided and stored in the data memory in this basic embodiment case.

[0046] The image processor unit 3 can multiply each weight W of the N×M gain matrix by each pixel P of the raw image data IMG i in the raw image data IMG RAW for each pixel P x, y as if the gain matrix were superimposed on the frame matrix shown in FIG. 2.

[0047] In the example of FIG. 2, the N×M = 2×2 gain matrix is periodically repeated over the entire size of the frame matrix and is, for example, a 6×4 frame matrix as illustrated for a very simplified example.

[0048] Each pixel P at each x,y position i, j is multiplied by the respective weight W at the same x,y position k . The result is a set of pixel values P for each y,x position processed by multiplying the pixel color value at the pixel position by the associated weight W k . i, j

[0049] The weights W of the N×M gain matrix k can be calibrated on a flat field reference image.

[0050] The use of N×M different gain values of the N×M gain matrix having the same size as the N×M color filter array matrix reduces the artificial periodic grid structure of N×M pixels caused by pixel crosstalk.

[0051] However, the crosstalk pattern can vary across frames. This can be addressed by defining an N×M gain matrix (gain lattice) that is not simply global as shown in FIG. 2, but rather varies. Improved embodiments provide different N×M gain matrices depending on the position within the frame.

[0052] FIG. 3 presents an example of spatial interpolation of a frame with different gain matrices S k positioned at support points S k for the frame.

[0053] In general, the number of U×V support points S n can be defined within the frame, i.e., the raw image data IMG RAW . Preferably, these support points S n can be arranged in a grid at the "macroscopic" pixel positions of the frame that are uniformly spread throughout the frame. The number U×V can be the whole divisor of the size N×M of the gain lattice if a set of gain lattices of uniform size N×M is used for the frame.

[0054] Based on the basic example of FIG. 2 with a simple 2×2 Bayer color filter array, two exemplary support points S1 and S2 are shown in the frame of FIG. 3, i.e., the raw image data IMG RAW for simplicity of presentation.

[0055] There is a set of gain matrices S k assigned to each respective support point S k . One gain matrix can be assigned to at least one support point S k .

[0056] Preferably, there is one N×M gain matrix for each of the U×V support points S k defined in the frame.

[0057] As illustrated, the support points S kEach is assigned to a pixel position in the grid of the frame and in the grid of the associated gain matrix. "Macroscopic" support points can be evenly spread throughout the entire frame.

[0058] Each pixel value in the frame is related to a specific support point S k and then can be weighted by the weight W k of the gain matrix related to the same support point S i, j This can be represented, for example, by an equation for an iterated 2×2 gain grid: For the N×M = 2×2 frame block for support point S1: P 33 = G 33 * W 11 P 34 = B 34 * W 13 P 43 = R 43 * W 12 P 44 = G 44 * W 14 For the N×M = 2×2 frame block for support point S2: P 53 = G 53 * W 21 P 54 = B 54 * W 23 P 63 = R 63 * W 22 P 64 = G 64 * W 24 The same applies hereinafter.

[0059] In a further improved embodiment, the weights in the "local" gain matrix between support points S k can be interpolated by using the weights W i in the neighboring gain matrix aligned with the support points.

[0060] For example, this can be performed using the following equation by interpolating the weights in the neighborhood gain grid for a specific color related to the pixel position: P 33 =G 33 *(W 11 +W 21 ) / 2 P 34 =B 34 *(W 13 +W 23 ) / 2 P 43 =R 43 *(W 12 +W 22 ) / 2 P 44 =G 44 *(W 14 +W 24 ) / 2 The same applies hereinafter.

[0061] Pattern artifacts caused by pixel crosstalk may also be affected by a specific color. The pattern may reappear, for example, on an object with saturated colors. The reason is that the interaction between white pixels and colored pixels depends on the color of the incident light.

[0062] To further reduce all pattern artifacts, a color adaptation gain grid can be applied.

[0063] Figure 4 presents a simplified example of color adaptation interpolation for a frame having a set of gain matrices selected for the solid color.

[0064] Instead of, or in addition to, having a single set of N×M gain matrices, a plurality of sets of gain matrices each assigned to a specific color can be provided by using the support points S in the N×M×U×V gain matrix k . Each set of gain matrices can be calibrated for one specific color, i.e., R = red, G = green, and B = blue.

[0065] Each pixel P i, jThe weight W for k can be interpolated among K sets of gain matrices based on the solid color at each pixel position P i, j .

[0066] The solid color of pixel position P i, j can be determined by calculating the average of the R, G, and B primary color pixels in the sliding window local neighborhood of the N×M pixels according to the N×M color filter array and the N×M gain matrix.

[0067] For example, the weight W for each of the K sets of gain matrices i can be calculated based on the local RGB weight average. The value K = 3 can be selected for calibration for near-monochromatic red (R), green (G), and blue (B) light.

[0068] The weight W i can be calculated from the weights in the K sets, and each of the K sets of gain matrices is assigned to a specific color. The weighting factors in the local gain matrix can be divided by the sum of the local RGB weighting factors in the matrix for three colors, namely green, red, and blue, according to, for example, the following formula: W i = RGB i / sum(RGB) [for i = 1, 2, 3]

[0069] This results in a weight sum of 1.

[0070] Optionally, the RGB white balance gain can be applied to the local RGB average.

[0071] According to a preferred embodiment of the present invention, exactly N×M pixels, i.e., each pixel of the N×M pixel set, are used in the sliding window local neighborhood. Otherwise, the measurement data would already include exactly that pattern which the algorithm is supposed to compensate for.

[0072] If there are known defective pixels or PDAF pixels in the frame matrix, they should be ignored.

[0073] This is illustrated by using a simple 2×2 Bayer color filter array, a set of K = 3 gain matrices defined and stored for local green, red, and blue, and a simplified example of a possible routine for color adaptation interpolation.

[0074] Each color gain matrix has weights W for the green matrix G1 , W G2 , W G3 , W G4 , weights W for the red gain matrix R1 , W R2 , W R3 , W R4 , and weights W for the blue gain matrix B1 , W B2 , W B3 , W B4 . These weights W Rn , W Gn , and W Bn are also called gain lattices.

[0075] Furthermore, the average weights W R , W G , and W B can be determined by calculating these weights from the local RGB average values.

[0076] The principle scheme is similar for higher orders in the N×M color filter array and the N×M gain matrix.

[0077] In addition, the support points S k can be defined in the frame matrix IMG RAW and each gain matrix as shown in FIG. 3. This results in a set of gain matrices for each color R, G, and B.

[0078] Each gain matrix is placed on the frame matrix, either periodically as shown in FIG. 2 or at support points S as shown in FIG. 3k Based on any of them, they are superimposed.

[0079] The pixel value P at each x, y position x, y is the raw image data IMG RAW , that is, the pixel value at each pixel position in the frame matrix, and the calculated weight W i and are calculated by.

[0080] The interpolated pixel value for the pixel position can be calculated, for example, by the following simplified formula: P 11 = G 11 * (W R * W R1 + W G * W G1 + W B * W B1 ) P 12 = B 12 * (W R * W R3 + W G * W G3 + W B * W B3 ) P 21 = R 21 * (W R * W R2 + W G * W G2 + W B * W B2 ) P 22 = G 22 * (W R * W R4 + W G * W G4 + W B * W B4 ) P 31 = G 31 * (W R * W R1 + W G * W G1 + W B * W B1 ) P 32 = B 32 * (WR *W R3 +W G *W G3 +W B *W B3 ) P 41 =R 41 *(W R *W R2 +W G *W G2 +W B *W B2 ) P 42 =G 42 *(W R *W R4 +W G *W G4 +W B *W B4 ) P 13 =G 13 *(W R *W R1 +W G *W G1 +W B *W B1 ) P 14 =B 14 *(W R *W R3 +W G *W G3 +W B *W B3 ) P 23 =R 23 *(W R *W R2 +W G *W G2 +W B *W B2 ) P 24 =G 24 *(W R *W R4 +W G *W G4 +W B *W B4 ) The same applies hereinafter. In particular, W R +W G +W BThis is the case when =1 is guaranteed. Otherwise, for example, it is desirable to normalize the above equation by dividing by the sum (W R +W G +W B ). This is illustrated for the first pixel position and can be adapted accordingly for all pixel positions: P 11 =G 11 *(W R *W R1 +W G *W G1 +W B *W B1 ) / (W R +W G +W B )

[0081] Furthermore, it is optional to apply non-linearity to the RGB average. This results in a gain matrix defined in the perceptual color space.

[0082] The saturation of the local RGB average can be increased as an additional option, preferably in order to always use a pure gain lattice set of mixing gains.

[0083] If the sum of the R, B, and G gain matrix sets is not equal to the gain matrix set calibrated for white light, the method may not function well enough for gray objects.

[0084] In particular, in this situation, the method can be further improved by using at least one of the strategies described below.

[0085] Preferably, for a color filter array including white pixels, such as an RGBW color filter array, the weighting method described above for the RGB color filter array can be supplemented.

[0086] An additional gain matrix or set of gain matrices calibrated for white light is used. From this, instead of the K = 3 RGB gain matrices as shown in FIG. 4, a set of K = 4 gain matrices for four “colors”, namely red, green, blue, and white, is defined and stored in the data memory.

[0087] The chroma from the local RGB average is calculated using an appropriate metric. This can be simply calculated for the medium by determining the difference between max(RGB) - min(RBG), i.e., the maximum value of the RGB pixel values. Suitable matrices or color representation models can be, for example, HSL (hue, saturation, lightness), HSV (hue, saturation, value), HSB (hue, saturation, brightness), or HSI (hue, saturation, intensity).

[0088] The chroma from the local RGB average can be calculated by using a lookup into a two-dimensional lookup table LUT of chroma values. The two-dimensional correlation can be, for example, {R / (R + G + b), B / (R + G + B)}.

[0089] The two-dimensional lookup table LUT has the advantage that it makes the chroma dependent on the hue so that the white grid functions well for some colors.

[0090] The weights can be calculated as follows by using a saturation coefficient sat: W i = sat * RGB i / sum(RGB) [for i = 1, 2, 3] W4 = 1 - sat [for white channel]

[0091] The result is a sum of weights of 1.

[0092] Optionally, the saturation value set can be modified by applying, for example, an offset, a coefficient or an exponent, or some combination thereof. This has the effect of preferring the white set over the saturation set.

[0093] Alternative representations of the RGB color model can be used by using a representation that includes hue, i.e., the visual attributes by the visual region appear to be similar to one of the perceived colors, e.g., one of red, yellow, green, and blue, or a combination of two of them.

[0094] A set of K = N + 1 gain matrices is defined and stored in the data memory along with the calibration for white and N different near-monochromatic hues.

[0095] The RGB average is converted to a color space with hue information, i.e., HSL, HSV, HSB, or HIS. Information about lightness, value, and intensity can be ignored.

[0096] Weight W i is calculated for N hues. This can be done, for example, by matching the measured hue. Preferably, something like Gaussian or nearest neighbor is determined to calculate the weight W i for the measured hue.

[0097] It is also possible to use a two-dimensional lookup with N weights W i for each entry. The two-dimensional coordinates of the lookup table can be, for example, {R / (R + G + B), B / (R + G + B)}.

[0098] Weight W i is calculated as W i = 1 - chroma for neutral gray.

[0099] Weight W i can be normalized so that the sum of the weights is 1.

[0100] Optionally, non-linearity can be applied to the hue and chroma.

Claims

Claim 1 A method for processing image data provided by an image sensor, the image data comprising a frame formed by an array of pixels, the pixel array being overlaid by a color filter array such that the pixels represent color information according to a specific color pattern defined by the color filter array, the method comprising: - multiplying each pixel of the frame by a respective gain coefficient predefined in a gain matrix stored for the pixel positions corresponding to the pixels in the frame when the gain matrix is positioned on the frame; - the gain matrix being smaller than the frame and being arranged at a plurality of different positions on the frame, and multiplying each pixel of the frame by the respective gain coefficient of the gain matrix. Claim 2 The method according to claim 1, characterized in that the gain coefficients assigned to the pixel positions of the gain grid on the flat field reference image are calibrated. Claim 3 The method according to claim 1 or 2, characterized in that the size of the gain matrix is the size of the color filter array pattern, which is repeated to form the color filter array of the size of the frame. Claim 4 The method according to any one of claims 1 to 3, characterized by periodically repeating and positioning the gain matrix on the frame such that the color pattern of the frame matches the color pattern assigned to the gain coefficients in the gain matrix, and multiplying each pixel of the frame by the respective gain coefficient of the periodically repeated gain matrix. Claim 5 The method according to any one of claims 1 to 3, characterized by providing a set of different gain matrices, each gain matrix being assigned to a respective position of the gain matrix on the frame for multiplying each pixel of the frame by the respective gain coefficient of the associated gain matrix overlaid. Claim 6 Allocating a selected gain matrix to relevant support points in the frame, wherein the support points at the defined support pixel positions of the selected gain matrix assign the gain coefficients for the pixel positions of the selected gain matrix to each pixel of the frame, and multiply the pixel values by the relevant gain coefficients, and allocating is characterized in that it coincides with the pixel positions of the support points in the frame, the method according to claim 5.

7. The method according to claim 6, characterized in that one specific gain matrix is provided for each support point in the frame.

8. The method according to claim 6 or 7, characterized in that the gain matrix between adjacent gain matrices located on each adjacent support point is interpolated.

9. - Providing a set of different gain matrices, each gain matrix being assigned to a respective color, and providing; - Determining the solid color for the matrix of pixels in the frame; - Determining the gain coefficients for the pixel positions of the matrix of pixels in the frame in the frame having the set of different gain matrices based on the determined solid color for the matrix of pixels. The method according to any one of claims 1 to 8.

10. Determining the solid color by calculating the average color value of the primary color pixels in the sliding window local neighborhood of the matrix of pixels in the location of the relevant gain matrix, wherein the size of the matrix of pixels is the size of the gain matrix, the method according to claim 9.

11. The method according to claim 9 or 10, characterized in that the gain coefficients between the sets of gain matrices are interpolated based on the determined solid color.

12. The method according to any one of claims 9 to 11, characterized in that the weights for the gain coefficients in the set of gain matrices are calculated based on the respective solid color average values of the colored pixel values in the matrix of relevant pixels in the frame.

13. An image processor unit for processing image data provided by an image sensor, wherein the image sensor comprises a sensor pixel array that provides a frame formed by an array of the pixels overlaid by a color filter array such that the pixels represent color information according to a specific color pattern defined by the color filter array. In the image processor unit, the image processor unit is configured to: - multiply each pixel of the frame by a respective gain coefficient defined in a gain matrix stored at a pixel position corresponding to the pixel in the frame when the gain matrix is positioned over the frame; - the gain matrix is smaller than the frame and is arranged at a plurality of different positions over the frame, and is characterized in that each pixel of the frame is multiplied by the respective gain coefficient of the gain matrix. An image processor unit.

14. The image processor unit according to claim 13, characterized in that the image processor unit is configured to process image data by executing the method steps according to any one of claims 1 to 12.

15. A computer program comprising instructions which, when the program is executed by a processing unit, cause the processing unit to perform the steps of the method according to any one of claims 1 to 12. A computer program.

Images