Pixel interpolation circuitry and pixel interpolation method

Abstract

The present disclosure generally pertains to pixel interpolation circuitry configured to: obtain, according to a predetermined pixel pattern, infrared pixel values for a plurality of first pixels configured to detect infrared light, the pixel pattern being indicative of a position of the plurality of first pixels on an image sensor; generate, based on the obtained infrared pixel values and in accordance with the pixel pattern, a first interpolated infrared pixel value for a second pixel being, in a normal operation mode, configured to detect light of a first color; and generate, based on the infrared pixel values and the first interpolated infrared pixel value and in accordance with the pixel pattern, a second interpolated infrared pixel value for a third pixel being, in a normal operation mode, configured to detect light of a second color.

Description

[0001] PIXEL INTERPOLATION CIRCUITRY AND PIXEL INTERPOLATION

[0002] METHOD

[0003] TECHNICAL FIELD

[0004] The present disclosure generally pertains to pixel interpolation circuitry and a pixel interpolation method.

[0005] TECHNICAL BACKGROUND

[0006] Generally, it is known as interpolation that a pixel value is determined based on a measured signal of another pixel. For example, an interpolated green pixel value (for a red pixel) may be determined based on surrounding green pixels. Also, known methods may perform the same interpolation scheme for all pixels that are to be interpolated.

[0007] Although there exist techniques for interpolating a pixel value, it is generally desirable to provide pixel interpolation circuitry and a pixel interpolation method.

[0008] SUMMARY

[0009] According to a first aspect, the disclosure provides pixel interpolation circuitry configured to: obtain, according to a predetermined pixel pattern, infrared pixel values for a plurality of first pixels configured to detect infrared light, the pixel pattern being indicative of a position of the plurality of first pixels on an image sensor; generate, based on the obtained infrared pixel values and in accordance with the pixel pattern, a first interpolated infrared pixel value for a second pixel being, in a normal operation mode, configured to detect light of a first color; and generate, based on the infrared pixel values and the first interpolated infrared pixel value and in accordance with the pixel pattern, a second interpolated infrared pixel value for a third pixel being, in a normal operation mode, configured to detect light of a second color.

[0010] According to a second aspect, the disclosure provides a pixel interpolation method comprising: obtaining, according to a predetermined pixel pattern, infrared pixel values for a plurality of first pixels configured to detect infrared light, the pixel pattern being indicative of a position of the plurality of first pixels on an image sensor; generating, based on the obtained infrared pixel values and in accordance with the pixel pattern, a first interpolated infrared pixel value for a second pixel being, in a normal operation mode, configured to detect light of a first color; and generating, based on the infrared pixel values and the first interpolated infrared pixel value and in accordance with the pixel pattern, a second interpolated infrared pixel value for a third pixel being, in a normal operation mode, configured to detect light of a second color.

[0011] Further aspects are set forth in the dependent claims, the drawings and the following description.

[0012] BRIEF DESCRIPTION OF THE DRAWINGS

[0013] Embodiments are explained by way of example with respect to the accompanying drawings, in which:

[0014] Fig. 1 depicts an illustrational image sensor according to the present disclosure;

[0015] Fig. 2 depicts the image sensor of Fig. 1, wherein formerly red and blue pixels are renamed to infrared pixels to show that an infrared value is interpolated for these pixels;

[0016] Fig. 3 depicts the image sensor of Fig. 1, wherein in addition to Fig. 2, also the formerly green pixels are renamed to infrared pixels to show that an infrared value is interpolated for these pixels;

[0017] Fig. 4 depicts an embodiment of a method in which a green diagonal gradients are used for interpolation;

[0018] Fig. 5 depicts an embodiment of a method in which green comer gradients are used for interpolation;

[0019] Fig. 6 depicts an embodiment of a method in which green border gradients are used for interpolation;

[0020] Fig. 7 depicts a diagram of an imaging device according to the present disclosure;

[0021] Fig. 8 depicts a block diagram of a method according to the present disclosure;

[0022] Fig. 9 depicts a block diagram of a method according to the present disclosure, wherein a first interpolated pixel value is determined based on first gradients;

[0023] Fig. 10 depicts a block diagram of a method according to the present disclosure, wherein second color pixel values are obtained and a diagonal gradient is determined; and

[0024] Fig. 11 depicts a block diagram of a method according to the present disclosure, wherein a weight is determined for interpolation. DETAILED DESCRIPTION OF EMBODIMENTS

[0025] Before a detailed description of the embodiments starting with Fig. 1 is given, general explanations are made.

[0026] As mentioned in the outset, known methods may apply the same interpolation scheme for all pixels. However, it has been recognized that it may be suitable to carry out a two-stage interpolation, wherein, in the second stage, already interpolated values are used.

[0027] It has further been recognized that it may be desirable to upscale an infrared channel of an RGBIR (red, green, blue, infrared) sensor to a native resolution of the sensor, for example for machine vision applications.. This recognizes that image sensing pixels, RGB, generally may have a higher resolution than the IR pixels. In embodiments, RGB pixels may not be of uniform or regular resolution, so it may be desirable to upscale am infrared channel up to a resolution of certain of the RGB pixels, for example a subset of RGB pixels or a sub-array of RGB pixels. However, it should be noted that the present disclosure is not limited to the case of RGB pixels since any combination of colors or color filters are envisaged. Also at least one tunable colorfilter may be applied, in some embodiments.

[0028] Moreover, it has been recognized that, according to the present disclosure, wide angle applications for infrared data may be enabled and chromatic aberration may be avoided.

[0029] It has further been recognized that for machine vision applications, edge preserving interpolation may be useful. This may be achieved by using local gradient information of an infrared channel, and in some embodiments, of a green channel. Gradient information of the green channel may be used indirectly, thereby decreasing an impact of chromatic aberration (in case the green channel is mis-aligned with a target edge of the infrared channel). In known methods, a green channel is used to create edges, but it has been recognized that such methods would create an edge at a wrong position because a green pixel and an IR pixel may be at different locations.

[0030] It has further been recognized that simple interpolation rules may enable the methods described herein to be implemented on-chip.

[0031] Therefore, some embodiments pertain to pixel interpolation circuitry configured to: obtain, according to a predetermined pixel pattern, infrared pixel values for a plurality of first pixels configured to detect infrared light, the pixel pattern being indicative of a position of the plurality of first pixels on an image sensor; generate, based on the obtained infrared pixel values and in accordance with the pixel pattern, a first interpolated infrared pixel value for a second pixel being, in a normal operation mode, configured to detect light of a first color; and generate, based on the infrared pixel values and the first interpolated infrared pixel value and in accordance with the pixel pattern, a second interpolated infrared pixel value for a third pixel being, in a normal operation mode, configured to detect light of a second color.

[0032] The pixel interpolation circuitry may be any entity or multitude of entities configured to process pixel values (discussed below). For example, the circuitry may include an image sensor or may be configured to communicate with an image sensor in order to obtain the pixel value(s). Hence, the circuitry may include at least one processor (e.g., CPU (central processing unit), GPU (graphics processing unit)), or the like. Also, a hybrid chip may be envisaged, such as a hybrid CPU-GPU, in some embodiments.

[0033] In some embodiments, infrared pixel values for a plurality of first pixels are obtained. The pixel values may be indicative of a light intensity of infrared light detected with each of the first pixels, as is commonly known. Accordingly, the first pixels may include (or correspond to) infrared pixels, wherein the present disclosure is not limited to any specific range of infrared light.

[0034] The pixel values may correspond to a current or voltage generated by the pixel that is read out and translated into an intensity value, or the like, as commonly known.

[0035] The infrared pixels may be provided according to a predetermined pixel pattern on an image sensor and thus, the obtaining of the infrared pixel values may also be achieved according to the predetermined pixel pattern. The pixel pattern may include any ordering of infrared pixels on an image sensor, and may indicate where on the sensor the infrared pixels are positioned (e.g., with respect to other (color) pixels, such as red and / or green, and / or blue).

[0036] In some embodiments, the obtained infrared pixel values are used to generate a first interpolated pixel value for a second pixel.

[0037] For example, according to the present disclosure, an infrared resolution sensor may be increased by interpolating infrared pixel values onto a pixel that would normally (in a normal operation mode) detect a different color. For example, taking into account the pixel pattern, on a position of a blue pixel, an infrared pixel value may be artificially generated by interpolating based on the obtained infrared pixel values.

[0038] In a simple example, if the blue pixel value lies between two infrared pixels, the first interpolated infrared pixel value may be generated based on an arithmetic mean value of the infrared pixel values of the two infrared pixels. The normal operation mode may refer to an operation of a single pixel or of a plurality of pixels to detect light of a predetermined wavelength / frequency (e.g., red, green, blue, or the like). Hence, the detection of light may be based on a color filter, wavelength / frequency filter, or the like. The (second) pixel in the normal operation mode is not configured to receive and / or process IR light and its IR pixel value may be obtained by interpolation.

[0039] In some embodiments, a second interpolated infrared pixel value is generated for a third pixel which would normally detect light of a second color (if the first one was blue, the second may be green, for example). Similarly to the first interpolated infrared pixel value, the second interpolated infrared pixel value is generated based on an interpolation with pixel values of adjacent pixels (or at least spatially close pixels, e.g., according to a predetermined distance of pixel positions on a sensor). However, the generation of the second interpolated pixel value is further based on the first interpolated infrared pixel value.

[0040] For example, if the third pixel is a green pixel, the second interpolated infrared pixel value may be generated based on an obtained infrared pixel value and based on an interpolated pixel value that was previously generated for the blue pixel (in case the green pixel lies between the infrared pixel and the blue pixel).

[0041] Hence, according to the present disclosure, a two-stage interpolation method is envisaged for increasing an infrared resolution of an image sensor up to the native resolution of the image sensor or the native resolution of a subset of pixels of the image sensor. However, the present disclosure is not limited to the case that infrared pixel values are obtained and interpolated and the skilled person may apply the teaching of the present disclosure to interpolation of other pixels (e.g., interpolating green pixel values on infrared pixels, or the like).

[0042] In some embodiments, the generation of the first interpolated infrared pixel value includes: determining which of two pairs of first pixels surrounding or either side of the second pixel have a higher pixel value gradient for example when pixel value gradient for a pair is calculated and compared to another pixel value gradient for another pair .

[0043] For example, a second pixel may be surrounded by four first pixels, e.g., at comers of the second pixel (in a simplified embodiment where the pixels are rectangular), such that one comer of each first pixel is adjacent to one comer of the second pixel.

[0044] In that case, a diagonal gradient may be determined for each pair of first pixels that are diagonally adjacent to the second pixel, for example where there is no intermediate pixel between their corners when arranged in an array and represented as rectangles. In some embodiments, the gradient may be determined based on subtracting the respective infrared pixel values of the pair of first pixels and determining the absolute value of the result, but the present disclosure is not limited to that case of determining the gradient. The present disclosure is also not limited to the case of determining which gradient is higher since it may likewise be determined which gradient is lower and / or whether the gradients are equal, but for simplicity, the present explications refer to the case that it is determined which gradient is higher.

[0045] In some embodiments, the first interpolated infrared value is set to a mean value of the two pixel values of which the gradient is higher.

[0046] In some embodiments, the first interpolated infrared value is set to a mean value of the two pixel values of which the gradient is lower.

[0047] In some embodiments, the first interpolated infrared value is set to a mean value of the four surrounding pixel values, if the gradients are (roughly) equal (or lie within a predetermined threshold)

[0048] Which mean value is chosen may depend on the circumstances and may be apparent to the skilled person.

[0049] In some embodiments, the generation of the second interpolated infrared pixel value includes: determining which of a pair of first pixels and a pair of interpolated second pixels have a higher pixel value gradient, wherein the pair of first pixels and the pair of interpolated pixels surround the third pixel.

[0050] In this case, two interpolated second pixels may be on opposing sides (edges) of the third pixel and two first pixels may be on opposing other sides (edges) of the third pixel. In that case, a gradient of the pair of first pixels may be compared with a gradient of the pair of (interpolated) second pixels and the second interpolated infrared value may be determined based on a determination which of the gradients is higher (or lower, or whether the gradients are roughly equal, as discussed herein).

[0051] In some embodiments, second color pixel values are used for the interpolation.

[0052] Hence, in some embodiments, the circuitry is further to: obtain second color pixel values indicative of the light of the second color; and generate the second interpolated infrared data further based on the second color pixel values.

[0053] In such embodiments, the intensity of the detected second color may be envisaged for determining the first interpolated infrared value, such that an accuracy may be increased. In some embodiments, the generation of the second interpolated infrared data further includes: determining a diagonal gradient for the second color pixels values.

[0054] If the second color pixel to be interpolated is surrounded by other second color pixels on its edges, the diagonal gradient may be determined for checking which mean value of two pairs of second color pixel values would come closest (or is below) to the actual second color pixel value that is measured.

[0055] In some embodiments, the circuitry is further configured to: obtain second color pixel values indicative of light of the second color; and determine a weight for the obtained infrared pixel values based on a gradient of the second color pixel values and weighting the obtained infrared pixel values with the determined weight, thereby obtaining weighted infrared pixel values.

[0056] In such embodiments, first pixels are weighted with gradients of second pixels that are neighboring the respective first pixel. The determined weight may be used in the generation of the first interpolated infrared data and / or the second interpolated infrared data. In that case, it may be possible to tune, if an artifact is created in favor of sharpness, for example, or more generally it may be possible to tune with regard to image quality qualifiers.

[0057] Hence, in some embodiments, the circuitry is further configured to: generate the first interpolated infrared pixel value based on the weighted infrared pixel values.

[0058] Also, in some embodiments, the circuitry is further configured to: generate the second interpolated infrared pixel value based on the weighted infrared pixel values.

[0059] In some embodiments, the first color corresponds to pixels with a predetermined first resolution on the image sensor.

[0060] In some embodiments, the second color corresponds to pixels with a predetermined second resolution on the image sensor, wherein the first resolution is lower than the second resolution.

[0061] However, it should be noted that the two resolutions may be the same in some embodiments and the colors may be changed according to the circumstances. For example, the first color includes at least one of red and blue and the second color includes green, but this may be exchanged and is discussed for exemplary purposes only and also other colors may be envisaged. It should be noted that also more or less than three different color pixels may be provided, in some embodiments (apart from IR). In some embodiments, the resolution of green may be higher than the resolution of red and blue since green may be dominant in the light spectrum of the sun, but as indicated above, other resolutions and combinations of the respective colors may be envisaged.

[0062] Some embodiments pertain to a pixel interpolation method including: obtaining, according to a predetermined pixel pattern, infrared pixel values for a plurality of first pixels configured to detect infrared light, the pixel pattern being indicative of a position of the plurality of first pixels on an image sensor; generating, based on the obtained infrared pixel values and in accordance with the pixel pattern, a first interpolated infrared pixel value for a second pixel being, in a normal operation mode, configured to detect light of a first color; and generating, based on the infrared pixel values and the first interpolated infrared pixel values and in accordance with the pixel pattern, a second interpolated infrared pixel value for a third pixel being, in a normal operation mode, configured to detect light of a second color, as discussed herein.

[0063] The method may be carried out with pixel interpolation circuitry according to the present disclosure.

[0064] In some embodiments, the generation of the first interpolated infrared pixel value includes: determining which of two pairs of first pixels surrounding the second pixel have a higher pixel value gradient, as discussed herein. In some embodiments the generation of the second interpolated infrared pixel value includes: determining which of a pair of first pixels and a pair of interpolated second pixels have a higher pixel value gradient, wherein the pair of first pixels and the pair of interpolated pixels surround the third pixel, as discussed herein. In some embodiments, the method further includes: obtaining second color pixel values indicative of the light of the second color; and generating the second interpolated infrared data further based on the second color pixel values, as discussed herein. In some embodiments, the generation of the second interpolated infrared data further includes: determining a diagonal gradient for the second color pixels values, as discussed herein. In some embodiments, the method further includes: obtaining second color pixel values indicative of light of the second color; and determining a weight for the obtained infrared pixel values based on a gradient of the second color pixel values and weighting the obtained infrared pixel values with the determined weight, thereby obtaining weighted infrared pixel values, as discussed herein. In some embodiments, the method further includes: generating the first interpolated infrared pixel value based on the weighted infrared pixel values, as discussed herein. In some embodiments, the method further includes: generating the second interpolated infrared pixel value based on the weighted infrared pixel values, as discussed herein. In some embodiments, the first color includes at least one of red and blue, as discussed herein. In some embodiments, the second color includes green.

[0065] The methods as described herein are also implemented in some embodiments as a computer program causing a computer and / or a processor to perform the method, when being carried out on the computer and / or processor. In some embodiments, also a non-transitory computer- readable recording medium is provided that stores therein a computer program product, which, when executed by a processor, such as the processor described above, causes the methods described herein to be performed.

[0066] Returning to Fig. 1, there is depicted an illustrational image sensor 1 including a plurality of pixels 2. The pixels include red pixels R (configured to detect red light), green pixels G (configured to detect green light), blue pixels B (configured to detect blue light), and infrared pixels IR (configured to detect infrared light). The red, green and blue pixels are configured to detect the respective color in accordance with a color filter, but it should be noted that the present disclosure is not limited to that case since also tunable filters may be envisaged according to the present disclosure.

[0067] Fig. 1 depicts the method of generating a first interpolated infrared pixel value with a plurality of pixels 3 depicted on the right of the image sensor 1. It should be noted that the plurality of pixels 3 is displayed only for illustrational purposes and the explications herein are also applicable to the remaining pixels of the image sensor 1. Moreover, it should be noted that a red pixel R is displayed in the middle of the plurality of pixels 3, but red pixels R and blue pixels B are likewise interpolated in the method depicted in Fig. 1, whereas green pixels G are interpolated in Fig. 2. Hence, for the red pixels R and for the blue pixels B, first interpolated infrared pixel values are generated.

[0068] In the plurality of pixels 3, the infrared pixels IR are given numbers which represent the respective infrared pixel value and which are used in the interpolation formulas below.

[0069] As shown below the image sensor 1, raw data including infrared pixel values IR1, IR2, IR3, and IR4 are obtained. The diagonal gradients (IR diag grad) are determined based on an absolute value of a difference of diagonally adjacent IR pixels, i.e.: gradl = |IR1 - IR4| grad2 = |IR3 - IR2| gradl, grad2, and the raw data are input into an R&B (red and blue) interpolation module configured to compare gradl and grad2. In this embodiment, it is determined whether gradl is larger than grad2 to determine an RB pixel value (in this case the red pixel value, but these explications also apply to blue pixel values such that the term “RB” is used herein):

[0070] If gradl > grad2: RB = (IR2 + IR3) / 2

[0071] Else if gradl < grad2: RB = (IR1 + IR4) / 2

[0072] Else RB = (IR1 + IR2 + IR3 + IR4) / 4

[0073] Hence, the value for the red or blue pixel RB is set to the mean value of the pair of infrared pixels IR whose gradient is smaller. If the gradients are (roughly) the same, the mean value of all four infrared pixels IR is used as the value for RB.

[0074] Fig. 2 depicts the image sensor 1, but the formerly red pixels R and blue pixels B are renamed to I which stands for interpolated infrared pixels. Of course, the pixels are still configured to detect red and blue light, but an infrared value is interpolated for the pixels, thereby increasing the infrared resolution of the image sensor.

[0075] Fig. 2 depicts a plurality of pixels 4, similar to the plurality of pixels 3, which serves as an illustrational basis for the method of generating a second interpolated infrared value for a green pixel G3.

[0076] The second interpolated infrared value is generated based on a pair of interpolated infrared values II and 12 and based on a pair of obtained infrared values IR2 and IR4 (corresponding to the values IR2 and IR4 of Fig. 1).

[0077] The RB interpolation (raw rb interpolation) that is output in the method of Fig. 1 is used as a basis in this embodiment and is fed into an IR HV grad module (IR HV grad: infrared horizontal vertical gradient) configured to determine a horizontal gradient (for the interpolated RB pixels II and 12) and a horizontal gradient (for the IR pixels IR2 and IR4). It should be noted that the horizontal gradient may for other green pixels be based on IR pixels and the vertical gradient may be based on RB pixels. The gradients are determined as follows:

[0078] Vertical gradient grad_v = |IR2-IR4|

[0079] Horizontal gradient grad h = |I1 -I2| grad v and grad h are fed, together with the raw rb interpolation data in a Green interpolation module configured to compare the gradients and interpolate the green pixel values, as follows: If grad h >grad_v: G3 = (IR2 + IR4) / 2

[0080] Else if grad_h < grad_v: G3 = (Il + 12) / 2

[0081] Else G3 = (IR2 + IR4 + Il + 12) / 4

[0082] Hence, the value for G3 is set to the mean value of the pair whose gradient is smaller. If the gradients are (roughly) the same, G3 is set to the mean value of all four surrounding pixels II, 12, IR2, and IR4.

[0083] As can be taken from Fig. 3, the formerly green pixels G are also now renamed to I, such that, based on the interpolation, the infrared resolution has reached the native resolution of the image sensor 1.

[0084] Hence, as can be seen in Figs. 1 to 3, the present disclosure uses a cascade of two (different) three-times-three (non-linear) kernels: In a first step, red and blue pixels are interpolated, and in a second step, green pixels are interpolated. However, the present disclosure is not limited to three-times-three kernels. For example, a five-times-five kernels may be used likewise. Outside of the kernels, no other pixel is used for the interpolation.

[0085] In this embodiment, the remaining green values are not used for interpolation, but Fig. 4 depicts the case in which a G diag grad module (green diagonal gradient module) is used for determining gradients of the surrounding green pixels with respect to the green pixel G3 to be interpolated, i.e., an absolute value of a difference of pixel value G3 and G1 is determined, thereby obtaining grad_G3Gl, and so on. Thereby, the information from the IR pixels may be further enhanced.

[0086] The remaining parts of Fig. 4 correspond to the embodiment of Fig. 2, such that repetitive description is omitted.

[0087] As indicated above, in some embodiments, pixel values are weighted for the interpolation. The following description of Figs. 5 and 6 is based on such a weighting.

[0088] The following discussion of Fig. 5 will be based on the plurality of pixels shown on top of Fig. 5.

[0089] In this embodiment, gradl and grad2 are determined as discussed above (also IR diagonal gradients) and it is compared whether one of the two gradients is much bigger than the other. In this context, “much bigger” is defined as: A » B <4> A > B * threshold. For example, if the threshold is 10, then 100 » 5, because 100 > 5*10.

[0090] The comparison is carried out as follows:

[0091] If gradl » grad2: Discard IR1 and IR4 from the interpolation If grad2 » grad 1 : Discard IR2 and IR3 from the interpolation

[0092] Else: All four IR pixels are used for the interpolation

[0093] Also, diagonal gradients of the green pixels G are determined as follows (also referred to as corner gradients in Fig. 5): grad_GlG2 = |G1-G2| grad_GlG4 = |G1-G4| grad_G2G3 = |G2-G3| grad_G3G4 = |G3-G4|

[0094] The IR pixels values IR1, IR2, IR3, and IR4 are weighted with the gradients as follows:

[0095] IR2 -A IR2*w2 wl, w2, w3, and w4 are the respective weights that are calculated as follows: wl = l / grad_GlG4 w2 = l / grad_GlG2 w3 = l / grad_G3G4 w4 = l / grad_G2G3

[0096] The first interpolated infrared value for the red pixel R corresponds to the sum of each weight multiplied by its corresponding infrared value divided by the sum of all weights. The first interpolated infrared is determined as follows (also referred to as R&B interpolation with G corner grad weighting in Fig. 5):

[0097] If gradl » grad 2: wSum = w2 + w3

[0098] I = (IR2*w2 + IR3*w3) / wSum

[0099] If grad2 » gradl : wSum = wl + w4 I = (IRl*wl + IR4*w4) / wSum

[0100] Else: wSum = wl+w2+w3+w4

[0101] I = (IRl*wl + IR2*w2 * IR3*w3 + IR4*w4)

[0102] As can be taken from Fig. 5, the raw data are input into the IR diag grad module (which determines gradl and grad2) and into the G comer grad module which determines the comer gradients of the green pixels G as discussed above. The respective gradients are output and input into a module configured to carry out R&B interpolation with G corner gradient weighting, according to the conditions above.

[0103] However, the present disclosure is not limited to the weighting as described under reference of Fig. 5. For example, the following weighting method may alternatively be used:

[0104] A tuning parameter g grad max may be introduced for that weighting method for determining the weights. Then, a weight wX may be determined as wX = max(g_grad_max - grad_X; 0).

[0105] Hence, the weight is then defined as the maximum of a difference between the tuning parameter and a gradient and zero, i.e., if zero is larger than the difference, wX is zero, and if not, then wX corresponds to said difference.

[0106] The interpolation may then be carried out as described under reference of Fig. 5, but with different weights.

[0107] In some embodiments, the further interpolation scheme (after the red and blue interpolation= is carried out as in Fig. 2 or in Fig. 4.

[0108] On the other hand, Fig. 6 depicts a further method for generating the second interpolated infrared data. Instead of using a diagonal gradient for the green pixel G2 (as in Fig. 4), green border gradients are determined in the module “G border grad”, based on the plurality of pixels shown on the top of Fig. 6, as follows: grad_GlG5 = |G1-G5| grad_G3G6 = |G3-G6| grad_GlG3 = |G1-G3| grad_G5G6 = |G5-G6| Then, the green interpolation module with green border gradient weighting is configured to carry out the following calculation for determining an interpolated value I for the green pixel G2: if grad h > grad_v*threshold

[0109] I = (IR2 / grad_GlG5 + IR4 / grad_G3G6) / (l / grad_GlG5 + l / grad_G3G6) if grad v > grad_h*threshold

[0110] I = (Il / grad_GlG3 + I2 / grad_G5G6) / (l / grad_GlG3 + l / grad_G5G6) else

[0111] I = (IR2 / grad_GlG5 + IR4 / grad_G3G6 + Il / grad_GlG3 + I2 / grad_G5G6) / / (l / grad_GlG5 + l / grad_G3G6 + l / grad_GlG3 + l / grad_G5G6)

[0112] Fig. 7 depicts a very simplified diagram of an imaging device 10 according to the present disclosure.

[0113] The imaging device 10 includes an image sensor 11 configured to detect light of different wavelengths (i.e., red, blue, green, and infrared), a lens 12 configured to focus incident light on the image sensor 11 and pixel interpolation circuitry 13 according to the present disclosure. The skilled person will appreciate that for simplification, other typical elements of an imaging device are omitted.

[0114] The pixel interpolation circuitry 13 is configured to obtain, according to a predetermined pixel pattern, infrared pixel values for a plurality of first pixels configured to detect infrared light, the pixel pattern being indicative of a position of the plurality of first pixels on the image sensor 11; generate, based on the obtained infrared pixel values and in accordance with the pixel pattern, a first interpolated infrared pixel value for a second pixel being, in a normal operation mode, configured to detect light of a first color; and generate, based on the infrared pixel values and the first interpolated infrared pixel values and in accordance with the pixel pattern, a second interpolated infrared pixel value for a third pixel being, in a normal operation mode, configured to detect light of a second color, as discussed herein.

[0115] Fig. 8 depicts a block diagram of a method 20 according to the present disclosure.

[0116] At 21, infrared pixel values are obtained for a plurality of first pixels, as discussed herein.

[0117] At 22, a first interpolated infrared pixel value is generated for a second pixel (a red pixel, in this embodiment), as discussed herein. At 23, a second interpolated infrared pixel value is generated for a third pixel (a green pixel, in this embodiment), as discussed herein.

[0118] Fig. 9 depicts a block diagram of a method 30 according to the present disclosure. 31 to 33 correspond to 21 to 23 of Fig. 8.

[0119] Additionally, the first interpolated infrared pixel value is determined based on first gradients, which are determined at 34, i.e., based on a determination which of two pairs of first pixels surrounding the second pixel have a higher pixel value gradient, as discussed herein.

[0120] Moreover, second gradients are determined at 35. That is, a determination is carried out which of a pair of first pixels and a pair of interpolated second pixels have a higher pixel value gradient, as discussed herein.

[0121] Fig. 10 depicts a block diagram of a method 40 according to the present disclosure. 41 to 43 correspond to 21 to 23 of Fig. 8.

[0122] Additionally, at 44, second color pixel values are obtained (green pixel values, in this embodiment). At 45, a diagonal gradient is determined for generating the second interpolated infrared pixel value at 43.

[0123] Fig. 11 depicts a block diagram of a method 50 according to the present disclosure. 51 to 54 correspond to 41 to 44 of Fig. 10.

[0124] Additionally, at 55, a weight is determined for the obtained infrared pixel values based on a gradient of the second color pixel values. Moreover, the first and the second interpolated infrared pixel values are generated based on the weight at 52 and 53.

[0125] It should be recognized that the embodiments describe methods with an exemplary ordering of method steps. The specific ordering of method steps is however given for illustrative purposes only and should not be construed as binding. For example the ordering of 34 and 35 in the embodiment of Fig. 9 may be exchanged. Also, the ordering of 42 and 44 in the embodiment of Fig. 10 may be exchanged. Other changes of the ordering of method steps may be apparent to the skilled person.

[0126] Please note that the division of the imaging device 10 into units 11 and 13 is only made for illustration purposes and that the present disclosure is not limited to any specific division of functions in specific units.

[0127] The methods described herein can also be implemented as a computer program causing a computer and / or a processor, such as the pixel interpolation circuitry 13 discussed above, to perform the method, when being carried out on the computer and / or processor. In some embodiments, also a non-transitory computer-readable recording medium is provided that stores therein a computer program product, which, when executed by a processor, such as the processor described above, causes the method described to be performed.

[0128] All units and entities described in this specification and claimed in the appended claims can, if not stated otherwise, be implemented as integrated circuit logic, for example on a chip, and functionality provided by such units and entities can, if not stated otherwise, be implemented by software.

[0129] In so far as the embodiments of the disclosure described above are implemented, at least in part, using software-controlled data processing apparatus, it will be appreciated that a computer program providing such software control and a transmission, storage or other medium by which such a computer program is provided are envisaged as aspects of the present disclosure.

[0130] Note that the present technology can also be configured as described below.

[0131] (1) Pixel interpolation circuitry configured to: obtain, according to a predetermined pixel pattern, infrared pixel values for a plurality of first pixels configured to detect infrared light, the pixel pattern being indicative of a position of the plurality of first pixels on an image sensor; generate, based on the obtained infrared pixel values and in accordance with the pixel pattern, a first interpolated infrared pixel value for a second pixel being, in a normal operation mode, configured to detect light of a first color; and generate, based on the infrared pixel values and the first interpolated infrared pixel value and in accordance with the pixel pattern, a second interpolated infrared pixel value for a third pixel being, in a normal operation mode, configured to detect light of a second color.

[0132] (2) The pixel interpolation circuitry of (1), wherein the generation of the first interpolated infrared pixel value includes: determining which of two pairs of first pixels surrounding the second pixel have a higher pixel value gradient.

[0133] (3) The pixel interpolation circuitry of (2), wherein the generation of the second interpolated infrared pixel value includes: determining which of a pair of first pixels and a pair of interpolated second pixels have a higher pixel value gradient, wherein the pair of first pixels and the pair of interpolated pixels surround the third pixel. (4) The pixel interpolation circuitry of (2) or (3), further configured to: obtain second color pixel values indicative of the light of the second color; and generate the second interpolated infrared data further based on the second color pixel values.

[0134] (5) The pixel interpolation circuitry of (4), wherein the generation of the second interpolated infrared data further includes: determining a diagonal gradient for the second color pixels values.

[0135] (6) The pixel interpolation circuitry of anyone of (1) to (5), further configured to: obtain second color pixel values indicative of light of the second color; and determine a weight for the obtained infrared pixel values based on a gradient of the second color pixel values and weighting the obtained infrared pixel values with the determined weight, thereby obtaining weighted infrared pixel values.

[0136] 7. The pixel interpolation circuitry of claim 6, further configured to: generate the first interpolated infrared pixel value based on the weighted infrared pixel values.

[0137] (8) The pixel interpolation circuitry of (6) or (7), further configured to: generate the second interpolated infrared pixel value based on the weighted infrared pixel values.

[0138] (9) The pixel interpolation circuitry of anyone of (1) to (8), wherein the first color corresponds to pixels with a predetermined first resolution on the image sensor.

[0139] (10) The pixel interpolation circuitry of anyone of (1) to (11), wherein the second color corresponds to pixels with a predetermined second resolution on the image sensor, wherein the first resolution is lower than the second resolution.

[0140] (11) An image sensor comprising the pixel interpolation circuitry of anyone of (1) to (10)

[0141] (12) A pixel interpolation method comprising: obtaining, according to a predetermined pixel pattern, infrared pixel values for a plurality of first pixels configured to detect infrared light, the pixel pattern being indicative of a position of the plurality of first pixels on an image sensor; generating, based on the obtained infrared pixel values and in accordance with the pixel pattern, a first interpolated infrared pixel value for a second pixel being, in a normal operation mode, configured to detect light of a first color; and generating, based on the infrared pixel values and the first interpolated infrared pixel value and in accordance with the pixel pattern, a second interpolated infrared pixel value for a third pixel being, in a normal operation mode, configured to detect light of a second color.

[0142] (13) The pixel interpolation method of (12), wherein the generation of the first interpolated infrared pixel value includes: determining which of two pairs of first pixels surrounding the second pixel have a higher pixel value gradient.

[0143] (14) The pixel interpolation method of (13), wherein the generation of the second interpolated infrared pixel value includes: determining which of a pair of first pixels and a pair of interpolated second pixels have a higher pixel value gradient, wherein the pair of first pixels and the pair of interpolated pixels surround the third pixel.

[0144] (15) The pixel interpolation method of (13) or (14), further comprising: obtaining second color pixel values indicative of the light of the second color; and generating the second interpolated infrared data further based on the second color pixel values.

[0145] (16) The pixel interpolation method of (15), wherein the generation of the second interpolated infrared data further includes: determining a diagonal gradient for the second color pixels values.

[0146] (17) The pixel interpolation method of anyone of (12) to (15), further comprising: obtaining second color pixel values indicative of light of the second color; and determining a weight for the obtained infrared pixel values based on a gradient of the second color pixel values and weighting the obtained infrared pixel values with the determined weight, thereby obtaining weighted infrared pixel values.

[0147] (18) The pixel interpolation method of (17) further comprising: generating the first interpolated infrared pixel value based on the weighted infrared pixel values.

[0148] (19) The pixel interpolation method of (17) or (18), further comprising: generating the second interpolated infrared pixel value based on the weighted infrared pixel values. (20) The pixel interpolation method of anyone of (12) to (19), wherein the first color corresponds to pixels with a predetermined first resolution on the image sensor.

[0149] (21) The pixel interpolation method of anyone of (12) to (20), wherein the second color corresponds to pixels with a predetermined second resolution on the image sensor, wherein the first resolution is lower than the second resolution.

[0150] (22) A computer program comprising program code causing a computer to perform the method according to anyone of (12) to (21), when being carried out on a computer.

[0151] (23) A non-transitory computer-readable recording medium that stores therein a computer program product, which, when executed by a processor, causes the method according to anyone of (12) to (21) to be performed.

Claims

CLAIMS1. Pixel interpolation circuitry configured to: obtain, according to a predetermined pixel pattern, infrared pixel values for a plurality of first pixels configured to detect infrared light, the pixel pattern being indicative of a position of the plurality of first pixels on an image sensor; generate, based on the obtained infrared pixel values and in accordance with the pixel pattern, a first interpolated infrared pixel value for a second pixel being, in a normal operation mode, configured to detect light of a first color; and generate, based on the infrared pixel values and the first interpolated infrared pixel value and in accordance with the pixel pattern, a second interpolated infrared pixel value for a third pixel being, in a normal operation mode, configured to detect light of a second color.

2. The pixel interpolation circuitry of claim 1, wherein the generation of the first interpolated infrared pixel value includes: determining which of two pairs of first pixels surrounding the second pixel have a higher pixel value gradient.

3. The pixel interpolation circuitry of claim 2, wherein the generation of the second interpolated infrared pixel value includes: determining which of a pair of first pixels and a pair of interpolated second pixels have a higher pixel value gradient, wherein the pair of first pixels and the pair of interpolated pixels surround the third pixel.

4. The pixel interpolation circuitry of claim 2, further configured to: obtain second color pixel values indicative of the light of the second color; and generate the second interpolated infrared data further based on the second color pixel values.

5. The pixel interpolation circuitry of claim 4, wherein the generation of the second interpolated infrared data further includes: determining a diagonal gradient for the second color pixels values.

6. The pixel interpolation circuitry of claim 1, further configured to: obtain second color pixel values indicative of light of the second color; and determine a weight for the obtained infrared pixel values based on a gradient of the second color pixel values and weighting the obtained infrared pixel values with the determined weight, thereby obtaining weighted infrared pixel values.

7. The pixel interpolation circuitry of claim 6, further configured to: generate the first interpolated infrared pixel value based on the weighted infrared pixel values.

8. The pixel interpolation circuitry of claim 6, further configured to: generate the second interpolated infrared pixel value based on the weighted infrared pixel values.

9. The pixel interpolation circuitry of claim 1, wherein the first color corresponds to pixels with a predetermined first resolution on the image sensor.

10. The pixel interpolation circuitry of claim 9, wherein the second color corresponds to pixels with a predetermined second resolution on the image sensor, wherein the first resolution is lower than the second resolution.

11. A pixel interpolation method comprising: obtaining, according to a predetermined pixel pattern, infrared pixel values for a plurality of first pixels configured to detect infrared light, the pixel pattern being indicative of a position of the plurality of first pixels on an image sensor; generating, based on the obtained infrared pixel values and in accordance with the pixel pattern, a first interpolated infrared pixel value for a second pixel being, in a normal operation mode, configured to detect light of a first color; and generating, based on the infrared pixel values and the first interpolated infrared pixel value and in accordance with the pixel pattern, a second interpolated infrared pixel value for a third pixel being, in a normal operation mode, configured to detect light of a second color.

12. The pixel interpolation method of claim 11, wherein the generation of the first interpolated infrared pixel value includes: determining which of two pairs of first pixels surrounding the second pixel have a higher pixel value gradient.

13. The pixel interpolation method of claim 12, wherein the generation of the second interpolated infrared pixel value includes: determining which of a pair of first pixels and a pair of interpolated second pixels have a higher pixel value gradient, wherein the pair of first pixels and the pair of interpolated pixels surround the third pixel.

14. The pixel interpolation method of claim 12, further comprising:obtaining second color pixel values indicative of the light of the second color; and generating the second interpolated infrared data further based on the second color pixel values.

15. The pixel interpolation method of claim 14, wherein the generation of the second interpolated infrared data further includes: determining a diagonal gradient for the second color pixels values.

16. The pixel interpolation method of claim 11, further comprising: obtaining second color pixel values indicative of light of the second color; and determining a weight for the obtained infrared pixel values based on a gradient of the second color pixel values and weighting the obtained infrared pixel values with the determined weight, thereby obtaining weighted infrared pixel values.

17. The pixel interpolation method of claim 16, further comprising: generating the first interpolated infrared pixel value based on the weighted infrared pixel values.

18. The pixel interpolation method of claim 16, further comprising: generating the second interpolated infrared pixel value based on the weighted infrared pixel values.

19. The pixel interpolation method of claim 11, wherein the first color corresponds to pixels with a predetermined first resolution on the image sensor.

20. The pixel interpolation method of claim 19, wherein the second color corresponds to pixels with a predetermined second resolution on the image sensor, wherein the first resolution is lower than the second resolution.

Images