Methods for generating infrared images

Abstract

The present disclosure provides a method for generating an infrared image. The method comprises the step of processing pixel values ​​provided at the output of an image sensor, the image sensor comprising a two-dimensional pixel array, the two-dimensional pixel array comprising a mixture of IR pixels and color pixels. The processing step comprises: - for each color pixel, determining (S3) a local color correlation ratio, which represents the infrared light receiving portion in the color pixel; and estimating (S4) an infrared portion value from the color pixel value based on the local color correlation ratio; and - generating (S5) an infrared image based on the pixel value from the IR pixel and the estimated infrared portion value from the color pixel.

Description

Technical Field

[0001] The present disclosure relates to the field of imaging systems for generating infrared (IR) images. Such imaging systems may be used, for example, in vehicles to provide surveillance and driver condition monitoring. Background Art

[0002] Industrial applications of various imaging devices are broadening every year as the capabilities of image processing algorithms and machine learning expand to solve complex tasks related to inspection, control, human-machine interaction, etc. In the automotive industry, recent safety standards require camera systems to be integrated into vehicles to provide surveillance and driver condition monitoring in both day and night conditions.

[0003] Most cameras or imaging systems have cut-off filters. For example, a monochrome IR image sensor is configured to completely block the visible spectral range, while an RGB image sensor is configured to completely block the infrared (IR) spectral range.

[0004] RGB-IR sensors offer significant advantages because they allow both day and night images to be captured with the same sensor. RGB-IR image sensors operate in two ranges: the visible spectrum range and the IR spectrum range. By typically allocating 25% of its pixel array pattern to infrared (IR) and 75% to RGB, an RGB-IR sensor can capture both RGB and IR images simultaneously. Figure 1 An example showing a small portion of a pixel array comprising 5×5 pixels, extracted from an RGB-IR sensor pixel array that may include several thousand pixels. The configuration of IR and RGB pixels is a pixel array pattern of 4×4 pixels, which is based on repeating in rows and columns to form the entire pixel array. The RGB-IR image sensor does not have any dedicated filters to improve the signal quality: it measures everything and extracts the image IR and RGB. This causes some optical problems because the signals in both the IR domain and the RGB domain are contaminated. All pixels in the pixel array of the RGB-IR sensor are able to receive IR signals. This means that not only the IR pixels receive IR signals, but also the RGB pixels receive IR signals. In addition, the IR pixels also receive a small amount of visible light signals. Figure 2 Represents the filter transmittance per wavelength for each type of pixel from a known RGB-IR image sensor. It includes four curves representing the filter transmittance characteristics associated with the filters for red, green, blue, and IR pixels, respectively. These curves can be provided by the sensor manufacturer.

[0005] Consider the curve corresponding to the red pixel. It defines a major region in the red spectral range and a smaller region in the IR spectral range under the curve. This is the same for each of the blue pixel curve and the green pixel curve. The curve for the IR pixel defines a major region in the IR spectral range and a very small region in the RGB spectral range at the bottom of the blue, green, and red pixel regions. Figure 2 As shown, each pixel of the RGB-IR image sensor measures a mixture of the IR signal and the RGB signal. This means that the intensity value at the output of each pixel includes a portion from the received IR signal and a portion from the received RGB signal.

[0006] In the automotive industry, infrared signals play a key role in automated image processing applications such as surveillance and driver monitoring. Typically, these applications require pure infrared images, and they cannot work on raw RGB-IR input. Therefore, it may be necessary to extract the infrared signal before performing further processing.

[0007] A known solution to extract infrared image data from the output data of an RGB-IR sensor is to retain only the IR pixels and discard the red, green and blue pixels. This approach is the easiest and most direct approach. However, this results in a loss of three quarters of the original resolution of the image sensor in the infrared image. This is a significant disadvantage for driver monitoring applications that require high-resolution images, such as for pupil tracking.

[0008] Another known solution for infrared image data extraction involves various image interpolations, such as nearest neighbor interpolation, linear interpolation or spline-based interpolation, to reach the original sensor resolution from one quarter. This solution solves the problem of generating infrared image resolution. However, it does not provide additional image detail levels because it uses the same one-quarter resolution input as the previous solution.

[0009] Therefore, there is a need to improve the situation. More precisely, there is a need to separate the two images, RGB and IR, from the data at the output of the RGB-IR sensor without losing quality, especially in infrared images. Summary of the invention

[0010] The present disclosure relates to a method of generating an infrared image, the method comprising the steps of processing a plurality of pixel values ​​provided at an output of an image sensor, the image sensor comprising a two-dimensional pixel array, the pixel array comprising a mixture of a plurality of IR pixels and a plurality of color pixels,

[0011] The processing steps include:

[0012] -For each color pixel,

[0013] determining a local color correlation ratio, the local color correlation ratio representing an infrared light receiving portion in the color pixel; and

[0014] estimating infrared partial values ​​according to color pixel values ​​based on the local color correlation ratio; and

[0015] - generating an infrared image based on a plurality of pixel values ​​from the plurality of IR pixels and an estimated plurality of infrared portion values ​​from the plurality of color pixels.

[0016] The present method is able to extract an infrared image from a multispectral image captured by an image sensor comprising a mixture of color pixels and IR pixels, such as an RGB-IR sensor. The extraction uses all pixels, not only IR pixels but also color pixels, which allows to preserve the original level of detail in the infrared image. The resolution of the infrared image is the same as the resolution of the original image at the output of the image sensor.

[0017] In an embodiment, the step of determining the local color correlation ratio for each color pixel may include: obtaining the local color correlation ratio according to a plurality of pixel values ​​of at least one IR pixel located adjacent to the color pixel in the pixel array.

[0018] In an embodiment, the step of determining the local color correlation ratio for each color pixel comprises obtaining the local color correlation ratio according to pixel values ​​of one or more color pixels having the same color as the color pixel and located adjacent to the at least one IR pixel in the pixel array.

[0019] Consider a target color pixel in a pixel array. For the target color pixel, a local color correlation ratio representing the infrared reception portion in the color pixel is determined based on the neighbors of the target color pixel, such as based on IR pixels adjacent to the target color pixel and "double-adjacent" color pixels having the same color (i.e., color pixels adjacent to adjacent IR pixels). Multiple IR pixels adjacent to the target color pixel can be used to estimate local infrared reception, while multiple "double-adjacent" color pixels adjacent to each adjacent IR pixel can be used to estimate local mixed color and infrared reception. The local color correlation ratio can be estimated based on the ratio between the local infrared reception and the local mixed color and infrared reception.

[0020] In an embodiment, the step of determining the local color correlation ratio for each color pixel includes:

[0021] - respectively determining a plurality of color correlation ratios relating to a plurality of IR pixels located adjacent to the color pixel in said pixel array;

[0022] - averaging the plurality of color correlation ratios.

[0023] Averaging the color correlation ratios determined for a plurality of IR pixels located adjacent to the target color pixel (e.g., all IR pixels adjacent to the target color pixel) allows refining the value of the local color correlation ratio, particularly in the case where there is an edge in the middle of the color pixel (e.g., the edge of an object sensed by an image sensor). In this case, the color correlation ratios determined for several adjacent pixels may be very different. In order to preserve the edge, the color correlation ratios within the adjacent region of the color pixel are averaged.

[0024] In an embodiment, the step of estimating the infrared portion value for each color pixel may comprise applying the determined local color correlation ratio to the pixel value of the color pixel.

[0025] In an embodiment, the method may further include a first pre-processing step before the processing step, wherein the first pre-processing step includes adjusting infrared transmittances of the plurality of color pixels in the pixel array to a target level.

[0026] In an embodiment, the step of adjusting the infrared transmittance of the plurality of color pixels to a target level comprises applying a color-dependent normalization factor to the plurality of color pixel values.

[0027] The first pre-processing step allows to make the IR transmittance or sensitivity of all pixels uniform so that the color pixels and the IR pixels have the same IR transmittance target level. This pre-processing step is mainly technically related to the sensor production technology. The individual pixels can have different filters, such as red, green, blue and IR filters in the case of an RGB-IR sensor, which amplify the received signal differently. The individual pixels have different sizes of intensity amplification in the IR spectrum. If the light source is sunlight, the amplitudes sensed by the red, green, blue and IR pixels in the IR spectrum may be different. The first pre-processing step allows to normalize the intensity levels with respect to the IR spectrum so that the power at each pixel is balanced. The first pre-processing step allows to equalize the imbalance between color and IR pixels in terms of IR transmittance or sensitivity.

[0028] In the case of an RGB-IR sensor, a red normalization factor, a green normalization factor, and a blue normalization factor may be predetermined and applied to a subset of red pixels, a subset of green pixels, and a subset of blue pixels, respectively. For IR pixels, advantageously, no normalization factor is applied, since the pixel values ​​from the color pixels are normalized to the level of the IR reception level of the IR pixel.

[0029] In an embodiment, the color-dependent normalization factor is determined by:

[0030] - irradiating a target with an infrared-only light signal, which reflects monotonically across the spectrum from the visible to the infrared;

[0031] - receiving, by the image sensor, an infrared light signal reflected by the target;

[0032] - average all IR pixel values ​​to determine the IR mean;

[0033] - For each color channel, average all color pixel values ​​to determine the color mean;

[0034] - Calculating each color-dependent normalization factor by calculating the ratio between the IR mean and the color mean.

[0035] Alternatively, the color-dependent normalization factor may be calculated according to the spectral transmittance characteristics of the image sensor.

[0036] In an embodiment, the method may further include: a second pre-processing step after the first pre-processing step, the second pre-processing step being performed for each IR pixel and comprising:

[0037] - determining a plurality of color component values ​​representing color light receiving components in the IR pixel in all color channels based on the pixel value of the IR pixel and a predetermined color-dependent IR pollution gain coefficient;

[0038] - filtering the pixel value of the IR pixel by removing the determined plurality of color portion values.

[0039] The method may include a second pre-processing step that enables filtering or cleaning of the output from the IR pixels that may be contaminated by visible light. In practice, light having all spectral ranges may not be properly filtered. For example, considering a light source that emits light in the IR spectrum as well as in the visible spectrum, the IR pixels may not completely cut off the visible spectrum and still receive some signal from the visible spectrum. As a result, each IR pixel may sense a small amount of visible light, for example about 15% to 20% of the intensity received in the visible spectrum.

[0040] The second preprocessing step aims to compensate for this phenomenon by estimating and removing the contamination of the visible spectrum from the IR pixel response. For this purpose, it is possible to evaluate Figure 2 The area under the IR filter curve in the visible spectrum shown in is used to determine the portion of this area in the response of the IR pixel. The pixel value at the output of each IR pixel is the sum of the value in the IR spectrum and the value in the visible spectrum. The purpose of the above preprocessing step is to eliminate the values ​​in the RGB spectrum.

[0041] In an embodiment, the step of determining the plurality of color component values ​​for each IR pixel comprises:

[0042] -For each color channel,

[0043] determining a local color pixel value from at least one color pixel of the color channel located adjacent to the IR pixel,

[0044] filtering the local color pixel values ​​by removing infrared light receiving portion values ​​estimated to be equal to the pixel value of the IR pixel;

[0045] The color portion value of the IR pixel is determined by applying a predetermined color-dependent IR contamination gain coefficient to the filtered local color pixel value.

[0046] Advantageously, the step of determining said local colour pixel value for each colour channel comprises averaging the colour pixel values ​​of a plurality of colour pixels located adjacent to the IR pixel.

[0047] In an embodiment, the method may further comprise the step of weighting the plurality of color portion values ​​of each IR pixel with a respective color related intensity weight in dependence on light conditions detected in an environment of the image sensor.

[0048] The intensity weights may depend on the light conditions in the environment surrounding the image sensor.The color component values ​​for each color channel may be weighted with a corresponding color-dependent intensity weight.

[0049] In an embodiment, the color-dependent IR pollution gain coefficient for each color channel may be determined by the following steps:

[0050] - irradiating a target reflecting monotonically in the spectrum from the visible spectrum to the infrared spectrum with a monochromatic light signal of that color;

[0051] - receiving, by the image sensor, a monochromatic light signal reflected by the target;

[0052] - averaging all IR pixel values ​​to determine a first color mean;

[0053] - averaging all color pixel values ​​associated with the color to determine a second color mean;

[0054] - calculating the color dependent IR pollution gain coefficient by calculating the ratio between the first color mean and the second color mean.

[0055] Alternatively, the color-dependent IR pollution gain coefficient is calculated according to the spectral transmittance characteristics of the image sensor.

[0056] The present disclosure also relates to an imaging system, the imaging system comprising:

[0057] an image sensor comprising a two-dimensional pixel array, the pixel array comprising a plurality of color pixels and a plurality of IR pixels;

[0058] An image processor for processing a plurality of pixel values ​​provided at an output of the plurality of pixels, the image processor being configured to perform the steps according to the aforementioned method.

[0059] Each color pixel may operate in one color channel, such as red or green or blue. The pixel array may include multiple color pixels for multiple color channels, such as red pixels, green pixels, and blue pixels.

[0060] A computer-readable data medium stores a computer program, which, when executed by a computer, causes the computer to execute the steps of the aforementioned method. BRIEF DESCRIPTION OF THE DRAWINGS

[0061] Other features, objects and advantages of the present disclosure will become more apparent from a reading of the detailed description of non-limiting embodiments made with reference to the accompanying drawings.

[0062] Figure 1 A portion of a pixel array from an RGB-IR image sensor is shown according to an example.

[0063] Figure 2 According to the example Figure 1 Characteristic curve of spectral filter transmittance of RGB-IR image sensor.

[0064] Figure 3 express Figure 1 A 3×3 pixel subarray.

[0065] Figure 4 Shows Figure 1 A portion of a pixel array with three color pixels labeled.

[0066] FIG. 5A to FIG. 5F is a flow chart relating to a method of generating an infrared image by processing pixel values ​​provided at an output of an RGB-IR image sensor, according to an embodiment.

[0067] Figure 6 A functional block diagram of an imaging system including an RGB-IR sensor according to an embodiment is shown. DETAILED DESCRIPTION

[0068] The present disclosure relates to an imaging system 300, the imaging system comprising:

[0069] - an RGB-IR image sensor 100 comprising a two-dimensional pixel array 110 having color pixels and IR pixels;

[0070] - An image processor 200 for processing pixel values ​​provided at an output of said pixels.

[0071] In the present disclosure, the term "pixel" refers to a light sensing element, which may include a pixel photosensor and a tiny filter disposed on the pixel photosensor to filter light by wavelength range. A color pixel includes a corresponding color filter disposed on the pixel photosensor to capture color information, and an IR pixel includes a corresponding IR filter disposed on the pixel photosensor to capture infrared information. The color filter and the IR filter form a mosaic or two-dimensional filter array.

[0072] By definition, in the present disclosure, a first pixel in pixel array 110 is positioned “adjacent” to a second pixel if the first pixel and the second pixel have a common edge or corner.

[0073] exist Figure 1 An example of an RGB-IR color filter array (CFA) is shown in FIG. Figure 1 In the structure of the RGB-IR color filter array 110 in FIG. 1 , each 4×4 cell or pixel unit has two red (R), four green (G), two blue (B), and four infrared (IR) pixels. Each 4×4 cell unit is repeated in two dimensions of the array to form a two-dimensional array of pixels 110 of the RGB-IR sensor 100.

[0074] Different from Figure 1 Any other configuration of filter arrays than shown is possible.

[0075] Figure 2 Graphs representing filter transmittances at respective wavelengths of color filters, namely, a red filter RF, a green filter GF, and a blue filter BF, and an infrared filter IRF according to an illustrative example. Figure 2 The color-dependent filter transmittance curve is shown as:

[0076] - color pixels, here red R, green G and blue B pixels, which are sensitive in the infrared wavelength range,

[0077] -IR pixels, which are weakly sensitive in the visible wavelength range.

[0078] The image processor 200 is configured to process the pixel value I at the output of the color and IR pixel (i, j) i,j To generate images, such as color images and IR images, “i” and “j” represent indexes of columns and rows in the pixel array 110, respectively.

[0079] In this disclosure, reference is made to FIG. 5A to FIG. 5F The image processor 200 is configured to perform a method of generating an infrared (IR) image, the method comprising processing the color and pixel values ​​I of the IR pixels (i, j) provided at the output of the RGB-IR sensor 100i,j The processing steps include steps S3 to S5 described below.

[0080] The present method allows to generate an infrared image from all pixels (i, j), not only the IR pixels (i, j) (where (i, j) ∈ S IR ), also includes color pixel (i, j) (where (i, j)∈S R ∪S G ∪S B ), where S IR ,S R ,S G ,S B Represents the set of IR, red, green, and blue pixels respectively.

[0081] In the processing step, for each color pixel (i, j) in the pixel array 110 (where (i, j)∈S R ∪S G ∪S B ), the image processor 200 determines in step S3 the local color correlation ratio representing the infrared light receiving portion in the color pixel (i, j), and the local color correlation ratio for the red pixel (i, j) is recorded as For the green pixel (i, j), it is recorded as For the blue pixel (i, j), it is recorded as And in step S4, based on the local color correlation ratio or The infrared component value is estimated based on the pixel value of the color pixel (i, j), or the infrared component value of the pixel value of the color pixel (i, j). In an embodiment, in step S4, the local color correlation ratio or can be applied to the pixel value of the color pixel (i, j) to estimate the infrared value of the color pixel (i, j). For example, the color pixel value can be multiplied by the local color correlation ratio or The value at the output of step S4 is recorded as It corresponds to the infrared partial values ​​for the color pixels and the pixel values ​​for the IR pixels, optionally pre-processed as described below. Then, as explained in more detail later, in step S5, the image processor 200 generates an infrared image based on the pixel values ​​from the IR pixels and based on the estimated infrared partial values ​​of the color pixels.

[0082] In an embodiment, in step S3, for each color pixel (i, j) in the pixel array 110 (where (i, j)∈S R ,、(i,j)∈S G or (i,j)∈SB ), local color correlation ratio or The pixel value of at least one IR pixel (denoted as (x, y)) adjacent to the color pixel (i, j) in the pixel array 110 and the pixel value of one or more color pixels having the same color as the color pixel and located adjacent to the at least one adjacent IR pixel (x, y) in the pixel array 110 can be obtained. For example, Figure 5B As shown, in step S3, for each color pixel (i, j), the image processor 200 may:

[0083] - in step S30, a plurality of color correlation ratios related to a plurality of IR pixels (x, y) located adjacent to the color pixel (i, j) in the pixel array 110 are respectively determined, wherein the color correlation ratio related to each IR pixel is obtained based on the pixel value of the IR pixel (x, y) and the pixel values ​​of one or more color pixels having the same color as the color pixel (i, j) and located adjacent to the IR pixel (x, y) in the pixel array 110;

[0084] - In step S31, an average value of a plurality of determined color correlation ratios is calculated to determine a local color correlation ratio or

[0085] Preferably, in order to determine the color correlation ratio for a given color pixel (i, j), step S30 may be performed for each IR pixel (x, y) located adjacent to the color pixel (i, j) and / or for each color pixel located adjacent to each adjacent IR pixel (x, y) and having the same color as the color pixel (i, j). Therefore, in an embodiment, the color correlation ratio of the color pixel (i, j) is or The color correlation ratio of each adjacent IR pixel (x, y) may be an average value of the color correlation ratios of all IR pixels (x, y) located adjacent to the color pixel (i, j). In addition, the color correlation ratio of each adjacent IR pixel (x, y) may be estimated based on the pixel value from the adjacent IR pixel (x, y) and the pixel values ​​from all color pixels located adjacent to the adjacent IR pixel (x, y) and having the same color as the color pixel (i, j).

[0086] Steps S3 to S5 of the steps of the process will be described in more detail later based on an illustrative example.

[0087] In some embodiments, the method may comprise one or more pre-processing steps performed by the image processor 200 prior to the step of processing, in other words prior to steps S3, S4 and S5. The pre-processing steps may be optional.

[0088] In an embodiment, the first pre-processing step S1 has the function of adjusting the infrared transmittance or sensitivity of the color pixels (i, j) in the pixel array 110 to the same target level. The target level can be defined based on the IR transmittance or sensitivity of the IR pixels. In the pre-processing step S1, the amplitude or level of infrared light received by the color pixels can be balanced with the amplitude or level of infrared light received by the IR pixels.

[0089] For example, for each color channel R, G, B, a color-dependent normalization factor (denoted as ngainR for red, ngainG for green, and ngainB for blue) is applied to the pixel value at the output of the color pixel (i, j). Therefore, for each color channel R, G, B, the pixel value at the output of the color pixel (i, j) is multiplied by the color-dependent normalization factor ngainR, ngainG, or ngainB, and the color-dependent normalization factor is configured to adjust the IR transmittance or sensitivity of the color pixel (i, j) to a target level of IR sensitivity of the IR pixel. As a result, the pixel value provided by the image sensor 200 at the output of the first pre-processing step S1 can be modified or normalized for the color pixel. For each color or IR pixel (i, j), the pixel value of the pixel at the output of the first pre-processing step S1 (denoted as ) can be expressed as follows:

[0090]

[0091] in,

[0092] -I i,j is the raw pixel value at the output of the pixel, here unprocessed;

[0093] - ngainR, ngainG and ngainB are color-dependent normalization factors for red, green and blue respectively.

[0094] The color-dependent normalization factors ngainR, ngainG and ngainB may be determined in a preliminary step PS1.

[0095] exist Figure 5CIn a first variant of the preliminary step PS1 shown, a reflective target, such as a white target, which reflects monotonically in the spectrum from the visible spectrum to the infrared spectrum can be illuminated with only infrared light or a signal, and the RGB-IR sensor can receive the IR light reflected by the reflective target. Then, it is determined how the color pixels react to this IR light. The purpose of the preliminary step PS1 is to equalize the intensity of the IR light detected by all color and IR pixels. For this purpose, for each color channel R, G and B, the color-related normalization factors ngainR, ngainG and ngainB are determined from the values ​​at the output of the color and IR pixels, as explained in more detail below.

[0096] Preliminary step PS1 may be performed, for example, by the manufacturer of RGB-IR sensor 100 on RGB-IR sensor 100 or on a reference RGB-IR sensor having the same characteristics as RGB-IR sensor 100 , in particular having the same curve in filter transmittance at various wavelengths.

[0097] In a first variant, the preliminary step PS1 may include a step PS10 of irradiating a target that reflects monotonically from the visible spectrum to the infrared spectrum with an infrared-only light signal.

[0098] In step PS11 , the RGB-IR sensor 100 may receive infrared signals reflected by the target and then output pixel values ​​from color pixels and IR pixels.

[0099] Then, in step PS12, the image processor 200 may:

[0100] - calculating the average of the pixel values ​​at the output of all IR pixels to calculate the IR mean value E[IR];

[0101] - For each color channel, the average of the pixel values ​​at the output of all color pixels is calculated to calculate a color mean, which is denoted E[R] for the red channel, E[G] for the green channel, and E[B] for the blue channel.

[0102] In the next step PS13, the image processor 200 may calculate the ratio between the IR mean E[IR] and the color mean (i.e., E[R] for the red channel, E[G] for the green channel, and E[B] for the blue channel) for each color channel to determine the color-dependent normalization factor. The color-dependent normalization factor may be expressed as follows:

[0103]

[0104]

[0105]

[0106] in,

[0107] - ngainR, ngainG and ngainB are normalization factors for red, green and blue.

[0108] - is the IR mean value obtained from all IR pixels when the RGB-IR sensor receives only infrared light signals reflected by the reflective target;

[0109] - is the red mean value obtained from all red pixels when the RGB-IR sensor receives only infrared light signals reflected by the reflective target;

[0110] - is the green mean value obtained from all green pixels when the RGB-IR sensor receives only infrared light signals reflected by the reflective target;

[0111] - is the blue mean value obtained from all blue pixels when the RGB-IR sensor receives only infrared light signals reflected by the reflective target.

[0112] In a second variant, the color-related normalization factors ngainR, ngainG, and ngainB can be calculated based on the spectral transmittance characteristics of the RGB-IR sensor. For example, for each color channel R, G, and B, the peak value of the IR curve of the filter transmittance in the infrared wavelength range is divided by the peak value of the color curve of the filter transmittance in the infrared wavelength range to obtain the color-related normalization factors ngainR, ngainG, and ngainB.

[0113] In an embodiment, the method may include a second pre-processing step S2 performed by the image processor 200, which has the function of removing contamination in the visible spectrum from the IR pixel response. In other words, the purpose of step S2 is to filter or cleanse the pixel value of the IR pixel (i, j) by removing part of the value due to contamination of visible light.

[0114] Advantageously, the second pre-processing step S2 can be performed after the first pre-processing step S1, so that the pixel values ​​provided at the output of the first pre-processing step S1 is used as an input value in the second pre-processing step S2. This means that all pixel values ​​used as input values ​​in the second pre-processing step S2 have the same IR reception target level, in other words, have the same infrared sensitivity.

[0115] consider Figure 3 An IR pixel (i, j) is shown to illustrate the second pre-processing step S2.

[0116] Figure 5D The second pre-processing step S2 shown comprises for each IR pixel (i, j) (where (i, j)∈S IR):

[0117] - A first step S20 of determining the color component values ​​for all color channels R, G, B according to the pixel value of the IR pixel (i, j) and predetermined color-dependent IR pollution gain coefficients cgainR, cgainG and cgainB It represents the fraction of color light received or contaminated in this IR pixel (i, j); and

[0118] - A second step S21, by removing the corresponding color part values ​​of all color channels R, G, B determined in step S20 To filter the IR pixel (i, j) (where (i, j)∈S IR )

[0119] For each IR pixel (i, j) (where (i, j)∈S IR ), determine the color part value The first step S20 may include, for each color channel R, G, B:

[0120] - Step S200, determining a local color pixel value according to one or more color pixels located adjacent to the IR pixel (i, j), denoted as E[r i,j ], and for green it is E[g i,j ], and for blue it is recorded as E[b i,j ];

[0121] - Step S201, by removing the pixel value estimated to be equal to the IR pixel (i, j) The infrared light receiving part value determines the local color pixel value E[r i,j ]、E[g i,j ] and E[b i,j ] to filter;

[0122] - Step S202, by applying predetermined color-dependent IR pollution gain coefficients cgainR, cgainG and cgainB to the filtered local color pixel value, determining the IR pixel (i, j) (where (i, j)∈S IR ) Right now and Cleaned up the IR receiving section.

[0123] In step S200, for each color channel R, G, B, the local color pixel value (ie, E[r i,j ], for green E[g i,j ] and [b i,j]) can be calculated by calculating the average value of the color pixel values ​​from the color pixels located adjacent to the IR pixel (i, j). For example, for each color channel R, G, B, one or more adjacent color pixels may include all color pixels of the color channel located adjacent to the IR pixel (i, j).

[0124] consider Figure 3 The local green pixel value E[g i,j The local red pixel value E[r i,j The local blue pixel value E[b i,j ].

[0125] In step S201, for each color channel R, G, B, the local color pixel value (ie, E[r i,j ], for green E[g i,j ] and [b i,j ]) minus the pixel value of the IR pixel (i, j) to filter or purify it.

[0126] In step S202, for each color channel, a color partial value is extracted or obtained from the filtered local color pixel value generated in step S201. In an embodiment, the filtered local color pixel value is multiplied by the color-dependent IR contamination gain coefficients cgainR, cgainG, and cgainB to obtain the color part value

[0127] represents an IR pixel (i, j) (where (i, j)∈S IR ) in the color light receiving part It can be expressed as follows for all color channels R, G, and B respectively:

[0128] For (i,j)∈S IR

[0129] For (i,j)∈S IR

[0130] For (i,j)∈S IR

[0131] in,

[0132] - represents the red component value, green component value and blue component value for the IR pixel (i, j);

[0133] -E[r i,j ]、E[g i,j ] and [b i,j ] represents the local red pixel value, the local green pixel value and the local blue pixel value for the IR pixel (i, j);

[0134] - represents the pixel value of the IR pixel (i, j) after the first pre-processing step S1;

[0135] -cgainR, cgainG, and cgainB represent the red-related IR pollution gain coefficient, the green-related IR pollution gain coefficient, and the blue-related IR pollution gain coefficient.

[0136] The pixel value of the filtered IR pixel (i, j) In the second step S21, the pixel value of the IR pixel (i, j) is Subtract the corresponding color values ​​of all color channels R, G, and B from

[0137] In a specific embodiment, in the second step S21, the color part value The corresponding color-related intensity weight (denoted as w) between 0 and 1 can be used in the subtraction r 、w g 、w b ) to weight them. These intensity weights w r 、w g 、w b may depend on the light conditions detected in the environment of the RGB-IR sensor 100. As an illustrative and non-limiting example, the following intensity weights may be used in sunlight conditions: r =0.3, w g =0.4, w b = 0.3. The filtered pixel value of the IR pixel (i, j) with the visible light receiving part cleared It can be expressed as follows:

[0138] For (i,j)∈S IR

[0139] in,

[0140] - represents the pixel value of the IR pixel (i, j) without contamination in the visible spectrum;

[0141] - represents the pixel value of the IR pixel (i, j) after the first pre-processing step S1;

[0142] - Indicates the red part value, the green part value and the blue part value;

[0143] -w r 、w g 、w b represents the color-related intensity weight, where 0≤w r ,w g ,w b ≤1.

[0144] The color-dependent IR pollution gain coefficients cgainR, cgainG and cgainB may be determined in a preliminary step PS2.

[0145] In a first variant of the preliminary step PS2, for each color channel R, G, B, a target that reflects monotonically in a spectrum from the visible spectrum to the infrared spectrum (e.g., a white target) can be illuminated with a monochromatic color light signal within the wavelength range of the color channel and the RGB-IR sensor can receive the color light reflected by the reflective target. Then, it is determined how the color pixels and IR pixels of the color channel react to the color light. The goal of the preliminary step PS2 is to determine, for each color channel, the ratio between the color reception of the IR pixels and the color reception of the color pixels. For this purpose, for each color channel R, G, and B, the ratio is determined based on the values ​​at the output of the color and IR pixels when the RGB-IR sensor receives the reflected color light, as explained in more detail below.

[0146] Step PS2 may be performed, for example, by the manufacturer of the RGB-IR sensor 100 on the RGB-IR sensor 100 or on a reference RGB-IR sensor having the same characteristics as the RGB-IR sensor 100 , in particular having the same curve in filter transmittance at each wavelength.

[0147] In a first variant, the preliminary step PS2 may include a step PS20 in which, for each color channel R, G, B, a target reflecting monotonically from the visible spectrum to the infrared spectrum is irradiated with a monochromatic color light signal within the wavelength range of the color channel.

[0148] In step PS21, the RGB-IR sensor 100 may receive the color light signal reflected by the reflective target and then output pixel values ​​from the color pixels and the IR pixels. Then, in step PS22, the image processor 200 may:

[0149] - calculating the average of the pixel values ​​at the output of all IR pixels to calculate the IR mean value E[IR];

[0150] - Calculate the average of the pixel values ​​at the output ends of all color pixels having the same color as the color light signal to calculate the color mean, which is recorded as E[R] for the red channel, E[G] for the green channel, and E[B] for the blue channel.

[0151] In the next step PS23, the image processor 200 may calculate the ratio between the IR mean and the color mean (i.e., E[R] for the red channel, or E[G] for the green channel, or E[B] for the blue channel) to determine the color-dependent IR pollution gain coefficient cgainR or cgainG or cgainB.

[0152] The color-dependent IR pollution gain coefficient can be expressed as follows:

[0153]

[0154]

[0155]

[0156] in,

[0157] - cgainR, cgainG, cgainB are the red related IR pollution gain coefficient, the green related IR pollution gain coefficient and the blue related IR pollution gain coefficient;

[0158] - E[IR] is the IR average value obtained from all IR pixels when the RGB-IR sensor receives monochromatic light of a given color R, G or B reflected by a reflective target;

[0159] - E[R] is the average red value obtained from all red pixels when the RGB-IR sensor receives monochromatic red light reflected by the reflective target;

[0160] - E[G] is the green mean value obtained from all green pixels when the RGB-IR sensor receives monochromatic green light reflected by the reflective target;

[0161] - E[B] is the average blue value obtained from all blue pixels when the RGB-IR sensor receives monochromatic blue light reflected by the reflective target.

[0162] In the second variant, the color-dependent IR pollution gain coefficients cgainR, cgainG, and cgainB can be calculated based on the spectral transmittance characteristics of the RGB-IR sensor. For example, for each color channel R, G, B, the peak value or maximum value of the IR curve of the filter transmittance within the color wavelength range is divided by the peak value of the color curve R, G, or B of the filter transmittance within the color wavelength range to obtain the color-dependent IR pollution gain coefficients cgainR, cgainG, and cgainB.

[0163] Now it will be based on Figure 4 The illustrative example shown describes steps S3 to S5 in some detail.

[0164] In this embodiment, two preprocessing steps S1 and S2 are performed before steps S3 to S5. Therefore, the pixel value used as input in step S3 is the preprocessed pixel value at the output of steps S1 and S2, which is recorded as This means:

[0165] - Color pixel (i, j) (where (i, j)∈S R , or (i,j)∈S G , or (i,j)∈S B ) has been normalized by a color-dependent normalization factor ngainR, ngainG, or gnainB (depending on the color of the pixel), and

[0166] - the IR pixel (i, j) (where (i, j)∈S IR ) It has been filtered to remove the visible light receiving part.

[0167] Figure 4 represents a 5x5 pixel subarray from pixel array 110. In this subarray, columns and rows may be numbered from index 0 to index 4. The process of estimating infrared portion values ​​for color pixels (2,1), (1,2), and (2,2) will now be described.

[0168] Consider the green pixel (2, 1). In step S3, the image processor 200 determines a local green correlation ratio representing the infrared light receiving portion in the green pixel (2, 1). To this end, in step S30, the image processor 200 determines two green correlation ratios involving two IR pixels (1, 1) and (3, 1) located adjacent to the green pixel (2, 1). The green correlation ratio involving each adjacent IR pixel (1, 1) (or (3, 1)) is obtained based on the pixel value of the adjacent IR pixel (1, 1) (or (3, 1)) and based on the pixel values ​​of four "double-adjacent" green pixels (1, 0), (0, 1), (2, 1) and (1, 2) (or (3, 0), (2, 1), (3, 2) and (4, 1)) located adjacent to the adjacent IR pixel (1, 1) (or (3, 1)). Each green correlation ratio is a ratio between the filtered pixel value of the adjacent IR pixel and the normalized pixel value of the "double-adjacent" color pixel in response to which the mixed green and IR signals are included. The green correlation ratio involving each adjacent IR pixel (x, y) (here, each IR pixel in IR pixels (1, 1) and (3, 1)) can be expressed as follows:

[0169]

[0170] in,

[0171] -ρG x,y is the green correlation ratio for that IR pixel (x, y);

[0172] - is the pre-processed pixel value of the IR pixel (x, y);

[0173] - are the preprocessed pixel values ​​of the four green pixels (x-1, y), (x+1, y), (x, y-1), (x, y+1) located adjacent to the IR pixel (x, y).

[0174] Then, in step S31, the image processor 200 calculates an average value of a plurality of determined green correlation ratios based on the following formula to determine a local green correlation ratio for the green pixel (2, 1):

[0175]

[0176] Then, in step S4, the image processor 200 calculates the local green correlation ratio based on the local green correlation ratio determined above. Estimate the infrared value for the green pixel (2, 1) according to the following formula:

[0177]

[0178] Now, consider the green pixel (1, 2). In step S3, the image processor 200 determines a local green correlation ratio representing the infrared light receiving portion in the green pixel (1, 2). To this end, in step S30, the image processor 200 determines two green correlation ratios respectively related to the two IR pixels (1, 1) and (1, 3) located adjacent to the green pixel (1, 2). The green correlation ratio related to each adjacent IR pixel (1, 1) (or (1, 3)) is obtained based on the pixel value of the adjacent IR pixel (1, 1) (or (1, 3)) and based on the pixel values ​​of the four "double-adjacent" green pixels (0, 1), (1, 0), (2, 1) and (1, 2) (or (0, 3), (1, 2), (2, 3) and (1, 4)) located adjacent to the adjacent IR pixel (1, 1) (or (1, 3)). Each green correlation ratio is the ratio between the filtered pixel value of the neighboring IR pixel and the normalized pixel value of the "double-neighboring" color pixel whose response includes mixed green and IR signals. The green correlation ratio involving each neighboring IR pixel (x, y) can be expressed as follows:

[0179]

[0180] in,

[0181] -ρG x,y is the green correlation ratio for that IR pixel (x, y);

[0182] - is the pre-processed pixel value of the IR pixel (x, y);

[0183] - are the pre-processed pixel values ​​of the four green pixels (x-1,y), (x+1,y), (x,y-1), (x,y+1) located adjacent to the IR pixel (x,y).

[0184] Then, in step S31, the image processor 200 calculates an average value of a plurality of determined green correlation ratios to determine a local green correlation ratio for the green pixel (1, 2) based on the following formula:

[0185]

[0186] Then, in step S4, the image processor 200 calculates the local green correlation ratio based on the local green correlation ratio determined above. Estimate the infrared value of the green pixel (1, 2) according to the following formula

[0187]

[0188] Now, consider the blue pixel (2, 2). In step S3, the image processor 200 determines a local blue correlation ratio representing the infrared light receiving portion in the blue pixel (1, 2). To this end, in step S30, the image processor 200 determines four blue correlation ratios respectively related to the four IR pixels (1, 1), (3, 1), (1, 3) and (3, 3) located adjacent to the blue pixel (2, 2). The blue correlation ratio involving each adjacent IR pixel (1,1) or (3,1) or (1,3) or (3,3) is obtained based on the pixel value of the adjacent IR pixel (1,1) or (3,1) or (1,3) or (3,3) and based on the pixel values ​​of two "double-adjacent" blue pixels (0,0), (2,2); or (2,2), (4,0); or (2,2), (0,4); or (2,2), (4,4) located adjacent to the adjacent IR pixel (1,1) or (3,1) or (1,3) or (3,3). Each blue correlation ratio is the ratio between the filtered pixel value of the adjacent IR pixel and the normalized pixel value of the "double-adjacent" color pixel in response to the mixed blue and IR signals. The blue correlation ratio involving each adjacent IR pixel (here, IR pixels (1,1), (3,1), (1,3), (3,3)) can be expressed as follows:

[0189] For x=y

[0190] For x≠y

[0191] in,

[0192] -ρB x,y is the blue correlation ratio for the IR pixel (x, y);

[0193] - is the pre-processed pixel value of the IR pixel (x, y);

[0194] - are the preprocessed pixel values ​​of the blue pixels (x-1, y-1), (x+1, y+1), (x-1, y+1), (x+1, y-1) located adjacent to the IR pixel (x, y).

[0195] Then, in step S31, the image processor 200 calculates the average value of the plurality of determined blue correlation ratios based on the following formula to determine the local blue correlation ratio for the blue pixel (2, 2):

[0196]

[0197] Then, in step S4, the image processor 200 calculates the local blue correlation ratio based on the above determined local blue correlation ratio. Estimate the infrared value of the blue pixel according to the following formula

[0198]

[0199] The red correlation ratio can be calculated based on a similar formula to the above formula used to calculate the blue correlation ratio. Therefore, the following formula can be used to determine the red correlation ratio involving an IR pixel (x, y) located adjacent to a red pixel (i, j):

[0200] For x=y

[0201] For x≠y

[0202] in,

[0203] -ρR x,y is the red correlation ratio for the adjacent IR pixel (x, y);

[0204] - is the pre-processed pixel value of the neighboring IR pixel (x, y);

[0205] - are the preprocessed pixel values ​​of the red pixels (x-1, y-1), (x+1, y+1), (x-1, y+1), (x+1, y-1) located adjacent to the IR pixel (x, y).

[0206] In addition, the local red correlation ratio for the red pixel (i, j) The calculation is based on the following formula:

[0207]

[0208] Optionally, the color correlation ratio involving the IR pixel may be amplified, or possibly attenuated, for example by taking an exponent using the amplification factor γ as an exponent or power. Thus, the color correlation ratio involving the IR pixel (x, y) may be raised to the power of the amplification factor γ to provide an amplified color correlation ratio, which may be expressed as follows:

[0209] (ρR x,y ) γ ,(ρG x,y ) γ ,(ρB x,y ) γ .

[0210] This magnification factor γ is used to control the infrared image extraction part The brightness intensity. The amplification factor γ can be:

[0211] - greater than 1, to increase image contrast, which results in an infrared image that appears darker or less bright, or

[0212] - Less than 1 to reduce image contrast, which results in brighter-looking infrared images.

[0213] Color pixels located at the border of the array 110 (in other words close to the image border) do not have a complete neighborhood of 5×5 pixels available. In this case, ratios cannot be calculated based on neighboring pixels and double neighboring pixels. Therefore, any color pixel close to the image or pixel array border can inherit its color correlation ratio from the nearest color pixel with the same color for which the calculation according to step S3 is possible. For example, a green pixel close to the image border can inherit its green correlation ratio from the nearest green pixel. The same applies to red and blue pixels near the image border.

[0214] The present method of generating an infrared image from all pixels of an RGB-IR sensor can be implemented by hardware and software. In an embodiment, the method is implemented by a software application running on a processor such as an image processor 200. Therefore, the present disclosure also relates to:

[0215] - a computer program comprising instructions which, when said program is executed by a computer, cause said computer to carry out the steps of the method previously described;

[0216] - a computer-readable data medium comprising instructions which, when said program is executed by a computer, cause said computer to carry out the steps of the method previously described.

[0217] Furthermore, the method can be applied to any image sensor having a first IR pixel and at least a second pixel having sensitivity to signals in a wavelength range different from or separate from the IR wavelength range. In particular, the image sensor can have a mixture of IR pixels and color pixels. Each color pixel can have a color channel, such as red, green or blue, and the pixel array can include a plurality of color pixels with different color channels.

Claims

1. A method for generating an infrared image, the method comprising: processing a plurality of pixel values ​​provided at an output of an image sensor (100), the image sensor comprising a two-dimensional pixel array (110), the pixel array comprising a mixture of a plurality of IR pixels and a plurality of color pixels, The processing includes: For each color pixel, - determining a local color correlation ratio, the local color correlation ratio representing the infrared light receiving portion in the color pixel; and - estimating the infrared part value according to the color pixel value of the color pixel based on the local color correlation ratio; generating an infrared image based on a plurality of pixel values ​​from the plurality of IR pixels for the plurality of IR pixels and an estimated plurality of infrared partial values ​​from the plurality of color pixels for the plurality of color pixels; Before processing the pixel values, performing a first preprocessing, the first preprocessing comprising adjusting infrared transmittances of the plurality of color pixels in the pixel array to a target level; and A second preprocessing is performed after the first preprocessing, the second preprocessing being performed for each IR pixel and comprising: - determining a plurality of color component values ​​representing color light receiving components in the IR pixel in all color channels based on the pixel value of the IR pixel and predetermined color-dependent IR pollution gain coefficients; and - filtering the pixel value of the IR pixel by removing the determined plurality of color portion values.

2. The method according to claim 1, wherein: Determining the local color correlation ratio for each color pixel includes obtaining the local color correlation ratio according to a plurality of pixel values ​​of at least one IR pixel located adjacent to the color pixel in the pixel array.

3. The method according to claim 2, wherein: Determining the local color correlation ratio for each color pixel includes obtaining the local color correlation ratio based on pixel values ​​of one or more color pixels having the same color as the color pixel and located adjacent to the at least one IR pixel in the pixel array (110).

4. The method according to any one of claims 1 to 3, wherein: Determining the local color correlation ratio for each color pixel includes: - respectively determining a plurality of color correlation ratios involving a plurality of IR pixels located adjacent to the color pixel in the pixel array (110); - averaging the plurality of color correlation ratios.

5. The method according to claim 1, wherein: Estimating the infrared portion value for each color pixel includes applying the determined local color correlation ratio to the pixel value of the color pixel.

6. The method according to claim 1, wherein: Adjusting the infrared transmittance of the plurality of color pixels to a target level includes applying a color-dependent normalization factor to the plurality of color pixel values.

7. The method according to claim 6, wherein: The color-dependent normalization factor is determined in the following way: - irradiating a target with an infrared-only light signal, which reflects monotonically across the spectrum from the visible to the infrared; - receiving, by the image sensor, an infrared light signal reflected by the target; - average all IR pixel values ​​to determine the IR mean; - For each color channel, average all color pixel values ​​to determine the color mean; - Calculating each color-dependent normalization factor by calculating the ratio between the IR mean and the color mean.

8. The method according to claim 1, wherein: Determining the plurality of color component values ​​for each IR pixel comprises: -For each color channel, determining a local color pixel value from at least one color pixel of the color channel located adjacent to the IR pixel, filtering the local color pixel values ​​by removing infrared light receiving portion values ​​estimated to be equal to the pixel value of the IR pixel; The color portion value of the IR pixel is determined by applying a predetermined color-dependent IR contamination gain coefficient to the filtered local color pixel value.

9. The method according to claim 8, wherein: Determining a local color pixel value for each color channel includes averaging color pixel values ​​of a plurality of color pixels located adjacent to the IR pixel.

10. The method according to claim 1, further comprising: The plurality of color component values ​​for each IR pixel are weighted with a corresponding color-dependent intensity weight depending on light conditions detected in an environment of the image sensor.

11. The method according to claim 1, further comprising: The color-dependent IR pollution gain coefficients for each color channel are determined by performing the following operations: - irradiating a target reflecting monotonically in the spectrum from the visible spectrum to the infrared spectrum with a monochromatic light signal of that color; - receiving, by the image sensor (100), a monochromatic light signal reflected by the target; - averaging all IR pixel values ​​to determine a first color mean; - averaging all color pixel values ​​associated with the color to determine a second color mean; - calculating the color dependent IR pollution gain coefficient by calculating the ratio between the first color mean and the second color mean.

12. An imaging system (300), comprising: An image sensor (100), the image sensor comprising a two-dimensional pixel array, the pixel array comprising a plurality of color pixels and a plurality of IR pixels; An image processor (200) for processing a plurality of pixel values ​​provided at an output of the image sensor, the image processor being configured to: For each color pixel, the pixel value is processed by the following operations: - determining a local color correlation ratio, the local color correlation ratio representing the infrared light receiving portion in the color pixel; and - estimating the infrared partial value from the color pixel value of the color pixel based on the local color correlation ratio; and generating an infrared image based on a plurality of pixel values ​​from the plurality of IR pixels for the plurality of IR pixels and an estimated plurality of infrared partial values ​​from the plurality of color pixels for the plurality of color pixels; Before processing the pixel values, performing a first preprocessing, the first preprocessing comprising adjusting the infrared transmittance of the plurality of color pixels in the pixel array to a target level; as well as A second preprocessing is performed after the first preprocessing, the second preprocessing being performed for each IR pixel and comprising: - determining a plurality of color component values ​​representing color light receiving components in the IR pixel in all color channels based on the pixel value of the IR pixel and a predetermined color-dependent IR pollution gain coefficient; as well as - filtering the pixel value of the IR pixel by removing the determined plurality of color portion values.

13. The imaging system of claim 12, wherein: The image processor is further configured to determine the local color correlation ratio for each color pixel by obtaining the local color correlation ratio according to a plurality of pixel values ​​of at least one IR pixel located adjacent to the color pixel in the pixel array.

14. The imaging system of claim 13, wherein: The image processor is further configured to determine the local color correlation ratio for each color pixel by obtaining the local color correlation ratio based on pixel values ​​of one or more color pixels having the same color as the color pixel and located adjacent to the at least one IR pixel in the pixel array.

15. The imaging system of claim 12, wherein: The image processor is also configured to determine the local color correlation ratio for each color pixel by the following operation: obtaining the local color correlation ratio based on pixel values ​​of one or more color pixels having the same color as the color pixel and positioned adjacent to at least one IR pixel positioned adjacent to the color pixel in the pixel array.

16. The imaging system according to any one of claims 12 to 15, wherein: The image processor is further configured to determine the local color correlation ratio for each color pixel by: - respectively determining a plurality of color correlation ratios relating to a plurality of IR pixels located adjacent to the color pixel in said pixel array; - averaging the plurality of color correlation ratios. 17 . A non-transitory computer-readable data medium storing a computer program which, when executed by a computer, causes the computer to perform the method according to claim 1 .

Images