Method and apparatus for image processing, in particular for debayering

Abstract

The invention relates to a method for image processing, in particular for debayering, comprising at least the following steps: Providing raw data (S1) output by a Bayer sensor (1) with a matrix of pixels (Pij) and containing individual pixel values ​​(PWij), Color interpolation from the raw data (S1) in a chroma interpolator (8) and determination of a chroma signal (CbCr-ij), Luminance interpolation from the raw data (S1) in a luminance interpolator (9) and determination of a first luma signal (Y1ij) containing individual luminance values ​​(Y1ij), Detection of artifacts in an artifact detector (10) and determination of a control signal (CTR1-ij), Controllable low-pass filtering of the first luminance signal (Y1ij) with the control signal (CTR1), adapting the low-pass filtering of the individual luminance values ​​(Yij) depending on the control signal (CTR1), and determination of a second luminance signal (Y2ij), Receiving the chroma signal (CbCr-ij) and the second luminance signal (Y2ij) in a conversion matrix (14) and determining image values ​​(F-Rij, F-Gij, F-Bij) to the pixel values ​​(Pij).

Description

[0001] The invention relates to a method and a device for image processing, in particular for debayering image signals from a Bayer sensor.

[0002] Bayer sensors are often used for photosensors to capture color images; these consist of a sensor chip with a matrix arrangement of sensor pixels and color filters placed in front of the individual sensor pixels.

[0003] Thus, an R-filter, G-filter, or B-filter of the RGB color values ​​is placed in front of each sensor pixel, with its characteristic filter properties or absorption spectrum. For example, in the matrix, rows with GR and rows with BG arrangements alternate, so that the G color value is present in 50% of the pixels, and the R and B color values ​​are each present in 25% of the pixels. The vertical columns are arranged in a correspondingly alternating pattern.

[0004] The Bayer sensor outputs a raw data signal, specifically a stream of successive pixel values, which initially represent intensity signals or values ​​corresponding to their color filter. The undetected color values ​​of each pixel are generally subsequently determined by Bayer interpolation from surrounding sensor pixels of other colors. Thus, for example, the green value G and the blue value B for the red color pixel R are determined by interpolation from the surrounding pixels.

[0005] Because the Bayer pattern is not uniform, with varying numbers of color values ​​and a geometric arrangement resembling a checkerboard, interpolation can lead to the appearance of disruptive patterns. This occurs, for example, at certain angles or color transitions. These effects, also known as artifacts, are particularly noticeable at sharp edges where the scene reaches the sensor's resolution.

[0006] This effect, or these artifacts, can be removed by blurring the image using a low-pass filter (OLPF), as shown, for example, in US2001 / 0052938A1. Alternatively, these artifacts can also be removed using a digital low-pass filter. To completely remove these artifacts, the image must be reduced to a quarter of its original resolution; however, this results in a loss of detail. The blurring can then be improved by sharpening. Therefore, it is always a matter of finding the best compromise.

[0007] Such Bayer interpolation and debayering with a digital low-pass filter are known from US2006 / 104537A1.

[0008] US 2011 / 52053 A1 describes an image processing device that converts a digital image from initial values ​​into YCrCb color space values, wherein the image has an initial luminance Y layer and two initial Cr,Cb chrominance layers. The image processing device includes a first block that processes and modifies the initial luminance Y layer of the digital image to produce and output a modified luminance layer, and a further color artifact corrector block that operates in parallel with the first block A and modifies the initial values ​​of the luminance in the Y and the CR,Cb chrominance layers through pixel-by-pixel processing.

[0009] WO 2018 / 234616 describes a method comprising the following steps: receiving image data representing pixels of a two-dimensional image from an array of photosensors, each photosensor having a color filter to restrict the photosensor's sensitivity to a single color range out of three color ranges; furthermore, estimating the luminance of the image data for a first set of pixels restricted to a first color range using a first filter; separately estimating the luminance of the image data for a second and third set of pixels restricted to the remaining color ranges; applying a second luminance evaluation filter to the first set of pixels; and estimating the luminance for the second and third sets of pixels to estimate an image luminance.

[0010] The invention is therefore based on the objective of creating a method and a device for image processing that enable image processing with high resolution and low artifacts with relatively little effort.

[0011] This problem is solved by a method according to claim 1 and a device according to claim 14. The dependent claims describe preferred embodiments.

[0012] The device according to the invention is particularly intended for carrying out the method according to the invention, and the method according to the invention can in particular be carried out with the device according to the invention.

[0013] Thus, in particular, the raw data from the image sensor is processed separately in a chroma interpolator to determine the color values, especially as CbCr values, and separately in a luma interpolator to determine the luma values ​​or brightness values. This alone achieves the advantage that both color values ​​and brightness values ​​can be optimized independently and filtered as needed or according to the inventive method.

[0014] Furthermore, after an initial luma interpolation, a controlled low-pass filtering of the luma values ​​is performed, which is carried out by control signals from an artifact detector.

[0015] The luma values ​​and the control signal with the control signal values ​​preferably each represent a corresponding matrix with indexing i, j corresponding to the pixels i, j, i.e., pixel-related correction values ​​are determined and used for low-pass filtering.

[0016] This offers the advantage that the luma values ​​can be processed independently to address the artifacts, without involving the processing of the chroma values.

[0017] The process is characterized in particular by the following properties: - The method preferably processes brightness information (luma or y) and color information (chroma) in separate processing steps. After processing, conversion back to RGB is possible. - Separate processing allows each part to be optimized independently. - The artifact detector identifies the areas in the image that contain disturbances. A dynamic artifact filter, controlled by, for example, a difference calculator or as part of the artifact detector, removes artifacts only in the areas where problems exist. Other areas of the image retain maximum sharpness.

[0018] Other preferred features and advantages include: Brightness is processed as follows.

[0019] The luminance signal, or luma signal, is interpolated by an adjustable low-pass filter; that is, with a lower low-pass filtering level, many details remain, while with a higher low-pass filtering level, few details remain in the image. The control signal is an image mask that preferentially marks critical image areas.

[0020] At critical points, the image becomes less sharp, but this smooths out edges so that no image artifacts are visible. At non-critical points, the image has maximum sharpness, without any image artifacts.

[0021] According to a particularly preferred design, the control signal is generated by a filtered image as follows. i. Amount of RB ii. Normalized to Y by 1 / Y iii. The masking area is optimized by a filter, whereby the filter in particular performs edge detection and, if necessary, creates a blur.

[0022] The invention is explained in more detail below with reference to the accompanying drawings. These show: Fig. 1 a Bayer sensor, Fig. 2 a block diagram of a method and system for image processing according to the invention; Fig. 3. Another representation of the block diagram showing the respective signals; Fig. 4 a scheme of an image to be reproduced, as a matrix with image areas of different colors; Fig. 5 and Fig. 6 color charts CC1 to CC9 Fig. 3 in larger view

[0023] In Fig. 1 is an example of a Bayer sensor 1, shown as a photosensor, which for illustrative purposes is configured as a 9x9 matrix of individual pixels Pij with a characteristic color filter or Bayer filter 3 with a matrix of color values; real image sensors are generally larger. Thus, a color value, R, G, or B, with the characteristic filter property or absorption spectrum is provided in front of each pixel 2. In Fig. The positions of the matrix are indexed in the usual way, specifying the respective color value. Thus, in this matrix, rows with GR arrangement and rows with BG arrangement alternate, so that the color value G is present in 50% of the pixels, and the color values ​​R and B are each present in 25% of the pixels. The vertical columns are arranged in a similarly alternating pattern.

[0024] The Bayer sensor 1 outputs a raw data signal S1, which is specifically configured as a stream of successive pixel values ​​PWij, thus initially representing intensity signals, and is subsequently acquired and processed in an image processing device 5 according to the invention, so that an image signal S2 with the color values ​​F-Rij, F-Gij, F-Bij is subsequently output. The image processing device 5 specifically performs Bayer interpolation with debayering.

[0025] Fig. Figure 2 shows a first embodiment of the device 5 according to the invention, which is in Fig. 3 is described in more detail.

[0026] The pixel values ​​PWij initially represent intensity values ​​corresponding to their color filter. The undetected color values ​​of each pixel Pij are subsequently determined by Bayer interpolation from surrounding sensor pixels of the other color values. Thus, for example, the green value G43 and the blue value B43 for the red color pixel R43 are determined by interpolation from the surrounding pixels.

[0027] The image processing device 5 comprises a chroma interpolator 8 and a luminance interpolator 9, each of which receives the raw data signal S1, i.e., the pixel values ​​PWij. The chroma interpolator 8 performs color interpolation and subsequently outputs a chroma image CbCr, in particular as CbCr data (complementary blue, complementary red), thereby keeping the bandwidth and data volume low in the usual manner.

[0028] The luminance interpolator 9 outputs luma values ​​Y for the individual pixel values ​​in the usual manner, i.e., Y11 to Y99, with this luma signal Y being output to a controllable low-pass filter 12. An artifact detector 10 is provided, which, according to the more detailed description of the Fig. 3 preferably has a difference calculator 15 and an artifact filter 11 and determines an artifact signal AS from the chroma values ​​CbCr, or also from other color values ​​that are determined differently from the raw data, as described below, which is subsequently filtered by the artifact filter 11, which then outputs a control signal CTR1 to a controllable low-pass filter 12 in order to define a low-pass filter for the individual luminance values ​​Yj of the luminance signal Y.

[0029] The control signal CTR1 serves in particular to define the width of the low-pass filter in the luma signal Y for the individual luma values ​​Yij, i.e. the averaging over neighboring luma values ​​Yij. Without low-pass filtering or "zero low-pass filtering", the luma value Yij is taken directly. For higher low-pass filtering, the directly adjacent luma values ​​Yi-1, j, Yi+1, j, Yi, j+1, Yi, j-1, as well as Yi+1, j+1, ... can be used, and for stronger low-pass filtering, further limiting luma values ​​Yij can also be used.

[0030] The controllable low-pass filter 12 subsequently outputs a low-pass filtered luminance signal Y2ij to a conversion matrix 14.

[0031] The artifact detector 10 thus detects artifacts and outputs an artifact signal AS to an artifact filter 11, which generates a blurred artifact signal through edge detection and smoothing, which is then input as a control signal CTR1 to the controllable low-pass filter 12.

[0032] The conversion matrix 14 thus receives the chroma signal C1, or CbCr, and the second luminance signal Y2ij and performs a conversion – known as such – as the conversion matrix YCbCr in order to generate RGB values ​​of the matrix, i.e., R11 to R99, G11 to G99, B11 to B99, which in Fig. 1. 9x9 matrix shown.

[0033] Thus, the luma values ​​or brightness information Y and the chroma values ​​or color information CbCr are interpolated and determined in separate processing steps, with the controllable low-pass filter 12 being used additionally in the processing of the luma values.

[0034] The artifact detector 10 detects the areas in the image that contain disturbances.

[0035] According to Fig. 2. The determined control signal CTR1 is generated, in particular, as a blurry artifact signal.

[0036] To illustrate, in Fig. Figure 4 shows an image as a panel with twenty-four different colors and brightness levels, which is subsequently processed in the following figures. Here, artifacts and blurring may occur, especially at the edges or transitions between the monochrome square tiles.

[0037] Fig. Figure 4 shows the chroma signal CbCr as an image of the recorded panels with image areas.

[0038] The control signal CTR1 thus represents an image mask that marks critical image areas. At the critical points, which are in Fig. If the image is 6 times brighter, stronger low-pass filtering makes it less sharp, which smooths edges and thus reduces or completely prevents image artifacts. The dark areas in Fig. 6 are correspondingly uncritical points where the image shows maximum sharpness without image artifacts and which are therefore low-pass filtered or not filtered at all.

[0039] According to a preferred embodiment, the control signal CTR1 in the artifact detector 10, i.e., the difference calculator 15 and the artifact filter 11, is formed by the following determination: The difference Rij - Bij is calculated in the difference calculator 15, followed by normalization to the Y-value Y, i.e., division by Y: (R−B) / Y.

[0040] In the Arefakt filter 11, these values ​​are converted into a masking area through edge detection and smoothing. Reference symbol list 1 Bayer sensor 2 sensor pixels, Bayer pixels 3 Bayer filters 5 Image processing device 8 Chroma Interpolator 9 Luminescence Interpolator 10 Artifact Detector 12 controllable low-pass filters 11 Artifact Filters 14 Conversion matrix YCbCr to RGB 15 Difference Calculator CC1 Color Chart of S1 = (PWij) CC2 Color Chart von Cb=Cbij CC3 Color Chart von Cr=Crij CC4 Color Chart von AS=ASij CC5 Color Chart von CTR CC6 Color Chart von Yij CC7 Color Chart von F-Rij CC8 Color Chart von F-Gij CC9 Color Chart von F-Bij

Claims

[1] Image processing method, in particular for debayering, comprising at least the following steps: Providing raw data (S1) output by a Bayer sensor (1) with a matrix of pixels (Pij) and containing individual pixel values ​​(PWij), Color interpolation from the raw data (S1) in a chroma interpolator (8) and determination of a chroma signal (CbCr-ij), Luminance interpolation from the raw data (S1) in a luminance interpolator (9) and determination of a first luma signal (Y1ij) containing individual luminance values ​​(Y1ij), Detection of artifacts in an artifact detector (10) and determination of a control signal (CTR1-ij), Controllable low-pass filtering of the first luminance signal (Y1ij) with the control signal (CTR1), adapting the low-pass filtering of the individual luminance values ​​(Yij) depending on the control signal (CTR1), and Determination of a second luminance signal (Y2ij), Receiving the chroma signal (CbCr-ij) and the second luminance signal (Y2ij) in a conversion matrix (14) and determining image values ​​(F-Rij, F-Gij, F-Bij) to the pixel values ​​(Pij). [2] Method according to any one of the preceding claims, characterized by , that the raw data (S1) are currently generated by the Bayer sensor (1) and processed directly. [3] Method according to any one of the preceding claims, characterized by , that the pixel values ​​(PWij) of the raw data (S1) each contain intensity values ​​and information about the upstream color filter (G11, R21,...G99). [4] Method according to any one of the preceding claims, characterized by , that the detection of artifacts is based on: - the chroma signal (CbCr-ij), or - the chroma signal and the first luminance signal (Y1ij), or - from a further chroma signal determined from the raw data (S1) with or without the luminance signal (Y1ij). [5] Method according to any one of the preceding claims, characterized by , that in the step of artifact detection and determination of a control signal (CTR1-ij) first in the artifact detector (10), preferably in a difference imager (15) of the artifact detector (10), relevant areas, preferably with high color differences, are detected, and the artifact detector (10), preferably the difference generator (15), detects an artifact signal (ASij) and a subsequent artifact filter (11), which is preferably part of the artifact detector (10), in which the artifact signal (ASij) highlights and / or smooths the relevant areas, in particular edges, while forming the control signal (CTR1-ij). [6] Method according to claim 5, characterized by , that the artifact detector (10) includes a difference calculator (15) and a subsequent artifact filter (11), wherein the difference calculator (15) detects the relevant areas, in particular high color differences, and determines an artifact signal (ASij), and the artifact signal (ASij) in the subsequent artifact filter (11) highlights and / or smooths the relevant areas, in particular edges, forming the control signal (CTR1-ij). [7] Method according to any of the preceding claims, characterized by , that the controllable low-pass filter (12) performs a low-pass filtering width, in particular the extent of the inclusion of neighboring pixels (Pij), depending on the control signal (CTR1). [8] Method according to any one of the preceding claims, characterized by , that in the second luminance signal (Y2ij) affected areas are masked as artifacts by stronger low-pass filtering. [9] Method according to any one of the preceding claims, characterized by, that in the artifact detector (10), preferably the difference generator (15) and the artifact filter (11), the control signal (CTR1-ij) for a pixel (Pij) is determined on the basis of: - the amount of the difference between the R-value and B-value of the pixel value (PWij), or the amount of the difference between the Cr-value and Cb-value of the pixel value (PWij), and - a normalization to the brightness value (Yij). [10] Method according to claim 9, characterized by , that in the artifact detector (10), preferably in the artifact filter (11) of the artifact detector (10) The control signal (CTR1ij) for a pixel Pij is determined from the quotient (Rij - Bij) / Yij, with Rij as the red value of pixel Pij and Bij as the blue value of pixel Pij, or the quotient (Cr-ij - Cb-ij) / Yij, with Cr-ij as the red-green difference signal of pixel Pij and Cb-ij as the blue-green difference signal of pixel Pij [11] Method according to claim 9 or 10, characterized by , that the control signal (CTR1-ij) for a pixel value (Pij) is determined without including the G-value of the pixel value (Pij). [12] Method according to any of the preceding claims, characterized by , that the chroma signal (CrCbij) is determined solely on the basis of the raw data (S1), without including the first or second luma signal, and / or the first luma signal (Y1ij) is determined solely on the basis of the raw data (S1), without including the chroma signal (CrCbij). [13] Method according to any of the preceding claims, characterized by , that in the chroma interpolator (8) the chroma signal (CbCr-ij) is determined as a CbCr signal. [14] Device (5) for image processing of raw data signals (S1) of a Bayer sensor (1), in particular for debayering, wherein the device (5) comprises: a chroma interpolator (8) configured to receive the raw data (S1) and output a chroma signal (CbCr-ij), a luminance interpolator (9) configured to receive the raw data (S1) and output a first luma signal (Y1ij), an artifact detector (10) that is trained to receive the raw data (S1) and to generate a control signal (CTR1), a controllable low-pass filter (12) configured to receive the first luminance signal (Y1ij), wherein the controllable low-pass filter (12) is configured to perform a low-pass filtering of the pixel values ​​of the luminance signal (Y1) depending on the control signal (CTR), adapting the low-pass filtering of the individual luminance values ​​(Yij) depending on the control signal (CTR1), wherein the controllable low-pass filter (12) is configured to determine a second luminance signal (Y2ij), and a conversion matrix (14) which is configured to receive the chroma signal (CbCr) and the second luminance signal (Y2) and to form image values ​​(F-Rij, F-Gij, F-Bij) from them. [15] Device (5) according to claim 14, characterized by , that the artifact detector (10) exhibits: a difference calculator (15) trained to generate the artifact signal (ASij), and an artifact filter (11) configured to receive the artifact signal (AS) and to form the control signal (CTR1) [16] Device (5) according to claim 15, characterized by , that the artifact filter (11) is designed to shape the control signal (CTR1) by edge detection and smoothing.

Images