Method and image processor unit for processing image data of an image sensor - Patents.com

Abstract

A method for processing image data is described, the image comprising a matrix of pixels. The method includes the steps of: a) determining a difference between a current pixel value and pixel values ​​of respective neighboring pixels in a number of gradient directions; and b) encoding the current pixel by replacing it with the gradient direction having the minimum gradient difference determined in step a) for the current pixel. Thus, the gradient direction is explicitly coded into the output image data stream.

Description

[Technical field]

[0001] The present invention relates to a method for processing image data, the image comprising a matrix of pixels.

[0002] The invention further relates to an image processor unit for processing image data of an image sensor, the image sensor comprising a sensor area of ​​a matrix of pixels providing an image comprising a matrix of pixel data.

[0003] Furthermore, the invention relates to a computer program arranged to carry out the above-referenced methods. [Background technology]

[0004] In order to process image data from an image sensor connected to a controller, the image data must be transmitted over high-speed processing limited hardware resources. For example, in an automotive application, multiple cameras are connected to a central controller, but the connection is available via a high-speed serial link with limited bandwidth. It is desirable to reduce the data rate on the link while preserving high image quality.

[0005] Simple quantization, ie, reducing the bit depth of image data, is easy to implement but results in a visible degradation of image quality.

[0006] Transform-based encoding (eg, JPEG, MPEG) offers good image quality, but is very complex to implement and difficult to keep the bit rate stable.

[0007] Wavelet encoding (e.g., JPEG2000 or similar) is a transform-based encoding that requires a frame buffer, which introduces delays that can be significant in certain applications, such as automotive applications.

[0008] Block encoding (e.g. the bc7 format used in GPU texturing) offers good image quality and constant bitrate. It is a relatively simple decoding procedure. However, the encoding is quite difficult and complex.

[0009] F. Lebowsky, M. Bona: Extraordinary Perceptual Colour Stability in Low Cost, Realtime Color Image Compression Inspired by Structure Tensor Analysis describes a nonlinear method applied to image compression, where error quantities are processed in several complementary differential domains that allow advanced visual masking strategies. Structure and magnitude quantities are estimated for each pixel in the image by sequential raster scan order from top left to bottom right across the entire image, row by row. Over all local neighborhoods of the central pixel, the minimum gradient magnitude is determined along with its maximum gradient magnitude in the vertical direction. The minimum gradient magnitude is normalized by the sum of the minimum and maximum gradient magnitudes. Structure quantities are classified into three main categories in the normalized gradient domain: contours (triactories), zones (homogeneous regions), and extrema (local minima or maxima). For each classification, an integral error control loop is operated and the main categories are prioritized in a given order by contours, zones and extrema.

[0010] US8,958,635B2 discloses a method and a device for processing a digital image, where each pixel has at least one colorimetric component. The value of the corresponding colorimetric component is modified to obtain a modified value located inside or outside the colorimetric range associated with the corresponding colorimetric component. Thus, a user can indicate violations of the colorimetric space by assigning a unique reference value that is located in the colorimetric range and different from the restrictions to the modified values ​​of the image located outside the colorimetric space. Summary of the Invention

[0011] It is an object of the invention to provide an improved method for processing image data, an improved image processor unit and a computer program for processing image data of an image sensor.

[0012] This object is achieved by a method with the features of claim 1, an image processor with the features of claim 8 and a computer program with the features of claim 10. Preferred embodiments are set out in the dependent claims.

[0013] The method comprises: a) determining a gradient difference between a current pixel value and pixel values ​​of respective adjacent pixels in a number of gradient directions; b) encoding the current pixel by replacing it with the gradient direction having the minimum gradient difference determined in step a) for the current pixel.

[0014] Thus, for example, the image processor unit is configured to perform these steps a) and b) by a hardware implementation in an image data processor, e.g. an fpga, or by a suitable computer program comprising instructions for performing the above steps on pixel data or a pixel data stream by use of a data processor.

[0015] According to the present invention, the gradient direction is explicitly coded into the output data stream.

[0016] The method provides good image quality and can easily result in visually lossless image processing. The method requires a minimum amount of computational resources. No frame buffer is required. A minimum amount of line buffers and a minimum amount of processing power are required. The results are predictable and ideally a fixed data rate can be provided. The compression ratio can be selected as desired.

[0017] A delta flag may be included in the gradient direction used in step b) to encode the current pixel to indicate that the pixel data reflects a predictor for the current pixel. Thus, the use of the delta flag allows for providing the original pixel data, or alternatively, for indication by the delta flag predictor data.

[0018] The current pixel may be encoded by replacing the current pixel with a gradient direction, in the case where the minimum gradient difference for the current pixel exceeds a threshold, the gradient direction includes data that corresponds to the minimum gradient difference determined in step a) for the current pixel and includes an encoded value for the gradient difference determined in step a).

[0019] Thus, the gradient direction is provided at the output as determined in case the difference between the predictions at the current pixel does not exceed a certain threshold. Otherwise, if the difference exceeds a certain threshold, the bits used for the gradient difference are reduced by further encoding the gradient difference value, i.e., the delta vector, so that the bit length of the value for the gradient difference would require a higher number of bits. This allows providing a certain bit length for the pixel data at the output.

[0020] The method can be performed for each pixel in the image's matrix, considered as the current pixel, which is compared with a set of surrounding pixels. For example, the image can consist of a rectangular grid of pixels. The image can be formed as a single matrix, i.e., as grayscale with singlets of pixels, or it can be a higher-dimensional image, e.g., a color-coded image, comprising triplets of pixels in either RGB or YUV color space, e.g. (typically) with 8 bits per component. The image can be encoded and decoded as a stream of pixels in the usual English reading order, i.e., from left to right, top to bottom.

[0021] The method can be performed for exactly one current pixel in a group of N adjacent pixels. Thus, each run of N consecutive pixels forms a group, and the groups do not overlap. The group size N is an encoding quality parameter. The gradient difference (delta or "update") is encoded for only one of the pixels in the group. The index of the pixel within the group (update selector-index) can be encoded as a ceil(log2(N)) bit value.

[0022] Encoding the value for the slope difference can be performed by assigning the value to a predefined code in a table of codes. For example, the slope difference, i.e., delta code, can use non-linear quantization. The delta code can utilize two's complement wraparound. [Brief description of the drawings]

[0023] The invention will now be described by way of example embodiments in conjunction with the accompanying drawings, in which: [Figure 1] FIG. 1 is a block diagram of an image processor unit connected to an image sensor. [Diagram 2] FIG. 2 is a schematic of adjacent pixels and a current pixel in a pixel array for an image. [Diagram 3] FIG. 3 is the encoded data set for the current pixel X. [Figure 4] FIG. 4 is the encoded data set for N pixel groups. [Diagram 5] FIG. 5 is a signed lookup table for encoding the slope difference bits with a 5-bit signed code and a 5-bit wrapping code. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0024] FIG. 1 shows image data PA of an image sensor 1 (e.g., at least one camera).IN 1 shows a block diagram of an image processor unit 2 arranged to process pixel data PA at the output of the image sensor 1, for example in the form of a pixel data stream. IN The sensor area has a pixel matrix providing an image having a matrix of pixels.

[0025] The image processor unit 2 may be designed in the form of a microprocessor together with a computer program containing instructions, which when executed by the microprocessor, transmits the incoming pixel data PA to the processor unit. IN and encoding the current pixel by replacing the current pixel with the gradient direction. IN This may be performed for each current pixel of the group of N pixels, or for a selected current pixel of the group of N pixels.

[0026] FIG. 2 illustrates a current pixel X and four neighboring pixels A, B, C, and D.

[0027] The current pixel considered in the process is read from the current pixel X to the right of the adjacent pixel at X,Y location 3,2 in a left-to-right, top-to-bottom order as indicated by the arrows. Reading starts at the first column and proceeds for each pixel in the line, wrapping around to the next line at the first column location once the current pixel reaches the last column of the respective line.

[0028] In the enclosed example, for each current pixel X, a prediction direction is encoded, for example, as a 2-bit value. The four possible directions are "left" (indicated by the arrow between left pixel A and current pixel X), "upper left" (indicated by the arrow between adjacent pixel B and current pixel X), "up" (indicated by the arrow between adjacent pixel C and current pixel X), and "upper right" (indicated by the arrow between pixel D and current pixel X). As shown, the image consists of a rectangular grid of pixels, which may be triplets, for example, in either RGB, YUV or YCrCg color spaces, with (typically) 8 bits per component. Each of the pixels shown thus consists of a matrix of three pixel values ​​for the respective color RGB or YUV or some other suitable color scheme.

[0029] Alternatively to the embodiment described, each of the pixels of the matrix can be formed as a singlet for a grayscaled image, or can be of any higher order according to the respective color code scheme. The example described is based on a three component code for each pixel.

[0030] The pixel value difference between the current pixel X and each of the neighboring pixels A, B, C, D is determined for each color, respectively. The minimum gradient direction is determined from the gradient difference values ​​to quantize the minimum gradient direction into four directions: left, top-left, top, top-right. This can be encoded, for example, by a two-bit value 00 for left (A), 01 for top-left (B), 10 for top (C), and 11 for top-right (D), which specifies the gradient direction with the smallest difference value in the set of gradient differences determined for all these four directions.

[0031] Instead of keeping the value for the current pixel in the output pixel data, the current pixel can be replaced by the minimum gradient direction and an indicator (delta flag) indicating this replacement. Thus, the gradient direction is encoded in the pixel data stream. The receiver can use samples of neighboring pixel data in the indicated gradient direction as predictors for the current data and pixel value not in the data stream.

[0032] In the case where the difference between the prediction and the current pixel X exceeds a certain threshold, the difference between the current pixel value and the neighboring pixel value that has the smallest gradient direction relative to the current pixel X can be encoded as it forms the delta vector. Thus, the current pixel value is replaced by an indicator of the gradient direction and the delta flag, along with three delta vectors for each color RGB, YUB, or the like.

[0033] The data set for the current pixel may, for example, contain 3x8 bits for a pixel value having three colors, but the bit length can be reduced to 3 bits for the current pixel value, or to 21 bits per current pixel together with the delta vector.

[0034] The reduction in bit length for the delta vector to 6 bits per pixel can be performed, for example, by using a piecewise linear approximation to a logarithmic curve.

[0035] This method allows for a very simple implementation and provides good color quality. However, it tends to create "banding" artifacts with smooth gradients. The bitrate is variable and depends on the content per pixel. This is difficult to control, requires rate control algorithms, and is difficult to realize in hardware. Therefore, a lower limit on the bitrate (e.g. 3bpp) due to the need to send 2 bits per gradient selector and 1 bit for the delta flag is beneficial.

[0036] 4 shows the data structure for output pixel data for groups of N pixels. The groups of pixels in the image do not overlap. The group size N can be selected as needed and forms an encoding quality parameter.

[0037] For one selected current pixel X in the group, the delta or update is encoded. The index of the selected current pixel X in the group is provided as an update selector with a bit length of log2(N).

[0038] The delta, i.e. the difference between the pixel value for the current pixel and the nearest neighboring pixel with the smallest gradient difference for the respective color RGB or YUV or YCgCo, is added to the current pixel being affected. Thus, the current pixel X is no longer a direct copy from the neighboring pixel. Instead, it has a delta vector applied to it before the encoding / decoding continues.

[0039] The delta vector itself is encoded as three Q-bit values, one for each color component (e.g. RGB / YUV / YCgCo or similar), where Q is an encoding quality parameter and can be different for each component.

[0040] For each of the N pixels in a group of N pixels, a gradient direction is determined and provided in a delta set.

[0041] Preferably, the algorithm can be easily converted into a fixed-rate compressor that provides a fixed number of bits per pixel. This can be achieved by encoding the delta for exactly one pixel in each group of N adjacent pixels. This can be done by encoding the delta for exactly one pixel in each group of N (full bits)

number

[0042] A lookup table can be applied to determine the value to be added to the prediction from the delta code. The lookup table can be constructed such that lower differences are quantized less than higher differences. The effect that the addition may cause integer overflow can be exploited to construct a more efficient code.

[0043] For example, with Q=6 for all components, the data set illustrated in FIG. 4 is encoded for N pixel groups.

[0044] As can be seen, the number of pixels encoded for a group is constant for any chosen combination of N and Q parameters, resulting in a fixed number of bits per pixel, i.e., a perfectly constant bitrate.

[0045] This results in a pixel predictive encoding scheme, where the prediction direction is explicitly encoded. The prediction for one of the pixels in the group is modified using a delta code. The delta code uses non-linear quantization. The delta code can take advantage of, for example, two component wraparound. The choice of number N for the group size, Q=6 bits, determines the bit rate for the case of a three component image color code (e.g., RGB / YUV / YCgCo) as follows: [Table 1]

[0046] This modification allows for a fixed bit rate that is easily controllable. This solves the image quality problem. The forced delta in a smooth gradient eliminates banding.

[0047] Processing N pixels per group requires a significantly more complex encoder implementation. There is no need to compute and compare all N encoding variants for each N pixel group, because encoding the delta for the current pixel X changes the gradient and predictors for all further pixels. The encoder complexity depends on the bitrate and group size. The lower the bitrate, the larger the groups and the higher the encoder complexity. It is preferable to perform the method at or below a target of 1:4 compression (6 bpp) with reasonable effort.

[0048] It is possible to choose different group sizes for each color component, for example RGB or YUV, from each other.

[0049] A lower bitrate limit, say 2bpp, can reach very large group sizes.

[0050] The construction of the delta groups can be performed by encoding the slope difference, i.e., the wrapping code utilizes two's complement arithmetic as opposed to traditional signed codes.

[0051] This is shown in FIG. 5 with an exemplary 5-bit signed code for a signed + / -256 value range for 8-bit color depth and a 5-bit wrapping code.

[0052] For a pixel that receives a delta update, the difference between the predicted (copied from nearby) pixel value and the actual (ground two's) pixel value can have a value range of -255 to +255 (assuming 8-bit color depth), which needs to be encoded into Q bits.

[0053] Obviously, if Q is less than 9, this is always a lossy process. However, typical values ​​of Q are roughly 3 to 6. The mapping lookup table LUT from the Q-bit delta code to the added / subtracted difference is of great importance to image quality.

[0054] Because the differences between predicted and actual pixels are typically small, the delta lookup table allocates more entries for smaller differences than for larger differences. For example, a useful 5-bit lookup table such as that shown in Figure 5 can contain the values ​​[0,1,2,3,4,8,12,16,20,36,52,68,84,148,212,255] and the corresponding negative values.

[0055] However, something like a "signed" lookup table is the most efficient lattice representation, because depending on the pixel value, only parts of the lookup table make sense: in the extreme case, when the predictor is 0 or 255, only the positive or negative half of the lookup table is used, i.e. half the space is wasted.

[0056] To remedy this, the lookup table can be wrapped and the negative values ​​are instead two's complement positive values ​​as shown in Figure 5 for a 5-bit wrapping code. When adding the decoded delta to the predicted pixel integer overflow, the resulting truncation of the most significant bit is explicitly used as a tool.

[0057] For example, a useful 3-bit "wrapping" lookup table could contain the values ​​[0, 3, 6, 36, 128, 220, 250, 253]. If the predicted value is 100, the actual value is 97, and a code for the difference of 253 is encoded. When adding 100 to 253, the result is 353, which cannot be represented in 8 bits. By discarding the 8th bit, what remains is 97, which is exactly the target value. In the extreme cases of predicted values ​​0 and 255, the "wrapping" lookup table no longer wastes half the code space. There is certainly an unnecessarily detailed representation of pixel values ​​near the opposite end of the value range, values ​​that by definition cannot be reached.

[0058] A "perfect" highest quality encoder would have to try all possible delta pixel locations in a group and choose the encoding that shows the smallest visual difference from the original image. This mode has by far the best quality, but is very slow, especially for large group sizes. The complexity for each pixel is O(N) in group size. A simple encoder implementation that avoids multiple encodings uses heuristics to pre-select delta pixel indices that would produce potentially suboptimal decisions (and therefore non-optimal image quality), but takes significantly less time. The complexity per pixel is constant, i.e., O(1).

[0059] Even simpler encoders not only determine the delta pixel index in advance, but also the prediction direction, which results in a further reduction in quality for another small gain in speed.

[0060] The decoder is not affected by this: it can more or less decode each pixel individually without looking ahead.

[0061] The encoding of the delta code can be performed in any suitable manner, for example it can be any two's complement co-element lookup table with values ​​between -255 and +255 ("signed") or 0 and 255 ("wrapping"), or each value can have a lower number of significant bits.

[0062] The method can be performed for any number of color components, for example for one color component of a monochrome image, or for more than three color components of a multi-spectral image.

[0063] By allowing the update selector to have a value of N, i.e. an invalid index, that does not reference any pixel in the group and does not encode a delta in that case, the algorithm can be transformed into a variable rate encoding scheme: in uniform regions of the image, the encoder can choose not to send a delta at all.

[0064] The method requires low computational complexity, especially in the decoder. Only one line buffer is needed for encoding and decoding. The lossy compression preserves structure and object contours. No blocking or ringing artifacts occur. The prediction direction, which is directly encoded in the delta stream, is a coarsely quantized structure tensor that can be used as input for a set of subsequent image processing operations that require such information.

[0065] Due to the fact that only one pixel in a group can have a delta, a very clear pattern appears for very noticeable content, which forms irregular clusters of pixels with exactly the same color.

Claims

1. 1. A method for processing image data, comprising the steps of: An image comprises a matrix of pixels, a) determining a difference between a pixel value of a current pixel and pixel values ​​of respective adjacent pixels in a number of gradient directions; b) encoding the current pixel by replacing the current pixel with the gradient direction having the smallest gradient difference determined in step a) for the current pixel.

2. 2. The method of claim 1, further comprising: including a delta flag in a gradient direction used in step b) to encode the current pixel to indicate that the pixel data is a predictor for the current pixel.

3. 3. The method according to claim 1, further comprising: encoding the current pixel in case the minimum gradient difference for the current pixel exceeds a threshold value by replacing the current pixel with the gradient direction having the minimum gradient difference determined in step a) for the current pixel and with an encoded value for the gradient difference determined in step a).

4. 4. The method according to claim 1, wherein the method is performed for each pixel in the matrix of an image, each of which is considered to be a current pixel.

5. 4. A method according to claim 1, characterized in that the method comprises implementing a method in which exactly one pixel of a group of N adjacent pixels is considered as the current pixel.

6. 6. A method according to claim 1, further comprising encoding the pixel values ​​for the gradient difference by assigning them to predetermined codes in a code table.

7. 7. The method of claim 6, further comprising wrapping said code table by using two's complement positive values ​​instead of negative values.

8. 1. An image processor unit for processing image data of an image sensor, the image sensor comprising a sensor area of ​​a matrix of pixels providing an image comprising a matrix of pixel data, the image processor unit comprising: a) determining a difference between a pixel value of a current pixel and a pixel value of each of the neighboring pixels in a number of gradient directions; b) an image processor unit configured to encode the current pixel by replacing the current pixel with the gradient direction having the minimum gradient difference determined in step a) for the current pixel.

9. An image processor unit according to claim 8, characterized in that the image processor unit is arranged to process image data by carrying out the steps of the method according to any one of claims 1 to 7.

10. A computer program comprising instructions which, when executed by a processing unit, cause the processing unit to perform the steps of the method according to any one of claims 1 to 7.

Images