Method and apparatus for correcting the value of a correction target pixel of an image from an electronic camera

Abstract

The invention relates to a correction method for correcting the value of a correction target pixel in an image from an electronic camera. The correction method comprises determining a corrected value of the correction target pixel by means of a filter based on a weighted averaging of the value of the correction target pixel and an estimated value, wherein the weighted averaging is based on weight values ​​for the correction target pixel and for the estimated value. The weight values ​​are based on an inhomogeneity value, which is determined based on a deviation of the value of the correction target pixel from at least one surrounding value, which is based on the values ​​of a plurality of pixels in the surroundings of the correction target pixel.

Description

[0001] The invention relates to a correction method for correcting a value of a correction target pixel of an image from an electronic camera. The invention further relates to a corresponding correction device and an electronic camera comprising an image sensor and the correction device. The invention further relates to a computer device and a computer program product. BACKGROUND OF THE INVENTION

[0002] In image sensors, disturbing image artifacts occur when pixels exhibit electrical behavior that deviates in an inhomogeneous manner from the electrical behavior of the surrounding pixels. Particularly isolated, significant deviations are perceived by the viewer as disturbing image defects.

[0003] Therefore, it is the object of this invention to provide a method and a device for correcting images that are affected by inhomogeneity-based image errors, wherein the method and the device are preferably designed to dynamically detect, in particular, strong deviations of individual pixels and to correct them in such a way that the image is assessed by the viewer as being largely error-free (or at least improved) with regard to the aforementioned error sources.

[0004] Furthermore, it is a desirable side condition that the intervention in the image data is minimal so that the image data is distorted as little as possible.

[0005] DE 197 15 983 C1 relates to a method for correcting the gray values ​​of images from a digital infrared camera with a two-dimensional detector. First, a stationary correction is performed on a reference source, and correction coefficients are determined. For the stationary correction, an average gray value characteristic curve is determined for the detector, and correction coefficients are determined for each pixel by comparing it with the gray value characteristic curve for each pixel of the two-dimensional detector. For this purpose, a nonlinear approach is used to adapt the gray value characteristic curve of each pixel to the average gray value characteristic curve. The correction coefficients are stored in a memory for correcting the individual gray values ​​of each pixel during operation. The correction coefficients are dynamically improved during operation.To do this, the gray values ​​of an image are captured and filtered using a locally acting adaptive filter. The remaining aberration is then determined using the gray values ​​of the unfiltered image and the gray values ​​of the filtered image, and the correction coefficients are improved using the determined remaining aberration.

[0006] US 2002 / 0196354 A1 discloses an image processing method and apparatus for processing a signal from a monochrome or color sensor that may contain pixel defects or errors. Without prior knowledge of pixel defects, the processing method examines each pixel value and its neighboring pixel values. A series of tests is applied to the set of pixel values ​​to determine if the underlying pixel is defective. If the underlying pixel is found to be defective, the pixel value is replaced with an estimated value derived from the values ​​of the neighboring pixels. Otherwise, the pixel value remains intact.

[0007] US 2009 / 0040371 A1 discloses methods, systems, and devices for approximating vertical and horizontal correction values ​​in a pixel value correction calculation. Multiple vertical and / or horizontal correction curves are used, which can be applied to one or more color channels of the imager. The use of multiple correction curves enables a more accurate approximation of the correction curves for image pixels. SUMMARY OF THE INVENTION

[0008] The invention is based on the object of providing a correction method for correcting a value of a correction target pixel of an image from an electronic camera, which is suitable for meeting one or more, preferably as many as possible, of the aforementioned requirements. Furthermore, the invention is based on the object of providing a corresponding correction device, as well as a computer device and a computer program product. Finally, the invention is based on the object of providing an electronic camera comprising the correction device according to the invention.

[0009] According to a first aspect of the invention, there is provided a correction method for correcting a value of a correction target pixel of an image of an electronic camera, the correction method comprising: - Determining a corrected value of the correction target pixel by means of filtering based on a weighted averaging of the value of the correction target pixel and an estimated value, wherein the weighted averaging is based on weight values ​​for the correction target pixel and for the estimated value, wherein the weight values ​​are based on an inhomogeneity value determined based on a deviation of the value of the correction target pixel from at least one ambient value based on values ​​of a plurality of pixels in an ambient of the correction target pixel.

[0010] The present invention is based on the inventors' idea of ​​correcting pixel values ​​that deviate due to inhomogeneity using a filter. Correcting the value of a correction target pixel involves determining a corrected value of the correction target pixel by filtering. The filtering is based on a weighted average of the value of the correction target pixel and an estimated value, wherein the weighted average is based on weight values ​​for the correction target pixel and for the estimated value. The weight values ​​are based on an inhomogeneity value, which is determined based on a deviation of the value of the correction target pixel from at least one ambient value based on values ​​of a plurality of pixels in an ambient environment of the correction target pixel.Because the weight values ​​are based on the inhomogeneity value, they can be determined dynamically in such a way that the filtering is adapted as well as possible to the actual inhomogeneity.

[0011] According to an advantageous development of the invention, if the inhomogeneity value indicates a lower inhomogeneity, the weighting values ​​are selected such that the value of the correction target pixel is weighted more heavily than if the inhomogeneity value indicates a greater inhomogeneity, and the estimated value is weighted less heavily than if the inhomogeneity value indicates a greater inhomogeneity. In this way, a high-quality correction can be achieved because, if the existing inhomogeneity is lower, the value of the correction target pixel influences the corrected value more strongly (and the estimated value is less), and, if the existing inhomogeneity is greater, the value of the correction target pixel influences the corrected value less strongly (and the estimated value is more strongly). The value of the correction target pixel is therefore always corrected with the estimated value only as much as necessary, depending on the existing inhomogeneity.

[0012] According to an advantageous development of the invention, the weighted averaging is performed continuously depending on the inhomogeneity value. Compared to a discontinuous correction, which, for example, operates with a case discrimination, this has the advantage that small deviations in the value of the correction target pixel from the at least one surrounding value do not lead to significant jumps in the corrected value of the filtering target pixel. Such jumps in the corrected values ​​are often perceived as extremely disturbing by the viewer, especially in image sequences, and are referred to as blinking artifacts.

[0013] It is preferred that the inhomogeneity value has a value range from a minimum value, which indicates a minimum inhomogeneity, to a maximum value, which indicates a maximum inhomogeneity, wherein, if the inhomogeneity value is equal to the minimum value, the value of the correction target pixel is given a maximum weight, preferably with a weight value equal to 1, and the value of the estimated value is given a minimum weight, preferably with a weight value equal to 0, and, if the inhomogeneity value is equal to the maximum value, the value of the correction target pixel is given a minimum weight, preferably with a weight value equal to 0, and the value of the estimated value is given a maximum weight, preferably with a weight value equal to 1. In this way, it can be achieved with the weighted averaging that it cannot leave the region of the continuous transition between the value of the correction target pixel and the estimated value.

[0014] It is further preferred that the minimum value be 0 and the maximum value be 1, wherein the weight value for the correction target pixel is 1 minus the inhomogeneity value, and the weight value for the estimated value is equal to the inhomogeneity value. In this case, the weight values ​​can be determined particularly easily from the inhomogeneity value.

[0015] The terms "maximum value" and "minimum value" here refer to the specification of a minimum and maximum inhomogeneity. These values ​​can be defined arbitrarily, and it is not necessary, for example, for the maximum value to be a larger number than the minimum value. In an example definition, the value 1 could indicate a minimum inhomogeneity (minimum value), and the value 0 could indicate a maximum inhomogeneity (maximum value).

[0016] It is preferred that the environment of the correction target pixel be a 5x5 environment. The inventors have found that a 5x5 environment enables a good compromise between efficient processing, particularly with regard to determining the inhomogeneity value and the estimated value, and high-quality correction. If the correction pixel is designated P22, for example, the 5x5 environment of the correction target pixel consists of the 24 pixels P00, P01, P02, P03, P04, P10, P11, P12, P13, P14, P20, P21, P23, P24, P30, P31, P32, P33, P34, P40, P41, P42, P43, and P44. In each direction, the correction target pixel is thus surrounded by two pixels of the 5x5 environment.

[0017] According to an advantageous development of the invention, the at least one ambient value comprises a first value and / or a second value, wherein the inhomogeneity value is determined based on a comparison of the value of the correction target pixel with the first value and / or a comparison of the value of the correction target pixel with the second value. The values ​​of pixels with highly inhomogeneous properties often differ singularly from the values ​​of the surrounding pixels. Therefore, the inhomogeneity value can be determined in a simple manner by comparing the value of the correction target pixel with the first and / or second value, which are based on the values ​​of the majority of pixels in the environment of the correction target pixel.

[0018] According to an advantageous development of the invention, if the value of the correction target pixel is less than or less than or equal to the first value, and / or if the value of the correction target pixel is greater than or greater than the second value, the inhomogeneity value indicates that the correction target pixel does not exhibit any inhomogeneity. This reflects the heuristic that the value of a pixel that does not exhibit any inhomogeneity is usually less than or less than a large value and / or greater than or greater than a small value, wherein the values ​​are based on the values ​​of the majority of pixels in the vicinity of the correction target pixel. An inhomogeneity only exists if the value of the correction target pixel is greater than or greater than the first value, and / or if the value of the correction target pixel is less than or less than the second value.It is preferred that the inhomogeneity value be determined such that it increases continuously with increasing difference from the first value and / or with increasing difference from the second value. This ensures a continuous relationship between the strength of the inhomogeneity and the inhomogeneity value.

[0019] According to an advantageous development of the invention, the first value is determined based on the m1-t largest value of the pixels of the plurality of pixels in the vicinity of the correction target pixel and / or the second value is determined based on the m2-t smallest value of the pixels of the plurality of pixels in the vicinity of the correction target pixel, where m1 and m2 are natural numbers and are preferably equal to 1 or 2. As described, the values ​​of pixels with highly inhomogeneous properties often deviate only solitarily from the values ​​of the surrounding pixels. If the correction target pixel exhibits an inhomogeneity, the value of the correction target pixel should therefore typically be greater than or equal to the largest value (m1 equal to 1) and / or less than or equal to the smallest value (m2 equal to 1) of the pixels of the plurality of pixels in the vicinity of the correction target pixel.

[0020] Of course, in practice it can also happen that more than one pixel within a neighborhood exhibits a high level of inhomogeneity. For example, two or three pixels may be affected by a stronger inhomogeneity. However, since strongly inhomogeneous pixels are by definition rare, the probability of such an occurrence decreases significantly with an increasing number of pixels within a neighborhood affected by a strong inhomogeneity. In this respect, the cases in which two pixels within a neighborhood are affected by an inhomogeneity are particularly important. This situation can be taken into account by subjecting the values ​​of the pixels of the majority of pixels in the neighborhood of the correction target pixel to rank filtering (RF).For example, if m1 and / or m2 are set to 2, the comparison of the value of the correction target pixel with the first value and / or the second value is not performed with the largest value and / or the smallest value, but with the second largest value and / or the second smallest value. This can reduce the influence of a second pixel affected by an inhomogeneity that may be present in the vicinity of the correction target pixel. It is possible to set m1 and m2 to the same value or to set them so that they are different from each other.

[0021] According to an advantageous development of the invention, the first value is determined based on the n1-t largest extrapolation value of at least two extrapolation values ​​determined based on the plurality of pixels in the vicinity of the correction target pixel, and / or the second value is determined based on the n2-t smallest extrapolation value of the at least two extrapolation values, where n1 and n2 are natural numbers and are preferably equal to 1 or 2. The use of extrapolation values ​​determined based on the plurality of pixels in the vicinity of the correction target pixel, in addition to or alternatively to the (direct) use of the values ​​of the pixels of the plurality of pixels in the vicinity of the correction target pixel, is advantageous in situations in which an ascending or descending image structure ends in the correction target pixel.In such a case, the value of the correction target pixel can be greater than or equal to the largest value (n1 equal to 1) and / or less than or equal to the smallest value (n2 equal to 1) of the pixels of the majority of pixels surrounding the correction target pixel, without the correction target pixel being affected by an inhomogeneity. The use of extrapolation values ​​allows this case to be taken into account when determining the inhomogeneity value.

[0022] As described above, it can also happen in practice that more than one pixel within a surrounding area exhibits a high degree of inhomogeneity. This situation can also be taken into account here by subjecting the extrapolation values ​​to rank filtering (RF). For example, if n1 and / or n2 are set equal to 2, the comparison of the value of the correction target pixel with the first value and / or the second value is not carried out with the largest extrapolation value and / or the smallest extrapolation value, but with the second largest extrapolation value and / or the second smallest extrapolation value. This can reduce the influence of a second pixel affected by an inhomogeneity that may be present in the vicinity of the correction target pixel. It is possible to set n1 and n2 to the same value or to set them so that they are different from each other.

[0023] According to an advantageous development of the invention, the at least two extrapolation values ​​are each determined based on values ​​of pixels of the plurality of pixels surrounding the correction target pixel that converge directly toward the correction target pixel in different directions. In this way, ascending and / or descending image structures that end in the correction target pixel can be taken into account when determining the inhomogeneity value, regardless of their direction.

[0024] It is preferred that the different directions include the directions horizontally from the left and / or horizontally from the right and / or vertically from the top and / or vertically from the bottom and / or diagonally from the top left and / or diagonally from the bottom right and / or diagonally from the top right and / or diagonally from the bottom left. For example, for a linear extrapolation in the horizontal direction from the left, a first extrapolation value can be determined as E_1 = 2 . P21 - P20 and in the horizontal direction from the right, a second interpolation value can be determined as E_2 = 2 . P23 ​​- P24.

[0025] It is further preferred that the at least two extrapolation values ​​are determined based on an extrapolation selected from the group consisting of a linear extrapolation, a quadratic extrapolation, a cubic extrapolation, a higher-order polynomial extrapolation, a trigonometric extrapolation, and a logarithmic extrapolation. By selecting a suitable extrapolation, it can be adapted in the best possible way to the ascending and / or descending image structures expected in the image.

[0026] According to an advantageous development of the invention, the first value is determined based on the m1-t largest value of the pixels of the plurality of pixels surrounding the correction target pixel and the n1-t largest extrapolation value of the at least two extrapolation values, and / or the second value is determined based on the m2-t smallest value of the pixels of the plurality of pixels surrounding the correction target pixel and the n2-t smallest extrapolation value of the at least two extrapolation values. In this way, the comparison of the value of the correction target pixel with the m1-t largest and / or m2-smallest value of the pixels of the plurality of pixels surrounding the correction target pixel and the comparison of the value of the correction target pixel with the n1-t largest and / or n2-t smallest extrapolation value are combined in a suitable manner. It is possible to set m1, m2, n1, and n2 all to the same value or to set them so that they differ at least partially from one another.For example, m1 and n1 could be set to the value 2 and m2 and n2 could be set to the value 1 (or vice versa), or only one value, such as n2, could be set to the value 2 and the other values ​​could be set to the value 1.

[0027] Even if a pixel does not exhibit any inhomogeneity, it may happen, for example due to temporal or spatial noise, that the value of the pixel meets the aforementioned criteria and is thus erroneously assigned an inhomogeneity value that indicates a relevant inhomogeneity.

[0028] According to an advantageous development of the invention, the m1-t largest value of the pixels of the plurality of pixels surrounding the correction target pixel and / or the n1-t largest extrapolation value of the at least two extrapolation values ​​is / are additionally increased by a first offset value and / or the m2-t smallest value of the pixels of the plurality of pixels surrounding the correction target pixel and / or the n2-t smallest extrapolation value of the at least two extrapolation values ​​is / are additionally reduced by a second offset value, wherein the first offset value is equal to or different from the second offset value. As a result, values ​​that occur within the natural noise are not changed or only slightly changed by the filter.

[0029] According to an advantageous development of the invention, the inhomogeneity value based on the deviation of the value of the correction target pixel from the at least one ambient value is additionally scaled with a scaling factor, e.g., by multiplying its value by a factor during determination or dividing it by a reciprocal factor. Scaling allows the behavior of the filter to be adjusted such that, on the one hand, the pixels that are conspicuously inhomogeneous are adequately corrected, and, on the other hand, the pixels that are not conspicuously inhomogeneous experience little or no change from the filter.

[0030] According to an advantageous development of the invention, the estimated value is determined based on an interpolation, median calculation, or averaging of the values ​​of pixels in the vicinity of the correction target pixel. In particular, known interpolation methods can be used. In particular, it is preferred that the interpolation be selected from the group consisting of linear interpolation, quadratic interpolation, cubic interpolation, higher-order polynomial interpolation, piecewise interpolation, spline interpolation, trigonometric interpolation, and logarithmic interpolation.

[0031] According to an advantageous development of the invention, the interpolation is based on a one-dimensional interpolation or a two-dimensional interpolation.

[0032] According to an advantageous development of the invention, a direction estimate is determined which indicates an estimate for the direction of an image structure within the environment of the correction target pixel, and the interpolation is based on a one-dimensional interpolation in the direction of the image structure indicated by the direction estimate. The inventors have found that in images, one-dimensional interpolation often produces better results than two-dimensional interpolation, provided the one-dimensional interpolation is performed in the direction of an image structure present in the image. Such image structures can consist, for example, of edges of objects, as occur, for example, in the outer contour of letters, or in lines that, for example, make up a barcode.In such a case, the interpolation result is better if the interpolation is performed parallel to the edge or lines of the barcode, for example, than in another direction, especially orthogonal to it. The direction estimate serves as an indicator of whether, for example, a predominantly horizontal, vertical, or ascending or descending diagonal image structure is present in the vicinity of the correction target pixel.

[0033] According to an advantageous development of the invention, the direction estimate is determined based on a comparison of a weighted sum of the amounts of pairwise differences of the values ​​of pixels of the environment of the correction target pixel arranged in a first direction with a weighted sum of the amounts of pairwise differences of the values ​​of pixels of the environment of the correction target pixel arranged in a second direction.

[0034] According to an advantageous development of the invention, the directional estimate is determined based on a comparison of weighted sums of the absolute values ​​of pairwise differences of the values ​​of pixels in the vicinity of the correction target pixel arranged in a plurality of directions relative to one another. A plurality of estimates are determined based on one-dimensional interpolations in the plurality of directions. The estimate is based on a weighted average of the plurality of estimates. The weighted average is based on weight values ​​for the plurality of estimates, the weight values ​​being based on the directional estimate. This type of processing offers the advantage of avoiding the occurrence of blink artifacts in image sequences.

[0035] According to a further aspect of the invention, a correction device for correcting a value of a correction target pixel of an image of an electronic camera is provided, the correction device being designed to: - Determining a corrected value of the correction target pixel by means of filtering based on a weighted averaging of the value of the correction target pixel and an estimated value, wherein the weighted averaging is based on weight values ​​for the correction target pixel and for the estimated value, wherein the weight values ​​are based on an inhomogeneity value determined based on a deviation of the value of the correction target pixel from at least one ambient value based on values ​​of a plurality of pixels in an ambient of the correction target pixel.

[0036] According to a further aspect of the invention, an electronic camera is provided, the electronic camera comprising: - an image sensor with pixels for generating an image; and - the correction device according to claim 17 for correcting a value of a correction target pixel of the image.

[0037] According to a further aspect of the invention, a computer device is provided, wherein the computer device comprises a computing unit configured to carry out the correction method according to one of claims 1 to 16.

[0038] According to a further aspect of the invention, a computer program product is provided, the computer program product comprising code means for causing a computer device to execute the correction method according to any one of claims 1 to 16 when the computer program product is executed on the computer device.

[0039] It is understood that the correction method according to claim 1, the correction device according to claim 17, the electronic camera according to claim 18, the computer device according to claim 19 and the computer program product according to claim 20 have similar and / or identical preferred embodiments, in particular as defined in the dependent claims.

[0040] It is understood that a preferred embodiment of the invention may also be any combination of the dependent claims with the corresponding independent claim. BRIEF DESCRIPTION OF THE DRAWINGS

[0041] Preferred embodiments of the invention are described in more detail below with reference to the accompanying figures, wherein Fig. 1 shows schematically and exemplarily the structure and operation of an electronic camera, Fig. 2 schematically and exemplarily shows a pixel and a 5x5 environment of a pixel, Fig. 3 schematically and exemplarily shows a correction method for correcting a value of a correction target pixel of an image of an electronic camera according to an embodiment of the invention, Fig. 4 schematically and exemplarily shows values ​​for an otherwise homogeneous image area in which a pixel assumes an increased value due to an inhomogeneity, Fig. 5 shows schematically and exemplary values ​​for an image area that depicts a point-like bright spot, a so-called highlight, without inhomogeneity, Fig. 6 schematically and exemplarily shows the determination of extrapolation values ​​based on a plurality of pixels in the vicinity of the correction target pixel, Fig. 7 schematically and exemplarily shows pixels of the plurality of pixels in the vicinity of the correction target pixel, which run in different directions directly towards the correction target pixel, Fig. 8 schematically and exemplarily shows the increase of the first value by the first offset value or the decrease of the second value by the second offset value, Fig. 9 schematically and exemplarily shows details of the determination of the inhomogeneity value according to an embodiment of the invention, Fig. 10 schematically and exemplarily shows the determination of the weighted sum of the amounts of pairwise differences of the values ​​of horizontally arranged pixels, and Fig. 11 schematically and exemplarily shows details of the determination of the estimated value according to an embodiment of the invention. DETAILED DESCRIPTION OF THE EMBODIMENTS

[0042] In the figures, identical or corresponding elements or units are provided with identical or corresponding reference symbols. If an element or unit has already been described in connection with one figure, a detailed description may be omitted in connection with another figure. The terms "left," "right," "top," "bottom," etc., used within the description of the figures refer to the corresponding figure in an orientation with a normally legible figure name and reference symbols.

[0043] The structure and functionality of an electronic camera, e.g. a digital industrial camera, are shown schematically and exemplarily in the Fig. 1 and described in more detail, for example, in DE 10 2013 000301 A1. The camera 10 usually has an imaging optics 22, e.g., a lens. With the imaging optics, a scene is imaged as an image 30 onto an image sensor 31. The image sensor 31 has a regular arrangement of pixels, which is referred to as a pixel array. This is usually designed as a two-dimensional square grid. With the help of the pixels, the image sensor 31 generates pixel values, which in their entirety describe the image 30 as an image signal. Furthermore, the camera 10 usually comprises an image processing unit 32, which can include, for example, an FPGA, a processor (CPU), a microcontroller (MCU), an embedded processor, a graphics processor (GPU), and optionally also so-called accelerators. The image sensor 31 transmits the image signal in electrical form, e.g.analog or digital to the image processing unit 32, which creates an image 33 that is usually corrected in many respects and makes it available in electrical form for further processing via an interface 34.

[0044] Such cameras are often used to capture individual images of objects, for example, when materials or products are visually inspected for compliance with quality criteria in automated mass production. Such cameras are also used to capture sequences of images, for example, showing moving objects.

[0045] It is known, for example, from the EMVA Standard 1288, that image sensors often exhibit inhomogeneities. For example, the electrical properties of a pixel P22 differ significantly from the electrical properties of the pixels in the surrounding area of ​​the pixel, e.g. Fig. 2 schematically and exemplarily shown 5x5 environment P00, P01, P02, P03, P04, P10, P11, P12, P13, P14, P20, P21, P23, P24, P30, P31, P32, P33, P34, P40, P41, P42, P43 and P44.

[0046] Numerous inhomogeneities (non-uniformity) can be considered as causes. Common causes include dark signal non-uniformity (DSNU), photoresponse non-uniformity (PRNU), and dark current non-uniformity (DCNU). The latter effect is particularly important in today's widely used image sensors, especially for images taken in low-brightness scenes with long exposure times and / or high camera temperatures, e.g., in microscopy. Another important cause of inhomogeneities is radio telegraph signal noise (RTS noise). This is pixel-by-pixel noise caused by random jumps in the offset value of a pixel.

[0047] There are effects that affect the majority of an image sensor's pixels, such as DSNU and PRNU. Such effects produce a generally homogeneous spatial noise, referred to here as normal image noise. It reduces the perceived image quality, but usually does not lead to individual pixels contributing to these effects being perceived as individually disturbing.

[0048] In contrast, deviations caused by DCNU or RTS noise occur less frequently and usually affect individual pixels. Nobukazu Teranishi ("Dark Current and White Blemish," Image Sensors, London, 2012) and Wang et al. ("Random Telegraph Signal in CMOS Image Sensor Pixels," 2006) identified defects in the semiconductor crystal structure of the image sensor as the most likely causes.

[0049] DCNU-related deviations usually result in pixel values ​​that appear slightly brighter compared to pixel values ​​in the local environment. It is technically possible for pixels to assume a maximum possible value, but slight increases in brightness are much more common. Furthermore, due to the stochastic nature of the dark current, the deviation is subject to temporal fluctuation, which makes a so-called static correction with a pre-calculated correction value difficult. Image sensors are exposed to high-energy cosmic radiation, particularly during transport in aircraft, but also during normal storage and operation. This radiation can lead to further defects in the semiconductor structure and thus to a change in the dark current distribution. The number of pixels affected by a high dark current then increases.Therefore, a correction based on determining affected pixels at a specific point in time, for example, during production, can often only partially correct DCNU-related errors. It is, so to speak, blind to deviations that arise later.

[0050] Furthermore, the magnitude of a DCNU-related deviation usually depends linearly on exposure time and exponentially on temperature. This further complicates a statistical determination of the magnitude of a deviation.

[0051] RTS noise, in turn, is usually caused by defects in the crystal lattice of the image sensor's semiconductor crystal, which occur below the gate electrode of an amplifier transistor in the pixel. In image sensors that use correlated double sampling (CDS), as is often the case today, this results in the brightness of affected pixels deviating, with a certain probability, upwards or downwards by a certain amount superimposed with temporal noise. Here, too, determining affected pixels at a specific point in time, for example, during production, is virtually impossible, since the finite probability of occurrence makes it impossible to guarantee that the defect will actually occur at the time of detection.

[0052] Fig. 3 shows schematically and by way of example a correction method for correcting a value of a correction target pixel P22 of an image 30 of an electronic camera 10 according to an embodiment of the invention, the correction method comprising: - Determining a corrected value F of the correction target pixel P22 by means of filtering based on a weighted averaging of the value of the correction target pixel P22 and an estimated value S, wherein the weighted averaging is based on weight values ​​G for the correction target pixel P22 and for the estimated value S, wherein the weight values ​​G are based on an inhomogeneity value I which is determined based on a deviation of the value of the correction target pixel P22 from at least one ambient value which is based on values ​​of a plurality of pixels of an ambient U of the correction target pixel P22.

[0053] The correction process can be performed, for example, by the image processing unit 32 of the camera 10 described above. Alternatively, the correction process can also be performed outside the camera 10, e.g., on an external computer, a mobile phone, or specially designed hardware.

[0054] If the inhomogeneity value I indicates a smaller inhomogeneity, the weight values ​​G are chosen such that the value of the correction target pixel P22 is weighted more heavily than if the inhomogeneity value I indicates a larger inhomogeneity, and the estimated value S is weighted less heavily than if the inhomogeneity value I indicates a larger inhomogeneity.

[0055] In this embodiment, the weighted averaging is carried out continuously depending on the inhomogeneity value I. In particular, the inhomogeneity value I has a value range from a minimum value, which indicates a minimum inhomogeneity, to a maximum value, which indicates a maximum inhomogeneity. If the inhomogeneity value I is equal to the minimum value, the value of the correction target pixel P22 is given the maximum weight, here with a weight value G equal to 1, and the value of the estimated value S is given the minimum weight, here with a weight value G equal to 0. If the inhomogeneity value I is equal to the maximum value, the value of the correction target pixel P22 is given the minimum weight, here with a weight value G equal to 0, and the value of the estimated value S is given the maximum weight, here with a weight value G equal to 1.In this embodiment, the minimum value is 0 and the maximum value is 1, where the weight value G for the correction target pixel P22 is 1 minus the inhomogeneity value I, and the weight value G for the estimated value S is equal to the inhomogeneity value I. In other embodiments, however, the minimum value and the maximum value may also have other values.

[0056] In this embodiment, the at least one ambient value comprises a first value and a second value, wherein the inhomogeneity value I is determined based on a comparison of the value of the correction target pixel P22 with the first value and a comparison of the value of the correction target pixel P22 with the second value. In particular, if the value of the correction target pixel P22 is less than or equal to the first value, and if the value of the correction target pixel P22 is greater than or equal to the second value, the inhomogeneity value I indicates that the correction target pixel P22 does not have any inhomogeneity.

[0057] Fig. 9 shows schematically and exemplarily details of the determination of the inhomogeneity value I according to an embodiment of the invention.

[0058] In this embodiment, the first value KM is determined based on two values, namely the m1-t largest value UM of the pixels of the plurality of pixels U_i of the environment U of the correction target pixel P22 and the n1-t largest extrapolation value EM of at least two extrapolation values ​​E_i, which are determined based on the plurality of pixels U_i of the environment U of the correction target pixel P22 (see also Fig. 6). Similarly, the second value Km is also determined based on two values, namely the m2-t smallest value Um of the plurality of pixels U_i of the neighborhood U of the correction target pixel P22 and the n2-t smallest extrapolation value Em of the at least two extrapolation values ​​E_i. Here, m1, m2, n1, and n2 are natural numbers and are preferably equal to 1 or 2.

[0059] In other embodiments, the first value may also be based only on the m1-t largest value UM of the pixels of the plurality of pixels U_i of the environment U of the correction target pixel P22 or only on the n1-t largest extrapolation value EM of the at least two extrapolation values ​​E_i. Correspondingly, the second value may also be based only on the m2-t smallest value Um of the pixels of the plurality of pixels U_i of the environment U of the correction target pixel P22 or only on the n2-t smallest extrapolation value Em of the at least two extrapolation values ​​E_i.

[0060] As described, the values ​​of pixels with highly inhomogeneous properties often deviate solitarily from the values ​​of the surrounding pixels. Fig. Figure 4 schematically shows exemplary values ​​for an otherwise homogeneous image area in which a pixel P22 assumes an increased value due to an inhomogeneity. The inhomogeneity-based increase in the value of pixel P22 has no effect on neighboring pixels P20, P21, P23, and P24.

[0061] In contrast to the Fig. 4 shows the Fig. 5 schematic and exemplary values ​​for an image area that depicts a point-like bright spot, a so-called highlight, without inhomogeneity. In current digital cameras, the geometric dimension of the pixels is usually on the order of magnitude of the resolution of the imaging optics or less. As a result, light from a bright spot in the scene not only reaches a single pixel in isolation, but also affects one or more neighboring pixels.

[0062] If the first value is based only or also on the m1-t largest value UM of the pixels of the majority of pixels U_i in the vicinity U of the correction target pixel P22, and the second value is based only or also on the m2-t smallest value Um of the pixels of the majority of pixels U_i in the vicinity U of the correction target pixel P22, a high-quality correction can be performed in both cases. While in the first case, filtering leads to a substantial change in the value of the correction target pixel P22 compared to the original value, in the second case, filtering leads to no or only a slight change in the filtered value.

[0063] In this embodiment, the at least two extrapolation values ​​E_i 8 comprise extrapolation values ​​that are each determined based on values ​​of pixels of the plurality of pixels of the environment U of the correction target pixel P22 that converge in different directions directly toward the correction target pixel P22. This is shown schematically and exemplarily in the Fig. 7. The different directions include horizontal from left, horizontal from right, vertical from top, vertical from bottom, diagonal from top left, diagonal from bottom right, diagonal from top right, and diagonal from bottom left.

[0064] In this embodiment, the 8 extrapolation values ​​E_i are determined based on linear interpolation.A first extrapolation value E_1 in the horizontal direction from the left can then be determined as E_1 = 2 · P21 - P20, a second extrapolation value E_2 in the horizontal direction from the right can be determined as E_2 = 2 · P23 - P24, a third extrapolation value E_3 in the vertical direction from above can be determined as E_3 = 2 · P12 - P02, a fourth extrapolation value E_4 in the vertical direction from below can be determined as E_4 = 2 · P32 - P42, a fifth extrapolation value E_5 in the diagonal direction from the top left can then be determined as E_5 = 2 · P11 - P00, a sixth extrapolation value E_6 in the diagonal direction from the bottom right can be determined as E_6 = 2 · P33 - P44, a seventh extrapolation value E_7 in the diagonal direction from the top right can be determined as E_7 = 2 · P13 - P04 can be determined and an eighth extrapolation value E_8 in the direction diagonal from the bottom left can be determined as E_8 = 2 · P31 - P40.

[0065] In other embodiments, the at least two extrapolation values ​​may also be determined based on another extrapolation, e.g., a quadratic extrapolation, a cubic extrapolation, a higher-order polynomial extrapolation, a trigonometric extrapolation, or a logarithmic extrapolation.

[0066] In this embodiment, the m1-t largest value UM of the pixels of the plurality of pixels U_i of the environment U of the correction target pixel P22 and the n1-t largest extrapolation value EM of the at least two extrapolation values ​​E_i are additionally increased by a first offset value O. In a corresponding manner, the m2-t smallest value Um of the pixels of the plurality of pixels U_i of the environment U of the correction target pixel P22 and the n2-t smallest extrapolation value EM of the at least two extrapolation values ​​E_i are additionally reduced by a second offset value O, wherein the first offset value O is equal to the second offset value O (cf. Fig. 9).

[0067] Alternatively, in this embodiment it is also possible to first increase the values ​​KM or Km by the first offset value O or to decrease them by the second offset value O.

[0068] It is preferred that the first offset value O and the second offset value O are selected such that they are in the order of magnitude of the spatial noise naturally occurring when capturing images with the electronic camera, e.g. due to DSNU and PRNU, and / or temporal noise, e.g. due to photon and / or readout noise. This is shown schematically and exemplarily in the Fig. 8, where in the figure the first offset value O and the second offset value O are the same here.

[0069] Fig. 11 shows schematically and exemplarily details of the determination of the estimated value S according to an embodiment of the invention.

[0070] In this embodiment, the estimated value S is determined based on an interpolation of the values ​​of pixels of the environment U of the correction target pixel P22, wherein the interpolation is based on a one-dimensional interpolation.

[0071] In this case, a direction estimate R is determined which indicates an estimate for the direction of an image structure within the environment U of the correction target pixel P22, and the interpolation is based on a one-dimensional interpolation in the direction of the image structure indicated by the direction estimate R.

[0072] In this embodiment, the direction estimation value R is determined based on a comparison of a weighted sum of the amounts of pairwise differences of the values ​​of pixels of the environment U of the correction target pixel P22 arranged in a first direction with a weighted sum of the amounts of pairwise differences of the values ​​of pixels of the environment U of the correction target pixel P22 arranged in a second direction.

[0073] In this embodiment, the directional estimate R for the one-dimensional interpolation is determined such that the sum of the absolute values ​​of the pixel value differences is smaller in the interpolation direction than in other directions. Alternatively, it is also possible to determine the directional value R for the interpolation orthogonal to the direction in which the sum of the absolute values ​​of the pixel value differences is particularly high.

[0074] For example, as in the Fig. 10 schematically and exemplarily shown, the weighted sum of the amounts of pairwise differences of the values ​​of horizontally arranged pixels can be determined as: D1=1 / 8⋅(1⋅|P11−P12|+1⋅|P12−P13|+2⋅|P20−P21|+2⋅|P23−P24|+1⋅|P31−P32|+1⋅|P32−P33|)

[0075] In a corresponding manner, the weighted sum of the magnitudes of pairwise differences of the values ​​of vertically arranged pixels can be determined, for example, in a manner rotated by 90° as: D2=1 / 8⋅(1⋅|P11−P21|+1⋅|P21−P31|+2⋅|P02−P12|+2⋅|P32−P42|+1⋅|P13−P23|+1⋅|P32−P33|)

[0076] Similarly, weighted sums of the absolute values ​​of the differences between pixels arranged diagonally or ascendingly can be determined. The weighted sum of the absolute values ​​of pairwise differences between pixels arranged diagonally and ascendingly can be determined, for example, as: D3=1 / 4⋅(1⋅|P40−P31|+1⋅|P21−P12|+1⋅|P32−P23|+1⋅|P13−P04|) and the weighted sum of the magnitudes of pairwise differences of the values ​​of pixels arranged diagonally to each other can be determined as: D4=1 / 4⋅(1⋅|P00−P11|+1⋅|P12−P23|+1⋅|P21−P32|+1⋅|P33−P44|)

[0077] The determination of the directional estimate R can then be based, for example, on the comparison of two or more of D1, D2, D3 and D4.

[0078] In this embodiment, the direction estimate R is determined based on a comparison of weighted sums D1, D2, D3, D4 of the amounts of pairwise differences of the values ​​of pixels of the environment U of the correction target pixel P22 arranged in a plurality of directions relative to one another, wherein a plurality of estimates S1, S2, S3, S4 are determined based on one-dimensional interpolations in the plurality of directions, wherein the estimate S is based on a weighted average of the plurality of estimates S1, S2, S3, S4, wherein the weighted average is based on weight values ​​for the plurality of estimates S1, S2, S3, S4, wherein the weight values ​​are based on the direction estimate R. The plurality of weighted sums D1, D2, D3, D4 can be, for example, the D1, D2, D3 and D4 described above.

[0079] In the embodiments described above, the inhomogeneity value I based on the deviation of the value of the correction target pixel P22 from the at least one ambient value can be additionally scaled with a scaling factor, e.g., by multiplying its value by a factor during determination or dividing it by a reciprocal factor. Scaling allows the behavior of the filter to be adjusted such that, on the one hand, the pixels that are conspicuously inhomogeneous are adequately corrected, and, on the other hand, the pixels that are not conspicuously inhomogeneous experience little or no change from the filter.

[0080] While in the embodiments described above the interpolation is based on one-dimensional interpolation, in other embodiments the interpolation can also be based on two-dimensional interpolation. Alternatively, it is also possible for the estimated value S to be determined based on median or mean values ​​of the pixels in the vicinity U of the correction target pixel P22.

[0081] The embodiments described above relate to situations in which inhomogeneities can lead to both an increase and a decrease in a pixel value. In other embodiments, however, it is also possible for inhomogeneities to act in only one direction, i.e., to lead only (or essentially only) to an increase or only (or essentially only) to a decrease in the pixel value. The value of the correction target pixel (P22) can then be compared, for example, only with the first value or only with the second value.

[0082] In practice, color image sensors are also frequently used, which have a so-called mosaic filter, in which pixels are each provided with a color filter of one of a plurality of colors. An example of such a mosaic filter is provided by US 3,971,065 A, which proposes an arrangement of pixels referred to as the "Bayer Pattern" and is very frequently used. In such mosaic filters, the pixels form color planes with color filters of the same color, as is the case, for example, in the Fig.1A of US Pat. No. 3,971,065 A. In an advantageous refinement of such filters, the filtering can be performed in such a way that it is based only on those pixels in the surrounding area that belong to the same color plane as the correction target pixel, i.e., whose color filter has the same color as the color filter of the correction target pixel. This avoids disruptive influences of possible intensity differences between the different color planes caused by the color.

[0083] In order to have access to a sufficiently large number of input pixels, a 7x7 environment can preferably be used.

[0084] Furthermore, filtering can also be used for color images containing multiple color planes (color channels) by applying the filtering separately to at least one color plane. Such images can be captured, for example, with so-called three-chip cameras, in which a set of prisms separates the images of the three different color channels and projects them onto different image sensors, as described, for example, in FR 2 386 220 A1. Images with multiple color planes can also be obtained, for example, through color interpolation or demosaicing. An example of this can be found in DE 10 2013 000301 A1.

[0085] Further variations of the disclosed embodiments may be understood and practiced by one skilled in the art practicing the claimed invention from a consideration of the drawings, the specification, and the appended claims.

[0086] In the claims, the words “comprise” and “comprising” do not exclude other elements or steps, and the indefinite article “a” does not exclude a plurality.

[0087] A single unit or device can perform the functions of multiple elements listed in the claims. The fact that individual functions and / or elements are listed in different dependent claims does not mean that a combination of these functions and / or elements could not also be used to advantage.

[0088] The reference signs in the claims are not to be understood in such a way that the subject matter and the scope of protection of the claims are restricted by these reference signs.

[0089] In summary, a correction method for correcting a value of a pixel of an image of an electronic camera has been provided, the correction method comprising: - Determining a corrected value of the correction target pixel by means of filtering based on a weighted averaging of the value of the correction target pixel and an estimated value, wherein the weighted averaging is based on weight values ​​for the correction target pixel and for the estimated value, wherein the weight values ​​are based on an inhomogeneity value determined based on a deviation of the value of the correction target pixel from at least one ambient value based on values ​​of a plurality of pixels in an ambient of the correction target pixel.

Claims

[1] A correction method for correcting a value of a correction target pixel (P22) of an image (30) of an electronic camera (10), the correction method comprising: - determining a corrected value (F) of the correction target pixel (P22) by means of a filtering based on a weighted average of the value of the correction target pixel (P22) and an estimated value (S), wherein the weighted average is based on weight values ​​(G) for the correction target pixel (P22) and for the estimated value (S), - wherein the weight values ​​(G) are based on an inhomogeneity value (I) which is determined based on a deviation of the value of the correction target pixel (P22) from at least one ambient value which is based on values ​​of a plurality of pixels (U_i) of an environment (U) of the correction target pixel (P22). [2] Correction method according to claim 1, wherein, when the inhomogeneity value (I) indicates a smaller inhomogeneity, the weight values ​​(G) are selected such that the value of the correction target pixel (P22) is weighted more heavily than when the inhomogeneity value (I) indicates a larger inhomogeneity, and the estimated value (S) is weighted less heavily than when the inhomogeneity value (I) indicates a larger inhomogeneity. [3] Correction method according to claim 1 or 2, wherein the weighted averaging is carried out continuously as a function of the inhomogeneity value (I). [4] Correction method according to one of claims 1 to 3, wherein the at least one ambient value comprises a first value and / or a second value, wherein the inhomogeneity value (I) is determined based on a comparison of the value of the correction target pixel (P22) with the first value and / or a comparison of the value of the correction target pixel (P22) with the second value. [5] The correction method according to claim 4, wherein, when the value of the correction target pixel (P22) is less than or equal to the first value, and / or when the value of the correction target pixel (P22) is greater than or equal to the second value, the inhomogeneity value (I) indicates that the correction target pixel (P22) has no inhomogeneity. [6] Correction method according to claim 4 or 5, wherein the first value is determined based on the m1-t largest value (UM) of the pixels of the plurality of pixels (U_i) of the environment (U) of the correction target pixel (P22) and / or the second value is determined based on the m2-t smallest value (Um) of the pixels of the plurality of pixels (U_i) of the environment (U) of the correction target pixel (P22), wherein m1 and m2 are natural numbers and are preferably equal to 1 or 2. [7] Correction method according to claim 4 or 5, wherein the first value is determined based on the n1-t largest extrapolation value (EM) of at least two extrapolation values ​​(E_i) determined based on the plurality of pixels (U_i) of the environment (U) of the correction target pixel (P22) and / or the second value is determined based on the n2-t smallest extrapolation value (Em) of the at least two extrapolation values ​​(E_i), where n1 and n2 are natural numbers and are preferably equal to 1 or 2. [8] Correction method according to claim 7, wherein the at least two extrapolation values ​​(E_i) are each determined based on values ​​of pixels of the plurality of pixels of the environment (U) of the correction target pixel (P22) which converge in different directions directly towards the correction target pixel (P22). [9] Correction method according to claim 6 and one of claims 7 or 8, wherein the first value is determined based on the m1-t largest value (UM) of the pixels of the plurality of pixels (U_i) of the environment (U) of the correction target pixel (P22) and the n1-t largest extrapolation value (EM) of the at least two extrapolation values ​​(E_i) and / or the second value is determined based on the m2-t smallest value (Um) of the pixels of the plurality of pixels (U_i) of the environment (U) of the correction target pixel (P22) and the n2-t smallest extrapolation value (Em) of the at least two extrapolation values ​​(E_i). [10] Correction method according to one of claims 6 to 9, wherein the m1-t largest value (UM) of the pixels of the plurality of pixels (U_i) of the environment (U) of the correction target pixel (P22) and / or the n1-t largest extrapolation value (EM) of the at least two extrapolation values ​​(E_i) is additionally increased by a first offset value (O) and / or the m2-t smallest value (Um) of the pixels of the plurality of pixels (U_i) of the environment (U) of the correction target pixel (P22) and / or the n2-t smallest extrapolation value (EM) of the at least two extrapolation values ​​(E_i) is additionally reduced by a second offset value (O), wherein the first offset value (O) is equal to or different from the second offset value (O). [11] Correction method according to one of claims 1 to 10, wherein the inhomogeneity value (I) based on the deviation of the value of the correction target pixel (P22) from the at least one ambient value is additionally scaled with a scaling factor. [12] Correction method according to one of claims 1 to 11, wherein the estimated value (S) is determined based on an interpolation, a median or an averaging of the values ​​of pixels of the environment (U) of the correction target pixel (P22). [13] The correction method according to claim 12, wherein the interpolation is based on a one-dimensional interpolation or a two-dimensional interpolation. [14] Correction method according to claim 13, wherein a direction estimate (R) is determined which indicates an estimate for the direction of an image structure within the vicinity (U) of the correction target pixel (P22), and the interpolation is based on a one-dimensional interpolation in the direction of the image structure indicated by the direction estimate (R). [15] Correction method according to claim 14, wherein the direction estimate (R) is determined based on a comparison of a weighted sum of the amounts of pairwise differences of the values ​​of pixels of the environment (U) of the correction target pixel (P22) arranged in a first direction with a weighted sum of the amounts of pairwise differences of the values ​​of pixels of the environment (U) of the correction target pixel (P22) arranged in a second direction. [16] Correction method according to claim 14 or 15, wherein the direction estimate (R) is determined based on a comparison of weighted sums (D1, D2, D3, D4) of the amounts of pairwise differences of the values ​​of pixels of the environment (U) of the correction target pixel (P22) arranged in a plurality of directions relative to one another, wherein a plurality of estimates (S1, S2, S3, S4) are determined based on one-dimensional interpolations in the plurality of directions, wherein the estimate (S) is based on a weighted average of the plurality of estimates (S1, S2, S3, S4), wherein the weighted average is based on weight values ​​for the plurality of estimates (S1, S2, S3, S4), wherein the weight values ​​are based on the direction estimate (R). [17] Correction device for correcting a value of a correction target pixel (P22) of an image (30) of an electronic camera (10), the correction device being designed to: - determining a corrected value (F) of the correction target pixel (P22) by means of a filtering based on a weighted average of the value of the correction target pixel (P22) and an estimated value (S), wherein the weighted average is based on weight values ​​(G) for the correction target pixel (P22) and for the estimated value (S), wherein the weight values ​​(G) are based on an inhomogeneity value (I) determined based on a deviation of the value of the correction target pixel (P22) from at least one ambient value based on values ​​of a plurality of pixels (U_i) of an ambient (U) of the correction target pixel (P22). [18] Electronic camera (10), the electronic camera (10) comprising: - an image sensor (31) with pixels for generating an image (30); and - the correction device according to claim 17 for correcting a value of a correction target pixel (P22) of the image (30). [19] Computer device comprising a computing unit designed to carry out the correction method according to one of claims 1 to 16. [20] A computer program product comprising code means for causing a computer device to execute the correction method according to any one of claims 1 to 16 when the computer program product is executed on the computer device.

Images