Method for image correction of image distribution in digital image recording

Abstract

In order to carry out an improved image correction, it is provided that a number of NM multiplication influence factors and a number of NC convolution influence factors are taken into account in the image recording (1), said multiplication influence factors describing multiplication factors between image points (p) in the interference-reduced image (U) and the same image points (p) in the interference-affected image (F), each multiplicative influence factor is described in a vector-matrix representation as a single multiplicative matrix (Mi) in the form of a diagonal matrix having matrix elements for each pixel (p) on a diagonal line, and the multiplicative matrix M is determined as a matrix product of the single multiplicative matrixes (Mi) from a number NM of the single multiplicative matrixes (Mi), and wherein the convolution influence factors describe the influence of a plurality of image points (p) in the interference reduction image (U) on an image point (p) in the interfered image (F), and each convolution influence factor is described in the form of a single convolution matrix (Ci), each row of individual convolution matrices (Ci) describes how an image point (p) in the interfered image (F) is affected by a respective image point (p) in the interference-reducing image (U), and a convolution matrix C is determined as a matrix product of the individual convolution matrices (Ci) from a number of NC individual convolution matrices (Ci), and wherein the matrix product of the multiplication matrix M and the convolution matrix C is used as a correction operator (A) for image correction, i.e. A = MC.

Description

[0001] The present invention relates to a method for correcting a disturbed image F of an object to an interference-reduced image U of the object, wherein the image recording of the object is recorded (photographed) with an image sensor of a digital camera, and the image recording consists of a plurality (n) of image points, and contains a disturbed image F of the object due to various influencing factors, wherein the relationship between the recorded disturbed image F of the object and the interference-reduced image U of the object is described in a vector-matrix representation with an image interference operator A, wherein the disturbed image F and the interference-reduced image U are described as vectors comprising the image points, and the image interference operator A is described as a matrix of dimension n x n, and the interference-reduced image U is obtained from the recorded disturbed image F by means of an optimization with a predefined target function θ1 which evaluates the deviation between the disturbed image F and the transformed interference-reduced image AU.

[0002] In digital image recordings, unavoidable image interferences occur in the recorded image due to various influencing factors. Thus, a disturbed image is produced by the image recording, wherein the interference exists, for example, in the form of an image blur. In digital cameras, an image sensor with a light detector array is provided for each channel of the image recording (for example, 3 channels in an RGB sensor), wherein each light detector represents an image point (pixel). Commonly used image sensors are known CCD sensors or CMOS sensors. Image interferences can result from the properties of the optical system or the image sensor, but also from the movement of the camera or the recorded object during the exposure time. Thus, various image corrections are usually applied to the recorded image in order to correct the image interferences, wherein correction here usually only means reducing the image interferences, but not necessarily completely eliminating them.

[0003] For example, when recording an image with a camera from a moving vehicle, an image interference in the form of an image blur (so-called motion blur) occurs due to the movement of the vehicle during the exposure time. The image blur is produced because the camera moves relative to the object to be recorded during the exposure time, and thus one image point (in the case of digital recording: a pixel) of the camera is aligned with a plurality of different points on the object during the exposure time. The relative movement between the object to be recorded and the camera can be based on the movement of the vehicle, on the movement of the camera relative to the vehicle, on the movement of the object, or any combination of these movements. However, even if the camera is not installed in a moving vehicle, a relative movement between the camera and the object can occur, which likewise leads to a motion blur in the image. With significant technical effort during the image recording, this motion blur in the image can be limited, but not completely avoided.

[0004] In order to reduce motion blur, it is known, for example, to move the image sensor of the camera during the exposure time by means of a drive in coordination with the movement of the vehicle, i.e. for example with the airspeed of the aircraft or other flying device, so that each pixel of the image sensor remains as precisely as possible in alignment with a specific point of the object. This is also referred to as forward motion compensation. However, only the known forward movement of the vehicle can be compensated in this way. Other movements and accelerations, in particular the pitch, yaw or roll of the vehicle, as can occur in the case of an aircraft, for example in the case of a passage through turbulence, or as a result of vibrations, cannot be compensated in this way. In addition to this, forward motion compensation naturally increases the complexity and cost of the camera system. To some extent, disturbing and unexpected movements of the vehicle or the camera can be compensated by a stabilizing camera suspension, however, this also has technical limitations, so that the motion blur caused by the movement can only be insufficiently or not at all excluded thereby. Furthermore, such a camera suspension also increases the complexity and cost of the camera system.

[0005] In recent times, image sharpening methods have been used extensively, in particular for post-correcting motion blur in images. By means of such image sharpening methods, motion blur in recorded digital images can be compensated for post-hoc. Here, a convolution matrix, often referred to as a "blur kernel", is usually calculated, which maps a sharp image onto a blurred image by a mathematical convolution operation. This approach is based on the idea that the blurred recorded image and the sharp image hidden therein are connected by this blur kernel. If the blur kernel is known, the sharp image can then be calculated from the blurred image by a mathematical deconvolution operation (inverse operation of the convolution). The fundamental problem here is that the blur kernel is usually not known. In some methods, the blur kernel is derived from the blurred image, which is also referred to as blind deconvolution. In other methods, the blur kernel is determined from the known movement of the camera relative to the recorded object, which is also referred to as non-blind deconvolution. In order to sense the movement of the camera, suitable sensors, for example acceleration sensors, gyroscopic sensors, inertial sensors, etc., can be used on the camera itself or on the moving component in which the camera is located, for example the vehicle. By means of the known movement of the vehicle in a geographical coordinate system and, if necessary, by means of the known movement of the camera relative to the vehicle, the relative movement of the camera with respect to the object can be inferred, for example.

[0006] For example, a method for correcting motion blur in an image is known from WO 2023 / 186848 A1, wherein the method is unable to correct other image disturbances in the disturbed image.

[0007] However, in addition to motion blur, there are a series of other influencing factors which affect the recorded image with a camera and lead to image disturbances in the recorded image. Here, the recorded image can be influenced by the camera optical system, the image sensor or noise effects. In the field of the optical system, for example vignetting or lens blur are influencing factors for disturbances. In the field of the image sensor, for example different pixel offsets or different pixel gains of individual pixels are influencing factors for disturbances. Of course, effects such as saturation of individual pixels, defective pixels, etc. also have a disturbing influence on the recorded image. Furthermore, in photography, white balance and brightness adjustment are also common corrections. These image disturbances are also attempted to be corrected mathematically in order to obtain a disturbance-reduced image.

[0008] The quality requirements for image recordings are very high, mainly in the field of photogrammetry, for example for applications in the field of geoinformatics. It is important, for example, that the position, pose and contours of the recorded objects in the recorded image are not shifted or blurred. In such applications, image recordings are often made with moving vehicles, usually with aircraft, which can lead to the above-mentioned motion blur.

[0009] In order to improve the quality of digital image recordings, various image corrections are usually applied to digital image recordings today in order to at least reduce, ideally completely compensate, the image disturbances in the image recordings due to the various influencing factors. Common image corrections are the correction of varying pixel offsets and varying pixel gains, the correction of defective or saturated pixels, the correction of vignetting, white balance, brightness adjustment, the correction of motion blur, de-noising or the correction of lens blur.

[0010] The different image corrections are applied to the recorded image in succession. This has disadvantages, however. For example, certain image correction methods enhance the noise or image defects in the image data, which can also have been introduced by previous correction steps. Due to the successive correction of the image disturbances caused by the individual influencing factors, the image disturbances in the recorded image are often not corrected accurately enough.

[0011] Therefore, there is a need for improved image corrections of the various disturbing influencing factors in images recorded with digital cameras in order to achieve high-quality image recordings, in particular high-resolution and high-detail-fidelity image recordings.

[0012] This is addressed using the method of the invention as described in claim 1. The advantage of the method according to the invention lies in the use of a unified image interference model that incorporates all influencing factors, thereby ensuring that all factors participate in the optimization process to obtain an interference-reduced image. This results in improved image recording, such as better image sharpness, because the method according to the invention implicitly considers the possible interdependencies between the various correction steps. Fine structures and details are also clearly visible in the interference-reduced image, especially. If noise is taken into account, the method according to the invention achieves better noise control, particularly in both shadow and bright areas of the image recording, while preserving fine structures and details.

[0013] A particular advantage of the method according to the invention is that a point spread function can be used as a single convolution matrix to correct lens blur. Therefore, lens blur can be directly incorporated into the image blur model and considered directly during the optimization process. This results in images where image interference is corrected particularly well.

[0014] Furthermore, the method according to the invention also allows for consideration of other effects, such as invalid pixels in the image recording, or transformations from a particular image sensor type to other specific formats (such as RGB). These other effects thus become part of the unified model.

[0015] A particularly advantageous approach is that the image record is divided into pixel regions, within which spatially invariant image interference caused by convolutional factors is assumed. Consequently, the system of equations derived from the optimization problem simplifies to a system of locally linear equations, which can be solved more efficiently and with less computation.

[0016] The following will refer to Figure 1 and Figure 2 To explain the invention in more detail, Figure 1 and Figure 2 The advantageous design features of the invention are illustrated, illustrative, and non-limiting. The accompanying drawings show:

[0017] Figure 1 The diagram illustrates the interrelationships in the case of image recording of objects, and

[0018] Figure 2 Image correction according to the present invention is illustrated.

[0019] The present invention is based on an image record 1 recorded using a digital camera 3 (specifically, using the image sensor 4 of the digital camera 3), which has a specific image resolution in the form of a plurality of (n) image points (pixels). For example, the image record 1 is recorded from a moving vehicle 5, but this is not a necessary prerequisite for the image correction according to the present invention.

[0020] Figure 1 The image schematically shows a vehicle 5 (here, an air vehicle, such as an airplane or drone), which is in motion (here, on the ground), and an image recording 1 of an object 2 (here, a terrain structure) is being taken using a digital camera 3 mounted on the vehicle 5. Figure 1 The diagram illustrates, by way of example, an image recording 1 recorded by a digital camera 3 from a vehicle 5. Image recording 1 includes an image region Ω containing all the image points p of image recording 1. Image recording 1 is a digital image recording, for example, having a width of B pixels and a height of H pixels (digital image points p), which also defines the image resolution. Image recording 1 is generated on the image plane of the digital camera 3, on which an array of photodetectors is arranged on an image sensor 4, forming the image points p. Image sensor 4 is, for example, an RGB sensor or a Bayer mode sensor.

[0021] The position and orientation of the digital camera 3, and the field of view 8 set by the optical unit 7 (using at least one lens), are correlated with the shape of the object 2 to determine the recording area 6, which in turn... Figure 1 The text is schematically drawn using dashed lines. For clarity, Figure 1 Object 2 is shown as a two-dimensional line. However, it is clear that the recording area 6 is generally a surface that corresponds to a roughly rectangular projection onto object 2.

[0022] Figure 1 The image schematically illustrates a digital image recording 1 made by a digital camera 3 at the indicated location, wherein, in the image region Ω (pixels Ω) ij A point p in the image 1 can be assigned to an object point G in the recording area 6 due to the optical characteristics of the camera system. However, in reality, this assignment is not always explicit, because during the exposure time required for recording, the position of the vehicle 5 and / or camera 3 may continue to move along the direction of movement of the vehicle 5, and / or be affected by other translational and / or rotational motions, such as due to vibration, turbulence, etc. In particular, rotation about an axis of the vehicle coordinate system (in the case of an aircraft, this is often referred to as roll, pitch, and yaw) can cause a very significant displacement of the recording area 6 on the object 2. Here, the position of object point G in the image recording 1 (which corresponds to...) is... Figure 1 The image point p at the start of recording shifts during the exposure time (this corresponds to...). Figure 1 (Image point p' in the image region Ω). In other words, image point p in the image region Ω may receive light from different object points G during exposure. This in particular causes motion blur in the recorded image.

[0023] As mentioned above, other influencing factors may also cause the imaging of object point G to image point p in image record 1 to deviate from theoretical expectations. Therefore, image record 1, recorded by the camera during the exposure time, may also be affected by further or other influencing factors, as described below.

[0024] These influencing factors resulted in image record 1 containing an disturbed image F of the recorded object 2.

[0025] The invention will now be described with respect to an image recording 1 having one channel, without limitation, but it can certainly be extended to multiple channels. For example, for an RGB image sensor, there are three channels (red, green, and blue). A Bayer mode sensor, for example, has one channel, in which different pixels receive light of different wavelengths due to a color filter above the image plane (e.g., still RGB).

[0026] As is well known, in order to correct image interference in image recording 1, the relationship between an unknown, interference-reduced image U and the interfered image F recorded using image recording 1 can be used, and its form is: The image interference operator A models specific image interference. Optionally (in brackets) represents the noise under consideration, such as normally distributed noise additionally recorded by the image sensor 4 of camera 3. It is white noise with a given standard deviation σ.

[0027] This representation can be achieved using a vector-matrix representation, which utilizes vectors F and U containing all n image points p, and a matrix A of dimension n x n. However, other representations can also be used, such as using matrices U and F.

[0028] This method, known in WO 2023 / 186848 A1, is used to correct motion blur, where the image interference operator A is the blur kernel (convolution matrix). The document describes that other camera effects on motion blur (such as vignetting and distortion) can also be considered in the blur kernel. Therefore, this does not address the correction of these effects, but only their influence on the motion blur model. However, this does not correct the effect of vignetting or distortion of the camera 3's optical system on the interfered image, but only corrects the effect of vignetting or distortion on motion blur. Thus, vignetting or distortion in the recorded image is not corrected. WO2023 / 186848 A1 thus describes image correction for image interference caused by a single influencing factor (motion). To correct image interference caused by other influencing factors, other image correction methods must be employed.

[0029] It is also known that, in the case of motion fuzziness, the system of equations (Each pixel p in the image has a corresponding equation) U cannot be solved directly. Therefore, in order to reconstruct the reduced-interference image U from the recorded disturbed image F, a formula of the form is usually constructed. The optimization problem is denoted by θ1, which is an arbitrary suitable objective function used to evaluate the deviation between the disturbed image F and the transformed, disturbance-reduced image AU, and minimizes this deviation to reconstruct the disturbance-reduced image U. θ2 is an optional regularization function used to suppress unwanted image characteristics, such as noise, during the optimization process. Thus, noise can be considered during the optimization process, for example, using the regularization function θ2.

[0030] Typically, optimization is performed iteratively using known optimization algorithms until a pre-defined termination condition is met, such as reaching a certain number of iterations or a certain residual (the deviation between the transformed perturbation-reduced image AU and the perturbed image F). The termination condition may also depend on the solver of the optimization problem.

[0031] The known implementation of this optimization problem is total variational (TV) regularization. The objective function is defined as follows: And the regularization function is defined as It has selectable or pre-given regularization parameters µ and γ, as well as known Nabla operators. . It is the Euclidean norm, and yes Find the summation norm for all image points p. The replacement can be implemented using the known Richardson-Lucy algorithm, as well as other known implementations.

[0032] Solvers for the above optimization problems are well known, such as the ADMM method (alternating direction multiplier method), the Split-Bregmann algorithm, or the Richardson-Lucy algorithm.

[0033] Thus, at least the motion blur in the recorded image F can be reduced so far, and an image U with reduced motion blur can be reconstructed from the recorded image F. However, other interfering factors affecting the recorded image cannot be reduced or even compensated for. Therefore, until now, additional image correction must be applied before or after motion blur correction, but this leads to the disadvantages described at the beginning. To overcome these disadvantages, the present invention employs the following method.

[0034] The various factors causing image interference in the recorded image 1 were categorized into different classes. At least four different classes were identified here: additive factors, multiplicative factors, convolutional factors, and noise factors.

[0035] Additive influencing factors include, for example, pixel offset, which is a known additive offset manifested at each pixel p in image sensor 4. Here, each pixel p in the disturbed image F is changed by an additive value in a defined manner, independent of other pixel p in image recording 1. Therefore, with respect to this influencing factor, the relationship between pixel p in the disturbance-reduced image U and the same pixel p in the disturbed image F is given by an additive addend.

[0036] Multiplicative influencing factors include, for example, white balance, brightness adjustment, vignetting, or pixel gain—that is, any effect that multiplies on pixel p of image sensor 4. Here, each pixel p in the disturbed image F is affected by a multiplicative factor in a defined manner, independent of other pixel p in image record 1. Therefore, with respect to this influencing factor, the relationship between pixel p in the disturbance-reduced image U and the same pixel p in the disturbed image F is given by a multiplicative factor.

[0037] Convolutional influencing factors include, for example, motion blur or lens blur. Convolutional influencing factors describe an effect where a pixel p in the disturbed image F is affected by different pixels p in the disturbance-reduced image U. For example, a pixel p in the disturbed image F is given by a weighted sum of all pixels p in the disturbance-reduced image U. For instance, due to motion or lens blur, a pixel p in the disturbed image may be exposed to different object points G during the exposure time; therefore, this pixel p must be assigned to different pixels p in the disturbance-reduced image to correct the image disturbance.

[0038] The noise influencing factor is the random effect of noise on a pixel p in the recorded disturbed image F (e.g., due to image sensor noise). The noise influencing factor can also be given individually for each pixel p in the disturbed image F (e.g., in the form of a random variable). Accordingly, the basic noise characteristics of pixel p can be given, such as noise that follows a normal distribution.

[0039] Furthermore, according to the present invention, invalid image points p, which do not contain valid information of the recorded image, can also be considered, as further described below. Invalid image points p are, for example, defective or saturated image points of the image sensor 4.

[0040] Additive influence factors can be corrected directly in the disturbed image F, for example, by simply subtracting the pixel offset from the disturbed image F for each pixel p, since this correction does not affect other types of image correction. For example, the calibration of the image sensor 4 used in camera 3 is known to be an additive influence factor. Therefore, according to the invention, additive influence factors are not considered, or it is assumed that they have been considered as needed in the disturbed image F.

[0041] Multiplicative influencing factors (such as white balance, brightness adjustment, vignetting, and pixel gain) can be expressed in the form of Let's consider the multiplicative image interference matrix M. Where N... M This represents the number of multiplicative influencing factors considered, and M... i This represents a single multiplication matrix for each multiplicative factor. In vector-matrix notation, each single multiplication matrix M... i It is a diagonal matrix with dimension n = B x H (i.e., the number of image points p, n). Since each multiplicative influencing factor is multiplicative, the multiplicative image interference matrix is ​​derived from the individual multiplication matrices M. i The matrix product (matrix multiplication) is the product of matrices. Therefore, in this case, the multiplicative image interference matrix M is the product of diagonal matrices of dimension n, and thus it is also a diagonal matrix of dimension n (the number of image points p).

[0042] For a specific digital camera 3 with an optical system and a specific image sensor 4, it can also be assumed that the multiplicative influencing factors are known (e.g., from camera or image sensor calibration).

[0043] Convolutional influencing factors (e.g., motion blur or lens blur) can be expressed in the form of The convolutional image interference matrix is ​​considered. Where N... C This represents the number of convolutional influencing factors considered, and C i A single convolution matrix represents each convolutional influencing factor, such as a fuzzy operator known in the art (e.g., from WO 2023 / 186848 A1). In vector-matrix notation, a single convolution matrix C i Each row describes how a pixel p in the disturbed image F is affected by the disturbance reduction image U. A single convolution matrix C i Each row of the convolutional image interference matrix C is thus filled with non-zero matrix elements, at least partially outside the diagonal. However, the matrix elements in each row may also be, and usually are, zero, because not every pixel p in the interfered image F is affected by all pixels p in the interference-reduced image U. This effect is usually limited to a few adjacent pixels p.

[0044] However, the method of this invention can now also integrate known point spread functions (PSFs) to correct lens blur, because the point spread function, like the blur kernel, can also be represented as a convolution matrix. PSFs can be obtained from lens manufacturer data or measurements. PSFs can also be provided by the lens manufacturer. In the case of measurements, a method similar to blind deconvolution can be used, i.e., deriving the PSF from the disturbed image F. Correcting lens blur using a compensated PSF is considered the most accurate lens blur correction.

[0045] Until now, lens blur has been corrected by a separate image correction using a known modulation transfer function (MTF). Unlike modeling lens blur using a modulation transfer function (PSF), MTF correction is based only on an approximation of the lens blur, which means that, by definition, it cannot correct all blur caused by the lens. This separate correction step using the MTF can now be omitted.

[0046] The PSF essentially demonstrates how an idealized point object is imaged through an optical system with at least one lens. Due to lens blur, a pixel p in the disturbed image F is affected by different pixels p in the disturbed image U. For example, a pixel p in the disturbed image F is given by a weighted sum of the pixels p in the disturbed image U. Therefore, the image disturbance introduced in image record 1 due to lens blur can be modeled as a convolution of the disturbed image U with the PSF. Thus, the PSF can be represented as a single convolution matrix C. i In vector-matrix notation, the single convolution matrix C of lens blur is... i Each line describes how a pixel p in the disturbed image F is affected by the disturbance in the reduced image U. The effect of lens blur is generally smaller in the central region of image record 1 than in the edge region.

[0047] Here, PSF may also depend on other influencing variables of the optical system, such as aperture setting. This can also be taken into account, such as the individual convolution matrix C that causes lens blur. i Depending on the aperture setting, or for different aperture settings, different single convolution matrices C are given for lens blur. i .

[0048] According to the present invention, the relationship between the interference-reduced image U and the interference-affected image F is extended. Therefore, the interference-affected image F no longer depends solely on a function of the operator A (in this case, the blur kernel) describing motion blur, i.e., F = g(U, A[, η]), but rather on a function of the multiplicative image interference matrix M and the convolutional image interference matrix C, i.e., F = g(U, M, C[, η]), for example... Any motion blur that might be considered will be passed through a single convolution matrix C. i It has an impact.

[0049] Similarly, as described above, an optimization problem can be proposed and solved from this, namely... However, the image interference operator A is now the matrix product (matrix multiplication) of the multiplicative image interference matrix M and the convolutional image interference matrix C, i.e. This leads to a global optimization, where all influencing factors can be considered at once. As mentioned above, optional noise factors, such as those indicated by square brackets, can be considered when necessary using the regularization function θ2.

[0050] Even without modeling noise, the regularization function θ2 can be selectively considered during optimization. For example, regularization using θ2 (in addition to considering noise) can be used to correctly identify invalid image points p or to interpolate invalid image points p. Even in the case of Bayer mode image sensors, regularization using θ2 (in addition to considering noise) can be used to reconstruct unknown RGB image points p. Therefore, the regularization function θ2 is not only related to considering noise.

[0051] The same solution algorithm as described above can be used.

[0052] Therefore, the various influencing factors causing image interference in the recorded disturbed image F can be directly considered in a single optimization problem in one step. Thus, the shortcomings of previous methods are eliminated, because all these image interferences are corrected in one step, rather than being processed sequentially as before. Here, by modeling using individual interference matrices, the potential interdependencies between the various image corrections are implicitly considered.

[0053] Of course, the same applies here: image correction reduces image interference in image record 1, but it is not usually completely eliminated, or only completely eliminated in special cases.

[0054] Therefore, the optimization problem can be written as follows: The form is θ1, where µ is a pre-defined regularization parameter. Here, k represents a specific norm, such as k=1, 2, but other objective functions θ1 can also be defined. θ2 can be defined as described above, or in other suitable forms.

[0055] To account for invalid image points, a selection matrix S can be additionally considered, which is essentially a diagonal matrix of dimension k in vector-matrix representation. The matrix element S[p,p] is defined as follows: for valid image points p, S[p,p] = 1; for invalid image points p, S[p,p] = 0. Invalid image points p must be considered in both the disturbed image U and the reduced-disturbance image F, which leads to… .pass , and This led to and ,For example This can be solved similarly by considering invalid image points p.

[0056] If Ω X Let p represent the set of invalid image points. Alternatively, invalid image points can be considered in the following way: the residuals of the optimization problem are evaluated only for valid image points. Mathematically, this can be expressed as: The form in which "Ω\Ω" X " indicates the considered image point p that does not contain invalid image points. Interpolation can be performed for invalid image points, or invalid image points can be automatically calculated in an appropriate manner when solving optimization problems by using additional regularization of θ2.

[0057] In the case of a multi-channel image sensor 4 (e.g., an RGB image sensor), such an optimization problem arises for each channel.

[0058] However, the present invention can also be used with other types of image sensors 4. A common type of image sensor 4 is the well-known Bayer-mode image sensor, which is equivalent to an image sensor 4 covered with a checkerboard pattern of color filters (usually red, green, and blue). Here, each pixel p of the image sensor 4 is filtered with a specific color. Therefore, in order to obtain an image record 1 with a resolution corresponding to the number n of pixel p for each channel of the image record 1, a large number of pixel points in each channel must be reconstructed (e.g., by interpolation). Now, the method of the present invention can even directly consider such a transformation.

[0059] For example, the transformation from RGB to Bayer mode can be described using vector-matrix representation through a transformation matrix B, where the matrix elements B[i,j] are defined as follows: if image point j is given by image point i in image record 1, then B[i,j] = 1; otherwise, B[i,j] = 0. This leads to... Where A = MBC. There exists a transformation matrix B that maps all three RGB channels to the Bayer mode sensor. Therefore, the dimension of matrix B is (n) x (3n) (i.e., n rows and 3n columns), where n is the number of image points p of the Bayer mode sensor. Clearly, this leads to an equivalent optimization problem similar to the one described above, but the transformation is directly included in the solution to the optimization problem. In this case, invalid image points p can also be considered as described above by choosing matrix S, i.e. .

[0060] Assuming the existence of spatially variable image interference caused by convolutional factors, the image interference is modeled separately for each pixel p. Due to the convolutional interference matrix C, this results in a very large set of equations, making optimization computationally expensive.

[0061] If we define d ≥ 1 pixel regions in image record 1, and assume that the image interference is spatially invariant in each of these regions, then the computational cost of image correction can be reduced. Therefore, in such pixel regions, it is assumed that the image interference caused by convolutional factors is the same for every pixel p in that region. This assumption can be made, for example, when the image interference, although spatially variable, changes very slowly in different pixel regions (as subsets of the entire set of all pixels p). However, the image interference may change in d different pixel regions, and these d pixel regions may partially overlap. Thus, for each pixel region with spatially invariant image blur in d pixel regions, the optimization problem can be solved more efficiently and independently of other pixel regions. Then, the interference-reduced image U can be derived from the solutions of each pixel region. Of course, this also allows us to consider at least one pixel region with spatially invariant image interference and at least one pixel region with spatially variable image interference.

[0062] Assuming the existence of spatially invariant (i.e., spatially constant) image interference, the optimization problem can be simplified for image regions with spatially invariant image interference. Here, for each image region with spatially invariant image interference among a number of d image regions, a linear system of local equations is obtained, in the form of: In this system, matrix multiplication is replaced by simple convolution. These locally linear equations can be solved with less computation.

[0063] Of course, this also applies to cases where the choice of matrix S or the transformation matrix B is considered, i.e. or or .

[0064] The method for image correction is implemented on the image correction unit 10, such as... Figure 2 As shown. The image correction unit 10 is computer hardware, on which the algorithm for solving the optimization problem is implemented as computer software. The image correction unit 10 obtains a single convolution matrix C for the convolutional influencing factors to be considered (such as motion blur, lens blur). i If necessary, it can be obtained separately for each defined image point region, and a single multiplication matrix M can also be obtained. i The image correction unit 10 then determines the convolutional image interference matrix C (for each pixel region if necessary) and the multiplicative image interference matrix M. Alternatively, the image correction unit 10 may also directly obtain the convolutional image interference matrix C (for each pixel region if necessary) and the multiplicative image interference matrix M. If necessary, the image correction unit 10 also obtains the selection matrix S and / or the transformation matrix B. Figure 2 (Not shown in the image). The image correction unit 10 acquires image record 1 with the disturbed image F as input, and determines image record 1 with the disturbed image U as output (i.e., the optimized solution) by solving an optimization problem.

[0065] Finally, it should be noted that matrix representation can be used equivalently to vector-matrix representation, where a point p is given as a matrix rather than a vector. However, such representation can always be converted to vector-matrix representation; therefore, this invention is described using vector-matrix representation, but it also includes and is equivalent to other such representations in any way. For example, in matrix representation, a single multiplication matrix M... i Each element of the matrix will describe the effect of multiplication on image point p, thus forming a single multiplication matrix M. i It is no longer a diagonal matrix. Then, the multiplicative image interference matrix M will be composed of a single multiplication matrix M. i The element-wise multiplication is used to derive the result. In the matrix representation, the convolutional image interference matrix C will be a tensor of the corresponding order, and the corresponding tensor product will be used instead of matrix multiplication. However, even if other representations different from the vector-matrix representation are used, the basic method according to the present invention remains the same.

Claims

1. An image correction method for correcting an distorted image (F) of an object (2) to a distorted image (U) of the object (2), wherein an image record (1) of the object (2) is recorded using an image sensor (4) of a digital camera (3), and the image record (1) consists of a plurality of, i.e., n, pixel points (p), and contains the distorted image (F) of the object (2) due to various influencing factors, wherein the relationship between the recorded distorted image (F) of the object (2) and the distorted image (U) of the object (2) is described in vector-matrix representation by an image interference operator (A), particularly using In the form that the disturbed image (F) and the disturbance-reduced image (U) are described as vectors comprising a plurality of n image points (p), and the image disturbance operator (A) is described as a matrix of dimension n x n, and the disturbance-reduced image (U) is obtained by means of optimization performed in the image correction unit (10). Obtained from the recorded disturbed image (F), where θ1 is the pre-given objective function for optimization, the objective function evaluating the deviation between the disturbed image (F) and the transformed disturbance-reduced image (AU), characterized in that... Consider that the number of images in the image record (1) is N. M There are N known factors influencing multiplication. C There are several known convolutional influencing factors, and the multiplication influencing factors describe the multiplication factor between a pixel (p) in the reduced-interference image (U) and the same pixel (p) in the disturbed image (F), and each multiplication influencing factor is described in vector-matrix notation as a single multiplication matrix (M) in diagonal matrix form. i The diagonal matrix has matrix elements on the diagonal for each image point (p), and the number is N. M A single multiplication matrix (M) i ) to determine the multiplication matrix M as the single multiplication matrix (M i The convolutional influencing factors describe the influence of multiple pixels (p) in the interference-reduced image (U) on a single pixel (p) in the interference-affected image (F), and each convolutional influencing factor is represented by a single convolutional matrix (C). i It is described in the form of ), where in vector-matrix notation, a single convolution matrix (C) i Each line of the image describes how a pixel (p) in the disturbed image (F) is affected by the pixels (p) in the disturbed image (U), and according to the number N C A single convolution matrix (C i ) to determine the convolution matrix C as the single convolution matrix (C i The matrix product of the multiplication matrix M and the convolution matrix C is used as the correction operator (A), i.e., A=MC.

2. The method of claim 1, wherein, The interference-reduced image (U) is obtained from the recorded interfered image (F) by means of an optimization performed in the image correction unit (10) with an additional, pre-specified regularization function θ2 .

3. The method according to claim 1 or 2, characterized in that, is used for the relationship between the recorded disturbed image (F) and the interference-reduced image (U), wherein the noise recorded with the image sensor (4) is described, and the optimization is solved in the form wherein θ2is a pre-given regularization function, by which the noise is taken into account.

4. The method according to any one of claims 1 to 3, characterized in that, As a single convolution matrix (C i ), at least one pre-defined point spread function is used to correct lens blur and / or a pre-defined blur operator is used to correct motion blur.

5. The method according to any one of claims 1 to 4, characterized in that, A Bayer pattern image sensor is used as image sensor (4) and the transformation from the Bayer pattern to the said interference reduced image (U) is described by a transformation matrix B, where in vector-matrix notation the matrix elements B[i,j] of the said transformation matrix B are defined as follows: B[i,j] = 1 if the image point j is given by the image point i in the said image recording, otherwise B[i,j] = 0, and the said image interference operator (A) is obtained by the matrix product A = MBC.

6. The method according to any one of claims 1 to 5, characterized in that, The invalid image points in the image recording (1) are taken into account by selecting a matrix S, wherein in the vector-matrix representation the selection matrix S is a diagonal matrix and the invalid image points are taken into account by the values of the matrix elements on the diagonal being zero and the invalid image points are taken into account in the disturbed image (F) or in the interference-reduced image (U) in the form or which leads to the optimization wherein , , and or to the optimization wherein .

7. The method according to any one of claims 1 to 4, characterized in that, In the image recording (1), there are assumed to be a number d of image point regions which have a spatially constant image disturbance caused by a convolution influencing factor, whereby for each of the image point regions the relationship between the disturbed image (F) and the disturbance reduction image (U) is by means of a convolution is simplified to The optimization is solved for each image point region, and the disturbance reduction image (U) is composed from the individual image point regions.

8. The method of claim 7, wherein, Additionally a selection matrix S for taking into account invalid image points and / or a transformation matrix B for transforming to other image sensors is considered. Additionally a selection matrix S for taking into account invalid image points and / or a transformation matrix B for transforming to other image sensors is considered.

Images