Method for correcting unsharp images in digital image recording

Abstract

In order to enable high-quality image recording (1) even in the presence of relative movements between the object (2) to be recorded and the camera (3), a method for image correction of unsharp images is proposed, in which the connection between an image point (p) in the unsharp image recording b(p) and an image point (p) in the sharpened image recording l(p) is modeled by a mathematical model, in which the relative movement between the camera and the object (2) during the exposure time (T) is taken into account and which contains a density function describing the influence of the exposure of the camera (3) during the exposure time, in which an image-point-dependent density function is used in the mathematical model, by means of which the different influences of the camera (3) on the exposure of the different image points (p) of the image recording (1) are taken into account during the image correction.

Description

[Technical field]

[0001] The present invention relates to a method for image correction of unsharp images in a digital image recording of an object, In this case, an image record is recorded by an image sensor of the camera, and the relative movement of the camera and the object during the exposure time of the image record changes the imaging of the object points of the object relative to the image points in the image record b(p) so that the imaging of the object points in the image record changes from the imaging of the object points at the exposure start time T S and the end of exposure time T E During the exposure, the image moves along the image trajectory, which causes a blurred image during image recording. In this case, the connections between the image points in the unsharp image record b(p) and the image points in the sharpened image record l(p) are modeled by a mathematical model, The model then takes into account the relative movement between the camera and the object during the exposure time by a transformation operator H and includes a density function ω describing the influence of the camera on the exposure during the exposure time, and to correct the image, a sharpened image at the image point is determined from the unsharp image record at the image point and the model, This relates to a method. [Background technology]

[0002] When recording images with a camera from a moving vehicle, the movement of the vehicle during the exposure time results in unsharp images. This is due to the fact that the camera moves relative to the object to be recorded during the exposure time, and thus the image points (pixels in the case of digital recording) of the camera are directed to different points of the object during the exposure time. In the case of commonly used digital cameras, an image sensor is provided for each channel (e.g. 3 channels in an RGB sensor) with an arrangement of photodetectors, whereby each photodetector represents one image point (pixel). Commonly used image sensors are the well-known CCD or CMOS sensors. The relative movement between the object to be recorded and the camera may be based on the movement of the vehicle, the movement of the camera relative to the vehicle, the movement of the object, or any combination of these. Such unsharp images can be reduced using considerable technical efforts during the recording of the images, but cannot be completely avoided.

[0003] To reduce unsharp images, it is known, for example, to shift the image sensor of the camera during the exposure time by driving it with a movement that is adjusted to the movement of the vehicle (i.e. the flight speed of, for example, an airplane or other aircraft), so that each pixel of the image sensor remains as accurately aligned as possible to a specific point of the object. This is also known as forward motion compensation. However, this can only compensate for the known forward movement of the vehicle. Other movements and accelerations, in particular the pitching, yawing or rolling of the vehicle, which may occur, for example, in the case of an airplane when flying through vibrations or turbulence, cannot be compensated for in this way. Apart from that, forward motion compensation naturally also increases the complexity and cost of the camera system.

[0004] Irregular and unexpected vehicle motion can be compensated to some extent by stabilized camera suspension, but this also has technical limitations. As a result, image blur caused by motion cannot be sufficiently corrected or cannot be corrected at all. Such camera suspension also increases the complexity and cost of the camera system.

[0005] In recent years, especially image sharpening processes have been widely used. Such image sharpening processes allow the subsequent compensation of unsharp images in the recorded digital image. In this case, a convolution matrix (often called a "blur kernel") is usually calculated, which converts the sharp image into an unsharp image by a mathematical convolution operation. This formula is based on the idea that the unsharp recorded image and the sharp image hidden in it are connected through the blur kernel. If the blur kernel is known, a sharp image can be calculated from the unsharp image by a mathematical deconvolution operation (the inverse operation of the convolution). In this case, the basic problem is that the blur kernel is usually not known. In some approaches, the blur kernel is derived from the unsharp image, also called blind deconvolution. In other approaches, the blur kernel is determined from a known movement of the camera relative to the recorded object, also called non-blind deconvolution. Acceleration sensors, gyro sensors, inertial sensors, etc. can be used in the camera to detect the camera movement. With the known movement of the vehicle in the geographic coordinate system and possibly the known movement of the camera relative to the vehicle, the relative motion of the camera with respect to the object can be inferred. In particular, there are many literature and known methods for identifying blur kernels and determining sharp images using blur kernels, some of which are described below.

[0006] In Non-Patent Document 1, the motion of the camera is first identified using an inertial sensor, from which a blur kernel is determined, and a sharp image is determined by the blur kernel using deconvolution.

[0007] Non-Patent Document 2 discloses the determination of a blur kernel for image sharpening, taking into account the camera motion during recording. Furthermore, a depth profile is created based on the sensor measurements and is taken into account when creating the convolution matrices for different image regions.

[0008] Non-Patent Document 3 describes the sharpening of aerial photographs. The pitch, yaw and / or roll of the aircraft during recording are measured by an inertial sensor and are taken into account when creating a blur kernel.

[0009] In non-patent document 4, it is intended to model the blur kernel as the sum of the chronological sequence of images that occurred during the camera movement and exposure time. The camera movement is also estimated from the image content, whereby a set of possible camera poses (positions and orientations) is defined and the influence of these poses on the unsharp image is described and weighted by a density function. The movement is determined by calculating the weights of the possible camera poses from this set. It is therefore described that the density function or the weights it describes are determined by image analysis of the recorded images. This is similar to the known blind deconvolution. However, this has the drawback that a very large amount of data has to be processed, which is also disadvantageous in the required calculation time. Apart from that, the quality of the extracted blur kernel also depends on the image content. In areas with little noticeable structure in the image, this method can fail, since the density function in these areas is not determined at all or can only be determined inaccurately.

[0010] Particularly in the field of photogrammetry, the demands on the quality of the image recording are very high. In this case, image recording is often carried out using moving vehicles, often using aircraft. Although the above-mentioned image sharpening processes give good results, they are not satisfactory or sufficient for many applications in the field of photogrammetry. This means that the image sharpening achievable by the known image sharpening processes is still not sufficient for applications, for example in the field of photogrammetry or geomatics. Another problem with the known image sharpening processes is that the position of the structures of the object, such as for example the contours of buildings, can be shifted in the recorded images. This is highly undesirable, since in the field of photogrammetry, in many applications, the position and location of the structures of the object are particularly critical. [Prior art documents] [Non-patent literature]

[0011] [Non-Patent Document 1] "Image Deblurring using Inertial Measurement Sensors," Neel Joshi et al., ACM SIGGRAPH 2010 Papers, SIGGRAPH'10, New York, NY, USA, 2010, Association for Computing Machinery [Non-Patent Document 2] "Accurate Motion Deblurring using Camera Motion Tracking and Scene Depth," Hyeoungho Bae et al., 2013 IEEE Workshop on Applications of Computer Vision [Non-Patent Document 3] "Automated Blur Detection and Removal in Airborne Imaging Systems using IMU Data," CA Shah et al., International Archives of the Photogrammetry, Remote Sensing and Spatial Information Sciences, Volume XXXIX-B1, 2012 [Non-Patent Document 4] "Fast Non-uniform Deblurring using Constrained Camera Pose Subspace", Zhe Hu et al., Proceedings British Machine Vision Conference 2012, p.136.1-136.11 Summary of the Invention [Problem to be solved by the invention]

[0012] There is therefore a need for an apparatus and a method which allows high-quality image recording, i.e. high image sharpness, with as little effort as possible when there is a relative movement between the object to be recorded and the camera. [Means for solving the problem]

[0013] This is achieved by using in the mathematical model an image point dependent density function in which the various effects of the camera are imaged for the exposure of different image points of the image record.

[0014] This allows the influence of the camera on the exposure of the image recording and thus on the blurred image to be imaged for each individual image point, at least for the affected image points, which allows the correction of the blurred image to a previously unknown extent.

[0015] It is particularly advantageous if the discretized image region is divided into a number of image sections and a mathematical model with an image point-dependent density function is used for the image points of at least one image section and a mathematical model with an image point-constant density function and a constant image trajectory is used for another image section, so that a sharpened image record can be determined for the image points of this image section by mathematical deconvolution, which requires less computing power. The sharpened image record can then be easily assembled from the sharpened image records of the individual image sections.

[0016] In a camera with a mechanical shutter device, the image point-dependent density function can be used to show the influence of the opening and closing movement of the mechanical shutter device on the exposure of various image points of the image recording. This allows the influence of the closing process of the shutter device or the opening process of the shutter device on the exposure of the image sensor to be imaged. This is particularly advantageous because the exposure at various image points becomes different during the closing or opening process due to the finite acceleration and speed of the shutter device. The fact that this influence can be better detected using the image point-dependent density function leads to improved and sharpened image recording.

[0017] In the following, the invention will be explained in more detail with reference to FIGS. 1 to 3, which show exemplary, schematic and non-limiting advantageous embodiments of the invention. [Brief description of the drawings]

[0018] [Figure 1] The relationship is shown when recording an image of an object on a moving vehicle. [Diagram 2] 1 illustrates the ideal closing motion of a mechanical shutter device. [Diagram 3] 4 shows an example of a density function of image points of an image recording. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0019] FIG. 1 diagrammatically depicts a vehicle 5 (here an aircraft, for example an airplane or a drone) moving (here above the ground) and making an image recording 1 of an object 2, here a terrain structure, by means of a camera 3 mounted on the vehicle 5. In FIG. 1, the image recording 1 taken from the vehicle 5 equipped with the camera 3 is exemplarily depicted as a schematic diagram. The image recording 1 has an image area Ω that includes all image points p of the image recording 1. The image recording 1 is a digital image recording with a pixel width B and a pixel (digital image point p) height H. The image recording 1 is made in the image plane 4 of the camera 3, in which an image sensor (not shown) is arranged with an array of photodetectors that constitute the image points p.

[0020] The invention is described without limiting its generality for an image record 1 having one channel of the light spectrum (a particular wavelength range of light), but can of course be generalized to several channels.

[0021] Using the geographic data, each point of the object 2 (in FIG. 1, object point G is shown as an example) can be assigned a well-defined spatial position in a spatially fixed geographic coordinate system (X,Y,Z). For example, the position of object point G in the geographic coordinate system can be expressed as a vector G=(X G ,Y G ,Z G ), and the position of the vehicle 5 (or the reference point of this vehicle 5) can be described by the vector F = (X F ,Y F ,Z F ) in a geographic coordinate system, where the vector F can be related to a given vehicle coordinate system.

[0022] The camera 3 is assigned a camera coordinate system (x, y, z) fixed to the substrate. The coordinate origin of the camera coordinate system (x, y, z) is usually located at the optical center of the camera 3, the optical axis of the camera 3 usually coinciding with the z-axis of the camera coordinate system. At a focal distance f>0 from the optical center, an image plane 4 of the camera 3 is located, on which the observed object 2 is imaged two-dimensionally. In this case, the image plane 4 is assumed to be aligned parallel to the xy plane of the camera coordinate system, and has a unique, local 2D image coordinate system, for example at a corner of the image area Ω.

[0023] The camera 3 with the image plane 4 may be fixedly mounted on the vehicle 5, so that the vehicle coordinate system may coincide with the camera coordinate system (x,y,z) and the position of the camera coordinate system with respect to the vehicle coordinate system does not change. In some cases, the camera 3 may be arranged on a mounting structure that is movable relative to the vehicle 5, for example a camera suspension that provides stability, which compensates to some extent for the movements of the vehicle 5. As a result, the orientation of the camera 3 with respect to the driving or flying direction remains as constant as possible. The camera coordinate system (x,y,z) can therefore be moved relative to the vehicle coordinate system. However, the position and orientation of the camera coordinate system (x,y,z) with respect to the vehicle coordinate system is assumed to be always known. To detect the movement of the camera 3 in space with respect to the vehicle 5, suitable motion sensors 8 may be provided, such as, for example, acceleration sensors, gyro sensors, inertial sensors, etc.

[0024] The position and orientation of the vehicle 5 in the geographic coordinate system (X,Y,Z) may be assumed to be known based on known movement data of the vehicle 5 (e.g., flight data of an aircraft), whereby the vehicle 5 can move and accelerate in all six degrees of freedom relative to the geographic coordinate system (X,Y,Z). However, the vehicle 5 may be equipped with suitable motion sensors (not shown), such as, for example, acceleration sensors, gyro sensors, inertial sensors, etc., to detect the movement of the vehicle 5 in space.

[0025] These relationships are well known, however, the arrangement (position and orientation) and assignment of these coordinate systems relative to one another may be quite different without affecting the invention.

[0026] It is known that a point of any coordinate system can be expressed as a point in another coordinate system, if the relationship of said coordinate systems to each other is known. This allows the position and orientation of the camera coordinate system (x,y,z) or the image coordinate system to be determined also with respect to the geographic coordinate system (X,Y,Z), for example by means of a well-known coordinate transformation. Via the known coordinate system relationships, each point in the geographic coordinate system (X,Y,Z), for example an object point G, can be unambiguously imaged on the image plane 4 of the camera 3, for example on an image point p in the image field Ω, and the image point p can also be specified in the geographic coordinate system (X,Y,Z).

[0027] The position and orientation of the camera 3 and its field of view 15 adjusted by the optical unit 13 together with the shape of the object 2 determine a recording area 14 which is shown diagrammatically in Fig. 1 by a combination of lines and dots. For the sake of clarity, the object 2 in Fig. 1 is depicted as a two-dimensional line. However, it is clear that the recording area 14 is a surface which corresponds to a substantially rectangular projection onto the object 2.

[0028] In Fig. 1 an image recording 1 is depicted diagrammatically, performed at the depicted position of the camera 3, whereby an image point p in the image field Ω can be assigned to an object point G in the recording field 14 due to the optical properties of the camera system. However, in practice this assignment is not necessarily unambiguous, since the position of the vehicle 5 and / or the camera 3 continues to move in the direction of movement of the vehicle 5 during the exposure time T required for the recording and / or is subject to other translational and / or rotational movements, for example due to vibrations, turbulence, etc. In particular, a rotation around one of the axes of the vehicle coordinate system (usually called roll, pitch and yaw in relation to aircraft) can result in a very large displacement of the recording field 14 on the object 2. In this case, the position of the object point G in the image recording 1 (which corresponds to the image point p in Fig. 1 at the start of the recording) can be displaced during the exposure time T (which corresponds to the image point p' in Fig. 1). In other words, the image point p in the image field Ω can receive light from different object points G during the exposure. This leads to image smearing in the recorded image, also called motion blur.

[0029] In FIG. 1, (exposure start T S From exposure end T E The movement of an object point G in the image field Ω during an exposure time T (between 1 and 2) is depicted by a dashed line in the form of an image trajectory BT. As mentioned above, the image trajectory BT is generated by the relative movement between the camera 3 and the object 2 during the exposure time T of the image recording 1.

[0030] The image trajectory BT of an image point p during an exposure time T can be determined to be known due to the known relative motion between the camera 3 and the object 2. For this purpose, known changes in the position, orientation and / or movement (velocity, acceleration) of the camera 3 with respect to the object 2 during the exposure time T can be evaluated.

[0031] The geo-data of the object point G can be obtained, for example, from a publicly available or proprietary database. Changes in the position, orientation and / or movement of the vehicle 5, e.g. flight altitude, flight speed, flight direction, during the exposure time T, can also be assumed to be known, e.g. from corresponding intervening vehicle data.

[0032] In the present invention, it is assumed that the image trajectory BT during the exposure time T is known or can be determined, which is due to the relative motion between the camera 3 and the object 2 during the exposure time T, which is assumed to be known.

[0033] By knowing exactly the course of the relative movement or image trajectory BT in the image area Ω, for the corresponding image point p (or for a defined image section of the image area Ω consisting of several image points p), a very effective model for imaging the motion blur can be created, which takes into account the real situation very well. In this case, not only the translational and rotational movements (velocity and acceleration) of the camera 3 in all directions relative to the geographic coordinate system can be taken into account, but also possible unsharpness due to the height of the ground, such as high-rise buildings, mountains, etc., depending on the distance of the object point G from the image plane 4. From this model, a sharpened image can then be determined from the unsharp image recording 1, as known from the prior art.

[0034] For this purpose, the relationship between an image point p in the unsharp image recording 1 and an associated object point G of the object 2 is modeled by a mathematical model. The model describes the relative movement of the image point p with respect to the object point G during the exposure time T, i.e. the image trajectory BT. In other words, the model indicates the object points G from which the image point p in the image recording 1 receives light during the exposure time T, i.e. the object points G from which the image point p is exposed during the exposure time T in the image recording 1. Furthermore, the model includes a density function ω that describes the influence of the camera on the exposure during the exposure time. The model, expressed as a function f, can therefore generally be described by the mathematical formula b(p)=f(ω,l,H,T,p[,η]).

[0035] In this case, the model of the unsharp image recording 1 naturally results from the sum of all the individual image points p, whereby each image point b(p) can be modeled by a model.

[0036] where b(p) describes the unsharp image of the image point P, T describes the exposure time, I describes the sharpened image to be reconstructed, and the transformation operator H, usually called homography in the technical literature, describes the movement performed by the image point p relative to the object 2 during the exposure time T. This corresponds to the image trajectory BT. Optionally (indicated by square brackets), the noise that may occur on the image sensor 4 during the exposure can also be taken into account.

[0037] For example, b(p) describes the total light intensity detected at image point p during exposure time T. Light intensity can be understood as the radiant energy that strikes a particular surface perpendicularly at a particular time.

[0038] The density function ω represents the effect of the optical system of the camera 3 on the light intensity reaching the image plane 4 of the camera 3. Thus, the density function number ω modifies the light intensity emanating from and representing the object 2 and detected by the image sensor of the camera 3.

[0039] The transformation operator H describes the movement of the camera 3 or the image plane 4 of the camera 3 relative to the object 2 recorded by the camera 3, i.e. the relative movement of the camera coordinate system (x,y,z) or the image coordinate system (x,y) with respect to the geographic coordinate system (X,Y,Z). The transformation operator H thus describes the image trajectory BT resulting from the relative movement between the camera 3 and the object 2.

[0040] The definition of the transformation operator H is well known, for example from the above mentioned technical papers. The transformation operator H is explained below by means of an exemplary embodiment.

[0041] In this embodiment, the transformation operator H includes known intrinsic and extrinsic parameters of the camera 3. The intrinsic parameters are at least the focal length f of the camera 3 and the position of the principal point h. The principal point h is the intersection of a normal to the image plane 4 that extends through the optical center of the camera 3, and typically corresponds to the intersection of the optical axis of the camera 3 with the image plane 4. The coordinates h0, h1 of the principal point h in the image coordinate system then describe the position of the principal point h with respect to the image coordinate system (x, y), for example at a corner of the image plane 4. The extrinsic parameters describe the rotation and translation of the camera 3 and thus the position and orientation of the camera coordinate system or the image coordinate system with respect to the geographic coordinate system. However, the extrinsic parameters may also take into account other known effects of the camera on the image, such as, for example, distortion, vignetting, aberrations, etc.

[0042] For example, the known intrinsic parameters of camera 3 are given by the matrix

number

[0043] The rotation of the camera 3 is described by a rotation matrix R(t) and the translation by a translation vector S(t), both over time.

[0044] Therefore, the transformation operator H is expressed as follows:

number

[0045] The transformation operator H is S and the end of exposure T E describes the position of image point p in the image coordinate system for each time instant t between t and t. The scene depth d is the normal distance of image point p from the image plane 4 to the object point G, and can also be determined from known geo-data and movement data of the vehicle 5.

number

[0046] The orientation of the camera 3 or the camera coordinate system at time t is expressed as three solid angles

number

number

number

[0047] The translation vector S(t) describes the position of the camera 3, or the position of the camera coordinate system, also relative to the reference coordinate system.

[0048] Therefore, the transformation operator H is the time t of imaging of an object point G in the three-dimensional geographic coordinate system relative to the two-dimensional image coordinate system (x,y) of the image plane 4 (exposure T S Start time and exposure T E or in relation to an image point p, the movement of the image point p during the exposure time T relative to the object 2, i.e. the image trajectory BT.

[0049] A known model of the unsharp image recording 1 is

number

[0050] It is seen that in this known model, the density function ω is assumed to be constant in the image plane 4, i.e. it does not vary from image point to image point in the image plane 4. The density function ω is therefore an image point constant density function.

[0051] However, in most cases, this does not reflect the real situation. This is because, for example, the shutter device (shutter) of the camera 3, whether it is a mechanical shutter, for example a central shutter, or an electronic shutter, for example a rolling shutter (reading a row or a column of pixels) or a global shutter (reading all pixels simultaneously), can generate different progressions of the density function ω in the image. In particular, with a mechanical shutter device, the influence of the shutter movement when opening and closing the shutter can affect the exposure of individual pixels of the image sensor of the camera 3 during the exposure time T. A mechanical shutter device has a finite opening and closing time and a chronological progression of the opening and closing operation, so that it takes a certain time before the shutter is actually completely opened and closed. This affects the intensity of the light reaching the image sensor. However, other devices of the optical unit of the camera, such as for example a center filter, a color filter, a lens, an aperture, the image sensor itself, etc., can also have an image point-dependent influence on the density function ω in the image plane 4. Such devices can affect the light intensity of the light reaching the image point p and / or the light intensity read out by the image sensor.

[0052] The effect of the shutter can be easily understood with the example of a mechanical central shutter. A central shutter opens from the inside to the outside. When open, the amount of light reaching the image sensor increases. Thus, the aging process affects the exposure. When closed, the opposite effect occurs. Such an effect also occurs with focal plane shutters or other shutter devices.

[0053] Similar effects can be caused by electronic shutter devices, optical filters, lenses, or apertures, which components of the optical unit can also affect the density function.

[0054] It has been recognised that these effects are image point dependent and therefore do not affect exposure across the image plane 4 or individual image points p within the image field Ω equally.

[0055] In order to take such influences into account in the reconstruction of a sharp image l, the density function ω, which was previously not simply modelled as time-dependent and image-point constant, is according to the invention additionally made image-point-dependent in the image plane 4. In this case, the image-point-dependence relates to the position of an image point p in the image field Ω or to the position of an image section consisting of several image points p in the image field. The density function ω thus represents the image-point-dependent and time-dependent influences on the exposure of various image points p during the exposure time T resulting from the design of the camera 3.

[0056] Thus, the above modeling of an unsharp image can be done, for example, by

number

[0057] However, the density function ω can be image point constant for a given image section of the image domain Ω, where this image section is a sub-region of the image domain Ω. It is assumed that the density function ω for each image point of such an image section is only time-dependent, but the density function ω can be different for the individual image sections, which means that the density function ω is also image point-dependent here.

[0058] The goal is to determine an unknown sharpened image l from a model of an unsharp image l. To this end, the image sensor is modeled on a set of discrete pixels Ω h Therefore, the model is first discretized.

[0059] The image area Ω of the camera 3 is given by the height of the image sensor, H, in pixels, and the width of the image sensor, W, in pixels, and the discretized image area Ω is h teeth,

number

number

[0060] Models such as the summed central rule or the summed trapezoidal rule, and the discretizations mentioned above.

number

number

number

[0061] The transformation operator H images an image point p in the unsharp image B to a specific position in the sharp image l. However, the change in this position in the sharp image l may be within a sub-pixel range. i,j This is of course not a problem for the continuous model above, since l can be evaluated everywhere. However, the pixel Ω i,j In order to simultaneously discretize the images B and L, i.e. to formulate equations for them, it is advantageous if an evaluation in the sub-pixel range is also possible for the discretized image L. For this purpose, a suitable interpolation, such as a bilinear interpolation, can usually be applied to discretize the sub-pixel range in the discretized image L. In this case, the discrete transformation operator H k not only arises from the numerical integration formula, but also includes these discretizations of the sharpened image L into the sub-pixel range.

[0062] Therefore, W k and H k are the discretized density functions and discretized transformation operators, which result from the application of numerical integration formulas and, if necessary, from discretizing the sharpened image L into the sub-pixel range (e.g. using known bilinear interpolation). Blur Operator

number

[0063] For such systems of linear equations, with or without noise, there are several known direct or iterative solution methods that can be applied to determine L (i.e., the sharpness of the image). Examples of solution methods are the well-known Richardson-Lucy algorithm or methods based on the well-known Total Variation (TV) regularization algorithm.

[0064] The Richardson-Lucy method calculates maximum likelihood estimates based on the underlying Poisson distribution as the probability distribution. The resulting iterative method is 0 =Contains B

number

number

number

[0065] Furthermore, TV regularization is included in the method to reduce the effect of any (white) noise in the determination of the sharpened image L, and the iterative rule includes a selectable or predefined regularization parameter λ.

number

[0066] In the case of spatially constant motion blur in the image record 1, e.g. negligible camera rotation during the exposure time T, and a constant density function ω of image points, the above system of linear equations transforms into a convolution operation B=A*L+η. In this case, each image point p describes the same image trajectory BT in the image record 1. The blur operator A can then be called a blur kernel, which describes a constant unsharp image that differs from the image-point-dependent unsharp image described above. Such equations can be solved much more efficiently than the above equations with spatially varying unsharp images.

[0067] In this case, the iterative rule for the Richardson-Lucy method above is

number

[0068] This can be exploited according to the invention by assuming a constant (in the sense of being uniform) motion blur and a constant density function ω of image points in a particular image section of the image recording 1. In this case, the image section Ω h is d=0, ,N D For -1, Ω d ⊂Ω h Ω, including overlapping or non-overlapping image sections d As a result, the formula B d =K d *L d +η d Or B d =A d L d +η d The linear, local equation for image N D These can be collapsed into a system of linear equations or solved individually. After solving the local systems of equations, the global solution L is given by d ,d=0,···,N D -1. The overlapping image sections Ω d If , the individual image sections Ω dA more even transition between can be achieved by appropriate blending.

[0069] The image point-dependent density function ω(p,t) can be assumed to be known for a particular camera 3, since it is derived from the design of the camera 3, in particular the design of the shutter device, but also from other optical units of the camera 3.

[0070] Thus, an advantage of the above-described inventive method for identifying the sharpened image L is that the image point-dependent density function ω(p,t) is known and therefore does not need to be determined, for example, from the image data in the process of determining the sharpened image L.

[0071] The method for image correction is carried out in a computing unit (not shown), e.g. a computer, microprocessor-based hardware, etc. In this case, the method is carried out in the form of program instructions (such as a computer program) which are executed on the computing unit. The computing unit also receives necessary information, such as data regarding the movement of the vehicle 5 and / or the camera 3 during the exposure. However, the method can also be implemented on an integrated circuit, such as a Field Programmable Gate Array (FPGA) or an Application Specific Integrated Circuit (ASIC), as the computing unit.

[0072] For image correction of motion blur in the images recorded by the camera 3 (wherein a relative motion occurs between the camera 3 and the recorded object 2 during the exposure), the computing unit receives data on the relative motion during the exposure (e.g. movement data from the vehicle 5 in a geographic coordinate system) and / or data on the movement of the camera 3 moving relative to the vehicle coordinate system, for example from the motion sensor 8. With these data on the relative motion and a density function ω(p,t) known for the camera 3, a system of linear equations, or several linear local systems of equations, as described above, can be determined, and then a sharpened image can be determined, as described above. The image correction is preferably performed offline, i.e. after the images have been taken, but may also be performed online immediately after the images have been taken.

[0073] In the following, an example of the possibility of determining the image point-dependent density function ω(p,t) is described. The determination of the density function ω(p,t) is preferably also performed in a computing unit. It should be noted that the density function ω(p,t) can be predetermined for a specific camera and then stored in a suitable manner for image correction.

[0074] To determine the density function ω, it is assumed that there is no relative motion between the camera 3 and the object 2 during exposure, for example by the camera 3 being mounted on a rigid test fixture and pointed fixedly with respect to the object 2 to be recorded. For this purpose, the camera 3 is pointed towards a bright surface that is as homogeneous as possible with the object 2, so that under ideal conditions the same light intensity should reach each image point p during exposure.

[0075] In this embodiment, a camera 3 is used that has a mechanical shutter device behind the aperture of the camera 3. In this embodiment, the camera 3 is electronically exposed, i.e. the time and duration at which the photodetector of the image sensor is activated can be electronically controlled. This allows exposure times T that are independent of the time limitations of the mechanical shutter device. This makes it possible to achieve exposure times T that are significantly shorter than those possible with the mechanical shutter device alone. This also makes it possible to open the shutter device before image recording and control the actual exposure using electronic exposure control. This makes image recording independent of the opening of the shutter device. Thus, for example, the exposure time T can be limited to a portion of the closing movement of the shutter device.

[0076] The image point-dependent density function ω(p,t) is intended to represent the opening and closing movement of a mechanical shutter device, e.g. a central shutter such as an iris diaphragm, and thus its influence on the exposure. In this case, the position dependence may be due to a not perfectly symmetric opening and closing movement of the shutter device. This effect may also depend on the set aperture, and as a result, the position dependence may also depend on other camera parameters, such as the set aperture. To determine the density function ω(p,t), measurements of the amount of light incident on the image sensor of the camera 3 are carried out at various exposure settings.

[0077] This is explained by way of example with reference to FIG. 2, which shows an idealized closing movement of a mechanical shutter device. Time T0 indicates the shutter start time, i.e. the time of the shutter command that closes the shutter device. T1 is the known mechanical shutter delay time, (T1-T0) is the time between the shutter start time and the start of the shutter action. Time T2 indicates the time when the shutter device is fully closed. In this case, times T0, T1, T2 are determined and known parameters of the shutter device, but can vary within normal tolerances. Ageing effects can also affect these times. S1 indicates the beginning of the exposure at the image sensor and S2 indicates the end of the exposure at the image sensor, which can be set in the camera 3, for example, using an electronic exposure control. The dashed area represents the actual exposure at the image sensor and thus defines the amount of light that reaches the image sensor during the exposure time T. It is assumed that the shutter device is fully open before time T0.

[0078] If the exposure time T in a series of recordings is changed, for example by changing the time T0, from the time when the shutter device is closed before the exposure start S1 of the image sensor 4 (indicated by a dashed line in FIG. 2) to the time when the shutter device is completely open during the exposure (indicated by a dotted line in FIG. 2), then depending on the actual exposure time T, which can be represented by (T2-S1), a progression of the amount of light reaching the image sensor, specifically the image point p of the image sensor, is obtained, whereby the exposure time can vary between 0 and (S2-S1). Depending on the diaphragm, the effective exposure time can be further shortened, for example when the open diaphragm section is smaller than the shutter opening. The change in exposure can of course also be carried out by changing the exposure start S1 while keeping the exposure time (S2-S1) constant, without changing the time T0 (and therefore also T1, T2).

[0079] To obtain a model of the shutter operation of the shutter device from this sequence of images, one can attempt the following.

[0080] Since there is no relative motion between the camera 3 and the object 2, the above transformation operator H reduces to an identity mapping, which is H(t,p)=p with p∈Ω. Therefore, the above model of an unsharp image is

number

number

[0081] Based on this expression of the density function ω(p,t), the exposure

number

number

[0082] For example, using forward, backward, or central differentials,

number

number

[0083] These observations can be stored as a density function ω(p,t) and interpolated between the time points of the observations. Observations

number

[0084] However, the observed value

number

[0085] Such curve fitting is well known and is usually solved by optimization. For this purpose, the function parameters α p Mathematical functions including

number

number

[0086] If the optimization is applied only to one image point p, it is also called a local optimization.

[0087] For example, the mathematical function

number

number

number

number

number

[0088] Mathematical Functions

number

number

number

[0089] The functions thus determined

number

[0090] Also, N different image points p in the image region Ω are p For each, N observations t From these observations, we can generate a global model for the density function ω(p,t) over the entire image domain Ω, since we can assume that the density function ω(p,t) in the normal case varies only slowly in the image domain. Then, we can generate an approximate density function

number

[0091] The optimization problem for global optimization is generally expressed as

number

[0092] Preferably, the density function ω(p,t) is normalized to the domain [0,1].

[0093] The density function ω(p,t) thus determined represents the influence of the closing process of the shutter device on the exposure of the image sensor 4 .

[0094] Depending on the embodiment of the camera 3 in determining the density function ω(p,t), other effects of the optical system of the camera 3 are also taken into account, such as aperture, filters, etc. For example, the density function ω(p,t) can be determined in this way for different aperture settings.

[0095] If the exposure is controlled by a shutter action, the density function ω(p,t) can be used as determined. If electronic exposure control is used, the part of the density function ω(p,t) that is critical for the exposure process, i.e. the part between T2 and S1, can be used based on the start time point S1 of the exposure (shown in FIG. 3), which is related to the time point T0 or T1.

[0096] In this way, the opening movement of the shutter device can also be modeled. This can also, if desired, describe the effect of the aperture movement on the exposure of the image sensor 4. However, in electronic exposure control, the iris movement usually has no effect on the exposure.

[0097] The shutter device may also be subject to environmental effects (pressure, temperature, etc.), ageing effects, random fluctuations in the shutter mechanism, etc. Thus, the density function ω(p,t) calibrated in a laboratory environment may not accurately reflect the actual conditions in real operation. In order to determine and possibly subsequently correct these deviations to the determined density function ω(p,t), the following can be attempted.

[0098] Using known image analysis of the images recorded with the camera 3, the motion blur occurring in the image can be measured and adapted values ​​for the function parameter α of the global model for the density function ω(p,t) can be derived. For this, suitable regions containing well-defined structures or characteristic forms can be searched for, typically using gradient-based methods for edge detection, and the blur kernel can be calculated by known methods. One possibility is to use so-called blind deconvolution. The blur kernel determined from the image data is then used as a basis for optimizing the function parameter α of the global model for the density function ω(p,t).

[0099] Cameras 3, particularly for use in the field of photogrammetry or geomatics, often monitor the shuttering of the shuttering device in order to detect malfunctions. A shutter monitoring unit provided for this purpose provides feedback on the movement of the shuttering device. An exemplary embodiment of the shutter monitoring unit is in the form of a constant light source, for example a light emitting diode, and a light sensor that detects the light from the light source. The shuttering device is arranged between the light source and the light sensor, so that the light sensor does not detect the light from the light source when the shuttering device is closed and detects the light from the light source when it is opened or closed according to the movement of the shuttering device. Of course, the shutter monitoring unit should be designed in such a way that the light from the light source does not affect the actual image recording or that the object light of the recorded object is not detected by the light sensor of the shutter monitoring unit. The light intensity detected from the light sensor cannot be directly assigned to the image point p of the image recording, but can be used to obtain feedback on the state of the shuttering device.

[0100] The light intensity detected by the light sensor shows a specific time course over the shutter operation. Certain information can be obtained from this time course. For example, the start of the opening and closing of the shutter device or the complete closing of the shutter device can be detected. Similarly, specific points in time of the shutter operation can be detected, for example at 20% and 75% of the shutter operation. From this, the steepness of the shutter operation can be derived, which gives information about the speed of the shutter device. By observing the shutter operation with the light sensor over the course of several shutters, conclusions can be drawn about the state of the shutter device or the influences on the shutter device. Of course, this should be assumed with respect to the first calibrated shutter operation. For example, if there is a time-dependent shift between the start of opening and closing or the end of opening and closing over the life of the shutter device, this can be linked to ageing or environmental influences. Such offsets can also be taken into account, for example, in the density function ω(p,t) in order to adapt the density function ω(p,t) to the current situation. The varying steepness of the shutter action can similarly be used to fit the shape of the curve of the density function ω(p,t).

Claims

1. 1. A method for image correction of unsharp images in a digital image record (1) of an object (2), comprising: In this case, the image recording (1) is recorded by the image sensor of the camera (3), and Due to the relative movement between the camera (3) and the object (2), during the exposure time (T) of the image recording (1), the imaging of an object point (G) of the object (2) onto an image point (p) in the image recording b(p) changes; As a result, the image of the object point (G) of the image recording (1) is formed at the exposure start time T Ｓ and the end of exposure time T Ｅ During the exposure time T between the image trajectory (BT) and the image trajectory (BT) moves along the image trajectory (BT), which causes an unsharp image in the image record (1), In this case, the connection between an image point (p) in the unsharp image record b(p) and an image point (p) in the sharpened image record l(p) is modeled by the mathematical model b(p)=f(ω, l, H, T, p[, η]), wherein the model is such that the relative motion between the camera and the object (2) during the exposure time (T) is taken into account by a transformation operator H and includes a density function (ω) describing the effect of the exposure of the camera (3) during the exposure time (T), and η optionally describing the noise occurring on the image sensor during the exposure, In order to correct the image, a sharpened image l(p) at the image point (p) is determined from the unsharp image record b(p) of the image point (p) and the model. In the method, The mathematical model uses an image point-dependent density function ω(p,t), By means of this image point-dependent density function, the different influences of the camera (3) on the exposure of the different image points (p) of the image recording (1) are taken into account during the image correction. A method characterized by:

2. Formula [Equation 1] is used, where η(p) arbitrarily describes the noise at an image point (p):

2. The method of claim 1 .

3. The blurred image record B[i,j] = b(p ｉ，ｊ ), the height in pixels H, and the width in pixels W of the image sensor. ｉ，ｊ ) is a discrete image region Ω ｈ of, [Equation 2] A set of pixels containing [Equation 3] is discretized into In that case, p ｉ，ｊ is the pixel Ω ｉ，ｊ indicates the geometric center of 2. The method of claim 1 .

4. the mathematical model is discretized by applying a numerical integration formula; Thereby, the mathematical model is a system of linear equations [Equation 4] is converted to M is the integral range [T Ｓ , T Ｅ ], and W ｋ denotes the discretized density function resulting from the integral formula, H ｋ denotes the discretized transformation operator arising from the integral formula, B shows the discretized blurred image recording; and L denotes the discretized sharpened image record; 4. The method of claim 3.

5. The discretized sharpened image record L is discretized into the sub-pixel domain; 5. The method of claim 4.

6. solving said system of linear equations to obtain a sharpened image record L; 5. The method of claim 4.

7. The system of linear equations comprising an iterative method is: L ０ = B and an iteration rule including predetermined termination criteria for the iteration [Equation 5] is solved by where λ is a predetermined regularization parameter.

7. The method of claim 6.

8. The iterative method produces a sharpened image record L ｋ , L ｋ＋１ until the relative change of successive estimates k, k+1 of At this time, the sharpened image L ｋ＋１ represents the sharpened image L obtained at the end of the iteration, 7. The method of claim 6.

9. The discretized image domain Ω ｈ is the number of image sections Ω ｄ d=0,...,N Ｄ A mathematical model having an image point-dependent density function, divided into −1, is used for the image points of at least one image section, so that for this image section d, the mathematical model B ｄ = A ｄ L ｄ [+η ｄ ], a system of linear equations is obtained:

4. The method of claim 3.

10. For the image points of at least one other image section, a mathematical model with a constant density function of image points and a constant image trajectory is used, so that for the image points of this image section, a mathematical formula B containing a mathematical convolution operator * is used. ｄ = A ｄ *L ｄ [+η ｄ ], and a system of linear equations for the sharpened image (L ｋ＋１ ) is determined by mathematical deconvolution, 10. The method of claim 9.

11. The sharpened image record L is the sharpened image record L of the image section. ｄ Can be combined from 11. The method of claim 10.

12. The camera (3) is held stationary and directed towards a given object (2), and the exposure of the given object (2) is [Equation 6] N including different starting times ｔ A series of individual image records (1) [Equation 7] is performed to determine the image point dependent density function ω(p,t), which results in a number of observations for one image point (p). [Equation 8] is obtained, 12. The method according to any one of claims 1 to 11.

13. the plurality of observed values [Equation 9] is stored for this image point (p) as an image-point dependent density function, 13. The method of claim 12.

14. The multiple observations are then fitted to the function parameter α ｐ is approximated by a predetermined mathematical function having the function parameters α ｐ is determined, and the determined function parameter α for the image point (p) is ｐ is stored as an image point-dependent density function ω(p,t), 13. The method of claim 12.

15. Using an image point dependent density function ω(p,t), the influence of the opening and closing movements of the mechanical shutter device of said camera (3) on the exposure of various image points (p) of the image recording (1) is shown, 2. The method of claim 1 .

Images