Method and device for correcting images

Abstract

The present invention relates to a method for correcting images, comprising the following steps: - optically scanning a document to produce a scan image of the document, - correcting the scan image by means of feature matching, wherein features of the scan image each comprising a feature vector are matched with corresponding features of a reference image of the document (first feature matching) and a homography matrix is determined, by means of which all pixels of the scan image are mapped to form a homography image, - adding the coordinates of the features in the homography image to the respective feature vectors and adding the coordinates of the features in the reference image to the respective feature vectors, and - new matching (second feature matching) of the features of the homography image with corresponding features of the reference image of the document, in which matching the coordinates associated with the feature vectors are taken into account, and - correcting the homography image according to the matching of the features of the homography image with the features of the reference image.

Description

[0001] Method and device for rectifying images

[0002] The invention relates to a method and a device for rectifying images.

[0003] Methods for image distortion correction often use feature matching. In feature matching, features are extracted from two images that are supposed to display at least some of the same content and then matched to each other. If one of the two images is a reference image, then based on this matching of features, the pixels of the image that is not the reference image can be shifted so that they are arranged exactly the same as, or in a similar arrangement to, the reference image. If the reference image is not distorted, then the non-reference image is corrected after this shift of the pixels.

[0004] US Pat. No. 6,711,293 B1 discloses a method for extracting scale-invariant features (SIFT features: scale invariant feature transform). Such features have the advantage that they are equally present in the scanned images, for example, from different camera viewing directions. These features can thus be correctly assigned to one another in images from different viewing directions.

[0005] There are different types of features such as ORB features or SURF features (Rublee Ethan et al. “ORB: An Efficient Alternative to SIFT or SURF.“ 2011 International Conference on Computer Vision, IEEE 2011).

[0006] EP 1 594 078 B1 discloses a system and a method for identifying points of interest (POIs) in each image at different resolutions. A point of interest is a point whose position within an image is defined by at least one property attributable to the pixels in a pixel neighborhood of a predefined size around the point. Furthermore, each point of interest is a point to which a unique orientation can be assigned based on at least one property attributable to the pixels. This can be a property attributable to the point that serves to define the point position or another property in a neighborhood around the point. Once the points of interest have been identified, a descriptor is created for each of the points of interest.This descriptor characterizes each point in a manner that is essentially invariant to changes in image position, orientation, and scale, as well as to changes in image size. Furthermore, the descriptor is essentially invariant to image scaling and to changes in the intensity of the pixels in the region around the point of interest.

[0007] US 2012 / 0148164 A1 discloses a method for comparing at least two images. The matrices are reused to enable faster processing, e.g. for moving images. The compared images are stitched together to create a panoramic image. This exploits the fact that two cameras which take several images in succession are often similarly related to one another, e.g. when two or more cameras are attached to an aircraft. Since the rectification required for a panoramic image is similar for each pair of images, it is possible to search for an associated feature point in an area for each subsequent set of images, which feature point was assigned using a previously determined homography matrix. If the two images of a first pair of images are compared, this information can be transferred to a subsequent pair of images.

[0008] These descriptors are also called feature vectors, where a feature is defined by the feature vector, the associated image coordinates and a principal direction, where the principal direction is calculated from the vicinity of the central point.

[0009] Such feature matching or feature assignment can be used for a wide variety of applications in image processing. Such feature assignment can be used, for example, for image stitching or for stabilizing video signals. It can also be used to track specific objects in a video signal or to generate 3D models from multiple images. When rectifying an input image with respect to a reference image, local displacements are determined from the paired features, which can include both linear displacements and rotations. These displacements are interpolated over the entire image domain, creating a "dense displacement field" for each pixel, as described, for example, in Lee Seungyong, Georg Wolberg, and Sung Yong Shin, "Scattered Data Interpolation with Multilevel B-Splines," IEEE Transactions on Visualization and Computer Graphics 3.3 (1997), pages 228 to 244.

[0010] There are also methods for rectifying images that do not rely on feature mapping. One such alternative method, for example, is based on mapping complete image sections of an input image to a reference image (template matching). Displacement vectors can also be generated from such mappings, which can then be interpolated to form a displacement field. One such method is described in Digital Image Processing, 4th edition, Rafael C. Gonzalez et al., Pearson Education Limited, 2018, p. 915. Scale- and rotation-invariant features (e.g., SIFT, ORB, or SURF) are therefore often used for rectification. The precision of rectification based on such feature mapping is limited, since an exact mapping of the features of the input image and the reference image is not always possible. Mismatches occur, for example, when certain patterns repeat in an image, as is the case with forms with tables.For example, the many intersection points in the table cannot always be clearly assigned.

[0011] Furthermore, with certain methods, such as template matching, the quality of the matching can depend heavily on the original orientation of the input image, which should ideally match the orientation of the reference image. If one wishes to position a document arbitrarily relative to an optical scanning device, it is not possible to correctly rectify the input image generated from this document by mapping its features to the features of a reference image.

[0012] Therefore, the object of the present invention is to provide a method and a device in which a scanned image can be rectified with high precision by means of a feature assignment.

[0013] The problem is solved by the subject matter of the independent patent claims. Advantageous embodiments are specified in the respective subclaims.

[0014] According to a first aspect of the present invention, a method for rectifying images is provided, comprising the following steps:

[0015] - -optically scanning a document to produce a scanned image of the document,

[0016] - -Distortion of the scanned image by means of feature matching, whereby features of the scanned image, each comprising a feature vector, are assigned to corresponding features of a reference image of the document (first feature assignment) and a homography matrix is ​​determined with which all pixels of the scanned image are mapped to form a homography image,

[0017] - -Adding the coordinates of the features in the homography image to the respective feature vectors and

[0018] - Adding the coordinates of the features in the reference image to the respective feature vectors and,

[0019] - -re-assigning (second feature assignment) the features of the homography image to corresponding features of the reference image of the document, whereby the coordinates assigned to the feature vectors are taken into account during the assignment, and rectifying the homography image in accordance with the assignment of the features of the homography image to the features of the reference image.

[0020] The homography matrix generated based on the initial feature assignment allows for global perspective distortion correction. The scanned image is corrected using the homography matrix through rotation, translation, and global perspective distortion correction. This allows the document itself to be positioned arbitrarily relative to the optical scanning device used for optical scanning. Homography mapping rotates and shifts the scanned image until its orientation essentially matches the orientation of the reference image. Distortions that vary locally, such as those caused by bending or folding the document, are not corrected in the homography image. Such homography mapping does not allow for the correction of locally varying distortions.

[0021] In the second feature assignment, by assigning the features of the homography image to the features of the reference image, the coordinates of the features of the homography image as well as the coordinates of the features of the reference image are taken into account. During such an assignment, the distance between the corresponding feature vectors is determined. The smaller the distance, the better the assignment. The feature pairs with the smallest distance are thus selected as feature pairs of the features of the homography image and the features of the reference image. By taking the coordinates of the features into account, the distance between the features of the homography image and the corresponding features in the reference image is small if the corresponding features are arranged at the same or a similar position in the two images.This allows features to be clearly assigned, even if identical or similar features occur repeatedly in the corresponding images. By spatially assigning these features, identical or similar features can be clearly distinguished from one another. However, such spatial assignment only makes sense if the image to be rectified is essentially aligned in the same way as the reference image. For example, if the image to be rectified is rotated by an angle of more than 45° relative to the reference image, the features to be assigned to each other would have completely different coordinates, and the coordinate information would have no significance for the similarity or assignment of the features.Only the alignment of the scan image by mapping it using the homography matrix to the homography image allows the meaningful use of the coordinates of the features as components of the feature vectors, since the scan image is aligned to the reference image by mapping it using the homography matrix.

[0022] After the second feature assignment, the features of the homography image and the reference image are assigned to each other so precisely that the homography image can be very precisely rectified according to the second feature assignment. Even locally varying rectifications, such as those caused by bending and folding a document, are possible. Rectification of the homography image is preferably performed using a freeform rectification method, for example, by interpolating and / or fitting a displacement field, as explained above (see discussion by Lee, Seungyong, et al.).

[0023] The features are preferably scale and / or rotation invariant features.

[0024] Freeform distortion can be performed using a dense displacement field. The dense displacement field is calculated by interpolation or fitting from the coordinates of the features or the image surroundings.

[0025] According to a further aspect of the present invention, a method for rectifying images is provided, comprising the following steps:

[0026] - -Assigning features of an input image of a document to be rectified, each of which comprises a feature vector, to corresponding features of a reference image of the document,

[0027] - -Assigning image sections of the input image to corresponding image sections of the reference image, and

[0028] - -Recognition of the input image according to the assignment of the features and the image sections using freeform rectification.

[0029] This aspect is based on the inventors' insight that a feature assignment includes many features in some areas, and few or even very few features in others, so that in areas where only a few features are present, the precision of freeform distortion correction can be very low. Rotation and scaling variations caused by physical deformation of the document can be eliminated by distorting the image. Such distortions cannot be corrected by assigning image sections. However, assigning image sections allows for unambiguous distortion correction even in areas where few features are present.The combination of using both feature assignment and image section assignment thus significantly improves the precision of the rectification compared to a method that is based solely on feature assignment or image section assignment.

[0030] Preferably, displacement data or displacement vectors and their support points relative to the reference image are determined from the assignments of the features and the image sections, and these are interpolated to form a displacement field. The coordinates of the features or specific points of the image sections in the input image can be used as support points for the displacement vectors. It may also be expedient to determine displacement vectors and their support points solely from the assignment of the features and to interpolate these to form a displacement field. First, the displacement vectors based on the assignment of the features are compared with corresponding displacement vectors based on the assignment of the image sections. Displacement vectors that deviate too greatly from the interpolation process are not used to form the displacement field.or that displacement vectors and their support points are determined solely from the assignment of the image sections and these are interpolated to form a displacement field, whereby first the displacement vectors based on the assignment of the image sections are compared with corresponding displacement vectors based on the assignment of the features, whereby displacement vectors that deviate too greatly are not used in the interpolation to form the displacement field.

[0031] The image distortion correction methods according to the first and second aspects explained above can also be used together.

[0032] By comparing the displacement vectors of the different assignments, displacement vectors are eliminated because these displacement vectors are judged to be imprecise due to the strong deviation from the neighboring displacement vectors of the other assignment.

[0033] Basically, when matching features or image sections of an image to be rectified with the features of the reference image, a distance between the corresponding features or image sections is determined and / or corresponding image sections are identified using a similarity measure, and the features or image sections with the smallest distance are assigned to each other in pairs. The distance can be a Euclidean distance, for example. Such a method is described in Digital Image Processing, 4th edition, Rafael C. Gonzalez et al., Pearson Education Limited, 2018, p. 915.

[0034] The method according to the second aspect can be used to rectify the homography image according to the first aspect, using the homography image as the input image. In this combined method, a first feature assignment is performed to create the homography image, a second feature assignment is performed to enable freeform rectification, and an additional image section assignment is performed to further improve the precision of the rectification. The second feature assignment and the image section assignment can be performed independently of one another and in any order, as well as simultaneously or quasi-simultaneously with one another. Using this method, which combines the first and second aspects of the present invention, the applicant has succeeded in performing pixel-precise rectification of crumpled documents arranged in any spatial relationship to an optical scanning device.The only prerequisite for this is that there is a reference image in which certain structures match those of the document. This could be, for example, a form, a lottery ticket, or something similar. The document may contain additional image elements, such as handwritten text. The document may also contain repetitive structures, such as tables or the like, which result in a large number of identical or similar features. These have often led to considerable problems with conventional rectification methods based on feature assignment. These problems can be reliably avoided through two-stage feature assignment, in which alignment is carried out first and only then is a precise assignment taking into account the spatial coordinates of the features.

[0035] The invention is explained in more detail below using the drawings. The drawings show:

[0036] Figure 1 a shows an embodiment of a scanning device for optically scanning documents in a roughly schematic simplified representation,

[0037] Figure 1 b shows a scanning unit of the scanning device from Figure 1A and a scanning device and an illumination device in their geometric assignment to the scanning area

[0038] The invention is explained in more detail below using a device for optically scanning documents (Figures 1a, 1b). Such a scanning device 1 comprises a scanning unit 2 and an evaluation unit 3.

[0039] In the present embodiment, the evaluation unit 3 is a conventional workstation or personal computer 4 with an input unit 5 and a display unit 6. The input unit 5 is a computer keyboard, and the display unit 6 is a computer screen.

[0040] However, the evaluation unit 3 can also be formed by any other multi-purpose computer, such as a computer tablet or a smartphone, or can be fully integrated into the scanning unit 2.

[0041] In the present embodiment, the evaluation unit 3 is designed in the form of a cuboid body with a base plate 7, two side walls 8, 9, and a top wall 10. The base plate 7 and the walls 8-10 define a cavity, with the base plate 7 forming a scanning area 11 on which a document 12 to be scanned can be placed. A scanning device 13 and an illumination device 14 are arranged in the top wall 10 (Fig. 1 b).

[0042] The scanning device 13 is a digital camera with a two-dimensional camera chip and a lens. The camera's viewing direction is directed toward the scanning area 11, allowing the entire scanning area to be optically scanned by the camera.

[0043] The illumination device 14 comprises light-emitting diodes as illuminants and is designed to emit white light. For this purpose, light-emitting diodes that emit white light can be provided, or light-emitting diodes of different colors can be provided that together generate white light. If there are several light-emitting diodes of different colors, it is possible to control them so that they can also emit light with different color ranges or spectral ranges by adjusting the light intensity of the differently colored light-emitting diodes accordingly. The illumination device can have reflectors (not shown), which are preferably designed to direct the light onto the scanning area 11 as diffusely as possible. Such illumination is also referred to as dark-field illumination.If primarily or exclusively non-reflective documents are to be scanned, then it may also make sense to provide bright field illumination, since with bright field illumination, a higher light intensity at the scanning area 11 is achieved than with dark field illumination at the same light intensity of the light source.

[0044] The surface of the scanning area 11 is preferably matt with a uniform color (especially white).

[0045] The surfaces of the side walls 8, 9 facing into the interior are preferably also matt and have a light color, in particular white, so that the light emitted by the lighting device and reflected by the side walls 8, 9 is directed into the scanning area 11 as diffusely as possible.

[0046] The evaluation unit 3 is connected to the scanning unit 2 via a data line 15 for bidirectional data transmission, so that the evaluation unit 3 can control both the scanning device 13 and the illumination device 14 and read the data acquired by the scanning device 13. The evaluation unit 3 thus serves not only to evaluate images captured by the camera 13, but also to control the entire process.

[0047] Other scanning devices can also be used within the scope of the invention, such as scanning devices that use a line-scan camera that moves parallel to the document being scanned during the scanning process, as is known, for example, from copiers, or even a digital camera that is mounted, for example, on a tripod in a specific spatial relationship to an evaluation area. It is also not necessary for the scanning unit to have an illumination device, since, in principle, a document could also be scanned using ambient light.

[0048] The scanning unit 2 according to the present embodiment has a specific, fixed geometric relationship between the scanning device 13, the illumination device 14, and the scanning area 11. Furthermore, the tubular design of the scanning device 1 with the two side walls, the top wall, and the base plate 7 minimizes interference from ambient light, yet the scanning area 11 is easily accessible from two sides, allowing a user to easily place a document 12 to be scanned in the scanning area 11.

[0049] Such scanning devices 1 are particularly suitable for scanning identification documents, such as identity cards, driving licenses or the like, or for scanning certain forms, such as lottery tickets, registration forms or the like.

[0050] The scanning device described above represents one possible embodiment. However, it can be adapted in a variety of ways, provided a camera and a scanning area are provided. For example, it is possible to provide a single, vertically extending, rod-shaped support between the scanning area and the ceiling wall instead of the side walls.

[0051] A method for rectifying images is explained below, which is carried out using documents scanned with the scanning device 1 explained above.

[0052] The process begins with step S1 (Figure 3).

[0053] In the next step (S2) (Scanning), the document 12 is scanned by the scanning device 13. For this purpose, the document 12 is first placed as flat as possible on the scanning area 11 of the scanning unit 2.

[0054] The document 12 is illuminated by the illumination device 14 in order to accurately capture all details of the document 12.

[0055] Once the document 12 is in place and properly illuminated, a user initiates the scanning process by pressing a button or activating a button in the input unit 5. Automatic initiation is also conceivable, for example, by pressure sensors in the base plate 7, which detect that a document 12 is lying on the base plate 7, so that the scanning process can be initiated.

[0056] Alternatively, it is also conceivable that the scanning process is initiated at regular intervals, regardless of whether a document 12 is not located, not correctly located, or not entirely or completely within the scanning area 11. In this case, empty or incomplete information is removed in the subsequent process.

[0057] The scanning device 13 comprises a two-dimensional camera chip and a lens and is also referred to below as a camera. The camera is aligned so that its viewing direction is directed toward the scanning area 11, so that the entire document 12 can be captured in a single shot. The scanning device 13 records the image as document 12 and creates a digital representation, referred to as a scanned image 16 of the document 12.

[0058] After the scan image 16 has been captured, the data of the scan image 16 are transmitted to the evaluation unit 3 via a data line 15. The evaluation unit 3 evaluates the images 16 captured by the scanning device 13.

[0059] Subsequently, a first feature matching is carried out (step S3).

[0060] The features, also called features, keypoints or key points, are identified in the scan image 16 using the so-called SIFT algorithm (Scale-Invariant Feature Transform algorithm).

[0061] This involves identifying features that are unchanging, i.e. invariant, with respect to a change in scale. A scale space is generated which essentially comprises a series of images with different scales or resolutions, these images being calculated from the scan image 16. For each scale, the scan image 16 is convolved with Gaussian filters with increasing sigma values ​​to generate a series of blurred images. The Difference of Gaussians (DOG) is then calculated by subtracting two consecutive and sharp images. The result is a series of DOG images. Possible feature points are identified as local minima or maxima in the DOG images by comparing each pixel with its neighbors in the current scan image 16 and the neighbors in the scale images.

[0062] The feature point specifies the coordinates of the central point of a feature.

[0063] Potential feature points are verified by fitting a quadratic function to the local image patterns around the potential feature points. Features with low contrast or poorly localized to an edge are discarded. Low-contrast features are sensitive to noise, and feature points along edges are spatially unstable.

[0064] The final position of the feature points is determined by the maximum or minimum of the fitted quadratic function. Each identified feature point is assigned one or more orientations based on the local gradient direction of the scanned image 16. This ensures rotation invariance. For this purpose, the gradient magnitude and orientation are calculated for the area surrounding the feature point. An orientation histogram with 36 pins covering 360° is created from the gradient orientation of key points within a region and the key point. The maxima in this histogram determine the orientation of the feature point. The highest peak in the histogram and any subsequent local maximums within 80% of the highest peak are used to assign the orientation.After the feature points have been identified and the orientation assigned, a descriptor, also called a feature vector, is created for each feature to capture its local image appearance.

[0065] The descriptor provides a unique fingerprint for each feature point, enabling accurate matching between two images. The descriptor is essentially a histogram of the gradient orientation in a region surrounding the feature point, providing a robust representation of the local image texture.

[0066] To create a descriptor, the region around the keypoint is first divided into a 4 x 4 grid of subregions. For each subregion, the gradient strength is calculated based on the orientation of the pixels. An orientation histogram with 8 pins is then created for each subregion, corresponding to a total of 128 bins, or dimensions. The histograms capture the dominant gradient direction in the local neighborhood of the feature point. The histogram values ​​form the elements of the descriptor vector. This descriptor vector is normalized to increase its robustness to illumination and contrast variations. The normalized vector is the descriptor for the feature points and provides an invariant representation of the local image structure.

[0067] This feature recognition, as just described, is also performed on a reference image 17.

[0068] Subsequently, both feature points of the scan image 16 and the reference image 17 are compared with each other.

[0069] When comparing the feature points between the scan image 16 and the reference image 17, the descriptors are used to determine which feature points of one image correspond to the feature points of the other image. A distance metric is used for this purpose, whereby for each feature point in the scan image 16, the closest descriptor in the reference image 17 is searched for. The closest descriptor here means that it has the smallest Euclidean distance. If the distance is below a certain threshold, it is considered a potential match. In principle, a distance measure other than the Euclidean distance can also be applied.

[0070] Next comes the ratio test, in which the distance of a feature point in the scan image 16 to the second-closest feature point in the reference image 17 is calculated. The ratio of the distances between the nearest and second-closest feature points is then calculated. If the ratio is below a certain threshold, for example, 0.8, the match is considered valid.

[0071] After matches have been found between the scan image 16 and the reference image 17, the process is repeated by comparing feature points of the reference image 17 with the scan image 16. Only the feature points that match in both directions are retained.

[0072] The retained feature points are then filtered to remove outliers, such as those caused by noise. For this purpose, appropriate algorithms, such as the RANSAC algorithm, are used to detect outliers.

[0073] Step S4 follows, in which a homography matrix is ​​calculated.

[0074] The homography matrix describes the differences between the scan image 16 and a homography image 18. The homography image 18 is the image of the scan image 16 with the transformation extracted from the comparison of the detected features. The homography image 18 is then aligned and transformed such that it has the same alignment and distortion correction as the reference image 17. However, creases and markings in the scan image 16 are retained in the homography image 18.

[0075] A Direct Linear Transformation (DLT) is used to calculate the homography matrix. It requires at least four matching, non-collinear features from the scan image 16 and the reference image 17. However, more matches can be used, in which case a least-squares solution is calculated. Due to noise, false matches, and other factors, not all matches match perfectly, so the homography matrix must be determined using methods such as the RANSAC algorithm mentioned above. The RANSAC algorithm repeatedly selects a random subset of matching feature points and calculates the homography matrix. It then determines how many matches match this calculated homography matrix, and the homography matrix with the highest number of matches is selected as the final result.

[0076] Once the homography matrix is ​​calculated, it is used to transform the scanned image 16 into the homographic image 18. To do this, each pixel in the scanned image 16 is multiplied by the homography matrix to obtain the transformed point in the homographic image 18. This mapping step represents a perspective homographic rectification (step S5). However, this perspective rectification does not allow for inhomogeneous rectification, such as occurs when scanning a crumpled document.

[0077] During this rectification, the pixels of the homography image 18 are interpolated unless the pixels of the scan image 16 are uniquely transformed to these new pixels. The most common interpolation methods include nearest neighbor, bilinear, and bicubic interpolation.

[0078] During this perspective correction, a scanned image 16 can also be adjusted in size and orientation, whereby individual parts of the captured scanned image 16 can be cropped. However, the system is designed so that the information content of the scanned image 16 relating to the document 12 is not cropped. Edge detection is performed to identify the edges of the document 12. The scanned image 16 is then cropped so that all edges of the document 12 are still included in the scanned image 16.

[0079] The homography image 18 can now be sharpened using appropriate known algorithms. Algorithms for contrast adjustment or smoothing artifacts created during the transformation are also known and can be applied here.

[0080] In the sixth step (S6), the (location) coordinates of the features in the homography image 18 are added to the respective descriptors, previously also called feature vectors. The coordinates of the features are also added to the respective descriptors (feature vectors) in the reference image 17. This addition of coordinates is called stamping.

[0081] In the subsequent step S7, the features of the homography image 18 are again assigned to corresponding features of the reference image 17, whereby the assignment takes into account the coordinates assigned to the descriptors (2nd feature matching). The descriptors comprise 130 dimensions, of which 128 dimensions originate from a 4x4 grid of subregions with 8 bins each (see above). The additional two dimensions are the coordinates. These two additional dimensions are weighted tenfold, so that the two coordinates have the same weight as the remaining 128 dimensions. The coordinates are therefore given the same weight as the pixels of the features. By taking the coordinates into account, the location of the features is weighted much more heavily than in the 1st feature matching, which means that only the features that are close to each other in the reference image and the scan image are actually assigned to each other.However, this is only reliably possible because perspective correction has been performed beforehand, in which the reference image and the scanned image are aligned to each other so that the corresponding features are located at similar positions in the respective images. Without this alignment, the two images could, for example, be rotated 90° to each other, causing corresponding features to be located in completely different locations. Taking the spatial coordinates into account in the feature vector would cause significant mismatches.

[0082] Subsequently, displacement vectors are calculated for each feature in both the homography image 18 and the reference image 17 (step S8), which is referred to as vectoring.

[0083] The displacement vectors are calculated using two methods.

[0084] The first method utilizes previously linked features. After matching features between the homography image 18 and the reference image 17, the spatial difference between each pair of matching features is calculated. This difference is called the displacement vector, which essentially points from the position of a feature in the homography image 18 to its corresponding match in the reference image 17.

[0085] Each displacement vector includes the features of length and direction. The length of the displacement vector indicates the amount of spatial displacement between the two matching features. The direction indicates the angle by which the feature has been shifted between the two images.

[0086] In addition to the features, image sections of the homography image 18 and the reference image 17 are also compared. For this purpose, a process called template matching is performed. Template matching complements the feature comparison, as there may be areas in the images where few or no features occur. For these areas, template matching is a very good complement. The disadvantage of template matching is that it is not scale- and rotation-invariant like the feature-based method.

[0087] During template matching, an initial image, such as the homography image 18, is divided into image sections. For example, the homography image 18 can be divided into 64 image sections. During template matching, each of these image sections is shifted over the reference image 17, and a metric is calculated at each position. This metric indicates the extent to which the area of ​​the homography image 18 matches the reference image 17. The shifted image section is also referred to as a sliding window. If all image sections are shifted over the reference image 17, a grayscale image is created in which each pixel indicates how closely the respective area of ​​the image matches the template. The intensity of the individual images indicates the quality of the matches. Shift vectors can also be calculated from this. The position of the highest match, i.e. the maximum gray value, relative to the center of the grayscale image is determined.If the match is in the middle, it means there is no shift.

[0088] A displacement vector field is generated from the vectors from the feature comparison and the template matching. The displacement vectors are incorporated into the displacement vector field either directly or averaged. Vectors suitable for the displacement vector field are also generated from the image sections. In locations without vectors, the surrounding vectors are interpolated.

[0089] The displacement vector field forms the basis for freeform equalization. Step S9 follows, in which the freeform equalization is performed (freeform transformation).

[0090] First, control points are selected. These control points are chosen from the displacement vector field, ensuring a distribution as even as possible across the homography image 18 and the reference image 17. Both control points in the homography image 18 and matching control points in the reference image 17 are selected. The connection between the control points in these images is provided by the displacement vectors and the template matching data.

[0091] Using the selected control points, a system of linear equations is established. This system of equations is solved to determine weights for each control point. These weights determine the strength and direction of the transformation from each control point.

[0092] For each pixel in the homography image, a new position in a transformation image 19 is calculated based on the weights and radial basis functions of the control points. The pixel values ​​are transferred directly if they perfectly match the grid of the target image, i.e., the transformation image 19. Since this is not the case in most cases, an interpolation method, such as bilinear or bicubic interpolation, is used to determine the pixel values ​​in the transformed transformation image 19.

[0093] To avoid overfitting, especially when many control points are used, a regular realization term can be added to the transformation. This smooths the transformation and avoids high-frequency distortions. After applying the transformation, the transformed image 19 is post-processed by sharpening it. Other post-processing steps such as cropping or other image adjustments to improve quality are also conceivable.

[0094] For this purpose, for example, Thin Plate Splines (TPS) can be used. TPS are based on a radial basis function with U (r) = r 2 x log (r 2), where r is the distance between the pairs of control points. The principle of TPS is to find a smooth interpolation function that passes through the control points with the least bending. This involves calculating a "bending energy" of the splines, with the goal of minimizing the energy to ensure a smooth transformation. Alternatively, or in addition, bicubic splines can also be used for fitting.

[0095] The resulting transformation image 19 is the final image of document 12.

[0096] The method ends with step S10.

[0097] The first rectification, which can be carried out quickly and easily, enables a new assignment of the features taking the spatial coordinates into account. This achieves a feature assignment with a quality that allows a free-form rectification to be carried out using the assigned features. Without taking the spatial coordinates into account, there would be many incorrect assignments, which would cause additional errors during free-form rectification. Particularly in documents 12 that have many identical elements, such as boxes on gambling tickets, the quality of the feature assignment is improved to such an extent that a good quality free-form rectification is possible. The second rectification is therefore very precise and error-free, so that even crumpled documents can be processed reliably.

[0098] It has been shown that this method can achieve pixel-accurate rectification, even when scanning crumpled documents. This enables reliable automatic evaluation of the document's contents, allowing a large number of such documents to be scanned by machine and their contents automatically evaluated.

[0099] 1 scanning device

[0100] 2 scanning unit

[0101] 3 Evaluation unit

[0102] 4 workstations

[0103] 5 Input unit

[0104] 6 Display unit

[0105] 7 Base plate

[0106] 8 Side wall

[0107] 9 Side wall

[0108] 10 Ceiling wall

[0109] 11 Scanning range

[0110] 12 Document

[0111] 13 Scanning device

[0112] 14 Lighting equipment

[0113] 15 data line

[0114] 16 scan image

[0115] 17 Reference image

[0116] 18 Homography image

[0117] 19 Transformation image

Claims

International patent application Chromasens GmbH Claims 1 . A method for rectifying images comprising the following steps: - optically scanning a document to create a scanned image of the document, - Rectifying the scanned image by means of a feature assignment, whereby features of the scanned image, each comprising a feature vector, are assigned to corresponding features of a reference image of the document and, in accordance with this feature assignment, a homography matrix is ​​determined with which all pixels of the scanned image are mapped to form a homography image, - Adding the coordinates of the features in the homography image to the respective feature vectors and Adding the coordinates of the features in the reference image to the respective feature vectors and, - re-assigning the features of the homography image to corresponding features of the reference image of the document, taking into account the coordinates assigned to the feature vectors, - Rectification of the homography image according to the assignment of the features of the homography image to the features of the reference image.

2. Method according to claim 1, characterized in that the rectification of the homography image is carried out using a free-form rectification method.

3. Method according to claim 1 or 2, characterized in that the features are scaling and / or rotation invariant features.

4. Method according to one of claims 1 to 3, characterized in that the freeform equalization is carried out by means of a dense displacement field.

5. A method for rectifying images comprising the following steps: - Assigning features of an input image of a document to be rectified, each of which is described by a feature vector, to corresponding features of a reference image of the document, - Assigning image sections of the input image to corresponding image sections of the reference image, and - Rectification of the input image according to the assignment of the features and the image sections using freeform rectification.

6. Method according to one of claims 1 to 4, characterized in that displacement vectors and their support points are determined from the assignments of the features and the image sections, and these are interpolated or fitted to form a displacement field.

7. Method according to claim 5 and one of claims 1 to 4, characterized in that displacement data (or better displacement vectors?) and their support points are determined solely from the assignment of the features, and these are interpolated to form a displacement field, wherein firstly the displacement data based on the assignment of the features are compared with corresponding displacement data based on the assignment of the image sections, and in the case of excessive deviations of corresponding displacement data, these are not used in the interpolation to form the displacement field, and / or that displacement data (or better displacement vectors?) and their support points are determined solely from the assignment of the image sections, and these are interpolated to form a displacement field, wherein firstly the displacement data based on the assignment of the image sections are compared with corresponding displacement data,which are based on the assignment of the features, are compared, and if the corresponding displacement data deviate too greatly, these are not used in the interpolation to form the displacement field.

8. Method according to one of the preceding claims, characterized in that when assigning features or image sections of an image to be rectified with the reference image: - a distance between the corresponding features is determined, and / or - image sections that belong to one another are determined using a similarity measure, and the features or image sections that have the smallest distance are assigned to one another in pairs.

9. Method according to one of claims 1 to 4, characterized in that the rectification of the homography image is carried out according to one of claims 5 to 8, wherein the homography image is used as input image.

Images