Method and device for rectifying images

Abstract

The present invention relates to a method for rectifying images, comprising the following steps: - optical scanning of a document to generate a scan image of the document, - rectifying the scan image by means of feature matching, wherein features of the scan image, each comprising a feature vector, are assigned to corresponding features of a reference image of the document (first feature matching) and a homography matrix is ​​determined with which all pixels of the scan image are mapped to form a homography image, - adding the coordinates of the features in the homography image to the respective feature vectors, and - adding the coordinates of the features in the reference image to the respective feature vectors, and - reassigning (second feature matching) the features of the homography image to corresponding features of the reference image of the document.where the coordinates assigned to the feature vectors are taken into account during the assignment, and rectification of the homography image according to the assignment of the features of the homography image to the features of the reference image.

Description

The invention relates to a method and an apparatus for equalizing images.Methods for equalizing images often use feature matching. In feature matching, features in two images, which are intended to show at least area-wise identical content, are basically extracted and assigned to one another. If one of the two images is a reference image, then on account of this assignment of the features the pixels of the image which is not the reference image can be shifted in such a way that they are arranged exactly as or in a similar arrangement to that in the reference image. If the reference image is not distorted, then after this shifting of the pixels the non-reference image is equalized.U.S. Pat. No. 6,711,293 B1 discloses a method with which scaling-invariant features (SIFT features: scale invariant feature transform) can be extracted. Such features have the advantage that they are equally present in the images scanned in this case, for example from different viewing directions of the camera. These features can thus be correctly assigned to one another in images from different viewing directions.There are different types of features such as ORB features or SURF features (Rublee Ethane et al., "ORB: An Efficient Alternative to SIFT or SURF." 2011 International Conference on Computer Vision, IEEE 2011).EP 1 594 078 B1 discloses a system and a method with which points of interest (POIs) at different resolutions are identified in each image. An interesting point is a point whose position within an image is defined by at least one property attributable to pixels in a pixel neighborhood of a predefined size around the point. Moreover, each point of interest is a point that can be assigned a unique orientation that is based on at least one property to be assigned to the pixels. This may be a point-calculable property that serves to define the point position or another property in a neighborhood around the point. Once the points of interest are identified, a descriptor is created for each of the points of interest. This descriptor characterizes each point in a manner that is substantially invariant to changes in image location, orientation and scale, as well as to changes in image size. Furthermore, the descriptor is substantially invariable with respect to scaling of the image as well as changes in the intensity of the pixels in the region around the point of interest.These descriptors are also referred to as feature vectors, wherein a feature is defined by the feature vector, the associated image coordinates and a principal direction, wherein the principal direction is calculated from the vicinity of the central point.Such feature matching or feature assignment can be used in image processing for a wide variety of applications. Such feature mapping can be used, for example, for picture stitching or for stabilizing video signals. It can also be used to track particular objects in a video signal or to generate 3D models from multiple images. When equalizing an input image with respect to a reference image, local displacements are determined from the features assigned in pairs, which displacements can comprise both linear displacements and rotations. These displacements are interpolated over the entire image area, so that a "dense displacement field" arises for each image point, as is described, for example, in Lee Segynong, Georg Wolberg and Sung Yong Shin "Dispersed Data Interpolation with Multilevel B-Splines "IEEE Transactions on Visualization and Computer Graphics 3.3 (1997); pages 228 to 244.There are also methods for equalizing images that are not based on feature mapping. Such an alternative method is based, for example, on the assignment of complete image sections of an input image to a reference image (template matching). Displacement vectors can also be formed from such assignments, which can be interpolated to form a displacement field. Such a method is described in Digital Image Processing, 4th Edition, Rafael C. Gonzalez et al., Pearson Edition Limited, 2018, p 915.Scaling / rotation-invariant features (e.g. SIFT, ORB or SURF) are thus frequently used for the equalization. The precision of an equalization due to such a feature assignment is limited, since an exact assignment of the features of the input image and of the reference image is not always possible. For example, there are mismatches when certain patterns repeat in an image, such as in forms with tables. For example, the many crossing points cannot always be assigned unambiguously to the table.Furthermore, in certain methods, e.g. in template matching, the quality of the assignment can depend very much on the original orientation of the input image, which should preferably coincide with the orientation of the reference image. If a document is to be positioned at will with respect to an optical scanning device, it is not possible to correctly equalize the input image produced by this document with the aid of a feature assignment to the features of a reference image.Therefore, the object of the present invention is to provide a method and a device in which a scanned image can be equalized with high precision by means of a feature assignment.The object is achieved by the subject matters of the independent claims. Advantageous embodiments are specified in the respective dependent claims.According to a first aspect of the present invention, there is provided a method of equalizing images, comprising the steps of:optically scanning a document to produce a scan image of the document,equalizing the scan image by means of feature matching (feature matching), features of the scan image, which each comprise a feature vector, being assigned to corresponding features of a reference image of the document (first feature assignment) and a homography matrix being determined with which all pixels of the scan image are mapped to form a homography image,adding the coordinates of the features in the homography image to the respective feature vectors; andadding the coordinates of the features in the reference image to the respective feature vectors; and,re-assigning (second feature assignment) the features of the homography image to corresponding features of the reference image of the document, wherein the coordinates assigned to the feature vectors are taken into account during the assignment, andequalizing the homography image according to the assignment of the features of the homography image to the features of the reference image.The homography matrix generated on the basis of the first feature assignment allows global perspective distortion correction. The scan image is corrected by means of the homography matrix by rotation, translation and global perspective equalization. Thereby, the document itself can be arbitrarily arranged with respect to the optical scanning device with which the optical scanning is performed. The scanning image is rotated and shifted by the homography imaging to such an extent that it substantially corresponds to the orientation of the reference image with respect to the orientation. In the homography image, distortions which are locally differently marked, as arise, for example, by folding and folding the document, are uncorrected. Such homography imaging does not allow correction of locally different distortions.In the second feature assignment by the assignment of the features of the homography image to the features of the reference image, the coordinates of the features of the homography image and the coordinates of the features of the reference image are also taken into account. In such an assignment, the distance of the corresponding feature vectors is determined. The smaller the distance, the better the assignment applies. The feature pairs selected for the features of the homography image and the features of the reference image are therefore those feature pairs which have the smallest spacing. By taking into account the coordinates of the features, the distance of the features of the homography image to corresponding features in the reference image is thus small if the corresponding features in the two images are respectively arranged at the same or at a similar position. Features can thereby be assigned unambiguously, even if identical or similar features occur repeatedly in the corresponding images. By locally assigning these features, identical or similar features can be clearly distinguished from one another. Such a local assignment, however, makes sense only if the image to be equalized is aligned substantially the same as the reference image. If, for example, the image to be equalized is rotated by an angle of more than 45° with respect to the reference image, the features to be assigned to one another have completely different coordinates and the information of the coordinates would not have any relevance for the similarity or assignment of the features. Only the alignment of the scan image by the imaging by means of the homography matrix to form the homography image allows the meaningful use of the coordinates of the features as a component of the feature vectors, since the scan image is aligned to form the reference image by the imaging by means of the homography matrix.After the second feature assignment, the features of the homography image and of the reference image are assigned to one another so precisely that the homography image can be equalized very precisely according to the second feature assignment, wherein locally different equalizations, as caused by folding and folding of a document, are also possible.The equalization of the homography image is preferably performed with a free form equalization method, for example by interpolating and / or strobing a displacement field as explained above (see discussion of Lee, Segeyong, et al.).The features are preferably scaling and / or rotation-invariant features.The free-form equalization can be carried out by a dense displacement field. The dense displacement field is calculated by interpolation or fitting from the coordinates of the features or of the image environments.According to another aspect of the present invention, there is provided a method of equalizing images, comprising the steps of:assigning features of an input image of a document to be equalized, which features each comprise a feature vector, to corresponding features of a reference image of the document,assigning image sections of the input image to corresponding image sections of the reference image, andequalizing the input image according to the assignments of the features and the image sections by means of free-form equalization.This aspect is based on the discovery by the inventors that a feature assignment comprises many features in some areas, few features in some areas or even very few features, so that the precision of free-form equalization can be very low in the areas in which only few features are present. By de-emphasis using feature mapping, rotational and scaling variations caused by physical deformation of the document can be eliminated. Such distortions cannot be equalized by means of an assignment of image sections. The assignment of image sections, on the other hand, also allows a clear equalization in the areas in which few features are present. The combination of the use of both the assignment of features and the assignment of image sections thus substantially improves the precision of the equalization compared to a method which is based only on the assignment of features or on the assignment of image sections.Preferably, displacement data or displacement vectors and their interpolation points relative to the reference image are determined from the assignments of the features and the image sections, and these are interpolated to form a displacement field. The reference points of the displacement vectors can be, for example, the coordinates of the features or specific points of the image sections in the input image.It can also be expedient to determine displacement vectors and their interpolation points solely from the assignment of the features and to interpolate these to form a displacement field, wherein firstly the displacement vectors based on the assignment of the features are compared with corresponding displacement vectors based on the assignment of the image sections, wherein displacement vectors which differ too much are not used during the interpolation for forming the displacement field, or that displacement vectors and their interpolation points are determined solely from the assignment of the image sections and these are interpolated to form a displacement field, wherein firstly the displacement vectors based on the assignment of the image sections are compared with corresponding displacement vectors based on the assignment of the features, wherein displacement vectors which differ too much are not used during the interpolation for forming the displacement field.The methods for equalizing images according to the first and second aspects explained above may also be used in common.In this case, by comparing the displacement vectors of the different assignments, displacement vectors are eliminated, since these displacement vectors are judged to be inaccurate on account of the strong deviation from the adjacent displacement vectors of the respective other assignment.In principle, when assigning features or image sections of an image to be equalized to the features of the reference image, a distance between the corresponding features or image sections is determined in each case and / or image sections associated with one another are determined via a similarity measure, and the features or image sections are assigned to one another in pairs, which have the smallest distance in each case. The distance may be, for example, a Euclidean distance. Such a method is described in Digital Image Processing, 4th Edition, Rafael C. Gonzalez et al., Pearson Edition Limited, 2018, p 915.The method according to the second aspect can be used for equalizing the homography image according to the first aspect, wherein the homography image is used as an input image. Thus, in this combined method, a first feature assignment is performed to perform homography mapping, a second feature assignment is performed to enable free-form equalization, and an additional image portion assignment is performed to further improve the precision of equalization. The second feature assignment and the image section assignment can be carried out independently of one another and also in any desired sequence and simultaneously or quasi-simultaneously with one another.With this method, combining the first and second aspects of the present invention, the Applicant has made attempts to equalize crumpled documents, which are arranged in any local association with an optical scanning device, with pixel precision. The only prerequisite for this is that there is a reference image in which certain structures coincide with those of the document. This can be, for example, a form, a ticket or the like. Additional picture elements such as handwritten inserted text may be included in the document. The document may also include repeating structures such as tables or the like which result in a variety of the same or similar features. These have often caused considerable problems in conventional equalization methods based on feature assignment. By means of the two-stage feature assignment, wherein an alignment is first carried out and then only a precise assignment taking into account the location coordinates of the features, these problems can be reliably avoided.The invention is explained in greater detail below by way of example with reference to the drawings. The drawings show in: FIG. 1a shows an exemplary embodiment of a scanning device for optically scanning documents in a roughly schematically simplified illustration, FIG. 1 bshows a scanning unit of the scanning apparatus from FIG. 1A and a scanning device and an illumination device in their geometrical assignment to the scanning regionThe invention is explained in more detail below with reference to an apparatus for optically scanning documents (FIGS. 1 a, 1 b). Such a scanning device 1 comprises a scanning unit 2 and an evaluation unit 3.In the present exemplary embodiment, the evaluation unit 3 is formed by a conventional workstation or personal computer 4 having an input unit 5 and a display unit 6. The input unit 5 is a computer keyboard and the display unit 6 is a computer screen.The evaluation unit 3 can, however, also be formed by any other general-purpose computer, such as a computer tablet or a smartphone, or can be completely integrated into the scanning unit 2.In the present exemplary embodiment, the evaluation unit 3 is designed in the form of a cuboidal body having a base plate 7, two side walls 8, 9 and a top wall 10. The base plate 7 and the walls 8 to 10 define a cavity, wherein the base plate 7 forms a scanning region 11 on which a document 12 to be scanned can be arranged. A scanning device 13 and an illumination device 14 are arranged in the ceiling wall 10 (FIG. 1 b ).The scanner 13 is a digital camera having a two-dimensional camera chip and a lens. The camera is oriented with its viewing direction in the direction of the scanning region 11, so that the entire scanning region can be optically scanned by the camera.The lighting device 14 has light-emitting diodes as a lighting means and is designed such that it can emit white light. For this purpose, light-emitting diodes can be provided which emit white light or light-emitting diodes of different colors can be provided which together generate white light. In the case of a plurality of differently colored light-emitting diodes, it is possible to control these in such a way that they can also emit light having different color ranges or spectral ranges by setting the light intensity of the differently colored light-emitting diodes accordingly. The illumination device can have reflectors (not shown), which are preferably designed such that the light is directed as diffusely as possible onto the scanning region 11. Such illumination is also referred to as dark field illumination. If, above all or exclusively, non-specular documents are to be scanned, then it can also make sense to provide bright-field illumination, since with bright-field illumination at the same light intensity of the light source, a higher light intensity is achieved at the scanning region 11 than with dark-field illumination.The surface of the scanning region 11 is preferably configured matt with a uniform color (in particular white).The surfaces of the side walls 8, 9 facing the interior are preferably also configured to be matt and have a light color, in particular white color, so that the light emitted by the illumination device and reflected by the side walls 8, 9 is directed as diffusely as possible into the scanning region 11.The evaluation unit 3 is connected to the scanning unit 2 by means of a data line 15 for the bidirectional transmission of data in such a way that the evaluation unit 3 can actuate both the scanning device 13 and the illumination device 14 and can read out the data recorded by the scanning device 13. The evaluation unit 3 thus serves not only for evaluating images captured by the camera 13, but also for controlling the entire process.Within the scope of the invention, other scanning devices can also be used, such as scanning devices which use a line camera which is moved parallel to a document to be scanned during the scanning operation, as is known, for example, from copying machines, or else a digital camera which is arranged, for example, with a stand in a specific local relationship with an evaluation region. It is also not necessary for the scanning unit to have an illumination device, because in principle a document could also be scanned accordingly with ambient light.The scanning unit 2 according to the present exemplary embodiment has a specific, fixed geometric relationship between the scanning device 13, the illumination device 14 and the scanning region 11. Moreover, due to the tubular configuration of the scanning device 1 with the two side walls, the top wall and the base plate 7, interference due to ambient light is low, and nevertheless the scanning region 11 is easily accessible from two sides, so that a document 12 to be scanned can be easily placed in the scanning region 11 by a user.Such scanning devices 1 are suitable above all for scanning identity documents, such as, for example, identity cards, driver's license or the like, or for scanning specific forms, such as, for example, lottery tickets, login forms or the like.The scanning device explained above represents a possible embodiment. However, it can be modified in a wide variety of ways provided that a camera and a scanning region are provided. Thus, it is possible, for example, to provide instead of the side walls a single, vertically extending, rod-shaped carrier between the scanning region and the ceiling wall.Next, a method of equalizing images performed on documents scanned by the above-mentioned scanning device 1 will be explained.The method begins with step S 1 (FIG. 3 ).In the next step (S2) (scanning), the document 12 is scanned by the scanner 13. For this purpose, the document 12 is first placed as flat as possible on the scanning region 11 of the scanning unit 2.The document 12 is illuminated by the illumination means 14 to accurately capture all details of the document 12.Once the document 12 is in place and properly illuminated, a user initiates the scanning process by pressing a button or actuating a button in the input unit 5.In principle, automatic initiation is also conceivable, for example by pressure sensors in the base plate 7, which determine that a document 12 is located on the base plate 7, so that the scanning process can be initiated.Alternatively, it is also conceivable that the scanning process is initiated at regular intervals, regardless of whether a document 12 is not, not correctly, or not completely or completely in the scanning region 11. Empty or incomplete information is removed in this case in the later process.The scanning device 13 comprises a two-dimensional camera chip and a lens and is also referred to below as cameras. The camera is oriented such that its viewing direction is aligned with the scanning region 11 so that the entire document 12 can be captured with one recording. The scanner 13 takes the image as document 12 and produces a digital representation referred to as the scan image 16 of the document 12.After the scanning image 16 has been acquired, the data of the scanning image 16 are transmitted to the evaluation unit 3 via a data line 15. The evaluation unit 3 evaluates the images 16 recorded by the scanning device 13.Subsequently, a first feature matching (feature matching) is performed (step S 3).The features, also called features, key points or key points, are identified in the scan image 16 using the so-called SIFT algorithm (scale-invariant feature transform algorithm).In this case, features are identified which are invariable, i.e. invariant, with respect to a change in scale. A scale space is generated that substantially comprises a series of images with different scales or resolutions, these images having been calculated from the scan image 16. For each scale, the scan image 16 is convoluted with Gaussian filters as the sigma value increases to produce a series of unsharp images. The Gaussian difference (DOG=Differentiation of Gaussian) is then calculated by subtracting two consecutive and sharp images. The result is a series of DOG images. Possible feature points are identified as local minima or maxima in the DOG images by comparing each pixel to its neighbors in the current scan image 16 and the neighbors in the scale images.The feature point indicates the coordinates of the central point of a feature.Potential feature points are verified by fitting a quadratic function to the local image patterns around the potential feature points. Features that are low contrast or poorly located on an edge are discarded. Low contrast features are sensitive to noise and feature points along edges are not spatially stable.The final position of the feature points is determined by the maximum or minimum of the fitted quadratic function. Each identified feature point is assigned one or more orientations based on the local gradient direction of the scan image 16. This ensures the rotational invariance. For this purpose, the gradient size and the orientation are calculated for the environment of the feature point. An orientation histogram with 36 pins covering 360° is created from the gradient orientation of vertices within a region and the key point. The maxima in this histogram determine the orientation of the feature point. The highest peak in the histogram and any further local maximum that is within 80% of the highest peak is used to assign the orientation. After the feature points have been identified and the orientation assigned, a descriptor, also called a feature vector, is created for each feature to capture its local image appearance.The descriptor provides a unique fingerprint for each feature point, thus allowing for accurate matching between two images. The descriptor is essentially a histogram of the gradient orientation in an area around the feature point that provides a robust representation of the local image texture.To create a descriptor, the region around the keypoint is first divided into a 4×4 grid of subregions. For each sub-region, the gradient strength in the orientation of the pixels is calculated. An orientation histogram with 8 pins is then created for each sub-region, which corresponds to a total of 128 bins or dimensions. The histograms capture the dominant gradient direction in the local neighborhood of the feature point. The histogram values form the elements of the descriptor vector. This descriptor vector is normalized to increase its robustness to illumination and contrast variations. The normalized vector is the descriptor for the feature points and provides an invariant representation of the local image structure.This feature recognition, as has just been described, is also carried out on a reference image 17.Subsequently, both feature points of the scan image 16 and of the reference image 17 are compared with one another.When matching the feature points between the scan image 16 and the reference image 17, it is determined from the descriptors which feature points of an image correspond to the feature points of the other image.For this purpose, a distance metric is used, wherein for each feature point in the scan image 16 the closest descriptor in the reference image 17 is sought. The closest descriptor here means that it has the smallest Euclidean distance. If the distance is below a certain threshold, it is considered a potential match. In principle, a distance measure other than the Euclidean distance can also be used.The ratio test follows, in which the distance of a feature point of the scan image 16 to the second closest feature point of the reference image 17 is calculated. Then, the ratio of the distances of the closest and second closest feature points is calculated. If the ratio is below a certain threshold, for example 0.8, the match is considered valid.After matching between the scan image 16 and the reference image 17 has been found, the process is repeated by matching feature points of the reference image 17 with the scan image 16. Only the feature points that match in both directions are retained.Subsequently, the retained feature points are still filtered in order to remove outliers, which have been produced by noise, for example. For this purpose, corresponding algorithms, such as e.g. a RANSAC algorithm, are used for detecting outliers.Step S4 follows, in which a homography matrix is calculated.The homography matrix describes the differences between the scan image 16 and a homography image 18. the homography image 18 is the image of the scan image 16 with the transformation extracted from the comparison of the detected features. The homography image 18 is then aligned and transformed to have the same alignment and equalization as the reference image 17. wrinkle creases and marks in the scan image 16 remain, however, in the homography image 18.A direct linear transformation (DLT) is used to calculate the homography matrix. It requires at least four matching non-collinear features from the scan image 16 and the reference image 17, however, more matches may be used, and a least squares solution is then calculated. Due to noise, false matches, and other factors, not all matches perfectly match, so the homography matrix needs to be determined using methods such as the RANSAC algorithm mentioned above. The RANSAC algorithm repeatedly selects a random subset of matching feature points and calculates the homography matrix. It is then determined how many matches match this calculated homography matrix and the homography matrix with the highest number of matches is chosen as the final result.Once the homography matrix is calculated, it is used to transform the scan image 16 into the homography image 18. For this purpose, each image point in the scan image 16 is multiplied by the homography matrix in order to obtain the transformed point in the homography image 18. This step of imaging represents perspective homographic equalization (step S 5). However, this perspective equalization does not allow inhomogeneous equalization, as occurs during the scanning of a crumpled document.In this equalizing, the pixels of the homography image 18 are interpolated, provided that the pixels of the scan image 16 are not transformed unambiguously to this new pixel. The most common interpolation methods include nearest neighbor, bilinear and bicubic interpolation.In this perspective equalizing, a scan image 16 can also be adapted in terms of size and orientation, wherein individual parts of the recorded scan image 16 can be cut off. However, it is designed in this case such that the information contents of the scan image 16 relating to the document 12 are not cut off. In this case, edge recognition is carried out in order to recognize the edges of the document 12. The scan image 16 is now trimmed such that all edges of the document 12 are still contained in the scan image 16.The homography image 18 can now be sharpened by corresponding known algorithms. Algorithms for contrast adaptation or the smoothing of artifacts produced during the transformation are also known, which can be used here.In the sixth step (S 6), the (location) coordinates of the features in the homography image 18 are added to the respective descriptors, which were also previously called feature vectors. Also in the reference image 17, the coordinates of the features are added to the respective descriptors (feature vectors). This addition of the coordinates is referred to as stamping.In subsequent step S 7, the features of homography image 18 are again assigned to corresponding features of reference image 17, the assignment taking into account the coordinates assigned to the descriptors (2ndfeature matching). The descriptors comprise 130 dimensions, of which 128 dimensions originate from a 4x4 grid of sub-regions each having 8 bins (see above). The additional two dimensions are the coordinates. These two additional dimensions are weighted ten times so that the two coordinates have the same weight as the remaining 128 dimensions. The coordinates thus occupy the same weight as the pixels of the features.By taking the coordinates into account, the location of the features is weighted much more strongly than in the 1stfeature matching, which has the result that actually only the features lying close to one another in the reference image and scan image are assigned to one another. However, this is only possible reliably since perspective equalization has been carried out beforehand, in which the reference image and the scan image have been aligned with one another, such that the corresponding features are located in similar locations in the respective images. Without this alignment, the two bidlers could be arranged rotated by 90° with respect to one another, for example, as a result of which mutually corresponding features would be located at quite different locations. Taking into account the location coordinates in the feature vector would cause considerable misallocations.Subsequently, for each feature, displacement vectors are calculated in both the homography image 18 and the reference image 17 (step S8), which is referred to as vectoring.The displacement vectors are calculated from two methods.The first method uses the previously linked features. After matching features between homography image 18 and reference image 17, the spatial difference between each pair of matching features is calculated. This difference is referred to as a displacement vector that substantially points from the position of a feature in homography image 18 to its corresponding match in reference image 17.Each displacement vector comprises the features of length and direction. The length of the displacement vector indicates the amount of spatial displacement between the two matching features. The direction indicates the angle by which the feature was shifted between the two images.In addition to the features, image sections of the homography image 18 and of the reference image 17 are also compared with one another. For this purpose, a so-called template matching (for German: template matching) is carried out. Template matching supplements the feature comparison, as there may be areas in the images where few or no features occur. Template matching is a very good addition for these regions. The disadvantage of template matching is that it is not scaling and rotation invariant as is the method with the features.During template matching, a first image, such as homography image 18, is divided into image sections. For example, the homography image 18 can be divided into 64 image sections. During template matching, each of these image sections is shifted over the reference image 17 and a metric is calculated at each position. This metric specifies the extent to which the region of the homography image 18 corresponds to the reference image 17. The shifted image section is also referred to as a sliding window. If all image sections are shifted over the reference image 17, a gray-scale image results in which each pixel specifies how much the relevant region of the image corresponds to the original. The intensity of the individual images indicates the quality of the matches. Displacement vectors can also be calculated from this. In this case, the position of the highest agreement, i.e. of the maximum gray scale value, relative to the center of the gray scale image is determined. If the match is in the middle, it means that there is no displacement.From the vectors of the feature comparison and from the template matching, a displacement vector field is generated together. In this case, the displacement vectors enter the displacement vector field directly or averaged. From the image sections made, vectors are also formed which are suitable for displacement vector fields. At locations that do not have vectors, interpolation is performed from the surrounding vectors... / The displacement vector field is the basis for free-form equalization. Step S 9 follows, in which the free-form equalization is carried out (free-form transformation).First, control points are selected here. These control points are selected from the displacement vector field, wherein the most uniform possible distribution is given over the homography image 18 and the reference image 17. Control points in the homography image 18 as well as matching control points in the reference image 17 are selected. The connection between the control points in these images is given by the displacement vectors and the template matching data.Using the selected control points, a system of linear equations is established. This equation system is solved to determine weights for each control point. These weights determine the strength and direction of the transform from each control point.For each pixel in the homography image, a new position in a transform image 19 is calculated based on the weights and radial basis functions of the control points. The pixel values are transmitted directly if they perfectly match the grid of the target image, i.e. the transform image 19. Since this is not the case in most cases, an interpolation method, for example bilinear or bicubic interpolation, is used to determine the pixel values in the transformed transform image 19.To avoid overfitting, especially when many control points are used, a regular realization term may be added to the transformation. This smoothes out the transformation and prevents high-frequency deformations.After application of the transformation, the transformation image 19 is remachined by sharpening it. Other post-processing operations such as cropping or other image adaptations for improving the quality are also conceivable here.Thin plate splines (TPS), for example, can be used for this purpose. TPS are based on a radial basis function with U(r)=r 2×log(r 2), where r is the distance between the pairs of control points. The principle of TPS is to find a smooth interpolation function which passes through the control points with the least curvature. In this case, a "bending energy" of the splines is calculated, wherein the aim is to minimize the energy in order to ensure a smooth transformation.The transformation image 19 obtained is the final image of the document 12.The method ends with step S 10.The first equalization, which can be carried out simply and quickly, makes it possible to re-assign the features taking into account the location coordinates. This achieves a feature assignment with a quality which makes it possible to carry out free-form equalization on the basis of the features assigned in this way. Without considering the location coordinates, there would be many misallocateds which would cause additional errors in free form equalization. In particular in the case of documents 12 which have many identical elements, such as boxes on notes of play, the quality of the assignment of the features is improved here in such a way that free-form equalization is possible in good quality. The second equalization is thus very precise and error-free, so that even crumpled documents can be processed reliably.It has been found that pixel-accurate equalization can be achieved with this method even if crumpled documents are scanned in. This makes possible a reliable automatic evaluation of the contents of the document, so that a multiplicity of such documents can be scanned mechanically and their contents can be automatically evaluated.List of reference characters1 Scanning device 2 Scanning unit 3 Evaluation unit 4 Workstation 5 Input unit 6 Display unit 7 Base plate 8 Side wall 9 Side wall 10 Top wall 11 Scanning region 12 Document 13 Scanning device 14 Illumination device 15 Data line 16 Scanning image 17 Reference image 18 Homography image 19 Transformation imageReferences included in the specificationThis list of documents cited by the applicant has been produced in an automated manner and is only included for the better information of the reader. The list is not part of the German patent application or utility model application. The DPMA does not take any adhesion for any faults or omissions.Patent Literature citedU.S. Pat. No. 6,711,293 B1

[0003] EP 1 594 078 B1

[0005] Cited Non-Patent LiteratureRublee Ethane et al., "ORB: An Efficient Alternative to SIFT or SURF." 2011 International Conference on Computer Vision, IEEE 2011

[0004] Lee Segeyong, Georg Wolberg and Sung Yong Shin "Mapped Data Interpolation with Multilevel B-Splines "IEEE Transactions on Visualization and Computer Graphics 3.3 (1997); pages 228 to 244

[0007] Digital Image Processing, 4th Edition, Rafael C. Gonzalez et al., Pearson Edition Limited, 2018, p 915 [0008, 0026]Discussion of Lee, Segeyong,

[0017]

Claims

Method for equalizing images comprising the following steps: - optically scanning a document to generate a scan image of the document, - equalizing the scan image by means of a feature assignment, features of the scan image which each comprise a feature vector being assigned to corresponding features of a reference image of the document and a homography matrix being determined in accordance with this feature assignment, with which homography matrix all pixels of the scan image are mapped to form a homography image, - adding the coordinates of the features in the homography image to the respective feature vectors and adding the coordinates of the features in the reference image to the respective feature vectors, and - re-assigning the features of the homography image to corresponding features of the reference image of the document, wherein the coordinates assigned to the feature vectors are taken into account in the assignment, equalizing the homography image according to the assignment of the features of the homography image to the features of the reference image.Method according to claim 1, characterised in that the equalization of the homography image is carried out with a free form equalization method.Method according to claim 1 or 2, characterised in that the features are scaling- and / or rotation-invariant features.Method according to one of Claims 1 to 3, characterized in that the free-form equalization is carried out by means of a dense displacement field.Method for equalizing images comprising the following steps: - assigning features of an input image of a document to be equalized, which features are each described by a feature vector, to corresponding features of a reference image of the document, - assigning image sections of the input image to corresponding image sections of the reference image, and - equalizing the input image in accordance with the assignments of the features and the image sections by means of free-form equalization.Method according to one of Claims 1 to 4, characterized in that displacement vectors and their interpolation points are determined from the assignments of the features and the image sections, and these are interpolated or filtered to form a displacement field.Method according to Claim 5 and one of Claims 1 to 4, characterized in that displacement data (or better displacement vectors?) and their supporting points are determined solely from the assignment of the features, and these are interpolated to form a displacement field, wherein firstly the displacement data based on the assignment of the features are compared with corresponding displacement data based on the assignment of the image sections, and, if displacement data corresponding to excessive deviations, these are not used during interpolation in order to form the displacement field, and / or in that displacement data (or better displacement vectors?) and their supporting points are determined solely from the assignment of the image sections, and these are interpolated to form a displacement field, wherein firstly the displacement data based on the assignment of the image sections are compared with corresponding displacement data based on the assignment of the features, and if the displacement data corresponds to too great deviations, these are not used during interpolation to form the displacement field.Method according to one of the preceding claims, characterized in that, when features or image sections of an image to be equalized are assigned to the reference image, in each case: - a distance between the corresponding features is determined, and / or - mutually associated image sections are determined via a similarity measure, and the features or image sections are assigned to one another in pairs, which in each case have the smallest distance.Method according to one of Claims 1 to 4, characterized in that the equalization of the homography image according to one of Claims 5 to 8 is carried out, the homography image being used as input image.

Images