Method for creating a database for document classes, method for scanning and processing a document, and device for carrying out these methods

Abstract

The invention relates to methods and a device which allow the partially or fully automatic processing of documents, even if they have been creased or folded and are therefore no longer flat. Scanned images generated by scanning the documents can be automatically assigned to document classes and accordingly processed and in particular read out according to their class. In particular, this allows the processing of machine-printed documents into which individuals have inserted handwritten text.

Description

[0001] Method for creating a database for document classes and methods for scanning and

[0002] Processing a document and a device for carrying out these processes

[0003] The invention relates to a method for creating a database for document classes and a method for scanning and processing a document as well as a device for carrying out these methods.

[0004] There are methods for recognizing similar images based on features. So-called "Content-Based Image Retrieval (CBIR)" systems are known for finding similar images in large image databases. Examples of such CBIR methods are found in X. Wangming et al. (December 2008), "Application of Image SIFT Features to the Context of CBIR," in "2008 International Conference on Computer Science and Software Engineering" (Volume 4, pages 552-555), IEEE, and in KR Reddy et al. (2016), "A Comparative Study of SIFT and PCA for Content-Based Image Retrieval," Inter. Refereed J. Ing. Sci. (IRJES) 5 (11), 12-19. Such CBIR systems often result in mismatches, but these are not problematic, as these systems are designed to return multiple matching images in cases of doubt.

[0005] Methods for distorting images sometimes use feature matching. In feature matching, features are extracted from two images that are intended to show at least some of the same content and then matched to one another. If one of the two images is a reference image, then based on this match of features, the pixels of the image that is not the reference image can be shifted so that they are arranged in the same way or in a similar order to that in the reference image. If the reference image is not distorted, then after this shift of the pixels, the non-reference image is distorted. The matching of features is described, for example, in RC Gonzalez “Digital Image Processing”, 4th edition Pearson, pages 915, 916.

[0006] Such feature mapping can be used for a wide variety of applications in image processing. For example, it can be used for image stitching or for stabilizing video signals. It can also be used to track specific objects in a video signal or to create 3D models from multiple images. When rectifying an input image with respect to a reference image, local displacements are determined from the paired features. These displacements can include both linear displacements and rotations. These displacements are interpolated over the entire image domain to create a “dense displacement field” for each pixel, as described, for example, in Lee Seungyong, Georg Wolberg and Sung Yong Shin “Scattered Data Interpolation with Multilevel B-Splines” IEEE Transactions on Visualisation and Computer Graphics 3.3 (1997); pages 228 to 244.

[0007] US Pat. No. 6,711,293 B1 discloses a method for extracting scale-invariant features (SIFT features: Scale Invariant Feature Transform). Such features have the advantage that they are equally present in the scanned images from different viewing directions, for example, of a camera. These features can thus be correctly assigned to one another in images from different viewing directions.

[0008] There are other types of features, such as ORB features or SURF features (Rublee Ethan et. al. “ORB: An Efficient Alternative to SIFT or SURF“ 2011 International Conference on Computer Vision, IEEE 2011).

[0009] EP 1 594 078 B1 discloses a system and a method for identifying points of interest (POIs) in each image at different resolutions. A point of interest is a point whose position within an image is defined by at least one property attributable to the pixels in a neighborhood of pixels of a predefined size around the point. Furthermore, each point of interest is a point to which a unique orientation can be assigned based on at least one property attributable to the pixels. This can be a property attributable to the point that serves to define the point position or another property in a neighborhood around the point. Once the points of interest have been identified, a descriptor is created for each point of interest.This descriptor characterizes the point in a manner that is essentially invariant to changes in image position, orientation, and scale, as well as to changes in image size. Furthermore, the descriptor is essentially invariant to image scaling and to changes in the intensity of the pixels in the region around the point of interest. These descriptors are also called feature vectors, where a feature is defined by the feature vector, the corresponding coordinates, and a principal direction.

[0010] EP 1 575 002 A1 discloses a method for generating a reference image of a banknote. To generate this reference, multiple banknotes can be scanned.

[0011] DE 10 2011 077 895 A1 discloses a method by which documents are scanned in two different ways to determine a first feature reference set and a second feature reference set. These different feature reference sets can be acquired, for example, using different wavelengths. The feature reference sets can be used, among other things, for document recognition, wherein the feature reference sets are compared with a stored image.

[0012] DE 10 2015 111 120 A1 relates to a method for scanning surfaces with a stereo camera.

[0013] The present invention is based on the object of creating a method for creating a database for document classes, a method for scanning and processing a document, and a device for carrying out these methods, which enables a reliable assignment of optically scanned documents to corresponding document classes. In particular, the assignment should be as unambiguous as possible. The device should also be easy to operate.

[0014] According to a first aspect, the present invention relates to a method for creating a database for document classes, wherein each document class is defined by a plurality of features. In this method, a plurality of copies of a document type to be classified are scanned with a scanning device, wherein a scan image is generated for each copy of the document type (= document). Each scan image is aligned to a digital reference image of the document type to be classified. The scan images aligned in this way are superimposed on one another, wherein the individual pixels are averaged, thus generating a prototype image. From the prototype image, features are again determined, which are entered in the database as the database features defining the document class.

[0015] The method involves scanning multiple copies of a document type, i.e., multiple documents, and aligning, superimposing, and average the resulting scanned images. This produces a prototype image that is prototypical for the document type. Since the prototype image was generated by optical scanning with a scanning device, the prototype image reproduces the document type as perceived by the scanning device. In other words, this means that the prototype image also exhibits properties determined by the scanning method and the scanning device and thus does not have to exactly match the digital reference image. Such a prototype image is therefore more similar to an image of such a document scanned with the scanning device than the digital reference image.The features extracted from the prototype image are therefore more specific to the scanned images of these documents created with the scanning device than the corresponding features generated solely from the digital reference image. This creates a database that very reliably defines the individual document classes using database features. If documents are to be processed automatically, then the documents should be clearly assigned to a document class, as this is the only way to ensure that the content of the documents can be reliably and correctly extracted automatically. The purpose of a database for document classes is to identify documents and assign them to the document class. Additional information may be available for the individual document classes, which facilitates the extraction of the information contained in the documents. This additional information describes, for example,Fields with checkboxes, fields with machine-readable text or numbers. However, this only works reliably if a specific document is correctly assigned to a document class. In the event of an incorrect assignment, instructions for extracting from an incorrect document class would be used, which would then be unsuitable for the respective document. The features derived from the prototype image allow such a clear assignment of a document to a specific document class.

[0016] Preferably, a specific type of scanning device or a single specific scanning device is used to create a specific database. This means that the database characteristics are specific to the type of scanning device or to the single specific scanning device. Scanning devices for optically scanning documents can differ considerably, so that correspondingly different scanned images are produced even if the same document is scanned with different scanning devices. The differences can lie in the color or brightness sensitivity and in the sharpness with which the individual pixels are captured. By using a single type of scanning device or a single specific scanning device, such deviations are reduced or eliminated and the documents can be reliably captured.

[0017] The alignment of each scanned image can be performed using feature mapping or other methods, such as an optical flow method. Because the individual scanned images are aligned to the digital reference image, they can essentially be arranged in any position within the scanning device. Automatic alignment aligns the scanned images with the digital reference image. Alignment using feature mapping allows for very precise alignment of the scanned images to the digital reference image.

[0018] The database features can each comprise a feature vector that describes the respective feature and coordinates that define the location of the respective feature, whereby the feature vector is obtained from the prototype image and the corresponding coordinates are obtained from the reference image. By obtaining the features from the prototype image, they describe the individual features as they are recognized or seen by the respective scanning device. By assigning the coordinates from the reference image to the respective features, however, the exact coordinates of the reference image are used, since the corresponding coordinates of the features in the prototype image can be somewhat blurred or imprecise due to the averaging of the numerous images. This results in an optimal combination of information in the features, resulting in a very high quality of the features.

[0019] When averaging the pixels of the multiple scans aligned to the reference image, the pixels of the superimposed scans with the same coordinates are averaged. The term “averaging the pixels” means that the color values ​​and / or brightness values ​​of the respective pixels are averaging. Averaging can be carried out by equally weighted averaging of all pixels that have the same coordinates. The individual pixels can also be weighted differently. For example, it can be useful to analyze the distribution of the color values ​​and / or brightness values ​​of the pixels and exclude values ​​that deviate significantly from the mean. Averaging can also be performed by determining a median value for the pixels with the same coordinates in the superimposed scans.

[0020] Preferably, certain predefined areas of the aligned scan images are excluded. These are primarily areas with different content in the respective copies of the document type. Typical examples of such areas are fields in which document users are asked to enter certain individual information, such as their name or address. The characters that represent this information differ from document to document. It makes little sense to use these characters as a feature classifying the document class. This exclusion means that, at least in these areas, no features are extracted.

[0021] Meanings can be assigned to certain predefined areas. These meanings can be stored in a database and used when evaluating the content of the document by assigning the respective meaning to the information contained in these areas or by evaluating this information with knowledge of its meaning.

[0022] The "specific areas" that are not considered and to which specific meanings are assigned may be the same areas. But they may also be different or only partially the same areas.

[0023] These areas can be excluded before overlaying, for example, by cutting out the corresponding areas in the not-yet-aligned scan images. It is also possible to cut out these areas only once the individual scan images are superimposed. These areas can be "cut out," for example, by assigning a predetermined color value to these pixels, such as a color value representing white or a color value representing black. The following steps can be performed to align the scan images to the digital reference image:

[0024] - Rectifying and aligning the respective scanned image by means of a feature assignment (= first feature assignment), whereby features of the scanned image, each comprising a feature vector, are assigned to corresponding features of the reference image of the document and, in accordance with this feature assignment, a homography matrix is ​​determined with which all pixels of the respective scanned image are mapped to form a homography image,

[0025] - Adding the coordinates of the features in the homography image to the respective feature vectors and

[0026] - Adding the coordinates of the features in the reference image to the respective feature vectors and,

[0027] - re-assigning the features of the homography image to corresponding features of the reference image of the document (= second feature assignment), whereby the coordinates assigned to the feature vectors are taken into account during the assignment,

[0028] - Rectification of the homography image according to the assignment of the features of the homography image to the features of the reference image.

[0029] The homography matrix generated based on the first feature assignment allows for global perspective distortion correction. The scanned image is corrected using the homography matrix through rotation, translation, and global perspective distortion correction. This allows the document itself to be positioned arbitrarily relative to the optical scanning device used for optical scanning. Homography mapping rotates and shifts the scanned image until its orientation essentially matches the orientation of the digital reference image. Distortions that vary locally, such as those caused by bending and folding of the document, are not corrected in the homography image. Such homography mapping does not allow for the correction of locally varying distortions.

[0030] In the second feature assignment, by assigning the features of the homography image to the features of the reference image, both the coordinates of the features of the homography image and the coordinates of the features of the reference image are taken into account. During such an assignment, the distance between the corresponding feature vectors is determined. The smaller the distance, the better the assignment. The feature pairs with the smallest distance are thus selected as feature pairs of the features of the homography image and the features of the reference image. By taking the coordinates of the features into account, the distance between the features of the homography image and the corresponding features in the reference image is small if the corresponding features are arranged at the same or a similar position in the two images.This allows features to be clearly assigned, even if the same or similar features appear repeatedly in the corresponding images.

[0031] By spatially assigning these features, identical or similar features can be clearly distinguished from one another. However, such spatial assignment only makes sense if the scanned image is aligned in essentially the same way as the reference image. If, for example, the scanned image is rotated by an angle of more than 45° with respect to the reference image, the features to be assigned to one another have completely different coordinates and the coordinates would have no significance for the similarity or assignment of the features. Only by aligning the scanned image by mapping it to the homography image using the homography matrix does the coordinates of the features become meaningfully used as part of the feature vectors, since the mapping using the homography matrix aligns the scanned image to the reference image.

[0032] After the second feature assignment, the features of the homography image and the reference image are assigned to each other so precisely that the homography image can be rectified very precisely according to the second feature assignment, whereby locally different rectifications, such as those caused by bending and folding of a document, are also possible.

[0033] The rectification of the homography image is preferably performed using a freeform rectification method, for example, by interpolating a displacement field as explained above (see discussion by Lee, Seungyong, et al.).

[0034] The features are preferably scaling and / or rotation invariant features.

[0035] Preferably, the scanned images are rectified before superimposition, whereby the rectification can be performed by means of feature assignment and / or by means of image section assignment. Both rectification by means of feature assignment and rectification by means of image section assignment can be performed in combination, so that certain areas of the scanned images are rectified by means of feature assignment and other areas by means of image section assignment. These rectification methods each enable freeform rectification.

[0036] According to a further aspect, the present invention relates to a method for scanning and processing a document comprising the steps

[0037] - scanning a document with an optical scanning device, whereby a scanned image of the document is generated,

[0038] Classifying the scanned image into a document class, wherein features are extracted from the scanned image, and these features are compared with database features of a feature database defining a plurality of document classes, wherein the scanned image is assigned to the document class to which the best match with the database features of the respective document class is achieved.

[0039] Classifying documents using feature mapping allows for precise and accurate classification. This is especially true when using a feature database, as explained above.

[0040] Documents classified in this way can be registered with the digital reference. The digital reference can contain instructions according to which specific information from the document is to be read and processed accordingly. These instructions can, for example, include instructions specifying what type of information (e.g., name, address, telephone number, etc.) should be included in specific areas of the document. This information can then be automatically read and processed according to its meaning.

[0041] Preferably, a single type of scanning device or a single specific scanning device is used as the scanning device. In particular, the same type of scanning device or the same specific scanning device is used to create the feature database. This maintains the quality of consistent scanning, which ensures a correspondingly high quality of the matching.

[0042] The invention is explained in more detail below using the drawings. The drawings show:

[0043] Figure 1a shows an embodiment of a scanning device for optically scanning documents in a roughly schematic simplified representation,

[0044] Figure 1b shows a scanning unit of the scanning device from Figure 1A and a scanning element and an illumination device in their geometrical assignment to the scanning area,

[0045] Figure 2 shows a method for creating a database for document classes in a flowchart,

[0046] Figure 3 shows a scanned image and a digital reference image of a document, each after certain processing steps, and

[0047] Figure 4 shows a flowchart of a method for scanning and processing a document using a database generated using the method of Figure 2. The invention is explained in more detail below using a device for optically scanning documents (Figures 1a, 1b). Such a scanning device 1 comprises a scanning unit 2 and an evaluation unit 3.

[0048] In the present embodiment, the evaluation unit 3 is formed by a conventional workstation or personal computer 4 with an input unit 5 and a display unit 6. The input unit 5 is a computer keyboard, and the display unit 6 is a computer screen.

[0049] However, the evaluation unit 3 can also be formed by any other multi-purpose computer, such as a computer tablet or a smartphone, or can be fully integrated into the scanning unit 2.

[0050] In the present embodiment, the evaluation unit 3 is designed in the form of a cuboid body with a base plate 7, two side walls 8, 9, and a top wall 10. The base plate 7 and the walls 8-10 define a cavity, with the base plate 7 forming a scanning area 11 on which a document 12 to be scanned can be placed. A scanning element 13 and an illumination device 14 are arranged in the top wall 10 (Fig. 1 b).

[0051] The scanning element 13 is a digital camera with a two-dimensional camera chip and a lens. The camera's viewing direction is directed toward the scanning area 11, allowing the entire scanning area to be optically scanned by the camera.

[0052] The illumination device 14 comprises light-emitting diodes as illuminants and is designed to emit white light. For this purpose, light-emitting diodes that emit white light can be provided, or light-emitting diodes of different colors can be provided that together generate white light. If there are several light-emitting diodes of different colors, it is possible to control them so that they can also emit light with different color ranges or spectral ranges by adjusting the light intensity of the differently colored light-emitting diodes accordingly. The illumination device can have reflectors (not shown), which are preferably designed to direct the light onto the scanning area 11 as diffusely as possible. Such illumination is also referred to as dark-field illumination.If primarily or exclusively non-reflective documents are to be scanned, then it may also make sense to provide bright field illumination, since with bright field illumination, a higher light intensity at the scanning area 11 is achieved than with dark field illumination at the same light intensity of the light source.

[0053] The surface of the scanning area 11 is preferably matte and of a uniform color (particularly white). The surfaces of the side walls 8, 9 facing the interior are preferably also matte and of a light color, particularly white, so that the light emitted by the illumination device and reflected by the side walls 8, 9 is directed into the scanning area 11 as diffusely as possible.

[0054] The evaluation unit 3 is connected to the scanning unit 2 via a data line 15 for bidirectional data transmission, so that the evaluation unit 3 can control both the scanning element 13 and the illumination device 14 and read the data acquired by the scanning element 13. The evaluation unit 3 thus serves not only to evaluate images captured by the camera 13, but also to control the entire process.

[0055] Other scanning devices can also be used within the scope of the invention, such as scanning devices that use a line-scan camera that moves parallel to the document being scanned during the scanning process, as is known, for example, from copiers, or even a digital camera that is mounted, for example, on a tripod in a specific spatial relationship to an evaluation area. It is also not necessary for the scanning unit to have an illumination device, since, in principle, a document could also be scanned using ambient light.

[0056] The scanning unit 2 according to the present embodiment has a specific, fixed geometric relationship between the scanning element 13, the illumination device 14, and the scanning area 11. Furthermore, the tubular design of the scanning device 1 with the two side walls, the top wall, and the base plate 7 minimizes interference from ambient light, yet the scanning area 11 is easily accessible from two sides, allowing a user to easily place a document 12 to be scanned in the scanning area 11.

[0057] Such scanning devices 1 are particularly suitable for scanning identification documents, such as identity cards, driving licenses or the like, or for scanning certain forms, such as lottery tickets, registration forms or the like.

[0058] The scanning device described above represents one possible embodiment. However, it can be adapted in a variety of ways, provided a camera and a scanning area are provided. For example, it is possible to provide a single, vertically extending, rod-shaped support between the scanning area and the ceiling wall instead of the side walls.

[0059] The following describes a method for creating a database for document classes, which is carried out using documents scanned with the scanning device 1 explained above. The method begins in step S1 (Figure 2).

[0060] In step (S2), several documents 12 are scanned by the scanning element 13. For this purpose, a document 12 is first placed on the scanning area 11 of the scanning unit 2. The document 12 is illuminated by the illumination device 14 in order to accurately capture all details of the document 12 and to obtain the most uniform scanning conditions possible when scanning different documents. Once the document 12 is in place and correctly illuminated, a user initiates the scanning process, for example, by pressing a button on the scanning unit 2 or on the input unit 5.

[0061] This process is repeated for multiple documents, ie n documents 12, so that n scan images are generated from the respective documents.

[0062] With the scanning device 1, the individual documents are manually placed onto the scanning area 11. It is of course also possible to provide an automatic feeder for feeding the documents into the scanning area 11, so that the scanning of multiple documents can be carried out fully automatically.

[0063] A scanning process can also be initiated automatically, without having to press a button. For example, a proximity sensor can be provided to detect that an object is located in the scanning area 11. Following this, a first preliminary optical scan is performed to check whether the object is stationary in the scanning area 11, i.e., that it is not moving. If this is the case, it can be illuminated with the illumination device 14, and the actual optical scan can be performed.

[0064] In the present embodiment, the scanning element 13 is designed as a camera comprising a two-dimensional camera chip and a lens. The camera's viewing direction is directed toward the scanning area 11, so that, if possible, the entire scanning area and thus the entire document 12 can be captured with a single image. The scanned image 16 thus generated contains a representation of the document 12, which can initially be rotated and / or distorted as desired (Figure 3, top row).

[0065] After the scanning image 16 has been captured, the data representing the scanning image 16 are transmitted to the evaluation unit 3 via the data line 15. The evaluation unit 3 evaluates the scanning images 16.

[0066] During this evaluation, a first feature matching of a scanned image of a first document is performed (step S3). In this process, features, also referred to as keypoints, are identified and extracted from the scanned image 16 using a so-called SIFT algorithm (Scale-Invariant Feature Transform algorithm).

[0067] This involves identifying features that are unchanging, i.e. invariant, with respect to a change in scale. This is carried out, for example, by generating a scale space that essentially comprises a series of images with different scales (different scales) or resolution, these images being calculated from the sample image 16. For each scale, the sample image 16 is convolved with a Gaussian filter with increasing sigma value to produce a series of blurred images. The Difference Of Gaussian (DOG) is then calculated by subtracting two consecutive blurred images. The result is a series of DOG images. Possible feature points are identified as local minima or maxima in the DOG images by comparing each pixel with its neighbors in the current sample image 16 and the neighbors in the scale images.

[0068] A feature point specifies the coordinates of a central point of a feature.

[0069] Potential feature points are verified by fitting a quadratic function to the local image patterns around the potential feature points. Features with low contrast or poorly localized to an edge are discarded. Low-contrast features are sensitive to noise. Feature points along edges are spatially unstable.

[0070] The final position of the feature points is determined by a maximum or minimum of the fitted quadratic function. Each identified feature point is assigned one or more orientations that coincide with the local gradient direction of the scan image 16. This ensures rotation invariance. For this purpose, the gradient magnitude and orientation are calculated for the neighborhood of the feature point. An orientation histogram with 36 columns covering 360° is created from the gradient orientation of key points within a region and the key point. The maxima in this histogram determine the orientation of the feature point. The highest peak in the histogram and any subsequent local maximums within 80% of the highest peak are used to assign the orientation.After the feature points have been identified and the orientation assigned, a descriptor, also called a feature vector, is created for each feature to capture the local appearance of the feature.

[0071] The descriptor is a unique fingerprint for each feature point and thus enables a comparison between two images, as the corresponding feature points in the respective images can be assigned to each other. The descriptor can be viewed as a histogram of the gradient orientation in an area around the feature point, which provides a top-level representation of the local image texture. To create a descriptor, the region around the key point is first divided into, for example, a 4 x 4 grid of subregions. For each subregion, the gradient strength is calculated in the orientation of the image points. Then, an orientation histogram with 8 columns or bins is created for each subregion, corresponding to a total of 128 columns or dimensions. The histogram values ​​capture the dominant gradient direction in the local neighborhood of the feature point. The histogram values ​​form the elements of the descriptor or vector.This descriptor is normalized to increase its robustness to illumination and contrast variations. The normalized vector is the descriptor for the feature points and provides an invariant representation of the local image structure.

[0072] This feature recognition is also performed on a digital reference image 17. The digital reference image is a perfect digital representation of essential elements of the document 12. The individual documents 12 may contain additional information, particularly information added manually or mechanically by a user of the document. Otherwise, the content of the document 12 largely corresponds to the digital reference image 17.

[0073] Subsequently, the actual assignment of the feature points of the scanned image 16 and the reference image 17 takes place by comparing them with each other. When comparing the feature points between the scanned image 16 and the reference image 17, a distance (e.g., Euclidean distance) between the corresponding vectors is determined based on the descriptors, and the feature points of the two images whose distance is the smallest are assigned to each other. This can be carried out, for example, by first evaluating distances between feature points that are below a certain threshold value as potential matches. Subsequently, a ratio test can be performed in which the distance of a feature point of the scanned image 16 to the second-closest feature point of the reference image 17 is calculated. The ratio of the distances of the nearest and second-closest feature points is then calculated.If the ratio is below a certain threshold, for example, 0.8, the match is considered valid. After the match between features of the scanned image 16 and the reference image 17 has been found, the process is repeated by comparing feature points of the reference image 17 with the scanned image 16. Only those feature points that match in both directions can be retained.

[0074] The retained feature points can then be filtered to remove outliers caused, for example, by noise. For this purpose, the so-called RANSAC (Random Sample Consensus) algorithm can be used to identify outliers and errors. Other algorithms for outlier detection are also possible. Matching features between two images is well known. Other known matching methods can also be used, provided they reliably match similar features.

[0075] The feature assignment is used to calculate a homography matrix. The homography matrix allows for perspective rectification of the scanned image and alignment with the reference image. By applying the homography matrix to the scanned image 16, the scanned image is aligned (rotated or shifted) with respect to the reference image 17 and also rectified, creating a homography image 18 (Figure 3). This homography mapping or homography rectification can correct perspective distortions, but not inhomogeneous distortions within the image. Distortions such as those caused by folding or creasing a document cannot be eliminated with homography rectification.

[0076] A direct linear transform (DLT) is used to calculate the homography matrix. It requires at least four matching, non-collinear features from the sample image 16 and the reference image 17. However, more matches can be used, in which case a least-squares solution is calculated. Due to noise, false matches, and other factors, not all matches will match perfectly, so the homography matrix must be determined using methods such as the RANSAC algorithm mentioned above. The RANSAC algorithm repeatedly selects a random subset of matching feature points and calculates the homography matrix. It then determines how many matches match this calculated homography matrix, and the homography matrix with the highest number of matches is selected as the final result.

[0077] During feature determination in step S3, certain predefined areas can be hidden. These are usually areas that contain individual information in the individual documents and are therefore not assignable to one another. Hiding can be achieved, for example, by setting the color and / or brightness values ​​of these areas to a specific value, which corresponds, for example, to the color white. These hidden areas can thus be defined based on the features.

[0078] Once the homography matrix has been determined, it is used to transform the scanned image 16 into the homography image 18. For this purpose, each pixel in the scanned image 16 is multiplied by the homography matrix to obtain the transformed point in the homography image 18. This mapping step represents the perspective rectification mentioned above (step S4). During this perspective rectification, a scanned image 16 can be adjusted in size and orientation to the reference image 17, whereby individual parts of the recorded scanned image 16 can be cropped. However, this is designed such that the information content of the scanned image 16 relating to the document 12 is not cropped. For example, edge detection is performed to detect the edges of the document 12. The scanned image 16 is now cropped such that all edges of the document 12 still remain in the scanned image 16.

[0079] After the transformation, there may be 18 regions in the homography image that do not contain corresponding pixels from the distorted scan image. These regions are filled with neighboring pixels or other inpainting techniques.

[0080] According to this embodiment, the homography image 18 is sharpened using corresponding known algorithms. Known algorithms for contrast adjustment or for smoothing artifacts created during the transformation are applicable here. The (location) coordinates of the features in both the homography image 18 and the reference image 17 are then added to the respective descriptors. This addition of the coordinates is referred to as stamping (step S5).

[0081] In the subsequent step S6, the features of the homography image 18 are again assigned to corresponding features of the reference image 17, whereby the assignment takes into account the coordinates assigned to the descriptors (second feature assignment). The descriptors comprise, for example, 130 dimensions, of which 128 of these dimensions originate from a 4 x 4 grid of subregions, each with 8 columns or bins (see above). The additional two dimensions are the coordinates. These two additional dimensions can be weighted in the same way as all other dimensions combined. However, it also makes sense to weight the two additional dimensions (= coordinates) more heavily, for example with a weighting factor that results in at least five times the weighting or at least ten times the weighting of the coordinates compared to the other dimensions.The coordinates can also be weighted so heavily that they have the same weight as the remaining 128 dimensions. Thus, the coordinates can be included in the comparison of the features for their assignment with the same weight as the other dimensions.

[0082] By taking the coordinates into account, the location of the features is considered much more closely than in the first feature assignment, which means that only the features that are close to each other in the reference image and the scan image are actually assigned to each other. This is reliably possible here because a perspective correction was previously performed, in which the reference image and the scan image were aligned to each other so that the corresponding features are located in similar locations in the respective images. Without this alignment, the two images can be arranged, for example, rotated by 90° to each other, which would result in corresponding features being located in completely different locations. Taking the location coordinates into account in the feature vector would cause significant misassignments.Based on the newly assigned features, displacement vectors are calculated between assigned features of the homography image 18 and the reference image 17, which is also referred to as vectoring.

[0083] It may be that contiguous areas of the documents have no or very few features, so that no displacement vectors exist for these areas due to the feature assignment. In this case, it makes sense to assign image sections of the homography image 18 and the reference image 17 to each other and determine a corresponding displacement vector for the assigned image sections. The assignment of image sections is also referred to as template matching.

[0084] A displacement vector field is generated from the displacement vectors of the feature assignment and the image section assignment. The displacement vectors are included in the displacement vector field either directly or in an averaged or interpolated form. For areas where no displacement vectors are present, corresponding displacement vectors are interpolated.

[0085] Using this displacement vector field, the homography image 18 is subjected to freeform rectification. First, control points are selected using the displacement vector field, which are distributed as evenly as possible across the homography image 18 and the reference image 17. The connection between the control points in these images is given by the displacement vectors. A system of linear equations is established based on the control points. This system of equations is solved to determine weights for each control point. These weights determine the strength and direction of the transformation of each control point.

[0086] For each pixel in the homography image 18, a new position in a transformation image 19 is calculated based on the weights and a radial basis function of the control points. The pixel values ​​are transferred directly if they perfectly match the grid of the transformation image 19. Since this is not the case in most cases, the pixel values ​​for the transformation image 19 are interpolated. This can be done, for example, using bilinear or bicubic interpolation.

[0087] To avoid overfitting, especially when many control points are used, a regular realization term can be added to the transformation. This smooths the transformation and avoids high-frequency deformations. Further post-processing steps can be performed on the transformation image 19, for example by sharpening, cropping, or performing other image adjustments to improve quality. The transformation image 19 is thus a freeform-rectified scan image. Using the methods explained above with reference to steps S3 to S7, the scan image 16 is aligned and rectified with respect to the reference image 17, whereby a pixel-precise match can be achieved. In step S8, a check is made as to whether a scan image of another document is available that is to be aligned and rectified with respect to the reference image 17.If this is the case, the method flow proceeds to step S3 and the alignment and rectification of the further scanned image 16 is carried out.

[0088] If it is determined in step S8 that the scanned images of all documents are aligned and rectified, the method flow proceeds to step S9, in which the scanned images aligned and rectified in this way are superimposed. Here, the pixels at the same locations of the aligned scanned images 19 are averaged, and the resulting average value is entered as a point value at the corresponding location in a prototype image. The average value can be a mean value, a median value, or even a mean value in which the pixel values ​​of the different aligned scanned images 19 are weighted differently. The prototype image thus represents an image of the document 12 that is rectified and aligned with respect to the reference image 17 and shows the document 12 as it is captured by the scanning device 1.Unlike the reference image, the prototype image therefore contains the effects of the optical scanning by the scanning device 1. Compared to the reference image, the prototype image may contain certain blurriness and / or brightness and / or color deviations. These deviations may be minor, but even minor deviations can cause errors during further processing.

[0089] Features are extracted from the prototype image thus generated. The features are preferably SIFT features. However, other types of features, such as ORB features or SURF features, can also be used. The features are preferably scale- and / or rotation-invariant features.

[0090] The features comprise at least one descriptor or feature vector and coordinates that define the location of the features in the image. Preferably, the descriptor or feature vector is determined based on the prototype image and the associated coordinate is determined based on the reference image. This provides a description of the feature as seen by the scanning device 1, with the exact location of the reference image being used as the location. The features thus generated are stored in a database, with the features of such a prototype image representing a document class. The matching representations in the documents of a document class are contained in the respective prototype image. The features derived from this thus apply to all documents of a document class. They are thus representative of a specific document class.

[0091] The method ends with step S11. This method can be repeated for different document classes, with multiple copies of the documents of a document class being scanned and the scanned images being evaluated according to the above method for extracting the features. Such a database can be used to classify documents, as explained below using an exemplary method (Figure 4).

[0092] In this method, a document of any document class can be scanned with the scanning device 1 (step S13).

[0093] Features are extracted from the resulting scan image, and these features are compared with the feature groups of the different document classes. The comparison can be performed, for example, by calculating a distance, particularly the Euclidean distance. The document is then assigned to the document class with the smallest deviation from the corresponding features.

[0094] In the next step, the scanned image is registered, i.e. aligned, with the digital reference image of the corresponding document class. This can be carried out in a similar way to steps S3 to S7 of the method explained above. The database preferably contains processing instructions for the individual document classes, specifying how the individual documents are to be processed. These processing instructions typically include instructions as to which areas of the document are to be read out and which information they contain. For example, the areas in which a surname, first name, address, telephone number, email address, or the like are to be contained can be defined, with the type of respective information also being stored accordingly. This allows predetermined information content to be reliably read out and further processed.Due to the registration of the scan image with the digital reference image, the corresponding instructions defined on the basis of the reference image are immediately applicable to the respective area of ​​the scan image.

[0095] In step S16, the corresponding information is read from the scanned image based on these instructions and passed on for further processing.

[0096] The method is terminated with step S17.

[0097] The methods and device described above allow for the partially or fully automated processing of documents, even if they were bent or folded and are therefore no longer smooth. The documents can be assigned to document classes and processed accordingly, and in particular, read out. This allows, in particular, the processing of machine-printed documents into which people have inserted handwritten text.

[0098] 1 scanning device

[0099] 2 scanning unit

[0100] 3 Evaluation unit

[0101] 4 workstations

[0102] 5 Input unit

[0103] 6 Display unit

[0104] 7 Base plate

[0105] 8 Side wall

[0106] 9 Side wall

[0107] 10 Ceiling wall

[0108] 11 Scanning range

[0109] 12 Document

[0110] 13 scanning element

[0111] 14 Lighting equipment

[0112] 15 data line

[0113] 16 scanning image

[0114] 17 Reference image

[0115] 18 Homography image

[0116] 19 Transformation image

Claims

Claims 1 . Method for creating a database for document classes, wherein each document class is defined by a plurality of features, wherein a plurality of copies of a document type to be classified are scanned with a scanning device, wherein in each case a scanned image is generated, and each scanned image is aligned with a digital reference image of the document type to be classified, the scanned images aligned in this way are superimposed on one another, wherein the individual pixels are averaged so that a prototype image is generated, wherein features are again determined from the prototype image and are entered in the database as the database features defining the document class.

2. Method according to claim 1, characterized in that each scanned image is also rectified during alignment with the digital reference image, wherein the rectification is preferably carried out by means of a feature assignment and / or by means of an assignment of image sections.

3. Method according to claim 1 or 2, characterized in that one type of scanning device or a single specific scanning device is used to create a specific database.

4. Method according to one of claims 1 to 3, characterized in that the alignment of each scanned image is carried out by means of a feature assignment or by means of an image section assignment.

5. Method according to one of claims 1 to 4, characterized in that that the database features each comprise a feature vector describing the respective feature and coordinates defining the location of the respective feature, wherein the feature vector is obtained from the prototype image and the corresponding coordinates are obtained from the reference image.

6. Method according to one of claims 1 to 5, characterized in that certain predefined areas of the aligned scan images are not taken into account.

7. Method according to one of claims 1 to 6, characterized in that a meaning is assigned to certain predefined areas.

8. Method according to one of claims 1 to 7, characterized in that the following steps are carried out to align the scanned images with the digital reference image: - Rectifying and aligning the respective scanned image by means of a feature assignment, whereby features of the scanned image, each comprising a feature vector, are assigned to corresponding features of the reference image of the document and, in accordance with this feature assignment, a homography matrix is determined with which all pixels of the respective scanned image are mapped to form a homography image, - Adding the coordinates of the features in the homography image to the respective feature vectors and Adding the coordinates of the features in the reference image to the respective feature vectors and, - re-assigning the features of the homography image to corresponding features of the reference image of the document, taking into account the coordinates assigned to the feature vectors, - Rectification of the homography image according to the assignment of the features of the homography image to the features of the reference image.

9. Method according to claim 8, characterized in that the rectification of the homography image is carried out using a free-form rectification method.

10. Method according to one of claims 1 to 9, characterized in that the features are scaling and / or rotation invariant features.

11. Method according to one of claims 1 to 7, characterized in that before the scanning images are superimposed, they are rectified, wherein the rectification is carried out by means of a feature assignment and / or by means of an assignment of image sections.

12. A method for scanning and processing a document comprising the steps - scanning a document with an optical scanning device, whereby a scanned image of the document is generated, - Classifying the scanned image into a document class, wherein features are extracted from the scanned image and these features are compared with database features of a feature database defining a plurality of document classes, wherein the scanned image is assigned to the document class to which the best match with the database features of the respective document class is achieved.

13. Method according to claim 12, characterized in that the scanned image is registered to a reference image on the basis of a feature assignment to the corresponding database features.

14. Method according to claim 12 or 13, characterized in that the feature database is a database generated by the method according to one of claims 1 to 10.

15. Method according to one of claims 12 to 14, characterized in that a type of scanning device or a single specific scanning device is used as the scanning device, in particular a type of scanning device or a single specific scanning device which has also been used to create the feature database.

Images