Digital stamp localization and overlapping text removal method and apparatus

Abstract

To provide a method and device configured to devise an algorithm that accounts for stamp or seal color, density, and / or shape to accomplish stamp localization and segmentation.SOLUTION: In a form recognition system, a deep learning system may be trained to perform stamp localization for stamp removal to facilitate form recognition. A stamp mask identifies locations of stamps or seals on forms. When a stamp or seal overlaps with underlying text on a form, and a color or grayscale of the stamp or seal is sufficiently similar to that of the underlying text, a combination of the stamp mask and the line mask identifying pixels on the stamp may enable removal of the stamp or seal without degrading the underlying text in the form, and facilitate form recognition.SELECTED DRAWING: Figure 11

Description

[Technical field]

[0001] Aspects of the invention relate to document and form analysis, and more particularly to the placement and removal of ink stamps or seals within images, particularly images such as invoices, receipts, or official documents. As used herein, a reference to either a stamp or seal refers to either or both of these. A reference to either an invoice, receipt, or official document refers to any one or more of these. [Background technology]

[0002] In the field of document and form analysis, document processing with Robotic Process Automation (RPA) is increasingly being used, which includes both feature localization and form matching. Document and form analysis can present additional challenges when extracting correction information from a large number of forms to train the document processing system. Such challenges include image scanning noise, degradation in quality and color, image warping, and rotation.

[0003] A recent problem is one in which forms contain stamps. In the process of training a document processing system with documents, some training documents may contain such stamps. These stamps may appear in various locations around the document, such as on invoices and receipts. Often, stamps may appear in the header or footer area of ​​the document, or near the top or bottom of the document (for example, near the lines identifying the billing company and / or address). However, some stamps may appear in various locations on the document. Summary of the Invention [Problem to be solved by the invention]

[0004] It would be desirable to devise an algorithm that takes into account the color, density and / or shape of the stamp or impression to achieve stamp localization and segmentation. [Means for solving the problem]

[0005] Aspects of the present invention provide a deep learning based model for predicting the location of stamps in both color and grayscale form and for segmenting stamp pixels. In one embodiment, segmentation can be achieved using line masks.

[0006] Aspects of the invention utilize both color filter methods and generalized grayscale models, thereby improving the accuracy and efficiency of processing both color and grayscale forms. If the stamp or seal is grayscale and the underlying form text is also grayscale, the stamp or seal may be substantially indistinguishable from the text. More generally, if the stamp or seal is the same color as the underlying text, the stamp or seal may be substantially indistinguishable from the text. Stamp location prediction helps enable removal of grayscale stamps or seals.

[0007] In one embodiment, the stamp localization model provides two main channel outputs. The first channel can output the stamp or seal pixel segmentation map (stamp mask). The second channel can output a mask that estimates the location of the foreground text lines and includes regions that mask only the stamp text (line mask). The stamp mask allows for localization and detection of the stamp on the form. The line mask allows for further segmentation of the foreground text lines while preserving the original pixels of the text lines. In this way, a balance can be achieved between the stamp pixel estimation and the text line detection, improving the performance of stamp localization and line segmentation. [Brief description of the drawings]

[0008] Various aspects of the present invention will now be described in detail with reference to illustrative, non-limiting embodiments and with reference to the accompanying drawings, which are described as follows:

[0009] [Figure 1] FIG. 1 is an image of various types of forms bearing various stamps or seals. [Diagram 2] FIG. 2 is an image of various types of forms with various stamps or seals. [Diagram 3] FIG. 3 is an image of various types of forms with various stamps or seals. [Figure 4] FIG. 4 is an image of various types of forms with various stamps or seals. [Diagram 5] FIG. 5 is an image of various types of forms with various stamps or seals. [Figure 6] FIG. 6 is an image of various types of forms with various stamps or seals. [Figure 7] FIG. 7 is an image of various types of forms with various stamps or seals. [Figure 8] FIG. 8 is an image of various types of forms with various stamps or seals. [Figure 9] FIG. 9 is an image of various types of forms with various stamps or seals. [Figure 10] FIG. 10 is an image of various types of forms with various stamps or seals. [Figure 11] FIG. 11 is a flow chart illustrating aspects of operation according to one embodiment. [Figure 12] FIG. 12 is a diagram illustrating a semantic network according to one embodiment. [Figure 13] FIG. 13 is a high-level block diagram according to one embodiment. [Figure 14]FIG. 14 is a high-level block diagram illustrating a portion of FIG. 12 according to one embodiment. [Figure 15] FIG. 15 is a high-level block diagram illustrating a portion of FIG. 13 according to one embodiment. [Figure 16] FIG. 16 is a high-level chart illustrating the order of processing of invoices according to one embodiment. [Figure 17] FIG. 17 shows images of an original document, a result of processing according to one embodiment, and a further result of processing according to one embodiment. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0010] Aspects of the present invention address challenges that stamps or seals on documents can present to document processing systems, including training such systems. Such stamps or seals can be used for a variety of purposes. For example, Japanese invoices and other documents may use a seal, or hanko, as a form of acknowledgment or consent. Other types of invoices may use "PAID" or "REC'D" stamps to help the reader understand the status of the invoice (e.g., "REC'D" does not mean "PAID"), but "PAID" does mean that the invoice has been received.

[0011] The ink of a stamp or seal may be of various colors (e.g., red or blue, or both) or may be relatively monotone (e.g., black or grayscale). Stamps may include foreground text. Different companies may vary in the design and content of stamps. Such variations are useful for form identification and matching purposes.

[0012] Stamps may also cover foreground text, overlaying important target text information and useful location information for accurate form registration. As a result of such covering and / or overlaying, document processing systems may misidentify forms and / or misidentify information, such as keyword location and content, and match forms with other forms, potentially reducing the accuracy and quality of the overall system.

[0013] Stamps come in a variety of shapes, including squares, rectangles, other polygons, and circles. Some of these shapes can appear on invoices and receipts with text that is angled relative to the underlying document. This makes it appear as if the shape has been rotated at some angle, even slightly.

[0014] Documents with grayscale forms or grayscale stamps can be more difficult to process than those mentioned above, because the grayscale stamp may differ from the foreground text only in density. Furthermore, the grayscale form may contain pixel density information, which can make text and feature segmentation difficult. Also, overlapping stamp text and foreground text can make both difficult to read. In some cases, the foreground text may contain the same color (e.g., red) as the stamp text.

[0015] In some cases, logos with certain shapes, such as squares or circles, can be recognized as stamps. Dealing with such logos can complicate the development and implementation of algorithms to remove stamps or seals. Another shape that frequently appears on forms is the QR code. Usually QR codes do not affect other text on the form, but in this case comes the stamp.

[0016] It may be desirable to digitally remove stamps from invoices and receipts so that what is underneath the stamp can be read.

[0017] In the following description, aspects of the present invention address various of the aforementioned challenges by providing a deep learning based model that predicts both the location of a stamp and an appropriate mask for segmenting the stamp pixels. In the description herein, the terms "form," "document," and "digital document" may be used interchangeably.

[0018] FIGS. 1 through 10 show different examples of when a stamp or seal may appear on a document, such as an invoice or receipt. In some of these examples, the stamp or seal covers or overlaps the text of the underlying document. In still other of these examples, the stamp or seal appears separate from the text of the underlying document. In some of these examples, the stamp or seal appears slanted or oblique to the text of the underlying document. In various of these examples, the document may include different colors, and the stamp or seal may also include different colors, in some cases different from the color or colors of the document in which the stamp or seal appears.

[0019] In Figure 1, invoice 100 is in grayscale. The text 110 in the upper right corner of invoice 100 is black. Overlaid on part of the text 110 is a red square stamp 120 that contains some text and patterns.

[0020] In Figure 2, invoice 200 is formed in multiple colors including purple portions 210, blue portions 220, and grayscale / black portions 230. A red square stamp 240 overlays a portion of the invoice text.

[0021] In Figure 3, the invoice 300 is black and white with black text 310. Overlaid on part of the text 310 is a red square stamp 320 containing some text and patterns.

[0022] In FIG. 4, the bill of lading 400 is mostly black and white, but includes a red rectangular stamp 410 with a number and accompanying text (a blue logo 420 and a blue stamped date 430).

[0023] 5, purchase order 500 includes black text 510, magenta text 520 and a magenta square stamp 530 overlaying a portion of the magenta text 520. An additional red circular stamp 540 is just outside the field of purchase order 500.

[0024] In Figure 6, a purchase order 600 contains a number of blue areas 610 as part of the document's formatting. There is also magenta text 620 and black text 630. A red circular stamp 640 appears on the document without covering any text. A QR code 650 also appears on the document. The QR code 650 is square and at first glance appears to be a stamp. In one embodiment, the QR code is distinct from the stamp.

[0025] Stamps or seals to distinguish text do not only appear on Japanese documents. In FIG. 7, invoice 700 has black text. A black "PAID" stamp 710 is on invoice 700 but does not overlap the text. In FIG. 8, invoice 800 is shown with the same text as invoice 700, but with a grayscale "REC'D" stamp over a portion of the text. FIG. 9 shows invoice 900 with the same text as invoices 700 and 800, but with a red stamp 920 over a portion of the text 920, also in red. The border and header portion of invoice 900 are black, as shown at 930. The header portion of invoice 900 contains a logo that must be distinguished from stamp 910. In FIG. 10, invoice 1000 is shown, also with the same text as invoices 700, 800, and 900, but with magenta text 1010 and a blue "REC'D" stamp 1020 overlaying a portion of the magenta text. Invoice 1000 also includes a black border 1000.

[0026] Figures 1 through 10 provide a non-exhaustive list of form, form color, and stamp or seal color combinations that need to be differentiated for document processing. The stamps or seals in these figures appear in different locations on different forms. Among the challenges in processing such forms is recognizing the form type and identifying the appearance and location of the stamps on the various forms.

[0027] FIG. 11 shows a flow chart illustrating the operation of aspects of the invention according to one embodiment. At 1105, a digital document is input. Depending on the embodiment, the digital document may be an invoice, a bill of lading, a purchase order, or any other type of document that has a stamp or seal on it. The digital document may be input in a variety of ways that are not critical to the practice of the embodiment. The digital document may be the result of a scan or may be retrieved from storage. At 1110, it is determined whether the digital document contains color. Color may appear anywhere in the digital document, such as as part of text, as part of one or more tables in the digital document, as a stamp on the digital document, etc. In one embodiment, this determination may be the result of an RGB or CYMK pixel analysis.

[0028] In response to determining that the digital document contains color, the digital document is input into a stamp model at 1115 and the stamps are identified at 1120. The process cycles between 1125 and 1120 until all stamps have been identified. Once all of them have been identified, the stamp area of ​​each previously identified stamp is identified at 1130. Those skilled in the art will appreciate that if the stamp is the same color as the underlying text, it will be necessary to process the stamp in a grayscale manner even if the color is not black, white, or some type of grayscale.

[0029] In response to determining at 1110 that the digital document does not contain color, i.e., the digital document is black and white or grayscale, then at 1145 the image is input into a stamp model and the stamps are identified at 1150. The process cycles between 1150 and 1160 until all stamps have been identified. Once they have all been identified, at 1165 the stamp region of each previously identified stamp is identified.

[0030] In response to a determination at 1135 that one or more of the located stamps have the same color as the underlying text in the form, flow can proceed to 1165 where the located stamps can be identified as substantially equivalent to grayscale regions having the same color as the underlying text. In response to a determination that the colors are different, color filtering is performed within the stamp regions at 1140 such that the color stamps can be removed from the digital document at 1190.

[0031] As a second channel, at 1145, the image is input into the stamp model, after which line masking is performed on the foreground text of the digital document at 1155. At 1170, the generated line mask may be used to identify the boundaries of the stamp or seal within the digital document. These boundaries may be identified in response to identifying the grayscale or corresponding stamp regions at 1165.

[0032] At 1175, it is determined whether any of the stamp regions overlap with text in the underlying digital document. In response to a determination that there is overlap, at 1180, the generated line mask may be used to identify pixels of the overlapping stamp regions. The stamps may then be removed from the digital document at 1190. In response to a determination that there is no overlap, i.e., that the stamps occur in the digital document separately from other text in the digital document (e.g., for one or more of the stamps of Figures 5, 6, and 7), at 1190, the stamps may be removed from the digital document with one of several known techniques, including color filtering or line masking, or other masking and / or pixel removal techniques, as previously described.

[0033] After stamp removal at 1190, in an embodiment, the remaining digital document may be a form that may be used for form recognition or processing, or training a deep learning model used for stamp location and duplicate text removal.

[0034] It should be noted that while the flowchart of FIG. 11 illustrates an exemplary sequence of operations, the present invention is not limited to the particular sequence shown. Various elements of FIG. 11 may be performed in parallel or in a different order than the embodiment of FIG. 11 illustrates. Furthermore, a deep learning system according to an embodiment may determine that for some digital documents, an appropriate mix of stamp masking and line masking may be performed. For some digital documents, line masking may not be necessary and may be omitted. In such situations, only stamp masking may be required. For other digital documents, stamp masking may not be possible and may be omitted. In such situations, only line masking may be performed.

[0035] FIG. 12 is a high-level diagram illustrating a semantic network according to one embodiment. In FIG. 12, an input image 1210 is passed to an input network 1220, which may be a tensor network in one embodiment. An encoder network 1230, which may be a convolutional neural network (CNN), specifically a Resnet network in an embodiment, receives input from the input network 1220. A self-attention mechanism 1240 receives the output of the encoder network 1230 and may provide input to a decoder network 1250. An adder 1260 may sum the outputs of the decoder network 1250 according to a desired weighting of the outputs and provide an output to an output network 1270, which may also be a tensor network in some embodiments, resulting in an output 1280.

[0036] In one embodiment, the self-attention mechanism based on CNN features can adjust the learned weights in the encoder network 1230 to provide greater weighting to more important features. In one embodiment, correlations between individual pixels can be calculated to enable weight adjustment. In one embodiment, the self-attention mechanism can include an attention gate module, which can aggregate information from the encoder network 1230 and upsample information while adjusting the weights. In one embodiment, the network can utilize an implicit inverse attention module and a set of explicit edge attention guidance to establish a relationship between the region where the stamp may be located and the boundary of the located stamp.

[0037] In one embodiment, the self-attention mechanism 1240 can obtain far-reaching feature information and adjust the weights of feature points by aggregating correlation information of global feature points. Although the self-attention mechanism embodiment can improve the recognition accuracy of deep learning models, it may suffer from excessive time, slow training speed, and / or excessively large number of weighting parameters. One approach to reduce time is to use tensor decomposition, which can decompose high-rank tensors into linear combinations of low-rank tensors. Thus, for example, the input tensor network 1220 may have a rank of 3, while the output tensor network 1270 may have a rank of 2.

[0038] Resnet networks can provide many convolutional layers, potentially thousands. Common numbers of layers for such networks are 18, 34, 50, 101, and 152. In one embodiment, as few as 18 convolutional layers may be sufficient.

[0039] From the model output, there can be two main channel outputs. The first channel outputs a pixel segmentation map of the stamp or seal. The second channel outputs a mask of the area that estimates the location of the foreground text line and masks only the stamp text. The stamp mask (first channel) can be used to localize and detect the stamp in the form. The line mask (second channel) can then be used to further segment the foreground text line without damaging the original pixels of the text line. This solution balances the pixel estimation of the stamp and the detection of the text line, achieving high performance stamp localization and line segmentation.

[0040] 13 is a high-level block diagram of a computing system 1300 that may implement one or more deep learning systems 1200 to perform stamp location and text removal according to an embodiment. Depending on the embodiment, the imaging input 1310 may have a library of images that may include training images as well as images to be processed. In one embodiment, the imaging input 1310 may include a scanner, camera, or other imaging device for scanning bills or other documents as inputs and may provide training data for the one or more deep learning systems 1200. The processing system 1350 may be a separate system or may be part of the imaging input 1310 depending on the embodiment. The processing system 1350 may include one or more processors, one or more storage devices, and one or more solid-state memory systems (which, unlike storage devices, may include both non-transient and transient memory).

[0041] In one embodiment, the processing system 1350 may include the deep learning system 1200 that the stamp filter 1320 and the mask filter 1330 use to locate stamps and perform text removal, depending on the embodiment. In other embodiments, either the stamp filter 1320 or the mask filter 1330 may implement its own deep learning system 1200, or each of the stamp filter 1320 and the mask filter 1330 may implement its own deep learning system 1200. In an embodiment, each of the stamp filter 1320 and the mask filter 1330 may include one or more processors, one or more storage devices, and one or more solid-state memory systems (which, unlike storage devices, may include both non-transient and transient memory). In an embodiment, the additional storage device 1360 is accessible to one or more of the stamp filter 1320, the mask filter 1330, and the processing system 1350 via the network 1340, which may be a wired or wireless network, or, in an embodiment, the cloud.

[0042] In an embodiment, storage 1360 may include training data for one or more deep learning systems 1200 and / or may include stamp location and / or mask filtering results. Storage 1340 may store input images from imaging input 1310 and / or may store images to be processed and / or may store processed images with the stamp or seal removed.

[0043] If the network 1340 is a cloud system for communication, one or more portions of the computing system 1300 may be separate from other portions. In one embodiment, the network 1340 may be a cloud-based system, even if the various elements are collocated.

[0044] 14 is a high-level view of an apparatus for weighting nodes in a deep learning system according to one embodiment. As training of a deep learning system progresses according to one embodiment, the various node layers 1420-1,...,1420-N can communicate with a node weighting module 1410 that calculates weights for the various nodes, and a database 1450 that stores the weights and data. As the node weighting module 1410 calculates updated weights, these may be stored in the database 1450.

[0045] FIG. 15 is a high-level block diagram of an apparatus for operating a deep learning system according to one embodiment. In FIG. 15, one or more CPUs 1510 communicate with CPU memory 1520 and non-volatile storage 1550. One or more GPUs 1530 communicate with GPU memory 1540 and non-volatile storage 1550. Generally speaking, a CPU can be understood to have a certain number of cores, each with a certain function and capacity. A GPU can be understood to have a larger number of cores than a CPU, often significantly larger number of cores. In an embodiment, each GPU core may have a lower power and capacity than a CPU core, but can perform specialized functions in a deep learning system, allowing the system to operate more quickly than if the CPU cores were used.

[0046] Depending on the embodiment, one or more of the stamp filter 1320, the mask filter 1330, the processing system 1350, and the node weighting module 1410 may employ the apparatus shown in FIG.

[0047] Figure 16 is an example of a grayscale output according to one embodiment. Input is form 1605 of stamps 1610 and 1620. As shown in Figure 11, for each channel, output 1625 with located stamps 1630 and 1640 and output 1645 with mask 1650 are combined to provide output 1665 where stamp locations 1670 and 1680 are identified.

[0048] FIG. 17 shows the left side of FIG. 11 and the output from the first channel. Input form 1710 has a number of color and grayscale stamps, including stamps 1711-1715. 1720 is the output from the first channel of FIG. 11 and is stamped with located stamps, including located stamps 1721-1725. Final output 1730 has the locations of all stamps identified, including locations 1731-1735. Note that as FIG. 17 shows, not all of the stamps in input 1710 are identified in either 1720 or 1730.

[0049] Although the embodiments of the present invention have been described in detail above, those skilled in the art will understand that various modifications are possible within the scope and spirit of the present invention. In particular, the identification of specific modifications in the course of this description is not intended to be an exhaustive list. Rather, the identification of these modifications provides examples to inform those skilled in the art of the types of modifications being considered herein. Therefore, the scope of the present invention should be construed as being limited only by the scope of the following claims.

Claims

1. a) in response to an input of a digital document, determining whether the digital document includes color; b) in response to determining that the digital document includes color, using a deep learning system to identify one or more first stamps in the digital document and identify respective regions of the one or more first stamps; c) in response to determining that the digital document does not include color, using the deep learning system to identify one or more second stamps in the digital document and identify respective regions of the one or more second stamps; d) in response to determining that one of the one or more first and second stamps overlays underlying text in the digital document, using the deep learning system to determine whether a color of one of the one or more first and second stamps is sufficiently similar to a color of the underlying text in the digital document; e) in response to determining that the color of the one of the one or more first and second stamps is sufficiently similar to the color of the underlying text in the digital document, using the deep learning system to perform line masking to identify pixels of the one of the one or more first and second stamps in the digital document for removal.

2. The method of claim 1 , further comprising repeating d) and e) for all of the one or more first and second stamps.

3. f) in response to c), using the deep learning system to perform the line masking and identify the one pixel of the one or more second stamps in the digital document for removal; and The method of claim 1 , further comprising: g) repeating f) for all of the one or more second stamps.

4. h) in response to determining that the respective color of the one or more first stamps differs from a color of the underlying text in the digital document, performing color filtering within the region of the one of the one or more first stamps using the deep learning system; The method of claim 1 , further comprising: i) repeating h) for all of the one or more first stamps.

5. j) in response to identifying a pixel of the one of the one or more second stamps in the digital document, using the deep learning system to digitally remove the one of the one or more second stamps from the digital document; The method of claim 1 , further comprising: k) repeating j) for all of the one or more second stamps.

6. l) in response to determining that the one of the one or more first and second stamps does not overlay the underlying text in the digital document, digitally removing the one of the one or more first and second stamps from the digital document using the deep learning system; The method of claim 1 , further comprising: m) repeating l) for all of the one or more first and second stamps.

7. n) in response to h), digitally removing the one of the one or more first stamps from the digital document using the deep learning system; and The method of claim 4 , further comprising: o) repeating n) for all of the first stamps.

8. The method of claim 1 , wherein the one or more first stamps are color stamps.

9. The method of claim 1 , wherein the one or more second stamps are grayscale stamps.

10. 10. The method of claim 1, wherein the deep learning system comprises a system selected from the group consisting of a Convolutional Neural Network (CNN) and a Resnet network.

11. At least one processor; and a non-transitory memory containing instructions that, when executed, enable the machine learning system to perform a method, the method comprising: a) in response to an input of a digital document, determining whether the digital document includes color; b) in response to determining that the digital document includes color, using a deep learning system to identify one or more first stamps in the digital document and identify respective regions of the one or more first stamps; c) in response to determining that the digital document does not include color, using the deep learning system to identify one or more second stamps in the digital document and identify respective regions of the one or more second stamps; d) in response to determining that one of the one or more first and second stamps overlays underlying text in the digital document, using the deep learning system to determine whether a color of one of the one or more first and second stamps is sufficiently similar to a color of the underlying text in the digital document; e) in response to determining that the color of the one of the one or more first and second stamps is sufficiently similar to the color of the underlying text in the digital document, using the deep learning system to perform line masking to identify pixels of the one of the one or more first and second stamps in the digital document for removal.

12. The apparatus of claim 11 , further comprising repeating d) and e) for all of the one or more first and second stamps.

13. The method further comprising: f) in response to c), using the deep learning system to perform the line masking and identify the one pixel of the one or more second stamps in the digital document for removal; and The apparatus of claim 11 , further comprising: g) repeating f) for all of the one or more second stamps.

14. The method further comprising: h) in response to determining that the respective color of the one or more first stamps differs from a color of the underlying text in the digital document, performing color filtering within the region of the one of the one or more first stamps using the deep learning system; The apparatus of claim 11 , further comprising: i) repeating h) for all of the one or more first stamps.

15. The method further comprising: j) in response to identifying a pixel of the one of the one or more second stamps in the digital document, using the deep learning system to digitally remove the one of the one or more second stamps from the digital document; The apparatus of claim 11 , further comprising: k) repeating j) for all of the one or more second stamps.

16. The method further comprising: l) in response to determining that the one of the one or more first and second stamps does not overlay the underlying text in the digital document, digitally removing the one of the one or more first and second stamps from the digital document using the deep learning system; The apparatus of claim 11 , further comprising: m) repeating l) for all of the one or more first and second stamps.

17. The method further comprising: n) in response to h), digitally removing the one of the one or more first stamps from the digital document using the deep learning system; and 15. The apparatus of claim 14, further comprising: o) repeating n) for all of the first stamps.

18. The apparatus of claim 11 , wherein the one or more stamps are color stamps.

19. The apparatus of claim 11 , wherein the one or more stamps are grayscale stamps.

20. 12. The apparatus of claim 11, wherein the deep learning system comprises a system selected from the group consisting of a Convolutional Neural Network (CNN) and a Resnet network.

Images