Method and system for identifying field labels on a form - Patents.com

Abstract

To provide a method and a system for allowing identification of a field label in a written form and generation of an unwritten form after the identification in repetition of a comparatively small number without requiring use of AI technology or substantial compilation of a data set for training an AI system.SOLUTION: In a method for identifying a field label from a written form and compare two or several copies of forms of the same type, or differently written forms by using image processing, one text character string is generated on each text line on each written form, the text character string is converted into a vector, vectors from different written forms are compared with one another to identify a common word showing the field label. In one embodiment, a histogram is generated to show appearance frequencies of characters and words. A histogram numeric value also shows the field label.SELECTED DRAWING: Figure 5

Description

[Technical field]

[0001] Aspects of the present invention relate to processing images of forms, and more particularly, to processing images of completed forms to identify fields within the form. [Background technology]

[0002] Automatically finding field labels in form images is useful for automating document processing. To understand how the form content matches the form's fields, the form's field labels need to be known. In many cases, field labels can be identified on a blank form. However, a blank form is not always available. In that case, it is necessary to identify the field labels from an image of a filled-in form.

[0003] There are unique challenges in reading and comparing completed forms. For example, the same form may have different contents entered into each field in different ways. As another example, completed form images may be scaled to different degrees, resulting in different completed forms having different vertical and horizontal dimensions. Summary of the Invention [Problem to be solved by the invention]

[0004] Due to the aforementioned differences, two filled-in form images of the same type may differ significantly from each other not only in appearance but also in scale, rotation, font style, and / or font size. As a result, it may be difficult to directly apply image processing techniques such as image feature descriptors and image transformations to find fields in filled-in forms.

[0005] Furthermore, applying artificial intelligence (AI) techniques to the problems mentioned above, whether supervised or unsupervised, can be time consuming and difficult to accomplish. [Means for solving the problem]

[0006] In view of the foregoing, one object of the present invention is to identify field labels by comparing images of filled-in forms of the same type. Among other things, embodiments of the present invention provide: • A highly robust string encoding technique under variations in scale and rotation, font size and font style to solve image alignment of a line of text between two different images of a filled-in form. • A field label discrimination histogram hierarchical technique that can discriminate between field labels and filled-in content while tolerating optical character recognition (OCR) error tolerance. • Reducing the need for numerous images of completed forms to identify field labels.

[0007] Aspects of the present invention enable the identification of field labels in filled-out forms, and the generation of unfilled forms thereafter, in a relatively small number of iterations, without requiring the use of AI techniques or the compilation of substantial data sets to train an AI system. [Brief description of the drawings]

[0008] The foregoing and other features and aspects of the present invention will now be described in detail with reference to the accompanying drawings, which are described as follows:

[0009] [Figure 1] FIG. 1 is a high-level block diagram illustrating an exemplary embodiment of an architecture for implementing aspects of the present invention. [Figure 2A] FIG. 2A is a sample form with field labels. [Figure 2B] FIG. 2B is a sample form with field labels. [Diagram 3] FIG. 3 shows one of the lines of text in the form of FIGS. 2A and 2B. [Figure 4A]FIG. 4A is a flow chart illustrating an exemplary sequence of operations according to one embodiment. [Figure 4B] FIG. 4B is a flow chart illustrating an exemplary sequence of operations according to one embodiment. [Diagram 5] FIG. 5 shows an occurrence histogram of the characters of the words in the text line of FIG. [Figure 6] FIG. 6 is a sample form that may be created according to one embodiment. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0010] Figure 1 is a high-level block diagram of a system for implementing aspects of the methods described herein, according to one or more embodiments. In Figure 1, a computing device may provide a pair of synthetically generated forms (blank, partially filled in, or filled in) to a computing system 150. In one embodiment, a filled in form 110 is provided via a scanner 120, which may be connected to the computing system 150 directly or via a computing device 130. In one embodiment, a camera may provide the input image.

[0011] In one embodiment, computing system 150 processes the completed form to identify field labels. Computing system 150 may include one or more processors, one or more storage devices, and one or more solid-state memory systems (which, unlike storage devices, can include both non-transient and transient memory).

[0012] As part of determining the numeric location with the field label, computing system 150 may generate a bounding box around the text using bounding box generation system 160. In one embodiment, computing system 150 may include text scaling and alignment system 165 to compensate for different scales of images on different filled-in forms. In one embodiment, storage 175 may store input images that computing system 150 processes.

[0013] Computing system 150 may be at a single location, with network 155 enabling communication between various elements of computing system 150. Additionally or alternatively, one or more portions of computing system 150 may be remote from other portions, in which case network 155 may represent a cloud system for communication. In one embodiment, network 155 may be a cloud-based system, even if the various elements are collocated.

[0014] Additionally or alternatively, the processing system 190, which may include one or more of the processors, storage systems, and memory systems mentioned above, may implement one or more algorithms to resolve the location of the field labels.

[0015] To have line level detection, the image of the line is first extracted using a line segmentation algorithm. Since those skilled in the art will generally recognize available line segmentation algorithms, the details of the line segmentation algorithm will be kept brief and omitted here.

[0016] Figures 2A and 2B show two examples of the same form, each filled out differently and each with a different scale factor. In these figures, a bounding box is drawn around the text in the form with the field labels. Fields 205, 205' are name fields, filled in with the name "Jonas Jackson". Fields 210, 210' are birthdate fields, filled in with two digits for the month and day, and year, either four digits (Figure 2A) or two digits (Figure 2B). Fields 215, 215' are address fields, filled in with two lines of address. Fields 220, 220' are phone number fields, filled in with ten digits.

[0017] Fields 225, 225' are diagnosis fields and are filled in with a description of the diagnosis. Fields 230, 230' are procedure / treatment fields and are filled in with the identification of the procedure and / or treatment to be followed. Fields 235, 235' are start date fields and are filled in with the date the procedure and / or treatment is to be started. Fields 240, 240' are end date fields and are filled in with the date the procedure and / or treatment is to be ended. Similar to fields 210, 210', in FIG. 2A, start date fields 235, 235' and end date fields 240, 240' are filled in with two digits for the month and day, and either four digits (FIG. 2A) or two digits (FIG. 2B) for the year. Fields 245, 245' are fields for filling in the reason for the particular procedure and / or treatment in fields 230, 230'. Fields 250, 250' are fields for filling in the instructions for carrying out the procedure and / or treatment.

[0018] FIG. 2A also includes extraneous handwritten notations 255 that are not present in FIG. 2B.

[0019] One aspect of filled-in forms such as those of Figures 2A and 2B is that their fields are filled in differently depending on what is to be filled in those fields. Descriptions of various diagnoses, treatments / therapy, and / or treatment / therapy instructions may require different amounts of space. The consequences of these variously filled-in forms are discussed below.

[0020] Figure 3 shows an example of an image of the first row in each of the documents of Figures 2A and 2B. The first row 310 contains fields 205, 210 (Figure 2A), and the second row 320 contains fields 205', 210' (Figure 2B).

[0021] In one embodiment, after the line images of FIG. 3 are extracted from the filled-out form of FIG. 2A and FIG. 2B, each line image is digitized into a text string. The text string is then segmented into a set of words in an order based on the position of the word in the string. Repeated words in the text string can be identified by placing an additional character after the word. For example, if a word occurs twice in the text string, a _1 may be appended to the second occurrence. If the word occurs three times, a _1 is appended to the second occurrence and a _2 is appended to the third occurrence. In one embodiment, punctuation marks are not counted as characters in the text string.

[0022] For example, after the line image 310 of FIG. 3 is digitized, the text string may be segmented as follows: "start\day\02\03\2023\end\day_1\04\03\2023" After the line image 320 of FIG. 3 is digitized, the resulting text string may be segmented as follows: "start\day\07\08\23\end\day_1\11\08\23"

[0023] With the foregoing in mind, in one embodiment, a set of words may be encoded into a text string as follows: 1) The words in the text string can be numbered sequentially from left to right to form a numeric vector representation of the text string. For example, the text string from the lines image 310 can be encoded as the vector S1 = {1, 2, 3, 4, 5, 6, 2, 7, 8, 9}. That is, the word "day" appears twice in the vector and is assigned the same number. 2) Vector S1 may be encoded for text string 320 of FIG. 3, as may vector S2. Vector S2 may be generated as follows: If a word in vector S2 is the same as a word in S1, the position of that word is the same as the position of the corresponding word in vector S1 and is assigned the same number. In the current situation, vector S2={1, 2, 10, 11, 12, 2, 7, 13, 14, 15}. That is, the first, second, sixth, and seventh words in text string 320 are the same as those in text string 310, and therefore they are assigned the same number as those in the vector of text string 310. The third through fifth words and the eighth through tenth words in text string 320 are different from those in text string 310.

[0024] In one embodiment, if there are words in vector S2 that are not in vector S1, these words may be encoded to follow the number of the last position in vector S1. In one embodiment, if the second text string is shorter than the first text string, zero padding may be added to vector S2 to make vector S2 the same length as vector S1. Vector S2 may be encoded in this manner such that the length of the longest increasing subsequence of vector S2 may be a measure of similarity between vectors S1 and S2, where the longest increasing subsequence is a subsequence of a given sequence in which the elements of the subsequence are sorted, from lowest to highest, and the subsequence is as long as possible.

[0025] In the above comparison of two vectors for the same text line of two different filled-in forms, it is instructive to consider the performance of this comparison for all text lines of the two different filled-in forms. For this comparison, it is appropriate to manually select two copies of the same form that are filled in differently. In this way, the text of the two copies can be compared and field labels identified.

[0026] Looking back at Figures 2A and 2B, it can be seen that some of the fields contain more rows than others. For example, field 215 in Figure 2A has two rows, while field 215' in Figure 2B has three rows. Field 225 in Figure 2A has one row, while field 225' in Figure 2B has two rows. Field 250 in Figure 2A has three rows, while field 255' in Figure 2B has one row. The different number of rows in the two forms shown in Figures 2A and 2B, respectively, motivates the described approach to comparing row images of the two forms.

[0027] Now, looking at two different parts of the same form, after digitization and word segmentation, there may be M text strings from the first image (corresponding to an M line image) and N text strings from the second image (corresponding to an N line image). M and N may differ for various reasons. For example, as Figures 2A and 2B show, there may be additional lines of text in one or the other of the images. As mentioned earlier, artifacts may be present, as can be seen in the top part of Figure 2A with handwritten notation 255. As a result, the following two word matrices A and B, with M and N rows respectively, can be provided to represent the two document images above.

number

[0028] After the ith iteration of the process just described, we have two lines of text, l1 and l2, respectively, associated from the two images, and the next step is to align l1 and l2 before proceeding to the next line. Alignment of l1 and l2 may be necessary due to considerations such as differences in image scale and font size between the two lines of text. To determine the correct alignment, in one embodiment, each word in l1 and l2 may be segmented into characters or words. Since the lines of text have been matched based on vector matching, there is a high probability that there are common characters or words between the two lines of text, which allows scaling to be performed. If no common characters or words are found, the lack of common characters means that there is no common keyword between these two lines. In this case, the process can continue to process other pairs of lines of text, as previously described.

[0029] In one embodiment, after a certain number of failures of the process with respect to its ability to align the text lines, it can be presumed that the alignment has failed. Such failures may occur due to the image quality to which one or both of the filled-in forms are compared, or artifacts in one or both of the filled-in forms, or other technical reasons that would be generally understood by one of ordinary skill in the art. In that case, another filled-in form may be substituted for one of the filled-in forms just compared, or another pair of filled-in forms may be selected, and the process repeated.

[0030] If two text lines l1 and l2 have a common character or word between them, then the scale S of the two text lines x and S. y Groups of characters or words common to both l1 and l2 can be identified as seeds for computing l. For example, looking at the bottom of Figure 5, bounding boxes may be drawn around each word in each line of text.

[0031] The scale S calculated above x and S. yUsing l1, all the bounding boxes of words in l1 can be scaled to the same scale as l2. Then, as shown in FIG. 5, each word in l2 can be aligned to its corresponding word in l1 based on the location of its bounding box. In one embodiment, further alignment can be performed on the characters of each corresponding word between l1 and l2 by minimizing the edit distance through deletion and insertion operations. After alignment, a character histogram can be constructed, as shown in the center of FIG. 5. In the character histogram, the x-axis represents the characters of the words in the text line, and the y-axis represents the number of characters that occur in the word. Characters that occur more frequently are represented by higher lines.

[0032] Furthermore, a word histogram can be built on top of a character histogram, where the x-axis represents the words in a line of text and the y-axis represents the density of the underlying distribution of characters within the words, where the density is in the range [0,1], using the following formula:

number

[0033] The preceding description is based on the assumption that the two filled-in forms received at the start of the process can be matched, aligned, and scaled, and that character and word histograms are created to determine field labels from only those two forms. In some cases, additional filled-in forms need to be added to the comparison. In that case, looking at the three images with the constructed word matrices A, B, and C, and the identified row correspondences between matrices A and B and between matrices A and C, the following process can be provided: 1) Take a row of words from matrix A and from the corresponding rows of matrices B and C. 2) Align the word rows of matrices B and C with the word rows of matrix A. 3) Create a histogram using the three aligned rows. 4) Localize keywords based on the hierarchical histogram of this row. 5) Repeat steps 1 to 4 until all rows have been processed.

[0034] 4A and 4B illustrate a flow for the aforementioned process according to one embodiment. At 401, multiple versions of the same filled-in form are received. In the specifically described embodiment, two or three versions may be received at this point, but in any event the number is not large. At 403, image processing is performed to identify row images of each of the filled-in forms. At 405, text strings are created from the row images. At 407, a matrix of text strings is created for each filled-in form. At 409, the first iteration of comparison of the rows of each matrix begins with the variable m set to 1.

[0035] At 411, words that occur multiple times within the text strings are identified for the first matrix. At 413, a vector is created for each text string using the specified words. At 415, like words are identified within the text strings and assigned the same number in their position within the vector.

[0036] At 417, a decision is made to see if the process is far enough down the matrix to work on row images that are far enough down the matrix to be able to see a predetermined number of rows (x being an integer) before and after the row being worked on. If not ("No" at 417), then at 419 the vector for the mth row is used and flow proceeds to 429, which will be described. If the process is far enough along ("Yes" at 417), then at 421 a number of text strings from mx to m+x are selected for another matrix and at 423 vectors are created for each of those text strings using the identified words in each text string. As before, the same number is also assigned to the vector positions of each text string that contains the same word.

[0037] At 427, a best vector match may be found between row m of a first matrix and rows mx through m+x of another matrix. (Similar to the description of matrices A, B, and C, 427 may be a comparison between two matrices, or between three matrices, or in either case between very small numbers compared to the number needed to obtain a training set for training an AI-based system.) At 429, the text lines are aligned corresponding to the matched vectors.

[0038] Having matched the vectors, it is determined at 433 whether scaling is possible to give the text lines the same scale. If not ("No" at 433), then it is checked at 435 whether all rows of the first matrix have been processed. If not ("No" at 435), then m is incremented at 437 and the flow returns to 411. If all rows of the first matrix have been processed ("Yes" at 435), then at 439 either the same or a further version of the completed form is received and the flow returns to 403.

[0039] If scaling is possible ("Yes" at 433), then at 441 seeds are identified among the characters and / or words for each text line and a scale is calculated. At 443, the calculated scale is used to give the compared text lines the same scale. Again, the number of compared text lines may be two or three, depending on the number of matrices being compared in an embodiment, or perhaps a few more, but in either case much less than would be needed to obtain a training set for training an AI-based system.

[0040] After scaling is accomplished, a character histogram may be constructed for the characters in the text line at 445. Then, a word histogram may be constructed for the words in the text line at 447. After this, the field labels may be identified from the histogram at 449, with higher numbers indicating characters in words within the field label (445) and words within the field label (447) as represented along the Y axis of the histogram.

[0041] If all field labels for the form have been identified at 451 ("Yes" at 451), then a blank form with the field labels may be generated at 453. If all field labels have not been generated ("No" at 451), then a check is made at 455 to see if all rows of the first matrix have been processed. If not ("No" at 455), then m is incremented at 437 and the flow returns to 411. If all rows of the first matrix have been processed ("Yes" at 455), then a version of the same completed form or a further version is received at 439 and the flow returns to 403.

[0042] Although embodiments of the present invention have been described in detail with reference to various drawing figures, those skilled in the art will recognize that numerous modifications may be made within the scope and spirit of the invention. Accordingly, the present invention is limited only by the following claims.

Claims

1. a. responsive to receiving multiple images of a version of a completed form, processing the images to generate a text string for each line of each version; b. For each version, generating a matrix of multiple rows of text strings; For each line in each version, c. generating a best match between a vector encoding the text strings of the lines of one version and one of the vectors encoding the text strings of the lines within a predetermined number of lines of the other version; d. constructing a character histogram for the characters in the text string in the line indicating the frequency of occurrence of the characters; e. constructing, for a plurality of words of the text string in the line, a word histogram indicating a density of an underlying distribution of a plurality of characters within the word, The method further comprising: f. responsive to generating the best match, identifying a group of words from the word histogram constructed on the character histogram that have density values ​​higher than a predetermined value as field labels for the completed form; g. The computer-implemented method further comprising generating a blank form having the identified field labels.

2. 2. The computer-implemented method of claim 1, wherein the number of versions in a. is two or three.

3. 3. The computer-implemented method of claim 1 or 2, wherein the predetermined number of rows is three.

4. 3. The computer-implemented method of claim 1 or 2, wherein the processing comprises identifying line images of each completed form and generating a text string from each of the identified line images.

5. The computer-implemented method of claim 1 or 2, further comprising responsive to generating the best match, aligning respective text lines corresponding to a plurality of vectors of the best match.

6. 6. The computer-implemented method of claim 5, wherein said aligning comprises scaling the text of each of said lines of text such that the text of each of said lines of text is the same size.

7. The computer-implemented method of claim 6 , wherein the scaling comprises identifying like characters in the respective lines of text and performing scaling on the identified characters.

8. generating said best match, a. creating a vector using the identified words of the text string that encodes the text string in that line by assigning a number to each identified word of the text string; b. comparing the vector of the row in one version with a number of vectors within a predetermined number of rows of the vector of the row in another version; 3. The computer-implemented method of claim 1 or 2, further comprising: c. generating the best match in response to said comparing.

9. 9. The computer-implemented method of claim 8, further comprising: for each line in each version, identifying words that occur multiple times in each of the text strings; and for each word that occurs multiple times, assigning the same number to each position in the vector in which the word occurs.

10. The x-axis of the word histogram represents a number of words of a line of text, and the y-axis of the word histogram represents a density of an underlying distribution of a number of characters within the word; The density is in the range [0,1], using the formula: [0010] where ρ is the density of the underlying distribution of characters in a word, m is the number of text lines, n is the number of characters in the word, f ｉ 3. The computer-implemented method of claim 1, wherein i is the number of the i-th character in the word.

11. A processor; and a non-transitory memory storing instructions that, when executed by the processor, perform a method, The method comprises: a. responsive to receiving multiple images of a version of a completed form, processing the images to generate a text string for each line of each version; b. For each version, generating a matrix of multiple rows of text strings; For each line in each version, c. generating a best match between a vector encoding the text strings of the lines of one version and one of the vectors encoding the text strings of the lines within a predetermined number of lines of the other version; d. constructing a character histogram for the characters in the text string in the line indicating the frequency of occurrence of the characters; e. constructing a word histogram for a plurality of words of the text string in the line that indicates a density of an underlying distribution of a plurality of characters within the word; The method further comprising: f. responsive to generating the best match, identifying a group of words from the word histogram constructed on the character histogram that have density values ​​higher than a predetermined value as field labels for the completed form; g. The system further comprises generating a blank form having the identified field labels.

12. 12. The system of claim 11, wherein the number of versions in a. is two or three.

13. 13. The system of claim 11 or 12, wherein the predetermined number of rows is three.

14. 13. The system of claim 11 or 12, wherein the processing comprises identifying line images of each completed form and generating a text string from each of the identified line images.

15. 13. The system of claim 11 or 12, wherein the method further comprises aligning respective text lines corresponding to a plurality of vectors of the best match in response to generating the best match.

16. 16. The system of claim 15, wherein the aligning comprises scaling the text of each of the text lines so that the text of each of the text lines is the same size.

17. 17. The system of claim 16, wherein the scaling comprises identifying like characters in the respective lines of text and performing scaling on the identified characters.

18. generating said best match, a. creating a vector using the identified words of the text string that encodes the text string in that line by assigning a number to each identified word of the text string; b. comparing the vector of the row in one version with a number of vectors within a predetermined number of rows of the vector of the row in another version; 13. The system of claim 11 or 12, further comprising: c. generating the best match in response to the comparison.

19. 13. The system of claim 11 or 12, wherein the method further comprises: for each line in each version, identifying words that occur multiple times in each of the text strings; and for each word that occurs multiple times, assigning the same number to each position in the vector in which the word occurs.

20. The x-axis of the word histogram represents a number of words of a line of text, and the y-axis of the word histogram represents a density of an underlying distribution of a number of characters within the word; The density is in the range [0,1], using the formula: [0025] where ρ is the density of the underlying distribution of characters in a word, m is the number of text lines, n is the number of characters in the word, f ｉ 13. The system of claim 11 or 12, wherein i is the number of the i-th character in the word.

Images