Character recognition device and program

Abstract

This invention provides a character recognition device and program that enable the extraction of at least candidate item names from strings contained in various types of tables included in document images. [Solution] The image document processing server 1 includes a structured text creation unit 13 that creates structured text by structuring strings using text information including strings and coordinate information of strings in a table area extracted from a form image, an instruction generation unit 14 that generates an item extraction instruction by embedding the structured text created by the structured text creation unit 13 into an instruction statement, and an item name candidate acquisition unit 15 that acquires at least item name candidates output by the generation AI server 3 by inputting the item extraction instruction generated by the instruction generation unit 14 to the generation AI server 3 which generates and outputs information according to the input information.

Description

Technical Field

[0004] , , , , ,

[0005] , , ,

[0003] , , , , ,

[0001] The present invention relates to a character recognition device and a program.

Background Art

[0002] Conventionally, when registering documents with different formats issued by issuers such as estimates and invoices as data in a system, character input has been performed manually, which takes time and human costs. Therefore, various efforts have been considered to automatically perform character input in order to reduce costs. The use of OCR (Optical Character Recognition / Reader) is an example of this. However, with OCR, information is obtained piecemeal and the meaning of the text cannot be understood. Therefore, for example, there is one that uses a template for specifying the description position of a specific attribute in a document (for example, Patent Document 1).

Prior Art Documents

Patent Documents

[0003]

Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0004] In Patent Document 1, it was necessary to prepare a format for each document, which was complicated. In particular, non-standard documents such as estimates and invoices have tables, and item names and their contents are described, but there are some with complex table structures. There is a need to appropriately extract items and their contents from such complex tables.

[0005] Therefore, an object of the present invention is to provide a character recognition device and a program that enable extraction of at least candidates for item names from character strings included in various types of tables in a form image. [Means for solving the problem]

[0006] The present invention solves the above problem by the following means. The first invention is a character recognition device comprising: a structured text creation means that creates structured text by structuring a string using text information including a string in a table area extracted from a form image and coordinate information of the string; an instruction generation means that generates an item extraction instruction by embedding the structured text created by the structured text creation means into an instruction statement; and a result acquisition means that obtains at least an item name candidate output by a generation AI that generates and outputs information according to the input information by inputting the item extraction instruction generated by the instruction generation means to the generation AI. The second invention is a character recognition device of the first invention, comprising: an emphasis means for extracting candidate strings that can become item names from the strings contained in the text information and emphasizing the extracted candidate strings; and a string concatenation means for generating a group of characters by concatenating the strings that are within a predetermined range using the coordinate information of the strings contained in the text information and adding a specified symbol corresponding to the distance between the strings, wherein the structured text creation means uses the set of the group of characters generated by the string concatenation means as the structured text. The third invention is a character recognition device of the second invention, wherein the emphasis means performs morphological analysis on the string to extract words, and extracts a predetermined string containing a noun as the part of speech of the word as the candidate string. The fourth invention is a character recognition device of the second or third invention, comprising an item position confirmation means for confirming the position of an item name in the table area based on the coordinate position of the candidate string highlighted by the emphasis means, wherein the string concatenation means generates the character group with the predetermined range as a horizontal range when the item position confirmation means confirms that the position of the candidate string is in the vertical direction, and generates the character group with the predetermined range as a vertical range when the item position confirmation means confirms that the position of the candidate string is in the horizontal direction. The fifth invention is a character recognition device according to any of the first to fourth inventions, wherein the instruction generation means embeds the structured text into the instruction sentence which includes instructions to extract item names and item values ​​from the structured text, and the result acquisition means acquires the item name candidates and item value candidates output by the generating AI. The sixth invention is a character recognition device of the fifth invention, wherein the instruction statement includes extraction rules relating to the item name and the item value. The seventh invention is a character recognition device according to the fifth or sixth invention, further comprising an item name selection means that analyzes the item name candidates and item value candidates acquired by the result acquisition means and excludes predetermined strings classified as item name candidates from the item name candidates. The eighth invention is a character recognition device that, in any of the first to seventh inventions, comprises: a document image acquisition means for acquiring the document image; a table extraction means for extracting the table area from the document image acquired by the document image acquisition means; and a text information acquisition means for performing character recognition processing on the table area extracted by the table extraction means to acquire the text information. The ninth invention is a program for causing a computer to function as a character recognition device based on any of the first through eighth inventions. [Effects of the Invention]

[0007] According to the present invention, it is possible to provide a character recognition device and program that can extract at least candidate item names from strings contained in tables of various formats included in a document image. [Brief explanation of the drawing]

[0008] [Figure 1] This diagram shows the overall configuration of the image document processing system and the functional block diagram of the image document processing server according to this embodiment. [Figure 2] This flowchart shows the image document processing performed by the image document processing server according to this embodiment. [Figure 3] This figure shows an example of a table area in a document image according to this embodiment. [Figure 4] This figure shows an example of text information corresponding to a table area according to this embodiment. [Figure 5] This flowchart shows the process for obtaining candidate item names in the image document processing server according to this embodiment. [Figure 6] This flowchart shows the structured text creation process of the image document processing server according to this embodiment. [Figure 7] This diagram illustrates the structured text creation process of the image document processing server according to this embodiment. [Figure 8] This diagram illustrates the structured text creation process of the image document processing server according to this embodiment. [Figure 9] This diagram illustrates the structured text creation process of the image document processing server according to this embodiment. [Figure 10] This diagram illustrates the process of obtaining candidate item names in the image document processing server according to this embodiment. [Figure 11] This diagram illustrates the process of obtaining candidate item names in the image document processing server according to this embodiment. [Figure 12] This diagram illustrates the process of obtaining candidate item names in the image document processing server according to this embodiment. [Figure 13] This diagram illustrates the process of obtaining candidate item names in the image document processing server according to this embodiment. [Figure 14] This flowchart shows the process for determining the combination of image document processing servers according to this embodiment. [Figure 15] This diagram illustrates the combination determination process of the image document processing server according to this embodiment. [Figure 16] This diagram illustrates the combination determination process of the image document processing server according to this embodiment. [Figure 17] This diagram illustrates the combination determination process of the image document processing server according to this embodiment. [Figure 18]This is a diagram for explaining the combination determination process of the image document processing server according to the present embodiment. [Figure 19] This is a diagram for explaining the combination determination process of the image document processing server according to the present embodiment.

Embodiment for Implementing the Invention

[0009] Hereinafter, embodiments for implementing the present invention will be described with reference to the drawings. Note that this is merely an example, and the technical scope of the present invention is not limited thereto. (Embodiment) <Overall Configuration of Image Document Processing System 100> FIG. 1 is an overall configuration diagram of an image document processing system 100 and a functional block diagram of an image document processing server 1 according to the present embodiment. As shown in FIG. 1, the image document processing system 100 includes an image document processing server 1 (character recognition device, item combination device), a generation AI server 3, an OCR device 4, and a user terminal 5. The image document processing server 1, the generation AI server 3, the OCR device 4, and the user terminal 5 are connected via a communication network N.

[0010] The image document processing system 100 generates structured text from the text information obtained by performing character recognition processing on the table area included in the form image indicated by the form image data received from the user terminal 5. Then, the image document processing system 100 transmits an item extraction instruction including the generated structured text to the generation AI server 3, and obtains candidates for item names and item values from the generation AI server 3. Furthermore, the image document processing system 100 determines the combination of item names for item values based on the distance between the two character strings and the inclination based on the position between the two character strings from the reference position, and outputs the determined combination to the user terminal 5.

[0011] <Image Document Processing Server 1> Image document processing server 1 is a server that performs a series of operations on report image data and outputs combinations of item names and item values ​​of tablespaces contained in the report image. The image document processing server 1 comprises a control unit 10, a storage unit 20, and a communication interface unit 29. The control unit 10 is a CPU (Central Processing Unit) that controls the entire image document processing server 1. The control unit 10 works in cooperation with the aforementioned hardware to perform various functions by appropriately reading and executing the OS (Operating System) and various application programs stored in the memory unit 20.

[0012] The control unit 10 includes a form image acquisition unit 11 (form image acquisition means), a preprocessing unit 12 (table extraction means, text information acquisition means), a structured text creation unit 13 (emphasis means, item position confirmation means, string concatenation means, structured text creation means), an instruction generation unit 14 (instruction generation means), an item name candidate acquisition unit 15 (result acquisition means, item name selection means), a division unit 16 (division means), a table determination unit 17 (item position confirmation means, table determination means), a combination determination unit 18 (combination determination means), and an item combination output unit 19 (item combination output means). The document image acquisition unit 11 receives document image data, for example, from the user terminal 5. The document that forms the basis of the document image data includes, for example, the content of data to be input into an external system (not shown). The document image data is, for example, an image of a paper document created by a scanner (not shown), and is in PDF, JPEG, IMG, or other image data format.

[0013] The preprocessing unit 12 extracts table regions from the document image data received by the document image acquisition unit 11. The preprocessing unit 12 can detect table regions from image data using well-known techniques, such as deep learning-based object detection methods for images. If the detected table regions are not parallel in the vertical and horizontal directions, the preprocessing unit 12 performs a correction process to obtain table regions that are parallel in the vertical and horizontal directions. The correction process can be performed using well-known techniques.

[0014] Furthermore, the preprocessing unit 12 performs character recognition processing on the extracted table space to detect characters and obtains the detected character clusters as strings. More specifically, the preprocessing unit 12, for example, sends the form image data to the OCR device 4 and detects characters from the text data obtained from the OCR device 4. The preprocessing unit 12 then forms a string of characters according to the spacing between the detected characters. The spacing between characters to be formed into a string may vary depending on, for example, the font size. For example, if the font size is large, the preprocessing unit 12 will include adjacent characters in the string even if the spacing between characters is wider than when the font size is small. Furthermore, the preprocessing unit 12 obtains the string (text) and also obtains the coordinate position of the string.

[0015] The structured text creation unit 13 uses the text information, including the strings and their coordinate positions, detected by the preprocessing unit 12 to create structured text by structuring the strings. The structured text generation unit 13 extracts candidate strings that could become item names using natural language processing. Specifically, the structured text generation unit 13 first performs morphological analysis on the strings to extract words, and then extracts predetermined strings that contain nouns as the part of speech of the extracted words as candidate strings that could become item names. More specifically, the structured text generation unit 13 selects, for example, strings that end in a noun after removing suffixes and auxiliary symbols from the end of the word as candidate strings. Next, the structured text generation unit 13 highlights (marks) the candidate strings. One method of highlighting is to enclose the candidate strings in black brackets or the like.

[0016] The structured text creation unit 13 checks the position of the item name in the table area based on the coordinate position of the highlighted candidate string. If the structured text creation unit 13 confirms that the position of the candidate string is in the vertical direction, the structured text creation unit 13 determines that it is a table in which the item values ​​for the item names are in the horizontal direction (hereinafter also referred to as a "horizontal table"). Also, if the structured text creation unit 13 confirms that the position of the candidate string is in the horizontal direction, the structured text creation unit 13 determines that it is a table in which the item values ​​for the item names are in the vertical direction (hereinafter also referred to as a "vertical table"). The structured text creation unit 13 generates a group of characters by combining strings within a predetermined range using the coordinate information of the strings, and adding a specified symbol according to the distance between the strings. In this process, the structured text creation unit 13 can use a symbol such as an underscore as the specified symbol corresponding to the spaces between the strings.

[0017] For example, in the case of a horizontal table, the structured text creation unit 13 generates character sets for strings arranged horizontally. In other words, the structured text creation unit 13 generates character sets on a line-by-line basis. Furthermore, in the case of a vertical table, the structured text creation unit 13 generates character sets for strings arranged vertically. In other words, the structured text creation unit 13 generates character sets on a column-by-column basis. The structured text generation unit 13 then uses, for example, a set of characters generated on a line-by-line or column-by-column basis as structured text.

[0018] The instruction generation unit 14 generates an item extraction instruction by embedding the structured text created by the structured text creation unit 13 into the instruction statement. Here, the instruction statement is a prompt that includes instructions to extract item names and item values ​​from the structured text. A prompt refers to instructions that a user inputs to a conversational system, such as a generative AI.

[0019] The item name candidate acquisition unit 15 transmits the item extraction instruction generated by the instruction generation unit 14 to the generation AI server 3. The item name candidate acquisition unit 15 then receives the response to the item extraction instruction from the generation AI server 3 and acquires the item name candidates and item value candidates. Furthermore, the item name candidate acquisition unit 15 analyzes the acquired item name candidates and item value candidates and excludes predetermined strings classified as item name candidates from the item name candidates. Specifically, for example, the item name candidate acquisition unit 15 excludes items from the item name candidates if they have similar characters. Also, the item name candidate acquisition unit 15 excludes items that are clearly item values ​​and not item names from the item name candidates, for example, using rules.

[0020] The splitting unit 16 splits the strings identified as item name candidates by the item name candidate acquisition unit 15 into item name strings and the remaining strings into item value strings. The table determination unit 17 confirms the position of the item name in the table area based on the coordinate position of the item name string. Then, the table determination unit 17 determines the table format based on the position of the item name. If the position of the item name is vertical, the table determination unit 17 determines that it is a horizontal table. Conversely, if the position of the item name is horizontal, the table determination unit 17 determines that it is a vertical table.

[0021] The combination determination unit 18 determines the item name strings that form a pair with the item value string based on the distance between the two strings and the inclination of the item name strings relative to the item value string. In doing so, the combination determination unit 18 calculates the inclination of the item name strings relative to the item value string based on the table format determined by the table determination unit 17. Specifically, if it is determined to be a horizontal table, the combination determination unit 18 determines the item name string corresponding to the item value string based on the distance between the two strings and the inclination between the horizontal and vertical directions with the item value string as the reference position. If it is determined to be a vertical table, the combination determination unit 18 determines the item name string corresponding to the item value string based on the distance between the two strings and the inclination between the vertical and vertical directions with the item value string as the reference position.

[0022] The combination determination unit 18 may also determine the item name strings that form a pair with the item value strings to be the combination that minimizes the score calculated using the distance and slope between the two strings. In this case, if there are grid lines between the line segments connecting the item value string and the item name string, the combination determination unit 18 adds weights to the score according to the direction of the grid lines (vertical or horizontal) and the number of grid lines. The weights according to the direction of the grid lines may be predetermined, for example, depending on whether it is a horizontal or vertical table. Here, the combination determination unit 18 can detect grid lines included in the table area using, for example, well-known techniques such as the Hough transform.

[0023] The item combination output unit 19 outputs the combination of item name and item value determined by the combination determination unit 18 as the output target to the user terminal 5 as a processing result. The item combination output unit 19 may, for example, output the combination of item name and item value in a table format and also output a report image on the same screen. Details of each of these functions will be described later.

[0024] The memory unit 20 is a storage device such as a hard disk or semiconductor memory element for storing programs, data, etc., necessary for the operation of the image document processing server 1. The memory unit 20 comprises a program memory unit 21 and an instruction statement memory unit 22. The program storage unit 21 is a storage area for storing various programs to be executed by the image document processing server 1. The program storage unit 21 stores the character recognition processing program 21a and the item combination program 21b.

[0025] The character recognition processing program 21a is a program that performs various functions, such as obtaining candidate item names from a form image held by the control unit 10. The item combination program 21b is a program for executing various functions, such as splitting the item into item names and other items, which are stored in the control unit 10, and then outputting the item combinations. Note that the programs for executing various functions of the control unit 10 are not limited to those realized by the two programs as described above. For example, there may be a program for each function of the control unit 10, or for a plurality of functions, or there may be one program having all the functions of the control unit 10.

[0026] The instruction text storage unit 22 is a storage area that stores a template for an item creation instruction for obtaining an item name and candidate item values from structured text. The item creation instruction is an instruction, etc., input by the user to a dialogue-form system typified by a generative AI, called a prompt. The communication interface unit 29 is an interface for communicating with the generative AI server 3, the OCR device 4, the user terminal 5, etc. via the communication network N.

[0027] Here, the computer refers to an information processing device equipped with a control unit, a storage device, etc. The image document processing server 1 is an information processing device equipped with a control unit 10, a storage unit 20, etc., and is included in the concept of a computer. Also, there is no limit to the number of hardware components constituting the image document processing server 1. It may be composed of one or more as required. Also, the hardware of the image document processing server 1 may be configured to include various servers such as a web server, a DB (database) server, an application server, etc. as required, and may be composed of one server, or each may be composed of separate servers. Also, the image document processing server 1 may be, for example, a cloud.

[0028] <Generative AI Server 3> The generative AI server 3 is a server equipped with a generative AI as a language model that has learned a plurality of documents. The generative AI server 3 includes a control unit, a storage unit, a communication interface unit, etc., although not shown. Note that the generative AI server 3 may be, for example, a cloud.

[0029] <OCR Device 4> The OCR device 4 is a device that performs OCR-related processing to recognize characters from document image data. The OCR device 4 receives document image data from the image document processing server 1, performs character recognition processing on the document image to obtain text data, and transmits the text data to the image document processing server 1. The OCR device 4 is composed of, for example, a server. The OCR device 4 may also be composed of a personal computer (PC), etc. Although not shown in the figures, the OCR device 4 includes a control unit, a storage unit, a communication interface unit, etc.

[0030] <User Terminal 5> User terminal 5 is a terminal used by a user who wants to register various data contained in a document into an external system, for example. User terminal 5 can be composed of a PC, a tablet, or the like. Although not shown in the diagram, user terminal 5 includes a control unit, a storage unit, a display unit, an input unit, a communication interface unit, and the like.

[0031] The communication network N is the network between the image document processing server 1, the generation AI server 3, the OCR device 4, and the user terminal 5, and is, for example, a communication network such as an internet connection. The communication network N may also be a LAN (Local Area Network), etc. Furthermore, the communication network N may be wired or wireless.

[0032] <Explanation of the process> Next, we will explain the processing performed by the image document processing server 1. Figure 2 is a flowchart showing the image document processing of the image document processing server 1 according to this embodiment. Figure 3 shows an example of a table area 30 in a document image according to this embodiment. Figure 4 shows an example of text information 31 corresponding to the table area 30 according to this embodiment.

[0033] First, the user terminal 5 connects to the image document processing server 1 and transmits, for example, document image data such as an estimate or an invoice. In step S 11 of Figure 2 (hereinafter, "step S" will be simply referred to as "S"), the control unit 10 (document image acquisition unit 11) of the image document processing server 1 acquires the document image data.

[0034] In S12, the control unit 10 (preprocessing unit 12) extracts a table area from the document image indicated by the acquired document image data. Figure 3 shows an example of the extracted tablespace 30. Tablespace 30 is a tabular format in which the item values ​​for the item names are arranged horizontally. Unlike cases such as gender or relationship, where the item values ​​for the item names are located in the same horizontal position as the item names, tablespace 30 is a complex tabular format. In this embodiment, even with a complex tabular format as shown in table area 30, the combination of item name and item value can be obtained.

[0035] In S13 of Figure 2, the control unit 10 (preprocessing unit 12) detects characters in the extracted table space and obtains a string and coordinates from the detected characters. The preprocessing unit 12 can detect characters in the document image and obtain a string from the detected characters by, for example, sending the document image data to the OCR device 4 and obtaining text data from the OCR device 4. The OCR device 4 scans the document image from the top left to the right, and when it reaches the right edge, it moves down one row and scans from left to right, repeating this process until the scan reaches the bottom right of the document image, thereby obtaining a string as a series of characters.

[0036] Figure 4 shows an example of text information 31 obtained from the table area 30. Text information 31 has string information 31a, 31b, 31c, ... corresponding to each string. String information 31a also has a string 32a and coordinate information 33a. Coordinate information 33a includes the coordinates of each vertex of the rectangle surrounding the string, as well as the coordinate of the rectangle's center. Similarly, string information 31b has a string 32b and coordinate information 33b, and string information 31c has a string 32c and coordinate information 33c.

[0037] In S14 of Figure 2, the control unit 10 performs the process of acquiring candidate item names. The process of acquiring candidate item names will be described later. In S15, the control unit 10 performs item combination processing. The item combination processing will be described later. In S16, the control unit 10 (item combination output unit 19) outputs the determined item name and item value combination to the user terminal 5 as a processing result. The item combination output unit 19 may also output the form image data along with the determined item name and item value combination. After that, the control unit 10 terminates this process.

[0038] Next, we will explain the process for obtaining candidate item names. Figure 5 is a flowchart showing the process of obtaining candidate item names in the image document processing server 1 according to this embodiment. Figure 6 is a flowchart showing the structured text creation process of the image document processing server 1 according to this embodiment. Figures 7 to 9 illustrate the structured text creation process of the image document processing server 1 according to this embodiment. Figures 10 to 13 are diagrams illustrating the item name candidate acquisition process of the image document processing server 1 according to this embodiment.

[0039] In step S21 of Figure 5, the control unit 10 (structured text creation unit 13) performs structured text creation processing. Here, the structured text creation process will be explained based on Figure 6. In S31 of Figure 6, the control unit 10 performs morphological analysis on the string to extract words and extracts strings that can be used as item names as candidate strings. More specifically, the control unit 10 selects, for example, strings that end in a noun after removing suffixes and auxiliary symbols from the end of a word as candidate strings. Here, we will explain the candidate strings with specific examples. For example, morphological analysis of the string "Total Insurance Premium" yields "Total" + "Insurance" + "Fees". "Total" and "Insurance" are nouns, and "Fees" is a suffix. Therefore, removing the suffix "Fees" leaves "Insurance," which is also a noun. Thus, the string "Total Insurance Premium" is considered a candidate string as it could be an item name.

[0040] In S32, the control unit 10 emphasizes the candidate string. For example, it might say "≪Total Insurance Premium≫". Note that the emphasis can be anything other than the example given (for example, black brackets, etc.), and other symbols can be used as long as they emphasize the candidate string. In S33, the control unit 10 confirms the position of the item name in the table area from the coordinates of the highlighted candidate string. In S34, the control unit 10 generates a group of characters by combining strings within a predetermined range, including symbols corresponding to spaces, based on the position of the confirmed item name. In S35, the control unit 10 converts the generated set of characters into structured text. After that, the control unit 10 moves the process to S22 in Figure 5.

[0041] The processing from S33 onwards will be explained with specific examples. For example, a portion of the table area 30 shown in Figure 3 is shown in Figure 7(A). In the table area 30 in Figure 7(A), candidate strings are enclosed by dotted lines as shown in frame 30a. The positions of the candidate strings enclosed by dotted lines are in the vertical direction (e.g., "Date", "Phonetic Name"). The control unit 10 then determines that the table area 30 is a horizontal table and retrieves strings for each row within a predetermined range. At that time, the strings are sorted in ascending order by the Y coordinate (vertical direction) and X coordinate (horizontal direction) of the midpoint of the string's coordinate information, and the strings from the coordinate information until the next string wraps are treated as one row. In the range of the table area 30 shown in Figure 7(A), the control unit 10 determines that there are 4 rows.

[0042] The list 35 shown in Figure 7(B) is an extraction of the strings contained in each row of the tablespace 30. The control unit 10 treats the strings contained in one row as a single character group. Furthermore, the control unit 10 represents the distance between strings using, for example, an underscore (_). List 36, shown in Figure 7(C), is structured text for the portion contained in List 35. In this example, candidate strings are represented by square brackets. The distance between strings is represented by underscores. In this example, the number of underscores is determined by the multiplier of the difference in the X coordinates between strings (midpoints of each string) for each character group. Figure 8 shows an example of structured text 37. Structured text 37 extracts candidate strings using the morphological analysis described above. Therefore, options such as "Showa," "Heisei," and "Reiwa" are among the candidate strings.

[0043] The example above illustrates a table where item values ​​are arranged horizontally for each item name. This section explains, using a specific example, how to determine if a table has item values ​​arranged vertically for each item name. In the table area 40 shown in Figure 9(A), candidate strings are enclosed by dotted lines as shown in frame 40a. The positions of the candidate strings enclosed by dotted lines are in the horizontal direction (e.g., "basic amount", "additional special provisions"). The control unit 10 then determines that the table area 40 is a vertical table and retrieves strings for each column within a predetermined range.

[0044] More specifically, the control unit 10 may determine the table by the method described below. (1) The midpoints of the coordinate information of the candidate strings enclosed by the dotted line are aligned horizontally. (2) The candidate strings are listed within the first two lines, and the proportion of the line occupied by the candidate strings is 4 / 5 or more of the total including the other strings. (3) No candidate strings are listed in columns other than the first column. Figure 9(B) shows the information 42 obtained from the table area 40. According to information 42, conditions (1) to (3) above are satisfied. Therefore, the control unit 10 can determine that the table area 40 is a vertical table. Furthermore, for a vertical table, the control unit 10 acquires strings for each column within a predetermined range. At that time, it sorts the strings in ascending order based on the X coordinate (horizontal) and Y coordinate (vertical) of the midpoint of the string's coordinate information, and treats the strings from the coordinate information up to the next string as a single column.

[0045] In S22 of Figure 5, the control unit 10 (instruction generation unit 14) generates an item extraction instruction. Figure 10 shows an example of generating an item extraction instruction. An item extraction instruction is a statement of instructions for extracting item names and item values ​​from structured text 37 (see Figure 8). An item extraction instruction includes the instruction content and structured text 37. A template 51A of an item extraction instruction, which contains the instruction content excluding structured text 37, can be stored in advance in the instruction statement storage unit 22 and used.

[0046] The template 51A illustrated in Figure 10 includes an instruction area 51a, an example area 51b, a text setting area 51c, and an output example area 51d. Instruction area 51a is an area that describes the instructions given to the generating AI. The example area 51b is an area that describes examples that supplement the content of the instruction area 51a. The text setting area 51c is the area for setting the text specified in the instruction area 51a. The text setting area 51c includes the text replacement position 51e. The text replacement position 51e indicates the position where the structured text 37 is embedded. The output example area 51d is an area that describes examples that supplement the output method indicated in the instruction area 51a.

[0047] Note that template 51A is just an example, and is not limited to this template, as it is a command statement for outputting candidate item names and candidate item values ​​from structured text 37. For example, template 51B shown in Figure 11 may also be used. Template 51B has a rule description area 51f between the instruction area 51a and the example area 51b of template 51A. The rule description area 51f is an area in which extraction rules that supplement the content of the instruction area 51a are described.

[0048] In the rule description area 51f, you can describe, for example, whether the text of the item name and the item value have a parent-child relationship (a dependent relationship) (for example, in extraction rule 4). Furthermore, the rule description area 51f can also be used to describe, for example, the proximity of item names and item values ​​(e.g., extraction rule 3). Furthermore, the rule description area 51f can also be used to describe, for example, that an item does not exist based on its name alone (e.g., extraction rules 5 and 6). Note that the rule description area 51f shown in Figure 11 is just an example, and rules may be added, deleted, or modified as needed.

[0049] The control unit 10 (instruction generation unit 14) generates an item extraction instruction by pasting structured text 37 to the text replacement position 51e of template 51A or template 51B. In step S23 of Figure 5, the control unit 10 (item name candidate acquisition unit 15) transmits the generated item extraction instruction to the generation AI server 3. The generation AI server 3 inputs item extraction instructions to the generation AI and outputs candidate item names and candidate item values. The generation AI server 3 then transmits the output candidate item names and candidate item values ​​to the image document processing server 1, so in S24, the control unit 10 (item name candidate acquisition unit 15) receives the candidate item names and candidate item values.

[0050] When using templates 51A or 51B as described above, the "items" received from the generating AI server 3 correspond to candidate item names, and the "options" correspond to candidate item values. Figure 12(A) shows an example of output 55 from the generating AI server 3. Output example 55 is in the format shown in output example area 51d of Figure 10.

[0051] In S25, the control unit 10 (item name candidate acquisition unit 15) analyzes the received item name candidates and item value candidates to select an item name. First, the control unit 10 converts the received candidate item names and candidate item values, which are in the format shown in output example 55 in Figure 12(A), into a format that can be handled by the program. Figure 12(B) shows Table 56, which contains the extracted candidate item names and candidate item values. Next, the control unit 10 compares the candidate item names in Table 56 with the text information 31 obtained from the table area 30 (see Figure 4). If they match, it is used as the item name; otherwise, it is used as the item value.

[0052] Furthermore, if, for example, both the candidate item name and the candidate item value contain similar strings, the control unit 10 processes the string to treat it as the item value and exclude it from the item name. Furthermore, patterns that are clearly item values ​​rather than item names are pre-registered in the memory unit 20, and the control unit 10 converts items that match these patterns into item values. Examples of patterns that become item values ​​include "~ years old", "~ times", "~ person", etc.

[0053] Figure 12(C) shows the structured results 57 obtained as item names. Furthermore, Figure 13 shows the table area 30, reflecting the structured results 57, with the obtained item names enclosed in a dotted line. In this way, the control unit 10 can obtain item names from the table area 30. Subsequently, the control unit 10 moves the process to S15 in Figure 2.

[0054] Next, we will explain the item combination process. Figure 14 is a flowchart showing the item combination processing of the image document processing server 1 according to this embodiment. In S51 of Figure 14, the control unit 10 (splitting unit 16) splits the strings selected as item names by the item name candidate acquisition unit 15 into item name strings and the other strings into item value strings.

[0055] In S52, the control unit 10 (table determination unit 17) checks the position of the item name in the table area based on the coordinate position of the item name string and determines the table format. The table determination unit 17 determines the table format based on the position of the item name. If the position of the item name is in the vertical direction, the table determination unit 17 determines that it is a horizontal table. Also, if the position of the item name is in the horizontal direction, the table determination unit 17 determines that it is a vertical table. The control unit 10 may omit this process by reusing the table determination performed by the structured text creation unit 13. Alternatively, since the table determination performed by the structured text creation unit 13 is performed on candidate strings that could become item names, the control unit 10 may perform this process again using the strings selected as item names.

[0056] In S53, the control unit 10 (combination determination unit 18) determines the item name string for the item value string based on the distance and slope between the two strings. Here, we will explain how to determine combinations using a specific example. Figures 15 to 19 are diagrams illustrating the combination determination process of the image document processing server 1 according to this embodiment. In the table area 60 shown in Figure 15(A), the string enclosed by the dotted line is the item name string. As an example, we will explain the process of determining the item name corresponding to the item value string 62 "Corporate" in tablespace 60, as shown in Figure 15(B).

[0057] Table area 60 is a horizontal table. Figure 16 shows the method for determining item names in the case of a horizontal table. In the example table 71 in Figure 16(A), the score of the item name string 71b for the item value string 71a is performed by the method 72 shown in Figure 16(B). In Method 72, the distance (a) between the item value string 71a and the item name string 71b is the distance of the line segment connecting the midpoints of both strings. Then, in Method 72, the area (S) of the right triangle formed by the horizontal line (c) from the item value string 71a is calculated and used as the score. Furthermore, the scoring for the grid lines is performed using method 73, as shown in Figure 16(C). In horizontal tables, the item names and item values ​​are generally arranged horizontally. Therefore, according to method 73, a heavier weight is added when the value crosses a horizontal line. The control unit 10 then calculates a score for each item value string by comparing it with each item name string. The control unit 10 determines the item name string that minimizes the score obtained by adding the score for the grid line to the score calculated from the distance and slope as the item name for the target item value string.

[0058] Applying the method described in Figure 16 to the item value string 62 in tablespace 60 of Figure 15(B) results in the explanation 65 shown in Figure 15(C). In this example, the score is calculated by assigning a weight of 10 to cases where the line crosses vertical line 63 and a weight of 20 to cases where it crosses horizontal line 64. In this case, for the item value string 62 "Corporate" in tablespace 60, the score for the item name "Gender" is smaller than the score for the item name "Relationship". Therefore, the item name "Gender" is determined as a combination for the item value "Corporate".

[0059] Furthermore, if the method described in Figure 16 is applied to the item value string 66 "grandfather" in tablespace 60 of Figure 17(A), the result will be as shown in explanation 69 in Figure 17(B). In this example, the score is calculated by assigning a weight of 20 to cases where the line crosses horizontal line 67 and a weight of 10 to cases where it crosses vertical line 68. In this case, for the item value string 66 "Grandfather" in tablespace 60, the score for the item name "Relationship" is smaller than the score for the item name "Age". Therefore, the item name "Relationship" is determined as a combination for the item value "Grandfather". In this example, the item name "Age" is closer in terms of distance between the strings, but by considering the slope and borders, the correct item name, "Relationship," is determined.

[0060] The above explanation referred to horizontal tables. The same procedure can be applied to vertical tables as well. Figure 18(A) shows an example of tablespace 80. In tablespace 80, the strings enclosed by the dotted line are the item names. Therefore, tablespace 80 is a so-called vertical table, where the item values ​​for the item names are arranged vertically.

[0061] In the example table 81 in Figure 18(B), the score of the item name string 81b for the item value string 81a is performed by the method 82 shown in Figure 18(C). In Method 82, the distance (a) between the item value string 81a and the item name string 81b is the distance of the line segment connecting the midpoints of both strings. Then, in Method 82, the area (S) of the right triangle formed by the perpendicular line (b) from the item value string 81a is calculated and used as the score. Furthermore, the scoring for the grid lines is performed using method 83, as shown in Figure 18(D). In a vertical table, the item names and item values ​​are typically arranged vertically. Therefore, according to method 83, a heavier weight is added when the data crosses a vertical line. The control unit 10 then calculates a score for each item value string by comparing it with each item name string. The control unit 10 determines the item name string that minimizes the score obtained by adding the score for the grid line to the score calculated from the distance and slope as the item name for the target item value string.

[0062] Figure 19 shows the structured results 90 obtained from combinations of item names and item values. The structured result 90 is obtained by adding item values ​​(values) to each item name (key) of the structured result 57 shown in Figure 12(C).

[0063] Thus, according to this embodiment, the image document processing server 1 has the following advantages. (1) Using text information 31 containing strings and coordinate information of strings in a table area 30 extracted from a report image, structured text 37 is created by structuring the strings. An item extraction instruction is generated by embedding the created structured text 37 into a template 51A. The generated item extraction instruction is input to a generation AI server 3 that generates and outputs information according to the input information, thereby obtaining at least the item name candidates output by the generation AI server 3. Therefore, even if the table is in various formats and is included in the document image, the AI ​​server 3 can accurately obtain candidate item names from the strings contained in the table. As a result, there is no need to prepare a list of item names in advance, and candidate item names can be obtained efficiently.

[0064] (2) Candidate strings that can become item names are extracted from the strings contained in the text information 31, the extracted candidate strings are highlighted, and the coordinate information of the strings contained in the text information 31 is used to generate a group of characters by concatenating the strings that are within a predetermined range and adding a specified symbol corresponding to the distance between the strings, and the set of generated character groups is made into structured text 37. Therefore, even in the case of tabular formats with complex table structures, by highlighting candidate strings and using a group of characters formed by concatenating them with specified symbols corresponding to the distance between strings, the generation AI server 3 can correctly understand the positional relationships of strings in the table and obtain candidate item names.

[0065] (3) Morphological analysis is performed on the string to extract words, and predetermined strings containing nouns as part of speech are extracted as candidate strings. Therefore, strings that could become item names can be marked in advance.

[0066] (4) Based on the coordinate position of the highlighted candidate string, the position of the item name in the table area 30 is checked, and if it is confirmed that the candidate string has a position in the vertical direction, a character group is generated with the predetermined range as the horizontal range, and if it is confirmed that the candidate string has a position in the horizontal direction, a character group is generated with the predetermined range as the vertical range. Therefore, strings can be concatenated in the row or column direction to match the configuration of the tablespace 30.

[0067] (5) The structured text 37 is embedded in templates 51A and 51B, which contain instructions for extracting item names and item values, and the item name candidates and item value candidates output by the generating AI server 3 are obtained. Template 51B also includes extraction rules for item names and item values. Therefore, by embedding the structured text 37 into templates 51A and 51B, item creation instructions that the generation AI server 3 will handle can be generated, enabling the automatic generation of item creation instructions. Furthermore, by including extraction rules in template 51B, item names and item values ​​can be extracted from the structured text 37 with greater accuracy. In addition, not only item name candidates but also item value candidates can be obtained from the generation AI server 3.

[0068] (6) The acquired candidate item names and candidate item values ​​are analyzed, and predetermined strings classified as candidate item names are excluded from the candidate item names. Therefore, in the output of the generating AI server 3, strings that would normally be treated as item values ​​but could be included as item name candidates can be excluded from the item name candidates, making the item name candidates more accurate.

[0069] (7) The system acquires a report image, extracts a tablespace 30 from the acquired report image, and performs character recognition processing on the extracted tablespace 30 to obtain text information 31. Therefore, it is possible to extract the table area 30 from the document image and obtain the text information 31 contained in the table area 30.

[0070] (8) Using text information 31 which includes strings and coordinate information of strings contained in the table area 30 extracted from the form image, the strings determined to be item names are divided into item name strings and other strings are item value strings. The item name strings that form pairs with the divided item value strings are determined based on the distance between the two strings and the inclination of the item name strings based on the position of the item name strings with the item value string as the reference position, and the combination of the determined item name string and item value string is output. Therefore, the item name string corresponding to the item value string can be determined and output by considering not only the distance between the two strings but also their slope. As a result, even with tables of various formats included in report images, the item name and item value can be appropriately combined from the strings contained in the table.

[0071] (9) Based on the coordinate positions of the divided item name strings, the position of the item name in the table area 30 is confirmed, the tabular format is determined based on the confirmed position of the item name, and the slope based on the position of the item name string with the item value string as the reference position is calculated based on the determined tabular format. Therefore, by calculating the slope differently depending on the tabular format, it is possible to create a data format that takes the slope into account according to the tabular format.

[0072] (10) When it is confirmed that the item names are arranged vertically, it is determined that the table has the item values ​​corresponding to the item names arranged horizontally, and the item name string corresponding to the divided item value string is determined based on the distance between the two strings and the tilt between the horizontal and the item value string, with the item value string as the reference position. Furthermore, if it is confirmed that the item names are arranged horizontally, it is determined that the table has the item values ​​corresponding to the item names arranged vertically, and the item name string corresponding to the divided item value string is determined based on the distance between the two strings and the inclination in the vertical position relative to the item value string. Therefore, in a table where item values ​​corresponding to item names are located horizontally, the slope can be calculated as being between the horizontal and vertical directions, and in a table where item values ​​corresponding to item names are located vertically, the slope can be considered as being between the vertical and horizontal directions.

[0073] (11) For each divided item value string, the item name strings that form a pair are determined to be the combination that minimizes the score calculated using the distance and slope between the two strings. Therefore, since the combination is determined by showing the distance and slope as a score, it can be determined in an easy-to-understand way.

[0074] (12) When there is a grid line between the item value string and the item name string, a weight corresponding to the direction and number of grid lines is added to the score. Furthermore, the weight assigned to the direction of the grid lines differs depending on whether the direction of the item value is relative to the item name or not. Therefore, the presence or absence of grid lines can be taken into consideration, and the weighting of grid lines can be varied depending on the table format, and these factors can be added to the score. As a result, the combination of item names and item values ​​can be determined more accurately, taking the table format into account.

[0075] Although embodiments of the present invention have been described above, the present invention is not limited to the embodiments described above. Furthermore, the effects described in the embodiments are merely a list of the most preferred effects arising from the present invention, and the effects of the present invention are not limited to those described in the embodiments. The embodiments described above and the modified forms described later can be used in combination as appropriate, but a detailed explanation is omitted.

[0076] (Transformed form) (1) In this embodiment, an image of an application form was used as an example, but it is not limited to this. Other application forms, or images of documents such as quotations, invoices, or health examination reports may also be used. Furthermore, although a non-standard document was used as an example, it can of course be used for standard documents as well, and the same procedure can be followed for any type of document. In particular, even in table areas where item names and item values ​​are identified by color such as shading, and where there are no grid lines between item names and item values, or where the starting positions (horizontal and vertical) of item names are not aligned, it is possible to appropriately select item names and determine combinations of item names and item values.

[0077] (2) In this embodiment, an example has been described in which an item extraction instruction is sent to the generation AI server 3, and item name candidates and item value candidates are received from the generation AI server 3 to select an item name, but the embodiment is not limited to this. For example, the process of sending an item extraction instruction to the generation AI server 3 and receiving item name candidates and item value candidates from the generation AI server 3 may be performed multiple times. In this case, the response from the generation AI server 3 will be different each time, but for example, an item name candidate that appeared in two or more out of three attempts may be selected as the item name.

[0078] (3) In this embodiment, when determining the item name string to be combined with the item value string, the example described was one in which the weight for cases where the string crosses a grid line was set to a predetermined value, but the embodiment is not limited to this. For example, the weight may be set according to the slope, and different weights may be set depending on whether the item value is in the direction of the item name or not. Alternatively, the weight may be made variable. If the weights are to be variable, for example, a regression curve can be obtained that calculates two types of weights—heavy weights and light weights—from data in multiple tablespaces that have correct combinations of item names and item values, and the weights for the slope can be calculated using the obtained regression curve.

[0079] (4) In this embodiment, an example was described in which the output of item name and item value combinations is performed in real time in response to the receipt of report image data from the user terminal 5, but the system is not limited to this. The process can also be used in a system in which the item name and item value combinations are stored in a DB (not shown) and answers are made to questions from the user terminal 5 using the data in the DB.

[0080] (5) In this embodiment, an example was described in which the image document processing server sends a document image to the OCR device 4 and requests text conversion, but the embodiment is not limited to this. Before sending the document image to the OCR device 4, processing such as contrast adjustment or sharpening may be added. Doing so may further improve the accuracy of processing by the OCR device 4.

[0081] (6) In this embodiment, an example has been described comprising an image document processing server 1, a generation AI server 3, an OCR device 4, and a user terminal 5, but the embodiment is not limited to this. For example, the image document processing server and the OCR device may be implemented by the same server. Alternatively, for example, the configuration may consist of a client terminal having the functions of an image document processing server and an OCR device, and a generation AI server. [Explanation of symbols]

[0082] 1. Image document processing server 3. Generation AI Server 4 OCR device 5. User terminals 10 Control Unit 11. Form Image Acquisition Unit 12 Pre-processing section 13. Structured Text Creation Section 14 Instruction generation section 15 Item name candidate acquisition part 16 Division 17 Table judgment section 18. Combination Determination Section 19-item combination output section 20 Memory section 21a Character Recognition Processing Program 21b Item combination program 22 Directive sentence storage section 31 Text Information 37 Structured Text 51A, 51B templates 51f Rule description area 57, 90 structured results 72, 73, 82, 83 method 100 Image Document Processing Systems

Claims

1. A structured text creation means that creates structured text by structuring the string using text information that includes a string in a table area extracted from a form image and the coordinate information of the string, Instruction generation means that generates an item extraction instruction by embedding the structured text created by the structured text creation means into an instruction sentence, The result acquisition means inputs the item extraction instruction generated by the instruction generation means to a generation AI that generates and outputs information according to the input information, thereby acquiring at least the item name candidates output by the generation AI. A character recognition device equipped with the following features.

2. In the character recognition device according to claim 1, An emphasis means for extracting candidate strings that can become item names from the strings contained in the text information, and highlighting the extracted candidate strings, A string concatenation means that generates a group of characters by concatenating strings within a predetermined range using the coordinate information of the strings contained in the text information, by adding a specified symbol corresponding to the distance between the strings, Equipped with, The structured text creation means is a character recognition device that uses the set of characters generated by the string concatenation means as the structured text.

3. In the character recognition device according to claim 2, The enhancement means is a character recognition device that performs morphological analysis on the string to extract words, and extracts predetermined strings containing nouns as the part of speech of the words as candidate strings.

4. In the character recognition device according to claim 2, The system includes an item position confirmation means that confirms the position of the item name in the table area based on the coordinate position of the candidate string that has been highlighted by the emphasis means, The character recognition device comprises a character string concatenation means which generates the character group with the predetermined range as a horizontal range when the item position confirmation means confirms that the candidate string has a position in the vertical direction, and generates the character group with the predetermined range as a vertical range when the item position confirmation means confirms that the candidate string has a position in the horizontal direction.

5. In the character recognition device according to claim 1, The instruction generation means embeds the structured text into the instruction sentence, which includes instructions to extract item names and item values ​​from the structured text. The result acquisition means is a character recognition device that acquires the candidate item name and candidate item value output by the generating AI.

6. In the character recognition device according to claim 5, The instruction statement includes an extraction rule relating to the item name and the item value, and is a character recognition device.

7. In the character recognition device according to claim 5, A character recognition device comprising an item name selection means that analyzes the item name candidates and item value candidates acquired by the result acquisition means and excludes predetermined strings classified as item name candidates from the item name candidates.

8. In the character recognition device according to claim 1, A means for acquiring the aforementioned document image, A table extraction means for extracting the table area from the document image acquired by the document image acquisition means, A text information acquisition means performs character recognition processing on the table area extracted by the table extraction means to acquire the text information, A character recognition device equipped with the following features.

9. Computers, A structured text creation means that creates structured text by structuring the string using text information that includes a string in a table area extracted from a form image and the coordinate information of the string, Instruction generation means that generates an item extraction instruction by embedding the structured text created by the structured text creation means into an instruction sentence, The result acquisition means inputs the item extraction instruction generated by the instruction generation means to a generation AI that generates and outputs information according to the input information, thereby acquiring at least the item name candidates output by the generation AI. A program designed to function as such.

Images