Document recognition apparatus, document recognition method, and program

Abstract

To provide a document recognition device for improving accuracy in determining various kinds of documents.SOLUTION: In the document recognition system, a document recognition device 2 includes a character extraction unit 12 that extracts character information from document data, a specification unit 15 that specifies an item character string, which is a character string for specifying a document type, and a structure of the document data from the extracted character information, and a document determination unit 16 that determines the document type of the document data based on a combination of the item character string and the structure of the document data.SELECTED DRAWING: Figure 3

Description

[Technical Field]

[0001] The present invention relates to a document recognition device, a document recognition method, and a program. [Background technology]

[0002] Regarding the technology for extracting character information from documents using OCR technology, a technology is known in which the document type is determined and then character information corresponding to the document type is extracted. For this reason, various methods for determining the document type have been devised. Patent Document 1 discloses a technology for cluster classification by inputting character strings extracted from a document into a machine learning model. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Japanese Patent Publication No. 2022-187945 Summary of the Invention [Problem to be solved by the invention]

[0004] However, conventional technology can make mistakes when identifying various types of documents, such as contracts, invoices, delivery notes, purchase orders, estimates, receipts, and driver's licenses. However, there is a problem in that it is difficult to identify documents with similar content (invoices, delivery notes, purchase orders, estimates, receipts, etc.) even when using AI.

[0005] In view of the above-mentioned problems, the present invention provides a technique for further improving the accuracy of determining various types of documents. [Means for solving the problem]

[0006] A document recognition device according to one aspect of the present invention includes a character extraction unit that extracts character information from document data, an identification unit that identifies an item string, which is a string for identifying a document type, and a structure of the document data from the extracted character information, and a document determination unit that determines the document type of the document data based on a combination of the item string and the structure of the document data. [Effects of the Invention]

[0007] According to the present invention, it is possible to further improve the accuracy of determining various types of documents. [Brief explanation of the drawings]

[0008] [Figure 1] 1 is a diagram showing the overall configuration of a document recognition system according to an embodiment of the present invention; [Figure 2] 1 is a diagram illustrating a hardware configuration of a document recognition apparatus according to an embodiment of the present invention. [Figure 3] 1 is a diagram illustrating a functional configuration of a document recognition apparatus according to an embodiment of the present invention. [Figure 4] FIG. 2 is a flowchart of a document type determination process performed by a document recognition apparatus according to an embodiment of the present invention. [Figure 5] 1 is a diagram for explaining a method for determining whether a document is a fixed form document performed by a document recognition device according to an embodiment of the present invention; [Figure 6] FIG. 2 is a diagram for explaining a method for calculating a TF-IDF in a document recognition device according to an embodiment of the present invention. [Figure 7] FIG. 10 is a diagram showing an example of a template of appearance frequency of a specified word for each document type created by the document recognition apparatus according to an embodiment of the present invention. [Figure 8] FIG. 10 is a diagram showing an example of a template of part-of-speech ratios for each document type created by the document recognition device according to one embodiment of the present invention. [Figure 9] FIG. 4 is a flowchart of a second designated document specification process in the document recognition device according to one embodiment of the present invention. [Figure 10]1 is a diagram showing an example of document data in a document recognition apparatus according to an embodiment of the present invention; [Figure 11] FIG. 2 is a diagram showing an example of structure information in a document recognition apparatus according to an embodiment of the present invention. [Figure 12] FIG. 2 is a diagram showing an example of structure information in a document recognition apparatus according to an embodiment of the present invention. [Figure 13] FIG. 2 is a functional block diagram of a learning unit that generates a classifier in the document recognition device according to the embodiment of the present invention. [Figure 14] FIG. 10 is a flowchart of a modified example of the document type determination process performed by the document recognition apparatus according to an embodiment of the present invention. DETAILED DESCRIPTION OF THE INVENTION

[0009] DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS The following describes the preferred embodiments of the present invention with reference to the accompanying drawings. In the drawings, the same components are designated by the same reference numerals, and redundant explanations may be omitted.

[0010] <System configuration example> 1 is a diagram showing the overall configuration of a document recognition system 1 according to an embodiment of the present invention. As shown in FIG. 1, the document recognition system 1 includes a document recognition device 2, a user terminal 3, and a scanner device 4, which are capable of communicating with each other via a network 5.

[0011] The network 5 may be, for example, an in-house LAN. The network 5 may be configured using wireless communication such as Wi-Fi (registered trademark, omitted below). Furthermore, if the document recognition device 2 exists on a cloud, the network 5 may include a WAN or the Internet. For example, the user terminal 3 can transmit image data read by the scanner device 4 to the document recognition device 2.

[0012] The document recognition device 2 may be directly connected to the scanner device 4 in a one-to-one wired manner via a USB cable or the like. Even in the case of one-to-one connection, the document recognition device 2 and the scanner device 4 may communicate wirelessly. Known communication methods for this purpose include Wi-Fi Direct and Bluetooth (registered trademark).

[0013] The document recognition device 2 and the user terminal 3 may be any information processing device equipped with a communication function, such as a PC (Personal Computer), a server device, a smartphone, or a tablet PC.

[0014] The document recognition device 2 may extract character information by performing character recognition using, for example, OCR technology on document data, which is image data of a document read by the scanner device 4, and may allow the user to confirm or correct the results. The document recognition device 2 may also extract character information from document data transmitted from the user terminal 3.

[0015] The scanner device 4 is an optical reading device that reads a document and generates image data, which is document data, and transmits the data to the document recognition device 2. In this embodiment, the scanner device 4 scans documents. Although FIG. 1 illustrates the scanner device 4, image data for character recognition may be captured by a digital camera or the like. The image data captured by the digital camera may be transmitted via the network 5 or may be stored in a removable storage medium. When the user inserts the storage medium into the document recognition device 2, the document recognition device 2 can acquire the document data.

[0016] The scanner device 4 may also be a device called an MFP (Multifunction Peripheral) or a multifunction device. That is, the scanner device 4 may have a printer function, a copy function, a facsimile function, etc. in addition to a scanner function.

[0017] Furthermore, in FIG. 1, the document recognition device 2 and the scanner device 4 are separate devices, but the document recognition device 2 and the scanner device 4 may be integrated into one device (such as an MFP).

[0018] <Hardware configuration example> The hardware configuration of the document recognition device 2 according to this embodiment will be described with reference to Fig. 2. Fig. 2 is a diagram showing the hardware configuration of the document recognition device 2 according to one embodiment of the present invention.

[0019] As shown in FIG. 2, the document recognition device 2 includes a CPU 101, a ROM 102, a RAM 103, a HDD (Hard Disk Drive) 104, a HDD (Hard Disk Drive) controller 105, a display 106, an external device connection I / F (Interface) 108, a network I / F 109, a bus line 110, a keyboard 111, a pointing device 112, an optical drive 114 such as a DVD-RW (Digital Versatile Disk Rewritable), and a media I / F 116.

[0020] Of these, the CPU 101 controls the overall operation of the document recognition device 2. The ROM 102 stores programs used to drive the CPU 101, such as an IPL (Initial Program Loader). The RAM 103 is used as a work area for the CPU 101. The HDD 104 stores various data such as programs.

[0021] The HDD controller 105 controls the reading and writing of various data from and to the HDD 104 under the control of the CPU 101. The display 106 displays various information such as a cursor, menu, window, text, or image. The external device connection I / F 108 is an interface for connecting various external devices. In this case, the external device is, for example, a USB (Universal Serial Bus) memory or a printer.

[0022] The network I / F 109 is an interface for performing data communication using the network 5. The bus line 110 is an address bus, a data bus, or the like for electrically connecting the components such as the CPU 101.

[0023] The keyboard 111 is a type of input means having multiple keys used to input characters, numbers, various instructions, etc. The pointing device 112 is a type of input means used to select and execute various instructions, select a processing target, move a cursor, etc.

[0024] The optical drive 114 controls reading and writing of various data from and to a DVD-RW 113, which is an example of a removable recording medium. The optical drive 114 is not limited to a DVD-RW, and may be a DVD-R or the like. The media I / F 116 controls reading and writing (storing) of data from and to a recording medium 115, such as a flash memory.

[0025] <Example of functional configuration> Next, the functional configuration of the document recognition device 2 will be described with reference to Fig. 3. Fig. 3 is a diagram showing the functional configuration of the document recognition device 2 according to an embodiment of the present invention.

[0026] (scanner device) The scanner device 4 has a communication unit 41 and a reading unit 42. The reading unit 42 feeds documents such as forms one by one and generates image data with a predetermined resolution and gradation by scanning the document surface with a line sensor. Instead of the scanner device 4, a device equipped with a camera function such as a digital camera may acquire the image data of the document.

[0027] The communication unit 41 communicates with the document recognition device 2 using a communication protocol such as SNMP (Simple Network Management Protocol) or via a dedicated line such as a USB cable. The communication unit 41 transmits the image data generated by the reading unit 42 to the document recognition device 2.

[0028] (Document Recognition Device) The document recognition device 2 has an acquisition unit 11, a character extraction unit 12, a display control unit 13, an operation reception unit 14, an identification unit 15, and a document determination unit 16. The identification unit 15 has an item name / item value extraction unit 17, a table structure extraction unit 18, and a title extraction unit 19. The document determination unit 16 has a similarity judgment unit 20, a fixed form document identification unit 21, and a designated document identification unit 22. Hereinafter, the identification unit 15 and the designated document identification unit 22 will also be collectively referred to as a "classifier 23."

[0029] These units of the document recognition device 2 are functions or means realized by the CPU 101 of the document recognition device 2 executing program instructions. This program may be, for example, a native application dedicated to the scanner device, a general-purpose native application, or a web application as described below.

[0030] The acquisition unit 11 acquires the document data generated by the scanner device 4, for example, via the network 5. The acquisition unit 11 communicates with the scanner device 4 using a communication protocol such as SNMP (Simple Network Management Protocol). Additionally, the acquisition unit 11 may acquire document data generated by a device equipped with a camera function such as a digital camera, may acquire document data from the user terminal 3, or may read the document data from a storage medium.

[0031] Character extraction unit 12 extracts character information from document data. For document data read using scanner device 4, character recognition processing may be performed to extract character information, or for document data acquired from user terminal 3, characters contained in the document data may be read. The document data may be image data of a document generated by scanner device 4, or may be document data acquired from user terminal 3. Character extraction unit 12 extracts coherent character strings as character information, and can also extract coordinates that specify the position, such as the circumscribing rectangle, of each character string.

[0032] The display control unit 13 displays various screens on the displays of the document recognition device 2 and the user terminal 3. For example, the display control unit 13 displays document data and character information extracted by the character extraction unit 12. The display control unit 13 may also display document recognition results (determination results for contracts, invoices, delivery notes, order forms, estimates, receipts, driver's licenses, etc.).

[0033] The operation reception unit 14 receives various operations for the document recognition device 2. For example, the operation reception unit 14 receives an operation such as starting a process of determining the document type.

[0034] The identification unit 15 identifies, from the extracted character information, an item string, which is a string for identifying the document type, and the structure of the document data. Identification of the structure of the document data may be based on the positional relationship between the ruled lines and the character strings included in the document data. Alternatively, it may be based on the arrangement of the character strings included in the document data and the item strings. In other words, the identification unit 15 can identify the structure of the document data even if the document data does not include ruled lines.

[0035] The document determination unit 16 determines the document type of document data based on a combination of the item string and the structure of the document data. The combination of the item string and the structure of the document data includes the concept of their respective positional relationships. The positional relationships may be expressed by coordinates in the document data.

[0036] The document determination unit 16 may determine the document type of document data in multiple stages using a trained model in which feature amounts calculated based on the structure of document data and item strings for each document type are trained using the document type as training data. Note that the determination of the document type of document data using the trained model may be performed at all stages, at multiple stages among all stages, or at only one stage.

[0037] The document determination unit 16 determines document data that conforms to a predetermined format as a fixed form document. Furthermore, the document determination unit 16 determines, based on a combination of the structure of the document data and the item strings, document data that is determined to consist of only one document type among the document data that is not determined to be a fixed form document as a first designated document. Furthermore, the document determination unit 16 determines, based on a combination of the structure of the document data and the item strings, document data that is determined to include multiple document types among the document data that is not determined to be a fixed form document as a second designated document.

[0038] For example, a contract consisting of a single document type is a first designated document. Also, document data that contains a receipt and an invoice and is treated as a "receipt" is a second designated document because it contains both a receipt and an invoice.

[0039] The item name and item value extraction unit 17 extracts item names and item values ​​from document data. The extraction method will be described in detail later. The item names and item values ​​become input to a model generated by machine learning. The output of the model is the document type.

[0040] The table structure extraction unit 18 extracts a table structure from document data by detecting, for example, ruled lines. The table structure includes the entire table, ruled lines, field names and field values ​​in the table, and their position information. The table structure is also input to the model.

[0041] The title extraction unit 19 extracts titles from document data. The extraction method will be described in detail later. The titles are input to a model generated by machine learning. The output of the model is the document type.

[0042] The similarity determination unit 20 compares a prepared template with information obtained by forming a template from character information and table structure extracted from document data, and determines whether the document data is a designated document or not.

[0043] The standard document identification unit 21 determines whether the document data is a standard document such as a driver's license. The standard document identification unit 21 compares the standard document format prepared in advance for each standard document with the character information and table structure extracted from the document data, and determines that the document is the one in question if there is a certain degree of match or higher.

[0044] The designated document identification unit 22 inputs the title, item names, item values, and table structure into the model and determines the document type based on the model output. Since standard documents, other documents, and some designated documents have already been determined, the document type output is a document other than these. This improves the model's determination accuracy.

[0045] <Overall process flow> Next, a flow of document type determination processing performed by the document recognition device 2 will be described. Fig. 4 is a flow diagram of document type determination processing performed by the document recognition device 2 according to an embodiment of the present invention. As shown in Fig. 4, in this processing, document data is gradually determined to be one of a fixed form document, a first designated document, and a second designated document. A detailed description of each step will be given later.

[0046] The stage at which document data is judged is determined by the type of document. In Fig. 4, the types of documents to be classified are as follows: Standard document: Driver's license First designated document: contract Second designated document: invoice, quotation, delivery note, receipt, purchase order

[0047] First, the scanner device 4 reads a document or generates document data by acquiring the document data from the user terminal 3. The acquisition unit 11 acquires learning data (S101). The character extraction unit 12 first performs character recognition such as OCR on the document data.

[0048] Next, the standard document identification unit 21 identifies the standard document (S102), and determines whether the character information (character strings, coordinates) and table structure extracted from the document data correspond to a license, which is a standard document (S103).

[0049] If the standard document identification unit 21 determines that the document data is a standard document, such as a driver's license (Yes in step S103), the process proceeds to step S104, and the document determination unit 16 determines that the document data is a standard document, such as a driver's license. If the determination in step S103 is No, the process proceeds to step S105.

[0050] In step S105, the similarity determination unit 20 calculates the similarity between a template prepared in advance for the first specified document and the information obtained by converting the document data into the same format as the template, and determines whether the similarity is equal to or greater than a threshold (S105). If multiple templates are prepared, the document data is compared with all of the templates.

[0051] The designated document identification unit 22 determines the document type based on the similarity calculation result in step S105 (S106). The document determination unit 16 determines, among the document data that has not been determined as a fixed form document, document data that is determined to consist of only one document type, as a first designated document.

[0052] The document determination unit 16 determines that the document data consisting only of the contract is the contract, which is the first designated document (S107). If the document data is unknown, the document determination unit 16 processes the document as being of unknown type (S108).

[0053] The document determination unit 16 determines, as a second designated document, document data that is determined to include multiple document types among document data that is not determined to be a fixed form document. The designated document identification unit 22 and the specification unit 15 identify which document type among the multiple document types is included in the document data (S109). The designated document identification unit 22 determines the document type based on the identification result of step S109 (S110).

[0054] When it is identified that the document type included in the document data is a claim, the document determination unit 16 determines that the document type is a claim (S111). When it is identified that the document type included in the document data is a quotation, the document determination unit 16 determines that the document type is a quotation (S112). When it is identified that the document type included in the document data is a delivery note, the document determination unit 16 determines that the document type is a delivery note (S113).

[0055] When it is identified that the document type included in the document data is a receipt, the document determination unit 16 determines that the document type is a receipt (S114). When it is identified that the document type included in the document data is an order, the document determination unit 16 determines that the document type is an order (S115). And when the document data is unclear, the document determination unit 16 processes it as being unclear as the document type (S116).

[0056] Hereinafter, the processing of each step will be described in detail.

[0057] <S102 Standard Document Identification> The document determination unit 16 determines document data corresponding to a predetermined format as a standard document. A standard document refers to a document in which both the document type and the format are uniquely determined. For example, application forms used within a company or cards used for personal authentication (e.g., driver's license, my number card) are standard documents.

[0058] Here, referring to FIG. 5, the outline of the determination of standard documents will be described. FIG. 5 is a diagram for explaining a method for determining standard documents performed by the document recognition apparatus 2 according to an embodiment of the present invention.

[0059] FIG. 5(a) shows a driver's license as an example of a standard document. The standard document format is information obtained by extracting text, its coordinates, and formation from the standard document. In FIG. 5(a), text definition areas where name, address, and issuance are described and several ruled lines are shown.

[0060] Figure 5(b) shows the character information and coordinates of the text definition area. Figure 5(c) shows the ruled lines. The fixed-form document identification unit 21 acquires the character information, table structure, etc. from the coordinates defined in the fixed-form document format among the character information, table structure, etc. extracted from the document data. Then, the fixed-form document identification unit 21 determines whether this character information, table structure, etc. matches the text defined in the fixed-form document format.

[0061] Also, the fixed-form document identification unit 21 detects a straight line of a certain length or more from the document data by edge extraction or the like. The fixed-form document identification unit 21 performs template matching between this straight line and the ruled lines included in the fixed-form document format, and determines whether it conforms to the fixed-form document format based on whether the degree of match is a certain level or more. When the degrees of match of both the text and the ruled lines are above the threshold, the fixed-form document identification unit 21 determines that it is a fixed-form document associated with the fixed-form document format.

[0062] <S105 Specified Document Identification (Determination by Similarity)> The document recognition device 2 extracts character information from the document data in step S101. The similarity determination unit 20 extracts features from the character information, table structure, etc. for comparison with the template. The features are in the same format as the template. In this embodiment, as an example, the similarity determination unit vectorizes the character information using TF-IDF (Term Frequency-Inverse Document Frequency). The definitions of TF and IDF are as follows.

[0063] TF: Frequency of occurrence of a specified word in a document = Number of occurrences of the specified word in the document / Total number of occurrences of all words in the document IDF: Inverse document frequency (rarity of the specified word) = log (Number of documents (N) / Number of documents in which the word t appears) TF-IDF (Term Frequency-Inverse Document Frequency) = TF * IDF

[0064] Formula (1) is the calculation formula for TF, and formula (2) is the calculation formula for IDF. A large TF-IDF means that the word is important.

[0065]

number

[0066]

number

[0067] Here, a method for calculating TF-IDF will be described with reference to Fig. 6. Fig. 6 is a diagram for explaining a method for calculating TF-IDF in a document recognition device 2 according to an embodiment of the present invention.

[0068] Figure 6(a) shows the number of times a specified word appears in each document. As shown in Figure 6(a), the total number of times each word appears is counted for each document type. For example, in the case of the word "A" in a contract, the word "A" is searched for and counted in the contract. Note that the number of occurrences of one document type (e.g., contract) can be one or more, but the number of occurrences across each document type should be consistent or an average value should be used.

[0069] Figure 6(b) shows the frequency of occurrence (TF) of a specified word in each document. As shown in Figure 6(b), the frequency of occurrence of a word in each document is calculated for each document type. For example, for the word "Party A" in a contract, the frequency is 2 / (2+2+2+1+1)=0.25.

[0070] Figure 6(c) shows the inter-document rarity (IDF) of each word. As shown in Figure 6(c), the inter-document rarity of each word is calculated for each document type. The IDF is calculated for each word, based on the number of documents in which the word appears. The larger the IDF, the higher the rarity.

[0071] Figure 6(d) shows the importance (TF-IDF) of the specified word in each document. That is, TF-IDF is the value obtained by multiplying TF in Figure 6(b) by IDF in Figure 6(c).

[0072] The document recognition device 2 sets the TF-IDF created as shown in Figure 6(d) as a template. Templates may be created for each document, or one template may be created from multiple documents. When one template is created from multiple documents, it is difficult to accurately identify documents with similar layouts (such as invoices and estimates) by comparing them with templates alone. Documents with similar layouts are known, or are identified based on whether their TF-IDFs are similar.

[0073] In Figure 6(d), we assume that there is a contract system and an invoice system that combines templates for invoices, estimates, delivery notes, receipts, and purchase orders. Note that when combined, each TF-IDF value can be the average value of the values ​​for the same word. From the above, we assume that two templates, a contract system and an invoice system, have been generated.

[0074] TF-IDF contains the importance of each word used in a document, so it can be considered a feature vector that represents the characteristics of the document. Therefore, by comparing the similarity between the TF-IDF of document data and the TF-IDF created in advance for each document whose document type is known, the document type of the document data can be determined.

[0075] Fig. 7 is a diagram showing an example of templates of appearance frequencies of specified words for each document type created by the document recognition device 2 according to one embodiment of the present invention. Fig. 7 shows templates for contract-related documents and invoice-related documents. That is, the contract-related document template represents the characteristics of contracts, and the invoice-related document template represents the characteristics of invoices and estimates.

[0076] Figure 8 shows an example of templates of part-of-speech ratios for each document type created by a document recognition device 2 according to an embodiment of the present invention. Figure 8(a) shows templates for contract-related documents and invoice-related documents in the United States, and Figure 8(b) shows templates for contract-related documents and invoice-related documents in Japan. In other words, the document recognition device 2 can determine the document type based on the characteristics of contracts and the characteristics of invoices and estimates for each country.

[0077] The similarity determination unit 20 calculates the TF-IDF of the document data, and determines whether they are similar by calculating the cosine similarity or the like between this and the template. Equation (3) shows the calculation formula for the cosine similarity. cos(x,y) takes a value in the range from 1 to -1, and the closer it is to 1, the higher the similarity.

[0078] [Number]

[0079] <S109 Designated document identification (determination by classifier)> <C In the case of document data similar to the template summarized from the TF-IDF of a plurality of documents, the designated document identification unit 22 identifies which designated document it is among the plurality of types summarized.

[0080] FIG. 9 is a flowchart of the second designated document specifying process in the document recognition apparatus 2 according to an embodiment of the present invention. As shown in FIG. 9, the further classification (document specifying process) within the designated document has four steps S201 to S204. Hereinafter, the details of each step will be described while referring to FIG. 10. FIG. 10 is a diagram showing an example of document data in the document recognition apparatus 2 according to an embodiment of the present invention. The document data has a title, items, and a table structure for the convenience of explaining each step.

[0081] First, the title extraction in step S201 will be described.

[0082] The title extraction unit 19 extracts the title from the extracted character information. As extraction methods, methods such as comparison of the recognized character string with a dictionary, conditional branching using the height of the characters, and the position of the characters are known, and it is assumed that the extraction is performed by an existing method.

[0083] Since the character strings indicating the document type are known to be invoice, delivery note, order form, estimate, and receipt, the title extraction unit 19 searches for these character strings from the character information extracted from the document data. The title extraction unit 19 determines whether the character height of the character string that matches the search is higher than the height of the other character information, since titles are usually written in large characters. The title extraction unit 19 determines whether the coordinates of the character string that matches the search are located in the upper half of the entire document, since titles are usually written at the top of the document.

[0084] When the title extraction unit 19 finds character information that satisfies all of these three conditions, it determines that this character information is a title. It may also be determined that the character information is a title when it satisfies two or one of the three conditions.

[0085] Next, the extraction of item names and item values ​​in step S202 will be described.

[0086] The item name and item value extraction unit 17 extracts character strings corresponding to item names and item values ​​that indicate the document type from the extracted character information. The extraction method includes a method using a regular expression prepared in advance, and a method using a machine learning classifier 23 for item names and item values.

[0087] For example, in the case of an invoice, the item names that indicate the document type are "Total Amount," "Invoice Date," and "Invoice Number." Also, the item values ​​that indicate the document type are "2,000 yen," "2024 / 01 / 17," "AA-0123," "We are invoicing," etc.

[0088] Furthermore, the item name and item value extraction unit 17 extracts character strings corresponding to the item names and item values ​​that indicate the table structure from the recognized character information. The extraction method includes a method using a regular expression prepared in advance, a method using a machine learning classifier 23 for item names and item values, and the like.

[0089] For example, in the case of an invoice, the item names that indicate the table structure are "Contents," "Quantity," "Unit Price," "Amount," etc. Also, the item values ​​that indicate the table structure are "Spiny Lobster Hot Pot Set," "1," "1,000 yen," "10,000 yen," etc.

[0090] In Fig. 10, as an example of extracted item names and item values ​​using an invoice as an example, the item names and item values ​​indicating the document type are shown in rectangular frames. Also in Fig. 10, the item names and item values ​​indicating the table structure are shown in rectangular frames.

[0091] Next, the extraction of the table structure in step S203 will be described.

[0092] The table structure extraction unit 18 extracts ruled lines and acquires ruled line information from document data. The table structure includes ruled line information, item names and item values ​​that indicate the table structure, and combinations of these. The combinations include the concept of positional relationships. The positional relationships may be expressed by coordinates in the document data.

[0093] Finally, the determination in step S204 will be described.

[0094] As will be described later, the user may prepare a classifier 23 in advance that identifies documents to be classified (e.g., in the case of a group of invoices, invoice, estimate, receipt, delivery note, and order form). The classifier 23 may be any classifier that can classify document types, such as a gradient boosting decision tree or a support vector machine. The specified document identification unit 22 inputs structural information of the document data to the classifier 23 and obtains the result of identifying the type of document data from the classifier 23.

[0095] The structure information will be described with reference to Fig. 11. Fig. 11 is a diagram showing an example of the structure information in the document recognition device 2 according to one embodiment of the present invention. The structure information is information in which, for example, title extraction results, item names indicating document types, item value extraction results, table information, etc. are used as feature quantities. Fig. 11 shows an example of feature quantities input as the structure information. Examples of content and data are shown in association with each feature quantity.

[0096] Structure information for document types of receipts and invoices will be described with reference to Fig. 12. Fig. 12 is a diagram showing an example of structure information in a document recognition apparatus 2 according to an embodiment of the present invention.

[0097] Taking receipts and invoices as examples, the feature quantities are expressed as the ratio of the area of ​​the manuscript area to the area of ​​the text area, the number of characters that can fit on one line in the text area, the area of ​​the text area, and the number of characters that can fit on one line in the manuscript area. Figure 12(a) shows an example of feature quantities input as structural information. Examples of content and data are shown in association with each feature quantity. Figure 12(b) also shows an example of the manuscript area and the text area.

[0098] <About generating classifier 23> Here, the generation of the classifier 23 will be described with reference to Fig. 13. Fig. 13 is a functional block diagram of a learning unit 200 that generates the classifier 23 in a document recognition device 2 according to an embodiment of the present invention.

[0099] The learning unit 200 is realized by an arbitrary information processing device executing a program. The learning unit 200 has a function of generating a document type discrimination model. The learning unit 200 has a learning data acquisition unit 201, a learning data storage unit 202, and a model generation unit 203. The learning data acquisition unit 201 acquires learning data. The learning data is, for example, as follows: Input data: Document data structure Training data: Document type

[0100] The learning data acquisition unit 201 acquires learning data and stores it in the learning data storage unit 202. The learning data is a set of input data and training data, and multiple sets are prepared.

[0101] The training data storage unit 202 stores the training data acquired by the training data acquisition unit 201. The model generation unit 203 trains the training data using various machine learning algorithms to generate a classifier 23 (document identification model). The classifier 23 can also be expressed as correspondence information that associates structural information with document types. The document identification model of this embodiment is a classification model that classifies structural information. Classification models used in supervised learning include gradient boosting, neural networks, support vector machines, logistic regression, decision trees, and random forests. Classification models used in unsupervised learning include k-means, Gaussian mixture models, and EM algorithms.

[0102] As described above, machine learning is performed to generate the classifier 23. There are various methods for training and creating programs, but in this embodiment, training was performed using CATBoost.

[0103] In the learning phase of the classifier 23, structural information that identifies the document type is prepared, so the training data is a vector in which only the node corresponding to the relevant document type is "1" and all other nodes are zero. For example, suppose you want to identify invoices, estimates, and other documents. In the case of structural information that is known to be an invoice, the training data is a one-hot vector in which only the vector element corresponding to a contract is "1" and all other vector elements are "0."

[0104] <Modification of Document Type Determination Process> Next, we will explain a modified example of the document type determination process performed by the document recognition device 2. Fig. 14 is a flow diagram of a modified example of the document type determination process performed by the document recognition device 2 according to an embodiment of the present invention. As shown in Fig. 14, in this process, document data is gradually determined to be one of a fixed form document, a first designated document, a second designated document, and other documents.

[0105] The stage at which document data is judged is determined by the type of document. In Figure 14, the types of documents to be classified are as follows. Note that receipts can include receipts. Standard document: Driver's license First designated document: contract Second designated document: invoice, quotation, delivery note, receipt, purchase order Other documents: Warranty

[0106] First, the scanner device 4 reads a document or generates document data by acquiring the same from the user terminal 3. The acquisition unit 11 acquires learning data (S301). The character extraction unit 12 first performs character recognition such as OCR on the document data.

[0107] Next, the standard document identification unit 21 identifies the standard document (S302), and determines whether the character information (character strings, coordinates) and table structure extracted from the document data correspond to a standard document such as a driver's license (S303).

[0108] If the standard document identification unit 21 determines that the document data is a standard document, such as a driver's license (Yes in step S303), the process proceeds to step S304, and the document determination unit 16 determines that the document data is a standard document, such as a driver's license. If the determination in step S303 is No, the process proceeds to step S305.

[0109] The title extraction unit 19 searches the character information for character strings indicating the document type, such as invoice, delivery note, order form, estimate, and receipt, and determines the title based on the size and coordinates of these characters (S305).

[0110] The designated document identification unit 22 determines the document type based on the determination result based on the title in step S505 (S306). If the document type included in the document data is identified as an invoice, the document determination unit 16 determines the document type to be an invoice (S307). If the document type included in the document data is identified as an estimate, the document determination unit 16 determines the document type to be an estimate (S308). If the document type included in the document data is identified as a delivery note, the document determination unit 16 determines the document type to be a delivery note (S309).

[0111] If the document type included in the document data is identified as an order form, the document determination unit 16 determines the document type to be an order form (S310). If the document type included in the document data is identified as a contract, the document determination unit 16 determines the document type to be a contract (S311). If the document type included in the document data is identified as other, the document determination unit 16 determines the document type to be a warranty, which is another document (S312).

[0112] The document data determined to be a receipt may include a receipt document type. Therefore, the designated document identification unit 22 and the specification unit 15 identify whether the document data included in the receipt includes a receipt (S313). The designated document identification unit 22 determines the document type based on the identification result of step S313 (S314).

[0113] If the document type included in the document data is identified as a receipt, the document determination unit 16 determines the document type to be a receipt (S315).If the document type included in the document data is identified as a receipt, the document determination unit 16 determines the document type to be a receipt (S316).

[0114] In step S317, the similarity determination unit 20 calculates the similarity between a template prepared in advance for the first specified document and the information obtained by converting the document data into the same format as the template, and determines whether the similarity is equal to or greater than a threshold (S318). If multiple templates are prepared, the document data is compared with all of the templates.

[0115] The document determination unit 16 determines that the document data consisting only of the contract is the contract, which is the first designated document (S319). If the document data is unknown, the document determination unit 16 processes the document as being of unknown document type (S320).

[0116] The document determination unit 16 determines, as a second designated document, document data that is not determined to be a fixed form document and that is determined to contain multiple document types. The designated document identification unit 22 and the specification unit 15 identify which of the multiple document types is included in the document data (S321). The designated document identification unit 22 determines the document type based on the identification result of step S321 (S322).

[0117] If the document type included in the document data is identified as an invoice, the document determination unit 16 determines the document type to be an invoice (S323). If the document type included in the document data is identified as an estimate, the document determination unit 16 determines the document type to be an estimate (S324). If the document type included in the document data is identified as a delivery note, the document determination unit 16 determines the document type to be a delivery note (S325).

[0118] If the document type included in the document data is identified as an order form, the document determination unit 16 determines that the document type is an order form (S326). If the document data is unknown, the document determination unit 16 processes the document as being of unknown type (S327). If the document type included in the document data is identified as a receipt, the document determination unit 16 determines that the document type is a receipt (S328).

[0119] Similar to the process in step S313, the designated document identification unit 22 and the specification unit 15 identify whether the document data included in the receipt includes a receipt (S328). The designated document identification unit 22 determines the document type based on the identification result in step S328 (S329).

[0120] If the document type included in the document data is identified as a receipt, the document determination unit 16 determines the document type as a receipt (S330).If the document type included in the document data is identified as a receipt, the document determination unit 16 determines the document type as a receipt (S331).

[0121] <Action and effect> The document recognition device 2 according to this embodiment determines the document type in stages and determines the document type based on a combination of character strings and the structure of the document data. By determining the document type in stages, it becomes easier to classify various documents with high accuracy. Furthermore, by determining the document type based on a combination of character strings and the structure of the document data, it becomes easier to classify various documents with high accuracy. For example, by using character information and structure information for the determination, it is possible to classify documents with similar written content.

[0122] Therefore, the document recognition device 2 according to one embodiment of the present invention can further improve the accuracy of determining various types of documents.

[0123] Although the embodiments have been described above, the present invention is not limited to the above-described embodiments, and various modifications and improvements are possible within the scope of the present invention.

[0124] For example, aspects of the present invention are as follows. <1> A document recognition device comprising: a character extraction unit that extracts character information from document data; an identification unit that identifies an item string, which is a string of characters for identifying a document type, and a structure of the document data from the extracted character information; and a document determination unit that determines the document type of the document data based on a combination of the item string and the structure of the document data. <2> the document determination unit determines the document type of the document data in a plurality of stages using a trained model in which feature amounts calculated based on the structure of the document data and the item strings for each document type are trained using the document types as training data. <1> The document recognition device according to claim 1. <3> the document determination unit determines the document data corresponding to a predetermined format as a fixed form document, determines the document data that is determined to consist of only one of the document types among the document data that is not determined as the fixed form document based on a combination of the structure of the document data and the item strings as a first designated document, determines the document data that is determined to include multiple of the document types among the document data that is not determined as the fixed form document as a second designated document, and determines the document type of the document data that corresponds to the second designated document based on a similarity with the document type learned by the trained model, <2> The document recognition device according to claim 1. <4> A document recognition method executed by one or more computers, comprising: a step of extracting character information from document data; a step of identifying an item string, which is a string for identifying a document type, and a structure of the document data from the extracted character information; and a step of determining the document type of the document data based on a combination of the item string and the structure of the document data. <5> A program for causing a computer to execute the following processes: extracting character information from document data; identifying an item string, which is a string for identifying the document type, and the structure of the document data from the extracted character information; and determining the document type of the document data based on a combination of the item string and the structure of the document data. [Explanation of symbols]

[0125] 1 Document Recognition System 2. Document Recognition Device 12 Character extraction part 15 Specific section 16 Document Judgment Department

Claims

1. a character extraction unit that extracts character information from document data; an identification unit that identifies an item string, which is a string for identifying a document type, and a structure of the document data from the extracted character information; a document determination unit that determines the document type of the document data based on a combination of the item string and the structure of the document data; A document recognition device comprising:

2. The document determination unit determining the document type of the document data in a plurality of stages using a trained model in which feature amounts calculated based on the structure of the document data and the item strings for each document type are trained using the document types as training data; The document recognition device according to claim 1 .

3. The document determination unit determining that the document data that corresponds to a predetermined format is a fixed-form document; determining, based on a combination of the structure of the document data and the item character string, the document data that is determined to consist of only one of the document types among the document data that is not determined to be the fixed form document, as a first designated document; Among the document data that are not determined as the fixed form document, the document data that is determined to include a plurality of the document types is determined as a second designated document; determining the document type of the document data corresponding to the second specified document based on a similarity to the document type learned by the trained model; 3. The document recognition device according to claim 2.

4. 1. A document recognition method executed by one or more computers, comprising: extracting character information from document data; a step of identifying an item string, which is a string for identifying a document type, and a structure of the document data from the extracted character information; determining the document type of the document data based on a combination of the item string and the structure of the document data; A document recognition method comprising:

5. On the computer, A process of extracting character information from document data; A process of identifying an item string, which is a string for identifying a document type, and a structure of the document data from the extracted character information; a process of determining the document type of the document data based on a combination of the item string and the structure of the document data; A program to execute.

Images