Information processing system, item value extraction method, model generation method and program

Abstract

This information processing system comprises: a recognition result acquiring unit for acquiring a character recognition result of an image of interest; an item value candidate extracting unit for extracting a plurality of candidate character strings, which are character strings in the character recognition result of the image of interest that can be item values of items of interest for extraction; a feature generation unit which, with respect to each candidate character string, generates a feature amount based on a positional relationship between a plurality of item keywords and a candidate character string in the image of interest; a model storage unit storing a trained model generated by machine learning such that, when a feature amount based on the positional relationship between a character string and a plurality of item keywords in an image is input, the trained model outputs information indicating the validity of the character string being the item value of an item of interest for extraction; and an item value extracting unit that inputs the feature amount of each candidate character string in the image of interest to the trained model to extract the item value of the item of interest for extraction from the plurality of candidate character strings.

Description

[Technical Field]

[0001] The present disclosure relates to a technique for extracting item values contained in an image. [Background technology]

[0002] A conventional method has been proposed in which an extraction target area in a document image is specified, a character area containing an extraction term is extracted from the vicinity of the extraction target area and designated as an item name candidate area, and an extraction rule is generated based on the extraction target area and the item name candidate area. When there is one item name candidate area, the item name candidate area is designated as the item name area, and an extraction rule is generated based on the positional relationship between the extraction target area and the item name area. When there are multiple item name candidate areas and one item name area can be identified from the multiple item name candidate areas, an extraction rule is generated based on the positional relationship between the extraction target area and the identified item name area (see Patent Document 1).

[0003] In addition, a method has been proposed for determining the item values of an item value group by calculating an item value notation score for a character string detected and recognized from a form image, then calculating an item value candidate arrangement score that represents the appropriateness of the arrangement relationship between item values of different attributes for the arrangement relationship of the item value candidate pair, and further calculating an item value candidate pair score that represents the appropriateness of the pair of item values of different attributes based on the values of the item value candidate score and the item value candidate arrangement score (see Patent Document 2).

[0004] Furthermore, a method has been proposed that includes determining at least one possible target value using at least one scoring application that uses information from at least one training document, and applying the information to at least one new document using at least one scoring application to determine at least one value of the at least one target on at least one new document (see Patent Document 3). [Prior art documents] [Patent documents]

[0005] [Patent Document 1] Japanese Patent Publication No. 2020-20512 [Patent Document 2] Japanese Patent Application Laid-Open No. 2016-51339 [Patent Document 3] Special Publication No. 2013-509663 Summary of the Invention [Problem to be solved by the invention]

[0006] Conventionally, a technique for extracting information (item values) written in a document using OCR (Optical Character Recognition) has been proposed to extract item values from standard forms. In standard documents such as standard forms, the position (layout) of the items written is fixed, so by defining the reading position of the items by OCR in advance, it is possible to extract the desired information (item values).

[0007] However, when documents of the same type have various layouts (formats), the layout differs depending on the manuscript, so it is cumbersome to predefine the reading position of items by OCR for each layout, and it is difficult to extract item values using the conventional method described above.

[0008] In view of the above-mentioned problems, an object of the present disclosure is to extract item values from a document image even if the document has an undefined layout. [Means for solving the problem]

[0009] An example of the present disclosure is an information processing system including: a recognition result acquisition means for acquiring a character recognition result resulting from character recognition of a target image; an item value candidate extraction means for extracting, from the character recognition result of the target image, a plurality of candidate character strings that are character strings that can become item values of an item to be extracted; a feature generation means for generating, for each candidate character string, a feature based on the positional relationship within the target image between a plurality of item keywords that are keyword word strings for extracting item values of the item to be extracted; a model storage means for storing a trained model generated by machine learning so that, when a feature based on the positional relationship within the image between a character string and the plurality of item keywords is input, information indicating the validity of the character string as an item value of the item to be extracted is output; and an item value extraction means for extracting item values of the item to be extracted from the plurality of candidate character strings by inputting the feature of each candidate character string in the target image to the trained model.

[0010] The present disclosure can be understood as an information processing device, a system, a method executed by a computer, or a program executed by a computer. The present disclosure can also be understood as such a program recorded on a recording medium readable by a computer or other device, machine, etc. Here, a recording medium readable by a computer, etc. refers to a recording medium that stores information such as data and programs by electrical, magnetic, optical, mechanical, or chemical action and can be read by a computer, etc. [Effects of the Invention]

[0011] According to the present disclosure, it is possible to extract item values from a document image even if the document does not have a fixed layout. [Brief explanation of the drawings]

[0012] [Figure 1] 1 is a schematic diagram illustrating a configuration of an information processing system according to an embodiment. [Figure 2] FIG. 1 is a diagram illustrating an outline of the functional configuration of a learning device according to an embodiment. [Figure 3]FIG. 10 is a diagram showing an example of extraction of item value candidates (learning image 1) according to the embodiment. [Figure 4] FIG. 10 is a diagram showing an example of extraction of item value candidates (learning image 2) according to the embodiment. [Figure 5] FIG. 10 is a diagram illustrating an example of a correct answer definition table according to the embodiment. [Figure 6] FIG. 10 is a diagram illustrating an example of calculating a direction weight according to the embodiment. [Figure 7] 5A and 5B are diagrams illustrating an example of calculation of distance weights and direction weights in a learning image 1 according to the embodiment. [Figure 8] 10 is a diagram showing an example of calculation of distance weights and direction weights in a learning image 2 according to the embodiment. FIG. [Figure 9] FIG. 10 is a diagram illustrating an example of calculating an effectiveness score according to the embodiment. [Figure 10] 10 is a diagram showing an example of extraction of positional relationship information in a learning image 1 according to the embodiment. FIG. [Figure 11] FIG. 10 is a diagram illustrating an example of a positional relationship information list during a learning process according to the embodiment. [Figure 12] FIG. 10 is a diagram illustrating an example of a feature list during learning processing according to the embodiment. [Figure 13] FIG. 1 is a diagram illustrating an outline of a functional configuration of an information processing apparatus according to an embodiment. [Figure 14] 10A and 10B are diagrams illustrating examples of a positional relationship information list and validity scores for an extraction target image according to the embodiment. [Figure 15] 10 is a flowchart showing an outline of the flow of a learning process according to the embodiment. [Figure 16] 10 is a flowchart showing an outline of the flow of a category keyword determination process according to the embodiment. [Figure 17] 1 is a flowchart illustrating an overview of the flow of a trained model generation process according to an embodiment. [Figure 18] 10 is a flowchart showing an outline of the flow of an extraction process according to the embodiment. [Figure 19] FIG. 10 is a diagram illustrating an outline of the functional configuration of a learning device according to another embodiment. [Figure 20]FIG. 10 is a diagram showing an example of a correct answer definition generation screen (when "billing amount" is selected) according to the embodiment. [Figure 21] FIG. 10 is a diagram showing an example of a correct answer definition generation screen (when "payment deadline" is selected) according to the embodiment. [Figure 22] 10 is a flowchart showing an outline of the flow of a correct answer definition generation process according to the embodiment. DETAILED DESCRIPTION OF THE INVENTION

[0013] Hereinafter, embodiments of an information processing system, method, and program according to the present disclosure will be described with reference to the drawings. However, the embodiments described below are merely examples, and the information processing system, method, and program according to the present disclosure are not limited to the specific configurations described below. In implementing the present disclosure, a specific configuration according to the embodiment may be appropriately adopted, and various improvements and modifications may be made.

[0014] In this embodiment, an information processing system, a method, and a program according to the present disclosure will be described in terms of a system for extracting item values from a form image. However, the information processing system, a method, and a program according to the present disclosure can be widely used in technologies for extracting item values from document images, and the application of the present disclosure is not limited to the examples shown in the embodiment.

[0015] <System configuration> FIG. 1 is a schematic diagram showing the configuration of an information processing system 9 according to this embodiment. The information processing system 9 according to this embodiment includes one or more information processing devices 1, a learning device 2, and a document reading device 3 (3A, 3B), which are connected to a network and can communicate with each other. The learning device 2 generates a trained model for extracting item values of items (item attributes) to be extracted in a document. The information processing device 1 uses the trained model generated by the learning device 2 to extract item values of items to be extracted included in a target image.

[0016] In this embodiment, a form is used as an example of a document, but the document may be any document other than a form as long as it contains items (item values). Note that in this embodiment, "form" refers to forms in a broad sense, including ledgers, slips, supporting documents, etc. Furthermore, the form is not limited to forms of the same type that have different layouts depending on the manuscript (semi-standard form), but may also be forms of a type whose layout is determined in advance (standard form).

[0017] In this embodiment, an "item value" is a value corresponding to an item (item attribute) and is information (character string) entered (filled in) for the target item. For example, if the item is an invoice amount, the item value is a numeric character string such as "12,800" or "7,340," and if the item is a payment deadline, the item value is a date character string such as "August 2, 2021" or "March 5, 2022."

[0018] An "item name" is a name given to an item, and is the name that is written in the document (manuscript). For example, if the item (item attribute) is the invoice amount, item names such as "invoice amount," "total amount," or "total invoice amount" are written, and if the item is a payment due date, item names such as "payment due date," "transfer due date," or "payment due date" are written. In documents where the layout is not fixed, even if the item is the same, the item name and its position may differ depending on the manuscript (publishing company, etc.).

[0019] "Item attributes" are attributes defined for uniformly handling items that represent the same concept but may be assigned multiple different item names, regardless of the item names actually assigned in the document. Item attributes can be arbitrarily named (determined) by the user. For example, for items with names such as "billing amount," "total amount," or "billing total," the item attribute is determined to be "billing amount." For items with names such as "payment due date," "transfer deadline," or "payment due date," the item attribute is determined to be "payment deadline." As mentioned above, item names may differ depending on the manuscript, but item attributes are names (attributes) that can be used commonly across all manuscripts. In this embodiment, "item to be extracted" is synonymous with "item attribute to be extracted."

[0020] "Item keywords" are strings of words (keyword strings) that are written in a document (manuscript) and include item names, and serve as a guide for extracting the desired information (item values).Item keywords may include item names that are directly related to the item values, as well as item names that are less related and word strings other than item names.

[0021] The information processing device 1 is a computer including a central processing unit (CPU) 11, a read-only memory (ROM) 12, a random access memory (RAM) 13, a storage device 14 such as an electrically erasable and programmable read-only memory (EEPROM) or a hard disk drive (HDD), a communication unit 15 such as a network interface card (NIC), an input device 16 such as a keyboard or a touch panel, and an output device 17 such as a display. However, the specific hardware configuration of the information processing device 1 may be omitted, replaced, or added as appropriate depending on the embodiment. Furthermore, the information processing device 1 is not limited to a device consisting of a single housing. The information processing device 1 may be realized by multiple devices using so-called cloud or distributed computing technology, etc.

[0022] The information processing device 1 acquires and stores, from the learning device 2, a trained model and an item keyword list for extracting item values of extraction target items in a document (manuscript) of a predetermined document type, which is the document type from which item values are to be extracted. The information processing device 1 also acquires an image (extraction target image) of the document (manuscript) of the predetermined document type from the document reading device 3A. The information processing device 1 then extracts item values of the extraction target items from the extraction target image by using the trained model and the item keyword list. Examples of the document type (predetermined document type) from which item values are to be extracted include various document types such as invoices, purchase orders, delivery notes, slips, and expense books.

[0023] The document image is not limited to electronic data (image data) such as TIFF (Tagged Image File Format), JPEG (Joint Photographic Experts Group), or PNG (Portable Network Graphics), but may also be electronic data in PDF (Portable Document Format). Therefore, the document image may be electronic data (PDF file) obtained by scanning an original and converting it to PDF, or electronic data originally created as a PDF file.

[0024] The method for acquiring the extraction target image is not limited to the above-described example, and any method may be used, such as acquiring the image via another device, or acquiring the image by reading it from an external recording medium such as a USB (Universal Serial Bus) memory, an SD memory card (Secure Digital memory card), or an optical disk, or from the storage device 14. If the extraction target image is not acquired from the document reading device 3A, the information processing system 9 does not need to be equipped with the document reading device 3A. Similarly, the method for acquiring the trained model and the item keyword list is not limited to the above-described example, and any method may be used.

[0025] The learning device 2 is a computer equipped with a CPU 21, a ROM 22, a RAM 23, a storage device 24, a communication unit 25, etc. However, the specific hardware configuration of the learning device 2 can be omitted, replaced, or added as appropriate depending on the embodiment. Furthermore, the learning device 2 is not limited to a device consisting of a single housing. The learning device 2 may be realized by multiple devices using so-called cloud or distributed computing technology, etc.

[0026] The learning device 2 acquires an image (learning image) of a document of a predetermined document type (e.g., an invoice) from the document reading device 3 B. Then, the learning device 2 performs a learning process using the learning image to generate a trained model and an item keyword list for extracting item values of items to be extracted from documents of the predetermined document type.

[0027] The method of acquiring the learning images is not limited to the above example, and any method may be used, such as acquiring the images via another device, or acquiring the images by reading them from an external recording medium or the storage device 24. If the learning images are not acquired from the document reading device 3B, the information processing system 9 does not need to include the document reading device 3B. Furthermore, in this embodiment, the information processing device 1 and the learning device 2 are shown as separate devices (separate housings), but the present invention is not limited to this example, and the information processing device 9 may include a single device (housing) that performs both the learning process and the document type identification process.

[0028] The document reading devices 3 (3A, 3B) are devices that acquire document images by optically reading paper documents in response to scan instructions from a user, and are exemplified by scanners and multifunction peripherals. The document reading device 3A acquires an extraction target image by reading a form from which a user wants to extract item values (a form from which item values are to be extracted), such as an invoice that requires data entry. The document reading device 3B acquires multiple learning images by reading multiple forms of the same type (documents of a predetermined document type) that have different layouts. The document reading devices 3A and 3B may be the same device (housing). The document reading device 3 is not limited to a device having a function for transmitting images to other devices, and may be an imaging device such as a digital camera or a smartphone. The document reading device 3 does not necessarily have an optical character recognition (OCR) function.

[0029] FIG. 2 is a diagram illustrating an outline of the functional configuration of the learning device according to this embodiment. The learning device 2 functions as a device including an image acquisition unit 51, a recognition result acquisition unit 52, a format definition storage unit 53, an item value candidate extraction unit 54, a correct answer definition acquisition unit 55, an item keyword determination unit 56, a feature generation unit 57, a model generation unit 58, and a storage unit 59, by loading a program stored in a storage unit 24 into a RAM 23 and executing it by a CPU 21, thereby controlling the various hardware components of the learning device 2. Note that in this embodiment and other embodiments described below, the functions of the learning device 2 are executed by the CPU 21, which is a general-purpose processor. However, some or all of these functions may be executed by one or more dedicated processors. Furthermore, the functional units of the learning device 2 are not limited to being implemented in a device (a single device) consisting of a single housing, but may be implemented remotely and / or distributedly (e.g., on the cloud).

[0030] In this embodiment, a trained model and an item keyword list for extracting item values from a target form (form image) are generated based on the relationship between item values and item names in a general form (manuscript). Below, the concept of item value extraction based on the relationship (positional relationship) between item values and item names in a manuscript is explained.

[0031] Typically, the item name corresponding to an item value is written to the left or above the item value. Also, typically, the item name corresponding to an item value is often written near the item value. This is a relationship common to both standard and semi-standard forms. For example, if you want to extract the item value of the invoice amount item (item attribute "invoice amount"), item names corresponding to the item value, such as "total," "invoice amount," "payment amount," or "transfer amount," are written near the left side, and related keywords such as "amount" are written above, and related keywords such as "consumption tax," "subtotal," or "discount amount" are written diagonally.

[0032] Therefore, it is possible to determine the appropriateness of an item value candidate as a target item value (item value of an item to be extracted) based on the positional relationship between the item value candidate and the item keyword (a word sequence around the item value that is assumed to be related to the item value) written to the left or above the item value candidate. In other words, by statistically aggregating and learning what item keywords (word sequences) are written to the left or above the item value (item value candidate), how far away, and in what direction, it is possible to generate a trained model that can determine the appropriateness of the item value of the target item. In other words, by inputting as features what item keywords are written around the item value candidate, in what direction, and how far away they are, it is possible to generate a model that can identify the appropriateness of the item value candidate as an item value of the target item.

[0033] The image acquisition unit 51 acquires multiple learning images (sample images) to be used in the learning process. The image acquisition unit 51 acquires multiple images (image data) of the same type of document but with different layouts as learning images. When documents such as invoices are issued by different companies, the layout of the item positions and item names on the document may differ. Therefore, for example, images of multiple invoices issued by different issuers are used as learning images. For example, when a user instructs the document reading device 3B to read multiple invoices with different layouts, the image acquisition unit 51 acquires scanned images of the invoices as learning images. Note that the document images contain information from the documents as images.

[0034] The number of training images for each layout is arbitrary, and one or more training images can be used for each layout. Using multiple training images for one layout enables more accurate training. For example, if there is an invoice (such as an invoice issued by Company A) that is frequently handled in business, the number of training images can be adjusted according to the frequency (importance) of the layout handled, such as by increasing the number of training images for the layout of that invoice. This makes it possible to perform training tailored to the user's environment.

[0035] The recognition result acquisition unit 52 acquires character recognition results (character string data) for each training image. The recognition result acquisition unit 52 acquires character recognition results for the training image (hereinafter referred to as "full-text OCR results") by scanning the entire training image (entire region) using OCR. The full-text OCR results may have any data structure as long as they include character recognition results for each character string (character string image) in the training image. The method for acquiring the full-text OCR results is not limited to the above example, and any method may be used, such as acquiring the results via another device such as a character recognition device that performs OCR processing, or acquiring the results by reading them from an external recording medium or the storage device 24. In this embodiment, a character string is a string (a sequence of characters) consisting of one or more characters, and characters include hiragana, katakana, kanji, alphabets, numbers, symbols, etc.

[0036] The format definition storage unit 53 stores format definitions for the items to be extracted, which are used when extracting item value candidates. Specifically, in the item value candidate extraction process, character strings that match the format definition for the item to be extracted are extracted as item value candidates for the item to be extracted. Therefore, the format definition defines a string format related to the item to be extracted (the format of a string that can correspond to the item value of the item to be extracted) so that character strings that can be the item value of the item to be extracted are extracted as item value candidates. For example, in the case of a date-related item attribute "payment due date," a format related to "date" is defined as a string format related to "payment due date" so that character strings that can be the item value of "payment due date" are extracted as item value candidates. Furthermore, for example, the format definition for the amount-related item attribute "billing amount" defines a format related to "amount" as a string format related to "billing amount." Specific format definition examples are shown below.

[0037] For example, in the format definition for the item attribute "Payment deadline," the format for "date" is defined as "'￥d{4}[￥ / ￥.￥-]￥d{1,2}[￥ / ￥.￥-]￥d{1,2}'￥'|￥d{4}[year]￥d{1,2}[month]￥d{1,2}[day]'￥'|(JAN(UARY)?|FEB(LUARY)?|MAR(CH)?|APR(IL)?|MAY|JUNE?|JULY?|AUG(UST)?|SEP(TEMBER)?|OCT(OBER)?|NOV(EMBER)?|DEC(EMBER)?|JLY)[￥ / ￥.￥-]?￥d{1,2}(th)?[￥,￥ / ￥.￥-]?(￥d{4}|￥d{2})'" According to this example format definition, dates written in a variety of formats, such as dates written using slashes such as "2020 / 08 / 09", dates written using periods such as "2021.2.17", dates written in kanji such as "2020 July 24th", and dates written in English such as "JAN 23 2020", can be extracted as candidate item values (candidate strings) for the item attribute "Payment deadline".

[0038] As another example, the format definition for the item attribute "Amount invoiced" defines the format for "Amount" as "'\d{0,3}[.,]?\d{0,3}[.,]?\d{1,3}[.,]\d{0,3}'." This format definition example makes it possible to extract strings containing numbers in groups of three digits and separators such as commas or periods as item value candidates for the item attribute "Amount invoiced."

[0039] In this embodiment, a format definition created in advance by a user is exemplified, but the present invention is not limited to this example, and the format definition may be automatically generated based on the correct definition described below. Furthermore, the format definition is not limited to the format definition using the regular expressions described above, and may be defined using expressions other than regular expressions. Furthermore, in the above example, an item attribute and a corresponding format definition are associated with each item attribute to be extracted, but the present invention is not limited to this example, and multiple item attributes may be associated with one format definition. For example, a format (format definition) related to amounts may be associated with the item attribute "billing amount" and the item attribute "unit price."

[0040] The item value candidate extraction unit 54 extracts multiple candidate character strings (item value candidates) that are character strings that can be item values of the extraction target item from the character recognition results of each learning image. The item value candidate extraction unit 54 extracts character strings that match the format definition related to the extraction target item as item value candidates related to the extraction target item.

[0041] 3 and 4 are diagrams illustrating an example of extracted item value candidates according to this embodiment. FIG. 3 illustrates training image 1, which is a training image for an invoice. FIG. 4 illustrates training image 2, which is a training image for an invoice. FIGS. 3 and 4 illustrate a case where the item attribute to be extracted is "billed amount" and the format definition for "billed amount" is the format definition illustrated in the specific format definition. In this case, as indicated by the dashed lines in FIG. 3 , "199," "10," "1,990," "85," "20," "1,700," "3,690," "369," and "4,059" are extracted as character strings that match the format definition for the item attribute "billed amount" (item value candidates for "billed amount"). Similarly, as indicated by the dashed lines in FIG. 4 , "3,290," "1,200," "4,490," "449," and "4,939" are extracted as character strings that match the format definition for the item attribute "billed amount" (item value candidates for "billed amount").

[0042] The correct definition acquisition unit 55 acquires a correct definition in which one or more items to be extracted are associated with the item values of the items to be extracted in each learning image. In this embodiment, the correct definition acquisition unit 55 acquires the correct definition by inputting a correct definition generated (defined) by the user to the learning device 2. For example, the user determines the items to be extracted (item attributes) and extracts the item values of the items to be extracted that are written in each learning image by referring to each learning image. Then, the user generates a correct definition (correct definition table) by storing the items to be extracted and the item values of the items to be extracted in each extracted learning image in an associated manner, and inputs the generated correct definition to the learning device 2.

[0043] Fig. 5 is a diagram showing an example of a correct answer definition table according to this embodiment. In Fig. 5, "billing amount," "payment deadline," and "slip number" are shown as example item attributes to be extracted, but the item attributes are not limited to these and can be set arbitrarily. As shown in Fig. 5, the correct answer definition (correct answer definition table) stores item values (correct answer definition values) corresponding to each item attribute for each of a plurality of learning images (such as "Sheet_001.jpg," "Sheet_002.jpg," and "Sheet_003.jpg").

[0044] For example, "Sheet_001.jpg" is training image 1 (see Figure 3), and "Sheet_002.jpg" is training image 2 (see Figure 4). The item values (item values of the item names corresponding to the item attributes) of the item attributes "billing amount," "payment deadline," and "voucher number" written in the training images of Figures 3 and 4, respectively, are stored in the correct answer definition table shown in Figure 5. For example, the user stores in the correct answer definition table the item value "4,059" of the item name "total" included in training image 1 shown in Figure 3 as the correct answer defined value of the item attribute "billing amount," the item value "2021 / 7 / 25" of the item name "payment deadline" as the correct answer defined value of the item attribute "payment deadline," and the item value "BN0000868022" of the item name "voucher number" as the correct answer defined value of the item attribute "voucher number." In this manner, in this embodiment, the user inputs item values for the items to be extracted that are written in the learning image, and a correct answer definition (correct answer definition table) is generated.

[0045] The data structure for storing the item values (correct answer definition values) is not limited to a table format such as CSV (comma-separated values) format, and may be any format. Also, the method for acquiring the correct answer definition is not limited to the above-mentioned example, and any method may be used, such as a method of acquiring via another device, or a method of acquiring by reading from an external recording medium or the storage device 24.

[0046] The item keyword determination unit 56 determines multiple item keywords to be used to extract item values for items to be extracted. As will be described later, in the information processing device 1, after item value candidates for an item to be extracted are extracted, the validity of each item value candidate is determined based on the positional relationship between each item value candidate and multiple item keywords, and the most valid item value candidate is determined as the item value for the item to be extracted. Therefore, it is desirable that the item keywords be keywords that are useful for extracting item values for items to be extracted.

[0047] On the other hand, since the item names written on invoices and the like may vary (be different) depending on the issuing company, it is desirable to select as many keywords as possible as item keywords in order to handle the diverse manuscripts from each company. However, if keywords completely unrelated to the items to be extracted or irregular keywords are selected as item keywords, there are concerns that this may have a negative impact on the extraction of item values, bloating of the trained model, and a decrease in processing speed. Therefore, in this embodiment, keywords predicted to be useful for extracting item values from word strings written on the form are determined (selected) as item keywords. The method for determining item keywords will be described below. The item keyword determination unit 56 determines multiple item keywords for each item to be extracted.

[0048] First, the item keyword determination unit 56 identifies the position within the training image of the item value (defined correct answer value) of the item attribute to be extracted, which is stored in the correct answer definition. Then, the item keyword determination unit 56 extracts a word string surrounding the defined correct answer value whose position has been identified from the character recognition result of the training image as an item keyword candidate for the item attribute. In this embodiment, the word string is a string (a sequence of words) consisting of one or more words. Furthermore, the word string surrounding the defined correct answer value is a word string located within a predetermined range from the defined correct answer value, and is not limited to a word string close to the defined correct answer value, but may be a word string included in the entire area of the training image. The item keyword determination unit 56 performs this item keyword candidate extraction process for each training image.

[0049] For example, in the case of learning image 1 shown in Figure 3, the position of the correct definition value "4,059" of the item attribute "billing amount" stored in the correct definition is identified, and the word strings around it (e.g., "unit price," "amount," "subtotal," "total," etc.) are extracted as item keyword candidates. Also, for example, in the case of learning image 2 shown in Figure 4, the position of the correct definition value "4,939" of the item attribute "billing amount" stored in the correct definition is identified, and the word strings around it (e.g., "content," "amount," "subtotal," "total," etc.) are extracted as item keyword candidates.

[0050] In this embodiment, the item keyword determination unit 56 generates an item keyword candidate list containing item keyword candidates extracted from each learning image for each extraction target item. For example, first, a list of words (single words) surrounding the correct answer definition value and a list of word strings consisting of multiple words (word strings combining a word with words before and after that word) surrounding the correct answer definition value are generated, and then an item keyword candidate list is generated that stores the word strings contained in these lists. Note that the method for generating the item keyword candidate list is not limited to the example described above, and any method may be used.

[0051] In this way, by treating not only single words but also word strings consisting of multiple words as item keyword candidates (item keywords), even when an item name contains other item names, such as "Total" and "Sub Total," "Date" and "Due Date," or "Invoice Date," it is possible to distinguish and identify each and extract each as an item keyword candidate (item keyword). This makes it possible to prevent the intended item keyword from being confused with another keyword, which would have a negative impact on the extraction of item values.

[0052] The item keyword determination unit 56 determines (selects) item keywords (item keywords for extracting item values of the extraction target items) related to the extraction target items from the item keyword candidates related to the extraction target items extracted from each learning image. In other words, the item keywords related to the extraction target items are determined from the item keyword candidates (item keyword candidate list) related to the extraction target items, each of which is extracted from at least one learning image.

[0053] The item keyword determination unit 56 determines item keywords from the item keyword candidates based on the attributes of the item keyword candidates. The attributes of the item keyword candidates are, for example, at least one of the following: (1) the frequency of occurrence of the word string that is the item keyword candidate in the training image; (2) the distance between the item keyword candidate (area) and the correct answer definition value (area) in the training image; and (3) the direction from one of the correct answer definition value (area) to the other of the item keyword candidate (area) in the training image (e.g., the direction of the correct answer definition value as seen from the item keyword candidate). For example, the item keywords may be determined based on any one of these three attributes, or may be determined based on two or all of the three attributes. By determining the item keywords based on these attributes, it is possible to select keywords that are likely to be related (strongly related) to the item value as item keywords. Below, a method for determining item keywords based on each attribute will be described.

[0054] <Attribute (1) Item Keyword Candidate Occurrence Rate> Keywords that are commonly written in many manuscripts (learning images) are highly versatile and are predicted to be useful (effective) for extracting item values. Therefore, the item keyword determination unit 56 increases the likelihood that a character string that is commonly written in many manuscripts, i.e., an item keyword candidate that appears in many learning images, will be selected as an item keyword.

[0055] <Attribute (2) Distance between item keyword candidate and correct answer definition value> In many cases, an item name and an item value are written as a set, so it is expected that the two will be written close to each other. Therefore, a keyword written close to an item value is likely to be an item name that represents the item of the item value or an item name related to the item value, and is expected to be useful for extracting the item value. Therefore, the item keyword determination unit 56 increases the likelihood that an item keyword candidate with a smaller distance from the correct definition value in the learning image will be selected as an item keyword.

[0056] <Attribute (3) Correct Answer Definition Value and Direction from One to the Other of Item Keyword Candidates> In many cases, item names are written to the left or above the item value, aligned with the item value, so keywords written horizontally to the left or vertically above the item value, aligned with the item value, are likely to be item names that represent the item of that item value or item names related to that item value, and are predicted to be useful for item extraction. Therefore, the item keyword determination unit 56 increases the likelihood that an item keyword candidate that is horizontally to the left or vertically above the correct answer definition value in the learning image will be selected as an item keyword.

[0057] The item keyword determination unit 56 may calculate an effectiveness score for each item keyword candidate based on the attributes of the item keyword candidate, which indicates the effectiveness of the item keyword candidate as a keyword for extracting item values, and determine item keywords based on the effectiveness score. The item keyword determination unit 56 may, for example, select a predetermined number (e.g., 100) of item keyword candidates in descending order of effectiveness score, and determine the selected item keyword candidates as item keywords. Alternatively, by setting a predetermined threshold for the effectiveness score, item keyword candidates exceeding the predetermined threshold may be determined as item keywords.

[0058] The effectiveness score is calculated based on the attributes of the item keyword candidate. For example, in the case of attribute (1), the effectiveness score is calculated so that the more an item keyword candidate appears in more learning images, the higher the effectiveness score. In the case of attribute (2), the effectiveness score is calculated so that the closer the item keyword candidate is to the correct answer definition value, the higher the effectiveness score. In the case of attribute (3), the effectiveness score is calculated so that the more an item keyword candidate is located horizontally to the left or vertically above the correct answer definition value, the higher the effectiveness score.

[0059] The effectiveness score may be calculated based on at least one of the three attributes described above, and the calculation method is arbitrary. Below, as an example of calculating the effectiveness score based on the three attributes described above, a method using weighting (weight based on distance and weight based on direction (angle)) is shown. In this method, the effectiveness score (total effectiveness score) S of the item keyword candidate is calculated using the following formula 1.

[0060]

number

[0061] In the above formula 1, Si is the effectiveness score of training image i, xi is the occurrence count of item value keyword candidates in training image i, w1i is the weight of attribute (2) in training image i, w2i is the weight of attribute (3) in training image i, and N is the number of training images.

[0062] S i is the effectiveness score of training image i. The effectiveness score is the effectiveness score of the item keyword candidate calculated for each training image. The effectiveness score (total effectiveness score) S is calculated by summing the effectiveness scores for all training images.

[0063] xi is the appearance count (value indicating the degree of appearance) of the item keyword candidate in the training image. For example, the number (locations) where the item keyword candidate is detected in the character recognition results of the training image is input to xi. In many cases, a word string that is an item keyword candidate is predicted to appear only once in a single training image, in which case xi = 1. If the target item keyword candidate is not included in the character recognition results of the training image, xi = 0. Note that if the same item keyword candidate is detected multiple times in a single training image, the count number may be used as the number of detections, and the distance weight and direction weight for one of the multiple detections may be used. Alternatively, the effectiveness single score for the training image may be calculated by calculating and adding up the count number (count number (= 1) × distance weight × direction weight) for each detection.

[0064] In this embodiment, the appearance count of an item keyword candidate is the number of times the item keyword candidate is detected in a learning image (the number of detection locations), but this is not limited to this example and may be a numerical value indicating whether or not the item keyword candidate is detected in a learning image. In other words, even if an item keyword candidate is detected multiple times in a single learning image, xi may be set to 1. In this way, by using the appearance count described above, the total effectiveness score is the sum of the individual effectiveness scores for the number of learning images in which the item keyword candidate appears, making it possible to calculate the effectiveness score so that the more an item keyword candidate appears in more learning images, the higher the effectiveness score.

[0065] w1i is the weight of attribute (2) in training image i (hereinafter referred to as the "distance weight"). When the distance between the character string of the item value keyword candidate detected in the training image and the correct answer definition value is small, i.e., when the positions of the two are close, the value (weight) is calculated to be large. For example, when the detected item keyword candidate and the correct answer definition value are separated at the ends of the manuscript (training image), i.e., when the distance between them is equal to the length of the diagonal of the manuscript (the longest distance), the distance weight is set to the minimum value (e.g., 1). On the other hand, when the detected item keyword candidate and the correct answer definition value are adjacent, i.e., when the distance between them is the shortest, the distance weight is set to the maximum value (e.g., 10). Then, when the distance between them is between the minimum distance and the maximum distance, the distance weight is assigned according to the distance between them. For example, the distance weight is calculated to decrease linearly as the distance between them increases.

[0066] w2i is the weight of attribute (3) in training image i (hereinafter referred to as "directional weight"), and is calculated so that the value (weight) is larger when the character string of the item value keyword candidate detected in the training image is horizontally to the left or vertically above the correct answer definition value. Specifically, the directional weight is calculated based on the degree to which the item keyword candidate is horizontally to the left or vertically above the correct answer definition value. For example, the directional weight w2i in training image i is calculated using the following formula 2.

[0067]

number

[0068] In the above formula 2, w2hi is the first weight of attribute (3) in training image i, and w2vi is the second weight of attribute (3) in training image i.

[0069] The first weight w2hi (hereinafter referred to as the "first directional weight") of attribute (3) in training image i is a weight based on the degree to which the item keyword candidate is located horizontally to the left of the defined correct answer value in training image i. The first directional weight is calculated so that the value increases as the character string of the item value keyword candidate detected in the training image is located horizontally to the left of the defined correct answer value. For example, when the item keyword candidate is located horizontally to the left of the defined correct answer value, i.e., when the angle (hereinafter referred to as the "point-to-point angle") between the vector from the item keyword candidate toward the defined correct answer value and the horizontal right direction (x-axis) is 0 degrees, the first directional weight is set to a maximum value (e.g., 10). Then, as the vector tilts, the value of the first directional weight decreases, and the minimum value (e.g., 1) is set when the point-to-point angle is 45 degrees and -45 degrees, and the minimum value (e.g., 1) is also set when the point-to-point angle is outside the range of 0 degrees ± 45 degrees. The angle between two points is positive clockwise (the direction in which the angle increases).

[0070] The second weight w2vi (hereinafter referred to as the "second directional weight") of attribute (3) in training image i is a weight based on the degree to which the item keyword candidate is located vertically above the defined correct answer value in training image i. The second directional weight is calculated so that the value increases the closer the character string of the item value keyword candidate detected in the training image is located vertically above the defined correct answer value. For example, when the item keyword candidate is located vertically above the defined correct answer value, i.e., when the angle between the two points is 90 degrees, the second directional weight is set to a maximum value (e.g., 10). Then, as the vector tilts, the value of the second directional weight decreases, and the minimum value (e.g., 1) is set when the angle between the two points is 45 degrees and 135 degrees, and the minimum value (e.g., 1) is also set when the angle between the two points is outside the range of 90 degrees ± 45 degrees.

[0071] In this embodiment, the directional weight is calculated by calculating the angle between two points related to the correct definition value and the item keyword candidate. However, the present invention is not limited to this. For example, the directional weight may be calculated by determining which quadrant among the first to fourth quadrants the item keyword candidate is in when the correct definition value is set as the origin. For example, if the item keyword candidate is determined to be in the second or third quadrant, the first directional weight may be calculated to be higher. Furthermore, for example, if the item keyword candidate is determined to be in the first or second quadrant, the second directional weight may be calculated to be higher.

[0072] Fig. 6 is a diagram showing an example of calculating directional weights according to this embodiment. In Fig. 6, the horizontal axis represents the angle between two points, and the vertical axis represents the directional weights. In Fig. 6, the minimum value of the directional weights (first directional weight, second directional weight) is 1, and the maximum value is 10.

[0073] When the inter-point angle is between 0 and 45 degrees, as described above, the first directional weight is greater than the second directional weight, so w2i = w2hi. As a result, as shown in FIG. 6, in this angle range, the directional weight decreases linearly from a maximum value of 10 to a minimum value of 1. Furthermore, when the inter-point angle is between 45 and 90 degrees, as described above, the second directional weight is greater than the first directional weight, so w2i = w2vi. As a result, as shown in FIG. 6, in this angle range, the directional weight increases linearly from a minimum value of 1 to a maximum value of 10. Furthermore, when the inter-point angle is between 90 and 135 degrees, as described above, the second directional weight is greater than the first directional weight, so w2i = w2vi. As a result, as shown in FIG. 6, in this angle range, the directional weight decreases linearly from a maximum value of 10 to a minimum value of 1. Furthermore, when the angle between the two points is between 135 degrees and 315 degrees, as described above, the first directional weight and the second directional weight have a minimum value of 1, so w2i = w2hi = w2vi = minimum value 1. Furthermore, when the angle between the two points is between 315 degrees and 360 degrees, as described above, the first directional weight is greater than the second directional weight, so w2i = w2hi. As a result, as shown in Figure 6, in this angle range, the directional weight increases linearly from a minimum value of 1 to a maximum value of 10.

[0074] The minimum and maximum values of the distance weight and the directional weight can be adjusted (set) to any numerical value. The range of the two-point angle between the maximum and minimum values of the directional weight is not limited to ±45 degrees, but can be adjusted to any angle (range). In this embodiment, the two-point angle related to the correct definition value and the item keyword candidate is the angle formed by the vector pointing from the item keyword candidate to the correct definition value with the horizontal right direction, but is not limited to the angle with the horizontal right direction as long as it indicates the direction of the vector. The two-point angle may also be the angle formed by the vector pointing from the correct definition value to the item keyword candidate with the horizontal right direction.

[0075] Furthermore, to calculate the distance and angle between two points between the correct definition value and the item keyword candidate, any point within the area related to the correct definition value and the area related to the item keyword candidate within the training image may be used. For example, the upper left vertex of the circumscribing rectangle for each of the correct definition value and the item keyword candidate within the training image may be used. Specifically, the distance and angle between the two points may be calculated (extracted) using a vector pointing from the upper left vertex of the circumscribing rectangle of the item keyword candidate within the training image to the upper left vertex of the circumscribing rectangle of the correct definition value within the training image. Below are examples of calculating distance weights and directional weights for training image 1 and training image 2.

[0076] FIG. 7 is a diagram illustrating an example of distance weights and directional weights calculated for a training image 1 according to this embodiment. FIG. 7 illustrates example values of distance weights and directional weights for each item keyword candidate (e.g., "unit price," "amount," "subtotal," and "total") for the item attribute "bill amount" in the training image 1. As shown in FIG. 7, for example, the distances between each item keyword candidate "unit price," "amount," "subtotal," and "total" in the training image 1 and the correct answer definition value "4,059" for the item attribute "bill amount" are calculated as "88 mm," "72 mm," "36 mm," and "29 mm," respectively. Also, as shown in FIG. 7, for example, the angles (point-to-point angles) of vectors from each item keyword candidate "unit price," "amount," "subtotal," and "total" in the training image 1 toward the correct answer definition value "4,059" are calculated as "50 degrees," "90 degrees," "40 degrees," and "0 degrees," respectively.

[0077] Then, based on the above-mentioned distance weight and directional weight calculation method, the calculated distance and angle between the two points are converted into a distance weight and a directional weight, respectively. As shown in Figure 7, for example, the distance weights of each item keyword candidate "unit price", "amount", "subtotal", and "total" are calculated as "5", "6", "8", and "9". Also, the directional weights of each item keyword candidate "unit price", "amount", "subtotal", and "total" are calculated as "1", "10", "1", and "10".

[0078] FIG. 8 is a diagram illustrating an example of distance weights and directional weights calculated for a training image 2 according to this embodiment. FIG. 8 illustrates example values of distance weights and directional weights for each item keyword candidate (e.g., "contents," "amount," "subtotal," and "total") for the item attribute "bill amount" in the training image 2. As shown in FIG. 8, for example, the distances between each item keyword candidate "contents," "amount," "subtotal," and "total" in the training image 2 and the correct answer definition value "4,939" for the item attribute "bill amount" are calculated as "89 mm," "78 mm," "39 mm," and "30 mm," respectively. Also, as shown in FIG. 8, for example, the angles (point-to-point angles) of vectors from each item keyword candidate "contents," "amount," "subtotal," and "total" in the training image 2 toward the correct answer definition value "4,939" are calculated as "61 degrees," "90 degrees," "45 degrees," and "0 degrees," respectively.

[0079] Then, based on the above-mentioned distance weight and directional weight calculation method, the calculated distance and angle between the two points are converted into distance weight and directional weight, respectively. As shown in Figure 8, for example, the distance weights of each item keyword candidate "content," "price," "subtotal," and "total" are calculated as "5," "7," "8," and "9." Also, the directional weights of each item keyword candidate "content," "price," "subtotal," and "total" are calculated as "4," "10," "1," and "10."

[0080] Fig. 9 is a diagram showing an example of calculating the effectiveness score according to this embodiment. Fig. 9 shows an example of the numerical value of the effectiveness score of each item keyword candidate related to the item attribute "billing amount", calculated based on the distance weight and directional weight for learning image 1 and learning image 2 calculated in Fig. 7 and Fig. 8. As shown in Fig. 9, for each item keyword candidate, a distance weight and a directional weight are calculated for each learning image in which the item keyword candidate is detected, and the effectiveness scores (single effectiveness score, total effectiveness score) are calculated based on these calculated weights.

[0081] For example, as shown in Figures 7 and 8, the item keyword candidate "amount" is detected in each of training images 1 and 2, and the distance weight and directional weight are calculated for each. For example, since the distance weight for the item keyword candidate "amount" detected in training image 1 is "6" and the directional weight is "10," the effectiveness single score is calculated as effectiveness single score = 1 x 6 x 10 = 60 based on Formula 1. Similarly, since the distance weight for the item keyword candidate "amount" detected in training image 2 is "7" and the directional weight is "10," the effectiveness single score is calculated as effectiveness single score = 1 x 7 x 10 = 70 based on Formula 1.

[0082] 9, by summing the single effectiveness scores for the item keyword candidate "price" for each of learning images 1 to N, the total effectiveness score for the item keyword candidate "price" of the item attribute "billing amount" is calculated as "53,320." Similarly, for example, the total effectiveness scores for the item keyword candidates "total" and "content" related to the item attribute "billing amount" are calculated as "90,870" and "2,245," respectively. Note that the item keyword determination unit 56 may generate a table such as that shown in FIG. 9 (a table storing learning images, distance weights, direction weights, and effectiveness scores (single scores, total scores)) as the calculation results of the single effectiveness scores and total effectiveness scores.

[0083] Based on the effectiveness score (total effectiveness score) calculated by the effectiveness score calculation method or the like, the item keyword determination unit 56 determines multiple item keywords from the item keyword candidates. Furthermore, in this embodiment, the item keyword determination unit 56 generates an item keyword list including the determined item keywords for each item to be extracted. The generated item keyword list is stored in the storage unit 59. Note that the data structure for storing the item keywords is not limited to a list format, and may be any other format.

[0084] In conventional item value extraction methods for semi-structured forms, item value extraction rules are created based on the correspondence between item names and item values, but the item names (keywords) corresponding to the item values to be extracted are determined by skilled engineers or the like by observing the semi-structured form. However, as described above, according to this embodiment, the item keywords for extracting (identifying) item values can be automatically determined by the item keyword determination unit 56. This eliminates the need for the user to manually determine the item keywords, thereby reducing the user's workload.

[0085] The feature generation unit 57 generates feature quantities for each item value candidate related to the item to be extracted in the training image. The feature generation unit 57 generates feature quantities for the item value candidates based on the positional relationship between a plurality of item keywords related to the item to be extracted in the training image and the item value candidates. The feature generation unit 57 generates feature quantities for the item value candidates in each training image. In the training process described below, the feature quantities for each item value candidate are used as feature quantities (input for the trained model) for extracting item values.

[0086] The feature generation unit 57 generates features of item value candidates based on information indicating the positional relationship between multiple item keywords related to the items to be extracted in the training image and the item value candidates (hereinafter referred to as "positional relationship information"). The positional relationship information uses information indicating the distance between the item value candidate and the item keyword, and information indicating the direction from one of the item value candidate and the item keyword to the other. In this embodiment, the positional relationship information uses the distance (mm) between the item value candidate and the item keyword in the training image and the angle (point-to-point angle) (deg) of the vector from the item keyword in the training image to the item value candidate.

[0087] 10 is a diagram showing an example of extracted positional relationship information for the training image 1 according to this embodiment. As shown in Fig. 10, in the case of the item value candidate "3,690" for the item attribute "billing amount" in the training image 1, the length and direction of the vector (the distance and direction (angle between two points) between the two) pointing from each item keyword ("total," "subtotal," "amount," etc.) related to the item attribute "billing amount" to the item value candidate "3,690" are extracted as positional relationship information.

[0088] Note that the angle between two points relating to the item value candidate and the item keyword is not limited to an angle relative to the horizontal right direction, as with the angle between two points relating to the correct answer definition value and the item keyword candidate described above, and may also be the angle relative to a vector pointing from the item value candidate to the item keyword. Furthermore, any point within the area relating to the item value candidate and the area relating to the item keyword within the learning image may be used to calculate the distance and angle between two points between the item value candidate and the item keyword. The feature generation unit 57 generates a positional relationship information list that stores positional relationship information for each item to be extracted.

[0089] FIG. 11 is a diagram showing an example of a positional relationship information list during the learning process according to this embodiment. FIG. 11 shows the positional relationship information list for the item attribute "billing amount." As shown in FIG. 11, the positional relationship information list stores positional relationship information (distance and angle between two points) between each item value candidate for the item attribute "billing amount" extracted from each learning image and multiple item keywords related to the item attribute "billing amount." As shown in FIG. 11, for example, information indicating the positional relationship between item value candidates for the item attribute "billing amount" (such as "3,690," "4,059," and "1,990" (see FIG. 3)) in learning image 1 and multiple item keywords related to the item attribute "billing amount" (such as "total," "subtotal," "amount," and "billing amount") is stored.

[0090] The units of the positional relationship information (distance and angle between two points) are not limited to the units (mm, deg) shown in FIG. 11. Furthermore, since the item keyword "invoice amount" in the positional relationship information list in FIG. 11 is a keyword not included in learning image 1, the distance and angle between the item keyword "invoice amount" are set to the values "999" and "- (hyphen)" respectively, indicating no match, but the value indicating no match is not limited to these values, and any numeric value, character, symbol, etc. may be used. Furthermore, in this embodiment, a list format (table format) is exemplified as the data structure for storing positional relationship information, but the data structure may be any other format.

[0091] The feature generation unit 57 then generates feature quantities (distance feature quantities and directional feature quantities) for the item value candidate based on the extracted positional relationship information (the distance between them and the angle between two points). In this embodiment, the distance and the angle between two points are converted into distance feature quantities and directional feature quantities, respectively, depending on the likelihood that the item value candidate and the item keyword are related (associated) (the item keyword is a likely keyword related to the item value candidate). This makes it possible to learn feature quantities according to the strength of the relationship between the item value candidate and the item keyword, thereby enabling more accurate item value extraction.

[0092] <Generation of distance features> The distance feature is a feature based on information indicating the distance between an item value candidate and an item keyword. As described above, in many cases, an item name and an item value are written as a set, and therefore, it is predicted that the two will be written close to each other. Therefore, it is predicted that the smaller the distance between an item value candidate and an item keyword, the more likely the item value candidate and the item keyword are to be related. Therefore, the feature generation unit 57 generates (calculates) the distance feature so that its value increases or decreases according to the distance between them. In this embodiment, the value of the distance feature is calculated so that it increases as the distance between them decreases. For example, the distance feature has a maximum value of 100 points when the two are close to each other, decreases as the distance between them increases, and is set to a minimum value of 0 points when the two are located at opposite ends of the document.

[0093] <Conversion to directional features> The directional feature is a feature based on information indicating the direction from one of the item value candidate and the item keyword to the other. As described above, in many cases, the item name is written to the left or above the item value, aligned with the item value. Therefore, it is predicted that the item value candidate and the item keyword are more likely to be related when the item keyword is located horizontally to the left of the item value candidate or vertically above it. Therefore, the feature generation unit 57 generates (calculates) the directional feature so that its value increases or decreases depending on the degree to which the item keyword is located horizontally to the left and vertically above the item value candidate. In this embodiment, the directional feature is divided into two feature: a horizontal feature that increases or decreases depending on the degree to which the item keyword is located horizontally to the left of the item value candidate, and a vertical feature that increases or decreases depending on the degree to which the item keyword is located vertically above it.

[0094] In this embodiment, the horizontal feature value is calculated so that it increases as the item keyword is located horizontally to the left of the item value candidate. Similarly, the vertical feature value is calculated so that it increases as the item keyword is located vertically above the item value candidate. For example, the horizontal feature value is set to a maximum value of 100 points when the angle between the two points is 0 degrees (when the item keyword is located horizontally to the left of the item value candidate), and the value decreases as the vector between the two points tilts, reaching a minimum value of 0 points when the angle between the two points is 0 degrees ± 90 degrees. Note that the horizontal feature value is also set to a minimum value of 0 points when the angle between the two points is outside the range of 0 degrees ± 90 degrees. Similarly, the vertical feature value is set to a maximum value of 100 points when the angle between the two points is 90 degrees (when the item keyword is located vertically above the item value candidate), and the value decreases as the vector between the two points tilts, reaching a minimum value of 0 points when the angle between the two points is 90 degrees ± 90 degrees. Note that the vertical feature value is also set to a minimum value of 0 points when the angle between the two points is outside the range of 90 degrees ± 90 degrees. The maximum and minimum values of the feature amount can be adjusted (set) to any numerical values.

[0095] The feature generating unit 57 generates a feature list that stores feature amounts (distance feature amounts and direction feature amounts) of item value candidates based on the positional relationship information.

[0096] Fig. 12 is a diagram showing an example of a feature list during the learning process according to this embodiment. As shown in Fig. 12, the distance feature value is calculated to be larger as the distance between the two becomes smaller. Also, the horizontal feature value is calculated to be larger as the item keyword is located horizontally to the left of the item value candidate, and the vertical feature value is calculated to be larger as the item keyword is located vertically above the item value candidate.

[0097] In this embodiment, an example is shown in which the feature value is calculated to be large (high points) when the item value candidate and the item keyword are likely to be related, but the present invention is not limited to this example, and the feature value may be calculated to be small when the relationship is likely to be high. Furthermore, the feature value of the item value candidate that is input to the trained model may be the positional relationship information itself.

[0098] The model generation unit 58 performs machine learning (supervised learning) to generate a trained model for extracting item values of extraction target items for each extraction target item of a predetermined document type. The machine learning uses training data (a dataset of feature amounts and correct labels (teacher data)) in which feature amounts of candidate item values for each training image are associated with information (correct label) indicating whether the candidate item value is the item value (correct defined value) of the extraction target item.

[0099] Information indicating whether an item value candidate is a correct answer definition value (correct answer label) is information based on the correct answer definition acquired by the correct answer definition acquisition unit 55. For example, the item value candidate "4,059" for the item attribute "billing amount" in training image 1 shown in FIG. 12 matches the correct answer definition value for the item attribute "billing amount" in training image 1 in the correct answer definition shown in FIG. 5, so the correct answer label for the item value candidate is determined to be "correct (e.g., label 1)." Also, for example, the item value candidate "3,690" for the item attribute "billing amount" in training image 2 shown in FIG. 12 does not match the correct answer definition value for the item attribute "billing amount" in training image 1 in the correct answer definition shown in FIG. 5, so the correct answer label for the item value candidate is determined to be "incorrect (e.g., label 0)." By performing machine learning using the above-mentioned training data, it is possible to learn the feature quantities (positional relationships) of item values.

[0100] As a result, by inputting the feature amount of a character string in an image (the feature amount based on the positional relationship in the image between the character string and multiple item keywords related to the item to be extracted), it is possible to generate a classifier that can determine whether the character string is an item value of the item to be extracted. More specifically, by inputting the feature amount of a character string, it is possible to generate a classifier (trained model) that can output information indicating the validity of the character string being an item value of the item to be extracted. Note that the information indicating the validity of the character string being an item value of the item to be extracted is information (e.g., a label) indicating whether the character string is an item value of the item to be extracted and / or information (e.g., reliability, probability) indicating the likelihood that the character string is an item value of the item to be extracted. The generated trained model is stored in the storage unit 59.

[0101] Note that a classification learning model is used as the trained model, but the trained model may be any model, such as a discriminative model or a generative model. Furthermore, any machine learning method may be used, such as random forest, naive Bayes, decision tree, logistic regression, neural network, or the like. Furthermore, in this embodiment, training data is used in which, for each item value candidate, feature values based on the positional relationship between the item value candidate and multiple item keywords are associated with information indicating whether the item value candidate is an item value of an item to be extracted. However, this is not a limitation. For example, training data may be used in which, for each character string that is an item value (correct answer definition value) of an item to be extracted and other character strings included in each training image, feature values based on the positional relationship between the character string and multiple item keywords are associated with information indicating whether the character string is an item value of an item to be extracted.

[0102] The storage unit 59 stores, for each extraction target item, the item keyword list generated by the item keyword determination unit 56 and the trained model for extracting item values of item attributes generated by the model generation unit 58. The storage unit 58 may store the item keyword list and the trained model in association with each extraction target item.

[0103] FIG. 13 is a diagram illustrating an outline of the functional configuration of an information processing device according to this embodiment. A program stored in a storage device 14 is loaded into a RAM 13 and executed by a CPU 11, which controls each piece of hardware included in the information processing device 1. As a result, the information processing device 1 functions as a device including an image acquisition unit 41, a recognition result acquisition unit 42, a model storage unit 43, an item keyword list storage unit 44, a format definition storage unit 45, an item value candidate extraction unit 46, a feature generation unit 47, and an item value extraction unit 48. In this embodiment and other embodiments described below, each function included in the information processing device 1 is executed by the CPU 11, which is a general-purpose processor. However, some or all of these functions may be executed by one or more dedicated processors. Furthermore, each functional unit included in the information processing device 1 is not limited to being implemented in a device (the "device") consisting of a single housing, but may be implemented remotely and / or distributedly (for example, on the cloud).

[0104] The image acquisition unit 41 acquires a form image (hereinafter referred to as an "extraction target image") from which item values are to be extracted in the item value extraction process. In this embodiment, for example, when an original (document) to be extracted is read by the document reading device 3A in response to a scan instruction from a user, the image acquisition unit 41 acquires the scanned image that is the reading result as the extraction target image.

[0105] The recognition result acquisition unit 42 acquires the character recognition result (full text OCR result) for the extraction target image. Note that the processing in the recognition result acquisition unit 42 is similar to the processing in the recognition result acquisition unit 52, and therefore detailed description thereof will be omitted.

[0106] The model storage unit 43 stores a trained model generated by the learning device 2 for extracting item values of extraction target items for a predetermined document type. The model storage unit 54 stores a trained model for each extraction target item. Details of the trained model have been described in the explanation of the functional configuration of the learning device 2 (model generation unit 58), and therefore will not be described here.

[0107] The item keyword list storage unit 44 stores an item keyword list generated by the learning device 2 for extracting item values of extraction target items for a specified document type. The item keyword list storage unit 44 stores an item keyword list for each extraction target item. Details of the item keyword list have been described in the explanation of the functional configuration of the learning device 2 (item keyword determination unit 56), so further explanation will be omitted.

[0108] The format definition storage unit 45 stores format definitions related to the items to be extracted, which are used in the item value candidate extraction process. Details of the format definitions have been described in the explanation of the functional configuration (format definition storage unit 53) of the learning device 2, so explanation will be omitted here. Note that the format definitions stored by the format definition storage unit 45 are not limited to being the same as the format definitions stored in the format definition storage unit 53, and may be different format definitions from the format definitions stored in the format definition storage unit 53, as long as they define the character string format related to the items to be extracted.

[0109] The item value candidate extraction unit 46 extracts candidate character strings (item value candidates), which are character strings that can become item values of the extraction target items, from the character recognition results of the extraction target image. The item value candidate extraction unit 46 extracts character strings that match the format definitions related to the item attributes stored in the format definition storage unit 45 from the character recognition results acquired by the recognition result acquisition unit 42, as item value candidates for the item attributes. Note that the method of extracting item value candidates by the item value candidate extraction unit 46 is the same as the method described in the functional configuration of the learning device 2 (item value candidate extraction unit 54), and therefore a detailed description thereof will be omitted.

[0110] The feature generation unit 47 generates feature quantities for each item value candidate related to the extraction target item in the extraction target image. The feature generation unit 47 generates feature quantities for the item value candidates based on the positional relationship between a plurality of item keywords related to the extraction target item stored in the item keyword list storage unit 44 and the value candidates related to the extraction target item extracted by the item value candidate extraction unit 46. The method for generating feature quantities by the feature generation unit 47 is the same as the method described in the functional configuration (feature generation unit 57) of the learning device 2, and therefore detailed description thereof will be omitted.

[0111] The feature generation unit 47 also generates a positional relationship information list and a feature list for the extraction target image, but these lists are similar to the positional relationship information list and feature list (FIGS. 10 and 11) generated by the feature generation unit 57, and therefore detailed description thereof will be omitted. However, while the lists generated by the feature generation unit 57 (FIGS. 10 and 11) target item value candidates in each learning image, the feature list generated by the feature generation unit 47 is different and targets only item value candidates in the extraction target image.

[0112] The item value extraction unit 48 uses a trained model to extract (determine) an item value candidate that is likely to be the item value of an extraction target item from multiple item value candidates related to the extraction target item in the extraction target image. The item value extraction unit 48 inputs the feature amounts (distance feature amounts and directional feature amounts) of each item value candidate related to the extraction target item into the trained model related to the extraction target item, thereby determining whether each item value candidate is appropriate as the item value of the extraction target item. The item value extraction unit 48 outputs the determination result (extracted item value candidates). As described above, when the feature amounts of a character string are input into the trained model, information (label and / or probability) indicating the appropriateness of the character string as the item value of the extraction target item is output from the trained model. In this embodiment, the item value extraction unit 48 inputs the features of each item value candidate into the trained model to obtain information indicating whether each item value candidate is an item value of the item to be extracted (a label (e.g., label "1" if it is an item value of the item to be extracted, label "0" if it is not)) and information indicating the likelihood that each item value candidate is an item value of the item to be extracted (reliability, probability, etc.).

[0113] It should be noted that it can be determined that an item is an item value of an item to be extracted, for example, when the probability that it is an item value of an item to be extracted exceeds the probability that it is not an item value of an item to be extracted, or when it exceeds a predetermined threshold, etc. Therefore, the item value extraction unit 48 may obtain only the probability that it is an item value of an item to be extracted from the trained model, and determine whether or not it is an item value of an item to be extracted based on the obtained probability.

[0114] Furthermore, the item value extraction unit 48 calculates a validity score indicating the likelihood that the item value candidate is the item value of the extraction target item, based on information (reliability, probability, etc.) output from the trained model indicating the likelihood that the item value candidate is the item value of the extraction target item. Note that the validity score may be the information (probability, etc.) indicating the likelihood output from the trained model itself, or may be a numerical value (score) calculated based on the information (probability, etc.) indicating the likelihood. Below, an item value extraction method using the validity score will be described.

[0115] When there is one item value candidate determined to be valid as the item value of the extraction target item, the item value extraction unit 48 determines that item value candidate as the item value of the extraction target item. On the other hand, when there are multiple item value candidates determined to be valid as the item value of the extraction target item, the item value extraction unit 48 determines the item value candidate with the highest validity score among the multiple item value candidates as the most likely item value candidate as the item value of the extraction target item, and determines that item value candidate as the item value of the extraction target item. Note that the item value candidate with the highest validity score may be determined by comparing the validity scores of all item value candidates.

[0116] FIG. 14 is a diagram illustrating an example of a positional relationship information list and validity scores for an extraction target image according to this embodiment. FIG. 14 illustrates an example of a positional relationship information list relating to the positional relationships between multiple item value candidates (e.g., "4,000," "4,400," "1,800") related to the item attribute "billing amount" extracted from the extraction target image and multiple item keywords (e.g., "total," "subtotal," "amount," "billing amount") related to the item attribute "billing amount." When feature values based on the positional relationship information shown in FIG. 14 are input into a trained model, the trained model outputs a reliability score (validity score) for each item value candidate. As shown in FIG. 14, when the feature values of the item value candidates "4,000," "4,400," and "1,800" are input into the trained model, the validity scores for each item value candidate are calculated as "34," "97," and "13." The highest validity score, "4,400," is determined as the item value for the item attribute "billing amount."

[0117] <Processing flow> Next, the flow of the learning process executed by the learning device 2 according to this embodiment will be described. Note that the specific content and processing order of the processes described below are an example for implementing the present disclosure. The specific content and processing order may be selected as appropriate depending on the embodiment of the present disclosure.

[0118] 15 is a flowchart showing an outline of the flow of the learning process according to this embodiment. The process shown in this flowchart is executed when an instruction to scan a form (document) is received by the learning device 2, or when other triggers are received. Note that this flowchart may also be executed when an instruction from a user to acquire a form image stored in the storage device 24 is received, or when other triggers are received.

[0119] In step S101, a plurality of document images (learning images) are acquired. The image acquisition unit 51 acquires scanned images of documents (originals) of a predetermined document type (e.g., invoices) with different layouts. Then, the process proceeds to step S102.

[0120] In step S102, a correct definition is acquired. The correct definition acquisition unit 55 acquires a correct definition in which an item attribute (e.g., a billing amount) to be extracted for a predetermined document type (e.g., an invoice) is associated with a correct definition value of the item attribute in each learning image. Then, the process proceeds to step S103.

[0121] In step S103, a character recognition result (full-text OCR result) is acquired. The recognition result acquisition unit 52 performs character recognition on each learning image acquired in step S101, thereby acquiring a character recognition result for each learning image. Note that steps S102 and S103 are performed in no particular order. Also, steps S101 and S102 are performed in no particular order. Thereafter, the process proceeds to step S104.

[0122] In step S104, an item keyword determination process is performed. In the item keyword determination process, a plurality of item keywords are determined for extracting an item value related to one item attribute (e.g., "billing amount") from among the item attributes to be extracted. Details of the item keyword determination process will be described later with reference to FIG. 16. Then, the process proceeds to step S105.

[0123] In step S105, a trained model generation process is performed. In the trained model generation process, a trained model is generated for extracting an item value of one item attribute (e.g., "billing amount") from among the item attributes to be extracted. Details of the trained model generation process will be described later with reference to FIG. 17. Then, the process proceeds to step S106.

[0124] In step S106, it is determined whether the item keyword determination process (step S104) and the trained model generation process (step S105) have been executed for all extraction target items. The CPU 21 determines whether an item keyword list and a trained model have been generated for each extraction target item. Note that all extraction target items can be confirmed (recognized) by referring to the correct answer definition. If not all extraction target items have been processed (NO in step S106), the process returns to step S104, and the item keyword determination process and trained model generation process are executed for extraction target items that have not been processed (for example, the item attribute "payment deadline"). On the other hand, if all extraction target items have been processed (YES in step S106), the process shown in this flowchart ends.

[0125] Fig. 16 is a flowchart showing an outline of the flow of the item keyword determination process according to this embodiment. The process shown in this flowchart is executed when the process of step S103 in Fig. 15 is completed. Note that this flowchart illustrates an example of the process when the item attribute to be extracted is "billing amount." Note that the item keyword determination process corresponds to an analysis process that analyzes learning images.

[0126] In step S1041, the position of the correct answer definition value of the item to be extracted is identified in one of all the learning images. The item keyword determination unit 56 identifies, for example, the position in learning image 1 of the correct answer definition value "4,059" of the item attribute "billing amount" of learning image 1 in the correct answer definition (see FIG. 5). Thereafter, the process proceeds to step S1042.

[0127] In step S1042, in one of all the training images, a string of words around the defined correct value of the item to be extracted is extracted as a candidate item keyword for that item to be extracted. The item keyword determination unit 56 extracts candidate item keywords from the character recognition results of the training images. For example, the recognized character string for the character string image around the defined correct value "4,059" in training image 1, whose position was identified in step S1041, is extracted as a candidate item keyword for the item attribute "billing amount" (see FIG. 7). Then, the process proceeds to step S1043.

[0128] In step S1043, it is determined whether or not item keyword candidates related to the item attribute "billing amount" have been extracted for all learning images (whether the processes of steps S1041 and S1042 have been executed). CPU 21 determines whether or not item keyword candidates related to the item attribute "billing amount" have been extracted for each of all learning images. If processing has not been completed for all learning images (NO in step S1043), the process returns to step S1041, and processing is performed for a learning image that has not been processed (for example, learning image 2). On the other hand, if processing has been completed for all learning images (YES in step S1043), the process proceeds to step S1044.

[0129] In step S1044, item keywords for the item attribute "billing amount" are determined (generation of an item keyword list). The item keyword determination unit 56 selects multiple item keywords related to the item attribute "billing amount" from the item keyword candidates related to the item attribute "billing amount" for each learning image extracted in step S1042, and generates an item keyword list. The storage unit 59 then stores the generated item keyword list. Thereafter, the processing shown in this flowchart ends.

[0130] Figure 17 is a flowchart showing an outline of the flow of the trained model generation process according to this embodiment. The process shown in this flowchart is executed when the process of step S104 in Figure 15 (the process of Figure 16) ends. Note that this flowchart also illustrates an example of the process when the item attribute is "billing amount."

[0131] In step S1051, item value candidates for an item to be extracted are extracted from the character recognition result for one of all the learning images. The item value candidate extraction unit 54 extracts item value candidates for the item to be extracted using the format definition for the item to be extracted stored in the format definition storage unit 53. For example, the item value candidate extraction unit 54 extracts a word string that matches the format definition for the item attribute "billing amount" from the character recognition result for learning image 1 as an item value candidate for the item attribute "billing amount" in learning image 1. Then, the process proceeds to step S1052.

[0132] In step S1052, the position (location) of the item keyword related to the item to be extracted is identified in one of all the training images. For example, the feature generation unit 57 searches the character recognition result of training image 1 for a word string that matches an item keyword in the item keyword list related to the item attribute "billing amount," and identifies the position in training image 1 where the matching word string (item keyword) is written. Then, the process proceeds to step S1053.

[0133] In step S1053, for one training image out of all training images, feature quantities of item value candidates are generated based on the positional relationships between the item value candidates and multiple item keywords within the training image. The feature generation unit 57 generates feature quantities for each item value candidate extracted in step S1051 using the positions of the item keywords identified in step S1052. For example, the feature generation unit 57 generates feature quantities for each item value candidate for the item attribute "billing amount" in training image 1 based on the positional relationships between the item value candidate and multiple item keywords related to the item attribute "billing amount". Then, the process proceeds to step S1054.

[0134] In step S1054, it is determined whether feature quantities of item value candidates have been generated for all training images (whether the processes of steps S1051 to S1053 have been executed). CPU 21 determines whether feature quantities of each item value candidate related to the item attribute "billing amount" have been generated for each of all training images. If processing has not been completed for all training images (NO in step S1054), the process returns to step S1051, and processing is performed for training images that have not been processed (for example, training image 2). On the other hand, if processing has been completed for all training images (YES in step S1054), the process proceeds to step S1055.

[0135] In step S1055, a trained model for the item to be extracted is generated using the feature quantities and the correct definition (information indicating whether the item value candidate is the correct defined value). The model generation unit 58 generates a trained model for the item attribute "billing amount" using training data in which the feature quantities of the item value candidate for the item attribute "billing amount" generated in step S1053 are associated with information indicating whether the item value candidate is the correct defined value. The storage unit 59 then stores the generated trained model. The processing shown in this flowchart then ends.

[0136] As described above, it is possible to automatically generate a trained model and an item keyword list simply by using an image of a document of a specified document type, such as a semi-standard form, and a correct answer definition corresponding to that image.

[0137] 18 is a flowchart showing an outline of the flow of extraction processing according to this embodiment. The processing shown in this flowchart is executed when the information processing device 1 receives an instruction to scan a form (document), or when other such triggers are received. Note that this flowchart may also be executed when a user receives an instruction to acquire a form image stored in the storage device 14, or when other such triggers are received. Note that this flowchart also illustrates an example of processing when the item attribute is "billing amount."

[0138] In step S201, a document image (an image to be extracted) is acquired. The image acquisition unit 41 acquires a scanned image of a document (original) of a predetermined document type (for example, an invoice). Then, the process proceeds to step S202.

[0139] In step S202, the character recognition result (full-text OCR result) is acquired. The recognition result acquisition unit 42 performs character recognition on the extraction target image acquired in step S201 to acquire the character recognition result (full-text OCR result) for the extraction target image. Then, the process proceeds to step S203.

[0140] In step S203, item value candidates for the extraction target item are extracted from the character recognition results of the extraction target image. The item value candidate extraction unit 46 extracts word strings that match the format definition for the item attribute "billing amount" stored in the format definition storage unit 45 as item value candidates for the item attribute "billing amount". Then, the process proceeds to step S204.

[0141] In step S204, the position (location) of the item keyword related to the extraction target item in the extraction target image is identified. The feature generation unit 47 searches the character recognition result of the extraction target image for a word string that matches the item keyword in the item keyword list related to the item attribute "billing amount" and identifies the position in the extraction target image where the matching word string (item keyword) is written. Then, the process proceeds to step S205.

[0142] In step S205, feature quantities of the item value candidates are generated based on the positional relationship between the item value candidates and multiple item keywords within the extraction target image. The feature generation unit 47 generates feature quantities of each item value candidate related to the item attribute "billing amount" extracted in step S203 using the positions of the item keywords identified in step S204. Then, the process proceeds to step S206.

[0143] In step S206, the validity of each item value candidate for the item to be extracted is determined using the feature values of each item value candidate for the item to be extracted and the trained model. The item value extraction unit 48 inputs the feature values of each item value candidate for the item attribute "billing amount" generated in step S205 into the trained model for the item attribute "billing amount" stored in the model storage unit 43, thereby determining whether each item value candidate is valid as an item value for the item attribute "billing amount." In addition, the item value extraction unit 48 uses the trained model to calculate a validity score for each item value candidate, indicating the likelihood that the item value candidate is the item value in question. Then, the process proceeds to step S207.

[0144] In step S207, based on the validity score, an item value candidate that is likely to be the item value of the item to be extracted is selected (extracted). If there is one item value candidate that is determined to be valid as the item value in step S206, the item value extraction unit 48 determines that item value candidate as the most likely item value of the item to be extracted (the item value of the item to be extracted). On the other hand, if there are multiple item value candidates that are determined to be valid as the item value of the item to be extracted, the item value extraction unit 48 determines the item value candidate with the highest validity score calculated in step S206 as the most likely item value of the item to be extracted (the item value of the item to be extracted).

[0145] Then, the item value extraction unit 48 outputs the determined (extracted) item values. As a result, for example, the item values output by the item value extraction unit 48 are input into the system, thereby making it possible to automate (semi-automate) the input work of forms. Thereafter, the process proceeds to step S208.

[0146] In step S208, it is determined whether item values have been extracted for all extraction target items. The CPU 11 determines whether item values (likely item value candidates) have been extracted for each extraction target item. All extraction target items can be confirmed (recognized) by referring to the correct answer definition. If item values have not been extracted for all extraction target items (NO in step S208), the process returns to step S203, and processing is performed for extraction target items that have not yet been processed (for example, the item attribute "payment deadline"). On the other hand, if item values have been extracted for all extraction target items (YES in step S208), the process shown in this flowchart ends.

[0147] In this embodiment, an invoice is used as an example of a predetermined document type, and a learning process for extracting item values from an invoice and a process for extracting item values from an invoice are illustrated. However, the learning process may be performed for each of a plurality of predetermined document types. In this case, the learning device 2 generates a trained model and an item keyword list for each extraction target item for each of a plurality of predetermined document types (e.g., invoices, delivery notes, etc.). Furthermore, in this case, the information processing device 1 acquires a trained model and an item keyword list for each document type from the learning device 2, thereby becoming able to extract item values from documents (manuscripts) of various document types. Note that the trained model for which document type to use for the acquired extraction target image may be determined by a user who views the extraction target image (manuscript), or may be determined by the information processing device 1 having a function for automatically identifying the document type of the document (manuscript) shown in the extraction target image.

[0148] As described above, by performing extraction processing using an image of the document to be extracted, such as a semi-standard form, a trained model, and an item keyword list, it is possible to output the desired item value.

[0149] As described above, according to this embodiment, the learning device 2 can generate a trained model that can determine whether a character string (item value candidate) in an image is an item value of a target item from a feature based on the positional relationship between the character string (item value candidate) and multiple item keywords, making it possible to generate a model (extractor) that can extract item values even from an image of a document in which the item's position (layout) is not fixed (the layout is diverse). Furthermore, according to this embodiment, the information processing device 1 can determine the validity of each item value candidate in an extraction target image using a trained model that can determine whether a character string (item value candidate) is an item value of a target item from a feature based on the positional relationship between the character string (item value candidate) in an image and multiple item keywords, making it possible to extract item values even from an image of a document in which the layout is not fixed (extraction target image).

[0150] Furthermore, according to this embodiment, it is possible to easily generate an item value extractor (trained model) that can also handle document images with an undefined layout. Conventionally, there has been a desire to automatically extract data (item values) of items desired by users even from forms with diverse layouts. However, while the content (items) written on forms are generally the same regardless of the company that issued the form, the position of the content (form layout) often differs depending on the issuing company. In this case, a method of pre-defining the item reading position using OCR (layout definition) requires a layout definition to be created for each form layout. Therefore, in order to handle forms with diverse layouts, it is necessary to create a large number of layout format definitions for each trading partner company, which is not easy.

[0151] There is also a method called semi-standard form OCR, which manually defines and extracts extraction rules based on item name keywords that correspond to the item values of the desired issuer company name, payment invoice date, invoice amount, etc., as well as the relative positional relationship between these two. With this method, an operator (experienced person) who is familiar with forms manually creates extraction rules by observing the target semi-standard form. While this method is more versatile than the methods mentioned above, it requires knowledge and experience to find extraction rules for item name keywords and relative positional relationships, making it difficult to handle forms with a variety of layouts.

[0152] However, according to the present embodiment described above, by simply preparing target form samples (multiple learning images related to the same type of form but with different layouts) and a correct definition of the item value to be extracted (information indicating whether the item value candidate is the item value of the item to be extracted), it is possible to automatically (semi-automatically) create a trained model and item keywords that replace (equivalent to) the extraction rules for extracting item values. Therefore, even an ordinary worker can easily generate an extractor (trained model) that extracts item values from semi-standard forms. In other words, according to the present embodiment, it is possible to generate an extractor (trained model) that can also handle documents with an undefined layout, and by using this extractor, it is possible to extract item values from documents with an undefined layout.

[0153] Furthermore, during operation, item values can be extracted from an extraction target image simply by preparing a trained model and item keywords that can handle documents with no fixed layout (semi-structured forms) and the extraction target image, making it possible to extract item values from semi-structured forms easily (with little effort). Furthermore, in the past, even experienced users sometimes encountered inconsistencies in extraction rules when trying to handle a variety of bills, invoices, and the like. However, according to this embodiment, a trained model is generated by performing machine learning using sample images with various layouts, making it possible to extract item values from semi-structured forms with higher accuracy.

[0154] <Other embodiments> In the above-described embodiment, an example was shown in which a correct answer definition is generated by manually inputting a correct answer definition value by a user, but the method for generating a correct answer definition is not limited to the above-described method, and a method of generating a correct answer definition using a support tool for generating a correct answer definition may also be used. Also, in the above-described embodiment, a table-format correct answer definition is exemplified, but the format of the correct answer definition is not limited to a table format such as a CSV (comma-separated values) format (CSV file), and may be other formats. In this embodiment, an example is shown of a method of generating a correct answer definition in CSV format using a correct answer definition creation support tool (correct answer definition generation screen).

[0155] In the functional configuration of this embodiment described below, items that overlap with those described in the above-described embodiment are assigned the same reference numerals and will not be described again. Furthermore, the configuration of the information processing system 9 according to this embodiment is similar to the configuration of the information processing system 9 according to the above-described embodiment (FIG. 1), and therefore will not be described again. However, the learning device 2 according to this embodiment further includes an input device 26 such as a mouse, keyboard, or touch panel, and an output device 27 such as a display, in addition to a CPU 21, a ROM 22, a RAM 23, a storage device 24, and a communication unit 25.

[0156] 19 is a diagram showing an outline of the functional configuration of a learning device according to this embodiment. A program stored in storage device 24 is loaded into RAM 23 and executed by CPU 21, which controls the various hardware components of learning device 2. As a result, learning device 2 functions as a device including an image acquisition unit 51, a recognition result acquisition unit 52, a format definition storage unit 53, an item value candidate extraction unit 54, a correct answer definition acquisition unit 55, an item keyword determination unit 56, a feature generation unit 57, a model generation unit 58, and a storage unit 59, as well as a display unit 60, a designation acceptance unit 61, and a correct answer definition generation unit 62. The following describes the display unit 60, the designation acceptance unit 61, and the correct answer definition generation unit 62, which are different from the above-described embodiment.

[0157] The display unit 60 executes various display processes via the output device 27 of the learning device 2. For example, the display unit 60 generates and displays a correct definition generation screen for a user (a person generating a correct definition) to select an item value (a correct definition value) for an item to be extracted in order to generate a correct definition. The display unit 60 displays each learning image on the correct definition generation screen and displays the extracted item value candidates in the displayed learning image in a manner that allows the user to visually recognize that the item value candidates are item value candidates, for example, by surrounding the item value candidates in the displayed learning image with a red frame or a dotted frame. The display unit 60 also displays the OCR result character string (item value candidate) of the portion selected by the user as the correct definition value in a correct definition value table that displays the character string selected as the correct definition value. In this way, the display unit 60 is a user interface (UI, User Interface) (correct answer selection UI) for displaying learning images, extracted item value candidates, selected correct definition values, etc.

[0158] FIG. 20 is a diagram showing an example of a correct answer definition generation screen (when "billing amount" is selected) according to this embodiment. As shown in FIG. 20, the correct answer definition generation screen displays a learning image (correct answer definition image) (left side of the screen in FIG. 20) for which a correct answer definition is to be performed, and a correct answer definition value table (right side of the screen in FIG. 20). Note that the correct answer definition screen is not limited to the screen configuration shown in FIG. 20, and may have any screen configuration as long as it displays a correct answer definition image and item value candidates and allows a correct answer definition value to be selected from the displayed item value candidates.

[0159] 20 shows an example in which a correct answer definition value for the item attribute "billing amount" is determined for learning image 1. As shown in FIG. 20, on the correct answer definition generation screen, item value candidates for the item attribute "billing amount" are displayed in a dotted frame. As will be described later, when the user selects the item value candidate "4,059," the display unit 60 displays the OCR result character string "4,059" for the selected part in the column for the item attribute "billing amount" in the correct answer definition value table.

[0160] At this time, to make it easier for the user to recognize the item attribute (billing amount) for which the correct definition value is to be determined, an arrow or the like may be displayed at the corresponding item attribute in the correct definition value table, as shown in Fig. 20. Furthermore, when the correct definition value for the item attribute "billing amount" is selected (determined), the display unit 60 displays a screen for extracting the correct definition value for the next item attribute (for example, "payment deadline").

[0161] FIG. 21 is a diagram showing an example of a correct definition generation screen (when "payment deadline" is selected) according to this embodiment. The example in FIG. 21 shows an example in which a correct definition value for the item attribute "payment deadline" is determined for learning image 1. As shown in FIG. 21, on the correct definition generation screen, item value candidates for the item attribute "payment deadline" are displayed in a dotted frame. Also, as shown in FIG. 21, because the target item attribute has been changed from the state in FIG. 20, the focus (arrow) has moved to the corresponding item attribute, "payment deadline," in the correct definition value table. This allows the user to be prompted to select a correct definition value for the next item attribute.

[0162] The specification receiving unit 61 receives various inputs (specifications) from the user via the input device 26, such as a mouse. For example, the specification receiving unit 61 receives a user specification of one item value candidate as a defined correct value from among the item value candidates displayed on the correct definition generation screen. For example, the specification receiving unit 61 receives a specification related to the selection of a defined correct value when the user selects an item value candidate, which is the item value of the item to be extracted, with a mouse or the like. In the example of FIG. 20 , the item value candidate "4,059" is selected as the defined correct value by the user visually recognizing the item value candidates (arrow (pointer) location), and the OCR result character string "4,059" at the selected location is displayed as the defined correct value in the defined correct value table. Also, in the example of FIG. 21 , the item value candidate "2021 / 7 / 25" is selected as the defined correct value, and the OCR result character string "2021 / 7 / 25" at the selected location is displayed as the defined correct value in the defined correct value table.

[0163] The correct answer definition generating unit 62 generates a correct answer definition by using an item value candidate (OCR result character string at the specified location) for an extraction target item (item attribute) specified by the user as the item value (correct answer definition value) of the extraction target item in the learning image. The correct answer definition generating unit 62 generates a correct answer definition that stores, as a correct answer definition value, an item value candidate for each extraction target item whose specification is accepted by the specification accepting unit 61. Note that the correct answer definition is not limited to a CSV format and may be in another format. Furthermore, the correct answer definition generating unit 62 outputs the generated correct answer definition, and the correct answer definition acquiring unit 55 acquires the correct answer definition generated and output by the correct answer definition generating unit 62.

[0164] In the examples of Figures 20 and 21, after the user selects correct definition values for all item attributes, the user presses the "Confirm Correct Definition" button to generate a correct definition that stores the item values (correct definition values) for each selected item attribute. The item attributes for which the item values (correct definition values) are to be defined (extraction targets) may be determined in advance by the user and stored in the storage device 24 or the like as definition target item attribute data. The process of determining (selecting) the above-described correct definition value is performed for each definition target item attribute included in the definition target item attribute data. In this embodiment, the correct definition generation process is performed on the learning device 2, but this is not limited thereto and may be performed on a device other than the other learning device 2. In this case, the learning device 2 performs the learning process by acquiring a correct definition from another device on which the correct definition generation process has been performed.

[0165] FIG. 22 is a flowchart showing an outline of the flow of the correct answer definition generation process according to this embodiment. The process shown in this flowchart is executed when an instruction to scan a form (document) is accepted on the correct answer definition generation screen displayed on the learning processing device 2, or when such an instruction is accepted from a user to acquire a form image stored in the storage device 24. The process shown in this flowchart is executed prior to the learning process shown in FIG. 15 as pre-processing for the learning process. This flowchart also illustrates an example of processing when the item attribute is "billing amount."

[0166] In step S301, a plurality of document images (images for defining a correct answer, which are images for learning) are acquired. The image acquisition unit 51 acquires scanned images of documents (originals) of a predetermined document type (e.g., invoices) with different layouts. Then, the process proceeds to step S302.

[0167] In step S302, the character recognition result (full text OCR result) is acquired. The recognition result acquisition unit 52 performs character recognition on each correct answer definition image acquired in step S301, thereby acquiring the character recognition result (full text OCR result) for each correct answer definition image. Then, the process proceeds to step S303.

[0168] In step S303, item value candidates for the item attribute to be extracted are extracted from the character recognition results of the correct answer definition image. The item value candidate extraction unit 54 extracts item value candidates for the item to be extracted using the format definition for the item to be extracted stored in the format definition storage unit 53. For example, the item value candidate extraction unit 54 extracts a word string that matches the format definition for the item attribute "billing amount" from the character recognition results of the learning image 1 as an item value candidate for the item attribute "billing amount" in the learning image 1. Then, the process proceeds to step S304.

[0169] In step S304, the item value candidates are displayed on the correct answer definition generation screen. The display unit 60 displays the item value candidates on the correct answer definition generation screen so that the user can recognize which word strings the item value candidates extracted in step S303 correspond to (see FIG. 20). Thereafter, the process proceeds to step S305.

[0170] In step S305, a specification of a defined correct value from the user is accepted. The specification accepting unit 61 accepts the user's specification of the defined correct value. In the example of FIG. 20, the item value candidate "4,059", which is the item value of the item attribute "billing amount" in learning image 1, is specified by the user as the defined correct value, and the specification of the defined correct value is accepted. Furthermore, when the defined correct value is selected by the user, the display unit 60 displays the OCR result character string "4,059" of the selected part as the defined correct value in the defined correct value table on the defined correct value generation screen. Thereafter, the process proceeds to step S306.

[0171] In step S306, it is determined whether or not the specification of the correct definition value has been accepted for all extraction target items (item attributes) (whether the processes of steps S303 to S305 have been executed). CPU 21 determines whether or not the specification of the correct definition value has been accepted for each extraction target item. If not all extraction target items have been processed (NO in step S306), the process returns to step S303, and processing is executed for extraction target items that have not been processed (for example, the item attribute "payment deadline"). On the other hand, if all extraction target items have been processed (YES in step S306), the process proceeds to step S307.

[0172] In step S307, the correct answer definition value of each extraction target item (item attribute) in the learning image is determined. When the designation receiving unit 61 receives a user instruction to determine the item value candidates for all item attributes designated in step S305 as correct answer definition values, the correct answer definition generating unit 62 determines the correct answer definition value of each item attribute in the learning image (e.g., learning image 1). Then, the process proceeds to step S308.

[0173] In step S308, it is determined whether or not the correct answer definition value has been determined for all learning images. CPU 21 determines whether or not the correct answer definition value for each extraction target item (item attribute) has been determined for each of all learning images. If the correct answer definition value has not been determined for all learning images (NO in step S308), the process returns to step S303, a correct answer definition generation screen for an unprocessed learning image (e.g., learning image 2) is displayed, and subsequent processes are executed. On the other hand, if the correct answer definition value has been determined for all learning images (YES in step S308), the process proceeds to step S309.

[0174] In step S309, a correct answer definition is generated. The correct answer definition generation unit 62 generates and outputs a correct answer definition that stores the correct answer definition values of each item attribute for all learning images determined in step S307. Thereafter, the processing shown in this flowchart ends. Note that the correct answer definition acquisition unit 55 acquires the correct answer definition from the correct answer definition generation unit 62 (step S102 in FIG. 15).

[0175] In this embodiment, during the correct answer definition generation process, learning images and character recognition results for the learning images are acquired in the learning device 2, so the processes of steps S101 and S103 of the learning process in the learning device 2 in Fig. 15 may be omitted. Also, during the correct answer definition generation process, item value candidates are extracted in the learning device 2, so the process of step S1051 of the trained model generation process in the learning device 2 in Fig. 17 may be omitted. Also, in this embodiment, multiple correct answer definition images are acquired collectively in step S301, but the present invention is not limited to this, and the processes of steps S301 to S307 may be executed for each correct answer definition image.

[0176] As described above, in this embodiment, automatically extracted item value candidates are displayed, allowing the operator to generate a correct definition simply by selecting a correct item value from the displayed item value candidates, thereby making it possible to improve the efficiency of the work of generating a correct definition compared to a method in which the operator manually inputs a correct definition value.Furthermore, it is possible to improve the efficiency of the work of extracting item values (the work of generating a trained model and an item keyword list). [Explanation of symbols]

[0177] 1. Information processing equipment 2 Learning device 3 Document reader 9. Information Processing Systems

Claims

1. a recognition result acquisition means for acquiring a character recognition result obtained by character recognition of a target image; an item value candidate extraction means for extracting a plurality of candidate character strings that can be item values of the extraction target item from the character recognition result of the target image; a feature generating means for generating, for each candidate character string, a feature based on a positional relationship in the target image between the candidate character string and a plurality of item keywords, which are keyword word strings for extracting item values of the extraction target items; a model storage means for storing a trained model generated by machine learning so that, when a feature based on the positional relationship between a character string and the plurality of item keywords in an image is input, information indicating the validity of the character string as an item value of the item to be extracted is output; and an item value extraction means for extracting item values of the extraction target items from the plurality of candidate character strings by inputting the feature amounts of each candidate character string in the target image to the trained model; An information processing system comprising:

2. The trained model is a model generated by machine learning using training data in which, for each of a plurality of training images of documents having mutually different layouts, feature amounts based on positional relationships in the training image between candidate character strings, which are character strings that can be item values of the extraction target items, and the plurality of item keywords are associated with information indicating whether the candidate character strings are item values of the extraction target items. The information processing system according to claim 1 .

3. The feature amount is a feature amount based on information indicating a distance between a target character string and the item keyword for each of the plurality of item keywords, and a feature amount based on information indicating a direction from one of the target character string and the item keyword to the other.

3. The information processing system according to claim 1 or 2.

4. the feature based on the information indicating the distance is a feature that increases or decreases depending on the distance between the target character string and the category keyword; The information processing system according to claim 3 .

5. The feature based on the information indicating the direction is a feature that increases or decreases depending on the degree to which the topic keyword is located horizontally to the left and vertically above the target character string.

5. The information processing system according to claim 3 or 4.

6. the item value extraction means extracts the item value of the extraction target item based on information indicating a likelihood that each candidate character string is an item value of the extraction target item, the information being output from the trained model by inputting the feature amount of each candidate character string in the target image into the trained model; 6. The information processing system according to claim 1.

7. the item keyword is a word string determined from a word string contained in at least one training image among a plurality of training images used in the machine learning, based on the attributes of the word string; 7. The information processing system according to claim 1.

8. the attribute of the word string includes the frequency of appearance of the word string in the plurality of training images; The information processing system according to claim 7 .

9. The attribute of the word string includes a distance between the word string and the item value of the extraction target item in a learning image that includes the word string.

9. The information processing system according to claim 7 or 8.

10. The attribute of the word string includes a direction from one side of the word string to the other side of the item value of the extraction target item in the learning image containing the word string.

10. The information processing system according to claim 7.

11. The item keyword is determined based on the degree to which the word string is located horizontally to the left or vertically above the item value of the item to be extracted. The information processing system according to claim 10.

12. The word strings are determined as the item keywords in descending order of effectiveness scores indicating effectiveness of the word strings based on the attributes of the word strings.

12. The information processing system according to claim 7.

13. further comprising a format definition storage means for storing format definitions relating to the extraction target items; the item value candidate extraction means extracts, as the candidate character string, a character string that matches a format definition related to the extraction target item; The information processing system according to any one of claims 1 to 12.

14. a recognition result acquisition means for acquiring a character recognition result obtained by performing character recognition on a plurality of learning images of documents having different layouts; a feature generation means for generating, for each character string that is an item value of an item to be extracted and other character strings included in each learning image, a feature based on the positional relationship within the learning image between the character string and a plurality of item keywords that are keyword word strings for extracting the item value of the item to be extracted; a model generation means for generating a trained model by performing machine learning using training data in which, for each character string in each training image, the feature amount of the character string is associated with information indicating whether the character string is an item value of the item to be extracted; An information processing system comprising:

15. further comprising an item value candidate extraction means for extracting a plurality of candidate character strings that are character strings that can be item values of extraction target items from the character recognition results of each learning image; the character strings that are the item values of the extraction target items and the other character strings are the extracted plurality of candidate character strings; The information processing system according to claim 14.

16. The method further includes a correct definition acquisition means for acquiring a correct definition in which the extraction target items and the item values of the extraction target items in each learning image are associated with each other, the model generation means acquires information indicating whether the character string is an item value of the item to be extracted based on the correct answer definition; 16. The information processing system according to claim 14 or 15.

17. a display means for displaying candidate character strings for the extraction target items extracted from the character recognition results of the learning image in a manner that allows a user to visually recognize that the candidate character strings are candidates for item values; a designation receiving means for receiving a designation by a user for one of the displayed candidate character strings; and a correct definition generating means for generating the correct definition by using the specified candidate character string as an item value of the item to be extracted in the learning image, the correct definition acquisition means acquires the correct definition generated by the correct definition generation means; 17. The information processing system according to claim 16.

18. The computer a recognition result acquisition step of acquiring a character recognition result that is a result of character recognition of the target image; an item value candidate extraction step of extracting a plurality of candidate character strings that are character strings that can be item values of the extraction target item from the character recognition result of the target image; a feature generation step of generating, for each candidate character string, a feature based on a positional relationship in the target image between the candidate character string and a plurality of item keywords, which are keyword word strings for extracting item values of the extraction target items; a model storage step of storing a trained model generated by machine learning so that, when a feature based on the positional relationship between a character string and the plurality of item keywords in an image is input, information indicating the validity of the character string as an item value of the item to be extracted is output; an item value extraction step of extracting item values of the extraction target items from the plurality of candidate character strings by inputting the feature amounts of each candidate character string in the target image into the trained model; The field value extraction method to perform.

19. Computer, a recognition result acquisition means for acquiring a character recognition result obtained by character recognition of a target image; an item value candidate extraction means for extracting a plurality of candidate character strings that can be item values of the extraction target item from the character recognition result of the target image; a feature generating means for generating, for each candidate character string, a feature based on a positional relationship in the target image between the candidate character string and a plurality of item keywords, which are keyword word strings for extracting item values of the extraction target items; a model storage means for storing a trained model generated by machine learning so that, when a feature based on the positional relationship between a character string and the plurality of item keywords in an image is input, information indicating the validity of the character string as an item value of the item to be extracted is output; and an item value extraction means for extracting item values of the extraction target items from the plurality of candidate character strings by inputting the feature amounts of each candidate character string in the target image into the trained model; A program to function as a

20. The computer a recognition result acquisition step of acquiring character recognition results obtained by character recognition of a plurality of learning images for documents having different layouts; a feature generation step of generating, for each character string that is an item value of an extraction target item and other character strings included in each learning image, a feature based on the positional relationship within the learning image between the character string and a plurality of item keywords that are keyword word strings for extracting the item value of the extraction target item; a model generation step of generating a trained model by performing machine learning using training data in which, for each character string in each training image, the feature amount of the character string is associated with information indicating whether the character string is an item value of the item to be extracted; ,Model generation method.

21. Computer, a recognition result acquisition means for acquiring a character recognition result obtained by performing character recognition on a plurality of learning images of documents having different layouts; a feature generation means for generating, for each character string that is an item value of an item to be extracted and other character strings included in each learning image, a feature based on the positional relationship within the learning image between the character string and a plurality of item keywords that are keyword word strings for extracting the item value of the item to be extracted; a model generation means for generating a trained model by performing machine learning using training data in which, for each character string in each training image, the feature amount of the character string is associated with information indicating whether the character string is an item value of the item to be extracted; A program to function as a

Images