Information processing apparatus, information processing method, and program

Abstract

To appropriately extract a character string that increases burdens when estimating the character string by using a learned model, while reducing the burdens.SOLUTION: An information processing apparatus has: acquisition means that acquires a token string generated on the basis of a character string included in a document image; first determination means that determines a document class represented by the document image and a character string corresponding to a first item included in the document image by using a result obtained by inputting the token string to a learned model; and second determination means that applies the document class and the character string corresponding to the first item to a rule-based algorithm to determine a character string corresponding to a second item dependent on the first item included in the document image.SELECTED DRAWING: Figure 10

Description

[Technical field]

[0001] The present disclosure relates to a process for extracting data from document images. [Background technology]

[0002] There is a method to extract character strings (item values) corresponding to specific items from document images obtained by scanning documents. The extracted character strings are used as file names, input data for business systems, etc.

[0003] Patent Document 1 describes a method of assigning tags to each group of character strings extracted from a form using a trained model obtained by machine learning the correspondence between the position of an extracted character string and a tag to be assigned to the character string at that position.Then, it describes a method of generating structured data in which item names and item values ​​are associated with each other using the tagged character strings according to a format for each type of form. [Prior art documents] [Patent documents]

[0004] [Patent Document 1] JP 2020-13281 A Summary of the Invention [Problem to be solved by the invention]

[0005] Incidentally, it is conceivable to extract character strings such as the company name of the document's issuing destination and the company name of the document's issuing source from character strings in a document. However, depending on the document, for example, the company name of the billing destination may be the issuing destination company name, or the billing destination company name may be the issuing source company name. For this reason, when attempting to generate a trained model that estimates character strings such as the issuing destination and issuing source company names, it is not possible to generate them appropriately using a method such as Patent Document 1 that learns the correspondence between character strings, and further training data is required to cover all conditions. Therefore, the burden of costs or time required to generate a trained model increases. [Means for solving the problem]

[0006] The information processing device disclosed herein is characterized by having an acquisition means for acquiring a token sequence generated based on a character string included in a document image, a first determination means for determining a document type represented by the document image and a character string corresponding to a first item included in the document image using a result obtained by inputting the token sequence into a trained model, and a second determination means for applying the document type and the character string corresponding to the first item to a rule-based algorithm to determine a character string corresponding to a second item subordinate to the first item included in the document image. Effect of the Invention

[0007] According to the present disclosure, character strings that would be burdensome if estimated by a trained model can be appropriately extracted while minimizing the burden. [Brief description of the drawings]

[0008] [Figure 1] FIG. 1 is a diagram showing an example of the configuration of an information processing system. [Diagram 2] FIG. 1 is a diagram showing an example of a document. [Diagram 3] FIG. 2 is a diagram showing an example of the hardware configuration of an image forming apparatus, a learning apparatus, and an information processing apparatus. [Figure 4] FIG. 2 is a diagram showing a sequence of the information processing system. [Diagram 5] 6 is a flowchart illustrating the details of a process executed by the learning device. [Figure 6] FIG. 4 is a diagram for explaining a document image sample. [Figure 7] 10 is a flowchart for explaining details of a process for generating a document image token string. [Figure 8] 6A and 6B are diagrams for explaining a process of generating a document image token string. [Figure 9] FIG. 13 is a diagram for explaining the process of generating an item value token string. [Figure 10] 11 is a flowchart illustrating details of a process for determining a character string to be extracted. [Figure 11] 6A and 6B are diagrams for explaining a process for determining a character string to be extracted. [Figure 12] 6A and 6B are diagrams for explaining a process for determining a character string to be extracted. [Figure 13] 11 is a flowchart for explaining an extraction target character string determination algorithm. [Figure 14] 11A and 11B are diagrams for explaining the determination result of extraction target character strings. [Figure 15] 11 is a flowchart for explaining a process for determining a character string to be extracted. [Figure 16] 11A and 11B are diagrams for explaining the determination result of extraction target character strings. [Figure 17] 11A and 11B are diagrams for explaining the determination result of extraction target character strings. [Figure 18] FIG. 13 is a diagram showing priorities corresponding to the changed buyer position. [Figure 19] FIG. 11 is a diagram for explaining another example of extraction target items. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0009] DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS The present disclosure will be described with reference to the accompanying drawings. Note that the components described in the following embodiments are merely examples and are not intended to limit the technical scope of the present disclosure.

[0010] <First embodiment> [Information processing system configuration] 1 is a diagram showing an example of the configuration of an information processing system 100. As shown in FIG. 1, the information processing system 100 has, for example, an image forming device 110, a learning device 120, and an information processing device 130, and each device is connected to each other via a network 104. In the information processing system 100, the image forming device 110, the learning device 120, and the information processing device 130 may be configured to be connected to the network 104 in a multi-connection configuration rather than a single connection. For example, the information processing device 130 may be configured to be composed of a first server device having high-speed computing resources and a second server device having large-capacity storage, and connected to each other via the network 104.

[0011] The image forming apparatus 110 is realized by an MFP (Multi-Function Peripheral) having multiple functions such as printing, scanning, and faxing. The image forming apparatus 110 has at least an image acquisition unit 118 as a functional unit.

[0012] The image forming apparatus 110 has a scanner device 306 (see FIG. 3). The scanner device 306 optically reads a document 111 in which a character string is printed on a recording medium such as paper, and the image acquisition unit 118 performs predetermined image processing on the resulting data to generate a document image 113. Also, for example, the image acquisition unit 118 receives FAX data 112 transmitted from a FAX transmitter (not shown), and performs predetermined FAX image processing to generate the document image 113. The image acquisition unit 118 transmits the generated document image 113 to the information processing apparatus 130.

[0013] The image forming device 110 may be realized by a PC (Personal Computer) or the like other than the MFP having the above-mentioned scan or FAX function. For example, a document image 113 such as PDF or JPEG generated by using a document creation application running on the PC may be transmitted to the information processing device 130.

[0014] The learning device 120 has the functional units of a learning data generation unit 122 and a learning unit 123 .

[0015] The learning device 120 receives as input a plurality of document image samples 121 provided by an engineer, document type information for each document image sample, character string data included in each document image sample, and item value information to be extracted.

[0016] The learning data generation unit 122 generates an image document token sequence from each of the document image samples 121. In addition, an item value token sequence is generated by embedding item value information of the extraction target in each token of the generated image document token sequence. The document image token sequence and the item value token sequence will be described in detail later.

[0017] The learning unit 123 generates a document type estimation model, which is a trained model that estimates the document type of a document image using training data in which an image document token sequence and document type information are paired. The learning unit 123 also generates an item value estimation model, which is a trained model that estimates item values ​​included in a document image, using training data in which an image document token sequence and item value token sequence are paired.

[0018] The information processing device 130 has the functional units of an information processing unit 131 and a data management unit 135 .

[0019] The information processing unit 131 determines the document type of the document image 113 using the document type estimation model generated by the learning device 120. Also, using the item value extraction model, it determines a character string (item value) corresponding to a predetermined item name from character string data included in the document image 113. Then, the information processing unit 131 inputs the determined document type and item value to an extraction target character string determination algorithm in which a judgment condition for determining an issuer and an issuer is designed in advance by an engineer. Then, it determines an extraction target character string 114 that is a character string corresponding to an extraction target item. Also, when a user instructs correction, the information processing unit 131 corrects the extraction target character string 114 to a character string specified by the user. The information processing unit 131 can update information of the extraction target character string determination algorithm based on the character string specified by the user. Details of the processing of the information processing unit 131 will be described later.

[0020] The data management unit 135 stores data of the extraction target character string 114 determined by the information processing unit 131 .

[0021] The network 104 is realized by a LAN, a WAN, or the like, and is a communication unit that connects the image forming device 110, the learning device 120, and the information processing device 130 to one another and transmits and receives data between the devices.

[0022] [About items and strings to be extracted] There is a method for extracting character strings (item values) corresponding to a given item from document images created with different layouts, known as semi-standardized documents or non-standardized documents. For example, if multiple company names and names of people in charge are written in a document, it is possible to extract company names or names of people in charge of billing destinations and receiving destinations from the contents of the document by using a trained model obtained by machine learning the positional relationships of the character strings. On the other hand, if you try to extract character strings corresponding to the company names or names of people in charge of the document issuer and the document recipient as the extraction target character strings using a similar method, it may be difficult to extract the character strings.

[0023] FIG. 2(a) is a diagram showing an example of a document whose document type is an invoice. The company name of the issuer of the document in FIG. 2(a) is "AAA Optec" written under "From". For this reason, if a trained model that estimates the issuer or issuer by learning the contents of the character strings and the positional relationship of the character strings in a document such as FIG. 2(a) is generated, the trained model may output a character string near "From" as the issuer company name. On the other hand, there are documents such as the document in FIG. 2(b) in which the loading location is written under "From". For this reason, when the document in FIG. 2(b) is input to a trained model trained using a document such as FIG. 2(a), there is a risk that the model may fail to extract the character string indicating the issuer of the document.

[0024] Similarly, the name of the company to which the document is issued in the document in FIG. 2(a) is the buyer's company name in FIG. 2(a), "BBB Net," which is written under "Bill to." On the other hand, in the document type of a purchase order in FIG. 2(c), the buyer's company name in FIG. 2(c), which is near "Bill to," is the issuer's company name. In this way, depending on the document, the buyer's company name may be the document's issuer's company name, or the buyer's company name may be the document's issuer's company name. For this reason, when the document in FIG. 2(c) is input to a trained model trained using documents such as those in FIG. 2(a), there is a risk that the extraction of the document's issuer or issuer's character strings may fail.

[0025] To generate a trained model that can appropriately output the source and destination from any of the documents in FIG. 2(a), FIG. 2(b), and FIG. 2(c), it is necessary to train the trained model by further considering the document type and business content. For this reason, sufficient training data is required to cover all the conditions. Furthermore, an increase in training data increases the training time, etc. Therefore, there is a risk that the burden of generating a trained model, such as the cost and time, will increase.

[0026] Therefore, in this embodiment, a document type estimation model that estimates the document type of a document is generated as a trained model. In addition, an item value estimation model is generated that estimates a character string indicating a company name, etc., on the buyer's side (such as a billing destination) and a character string indicating a company name, etc., on the seller's side (such as a supplier) from character strings in the document. The character strings of the company name on the buyer's side and the company name on the seller's side are acquired as character strings that are candidates for the company names of the issuer and the publisher. The document type estimation model and the item value estimation model can be generated by learning the contents of character strings included in the document and the positional relationship of the character strings. Therefore, they can be generated while reducing the amount of training data and the training cost compared to trained models that estimate the issuer and the publisher.

[0027] In this embodiment, the document type determined using the document type estimation model and the candidate character strings for the source and destination determined using the item value estimation model are obtained. The document type and the candidate character strings are then applied to a rule-based algorithm (algorithm for determining character strings to be extracted) to determine the character strings for the destination and source of the document. By combining the trained model and the rule-based algorithm in this way, it is possible to appropriately determine the character strings to be extracted while reducing the burden.

[0028] [Hardware configuration of image forming device] FIG. 3 is a diagram showing an example of the hardware configuration of the image forming device 110, the learning device 120, and the information processing device 130 included in the information processing system 100 of FIG.

[0029] FIG. 3(a) is a diagram showing a hardware configuration of the image forming apparatus 110. As shown in FIG. 3(a), the image forming apparatus 110 has a CPU 301, a ROM 302, a RAM 304, a printer device 305, a scanner device 306, a storage 308, an external interface 311, and the like, which are connected to each other via a data bus 303. The CPU 301 is a control unit for controlling the overall operation of the image forming apparatus 110. The CPU 301 executes a startup program stored in the ROM 302 to start up the system of the image forming apparatus 110, and executes a control program stored in the storage 308 to realize functions of the image forming apparatus 110, such as printing, scanning, and faxing. The ROM 302 is realized by a non-volatile memory, and is a storage unit for storing a startup program for starting up the image forming apparatus 110. The data bus 303 is a communication unit for transmitting and receiving data between devices constituting the image forming apparatus 110. The RAM 304 is realized by a volatile memory, and is a storage unit used as a work memory when the CPU 301 executes a control program.

[0030] The printer device 305 is an image output device that prints and outputs an image on a storage medium such as paper. The scanner device 306 is an image input device that is used to optically read documents such as paper on which characters, diagrams, etc. are printed and generate document images. The document transport device 307 is realized by an ADF (Auto Document Feeder) or the like, detects documents placed on a document table, and transports the detected documents one by one to the scanner device 306.

[0031] The storage 308 is realized by an HDD (Hard Disk Drive) or the like, and is a storage unit for storing the above-mentioned control program and document images. The input device 309 is realized by a touch panel, hard keys, or the like, and accepts operation input from a user to the image forming apparatus 110. The display device 310 is realized by a liquid crystal display or the like, and is a display unit for displaying and outputting a setting screen of the image forming apparatus 110 to the user. The external interface 311 connects between the image forming apparatus 110 and the network 104, and is an interface unit for receiving FAX data from a FAX transmitter (not shown) and transmitting document images to the information processing apparatus 130.

[0032] [Hardware configuration of the learning device] 3(b) is a diagram showing the hardware configuration of the learning device 120. As shown in FIG. 3(b), the learning device 120 has a CPU 331, a ROM 332, a RAM 334, a storage 335, an input device 336, a display device 337, an external interface 338, and a GPU 339, which are connected to each other via a data bus 333.

[0033] The CPU 331 is a control unit for controlling the overall operation of the learning device 120. The CPU 331 starts the system of the learning device 120 by executing a boot program stored in the ROM 332. The CPU 331 also executes a learning program stored in the storage 308 to perform layout analysis of a document image sample, and generate a document type estimation model and an item value estimation model. The ROM 332 is realized by a non-volatile memory, and is a storage unit for storing a boot program for starting the learning device 120. The data bus 333 is a communication unit for transmitting and receiving data between devices constituting the learning device 120. The RAM 334 is realized by a volatile memory, and is a storage unit used as a work memory when the CPU 331 executes a program for generating document data and learning.

[0034] Storage 335 is realized by a hard disk drive (HDD) or the like, and is a memory unit for storing data of document image samples, etc. Input device 336 is realized by a mouse, keyboard, etc., and accepts operation input from an engineer to learning device 120. Display device 337 is realized by a liquid crystal display or the like, and is a display unit for displaying and outputting the setting screen of learning device 120 to the engineer. CPU 331 operates as a display control unit that controls the screen displayed on display device 337.

[0035] The external interface 338 connects the learning device 120 and the network 104, and is an interface unit that receives the document image sample 121 from a PC (not shown) or the like. It is also an interface unit for transmitting the document type estimation model and the item value estimation model to the information processing device 130. The GPU 339 is a calculation unit configured with an image processing processor. The GPU 339 performs calculations to generate a document type estimation model and an item value estimation model based on character string data included in a given document image, for example, in accordance with a control command provided by the CPU 331.

[0036] Each functional unit included in the learning device 120 shown in Fig. 1 is realized by the CPU 331 executing a predetermined program, but is not limited to this. Other hardware such as a GPU 339 for accelerating calculations or a field programmable gate array (FPGA) (not shown) may be used. Each functional unit may be realized by cooperation between software and hardware such as a dedicated IC, or some or all of the functions may be realized only by hardware.

[0037] [Hardware configuration of information processing device] Fig. 3(c) is a diagram showing a hardware configuration of the information processing device 130. As shown in Fig. 3(c), the information processing device 130 has a CPU 361, a ROM 362, a RAM 364, a storage 365, an input device 366, a display device 367, and an external interface 368. Each unit is connected to each other via a data bus 363.

[0038] The CPU 361 is a control unit for controlling the overall operation of the information processing device 130. The CPU 361 starts up the system of the information processing device 130 by executing a boot program stored in the ROM 362, and executes an information processing program stored in the storage 365. By executing the information processing program, the processing of the information processing unit 131 is performed.

[0039] The ROM 362 is realized by a non-volatile memory, and is a storage unit for storing a boot program that starts up the information processing device 130. The data bus 363 is a communication unit for transmitting and receiving data between devices constituting the information processing device 130. The RAM 364 is realized by a volatile memory, and is a storage unit used as a work memory when the CPU 361 executes the information processing program. The storage 365 is realized by an HDD (Hard Disk Drive) or the like, and is a storage unit for storing the above-mentioned information processing program, the document image 113, the item value estimation model, character string data, and the like.

[0040] The input device 366 is realized by a mouse, a keyboard, or the like, and is an operation unit that accepts operation input from a user or an engineer to the information processing device 130. The display device 367 is realized by a liquid crystal display, or the like, and is a display unit that displays a setting screen of the information processing device 130 to the user or the engineer. The CPU 361 operates as a display control unit that controls the screen displayed on the display device 367.

[0041] The external interface 368 connects the information processing device 130 and the network 104. The external interface 368 is an interface unit for receiving the item value estimation model and the document type estimation model from the learning device 120, and for receiving the document image 113 from the image forming device 110.

[0042] Each functional unit included in the information processing device 130 in Fig. 1 is realized by the CPU 361 executing a predetermined program, but is not limited to this. Other hardware such as a GPU (Graphics Processing Unit) or an FPGA (Field Programmable Gate Array) for accelerating calculations may also be used. Each functional unit may be realized by cooperation between software and hardware such as a dedicated IC, or some or all of the functions may be realized only by hardware.

[0043] [Sequence of trained model generation] Fig. 4 is a diagram showing a sequence of the information processing system 100 in Fig. 1. The symbol "S" in the description of each process means a step in the sequence, and the same applies to the following flowcharts. For convenience of explanation, the operation of a user or engineer is also explained using steps.

[0044] The steps in Fig. 4(a) show a flow when an engineer develops a trained model, and are a diagram explaining a flow in which the learning device 120 generates a document type estimation model and an item value estimation model. Details of S402 to S405 in Fig. 4(a) will be described later with reference to Fig. 5.

[0045] In S401, an engineer 400 of the information processing system 100 inputs a plurality of document image samples 121, which are samples of images representing a plurality of documents, to the learning device 120 in order to generate a document type estimation model and an item value estimation model. Each document image sample is given document type information indicating the document type by the engineer. In addition, each document image sample is given in advance by the engineer information on the item names to be learned by the item value estimation model and the character strings of those item names.

[0046] In S402, the learning device 120 generates learning data using a plurality of document image samples 121 and the document types of the plurality of document image samples 121. Then, a document type estimation model is generated by performing learning using the learning data. In this manner, a document type estimation model for estimating the document type of an invoice, estimate, purchase order, delivery note, etc. is generated.

[0047] In S403, the learning device 120 transmits the generated document-type estimation model to the information processing device 130. The information processing device 130 stores the document-type model in the storage 365.

[0048] In S404, the learning device 120 generates learning data using a plurality of document image samples 121 and an item name ID list 900 (see FIG. 9), etc. Then, an item value estimation model is generated by performing learning using the learning data. In this manner, an item value estimation model is generated for determining character strings (item values) corresponding to each item name, which is a character string that is a candidate for the issuing company name (person in charge name) and the issuing company name (person in charge name), which are the items to be extracted in this embodiment. In other words, the items to be extracted are items subordinate to the item names determined based on the item value estimation model.

[0049] In S405, the learning device 120 transmits the generated item value estimation model to the information processing device 130. The information processing device 130 stores the item value estimation model in the storage 365.

[0050] In S406, the engineer 400 registers in the information processing device 130 an extraction target character string determination algorithm in which information such as a judgment condition required for determining an extraction target character string is set.

[0051] [Sequence of process to determine strings to extract] Fig. 4(b) is a diagram for explaining a flow in which the information processing device 130 determines character strings indicating the issuing company name (person in charge name) and the issuing company name (person in charge name), which are character strings to be extracted, from character strings included in the document image 113. Details of S413 to S417 in Fig. 4(b) will be explained using Fig. 10.

[0052] In S411, the user 401 places a paper document (original) on the image forming apparatus 110 and instructs the image forming apparatus 110 to scan the document.

[0053] In S412, the scanner device 306 of the image forming apparatus 110 reads the placed paper document, and the image acquisition unit 118 generates a document image which is an image of the scanned document. Then, the image acquisition unit 118 transmits the generated document image to the information processing apparatus 130.

[0054] In S413, the information processing device 130 executes character recognition processing (OCR processing) on ​​the document image 113 transmitted in S412, and layout analysis processing for analyzing the layout within the document image.

[0055] In S414, the information processing device 130 determines the document type of the document image 113 using the document type estimation model.

[0056] In S415, the information processing device 130 uses the item value estimation model to determine character strings (item values) that are candidates for extraction target character strings from the character strings recognized from the document image 113.

[0057] In S416, the information processing device 130 applies the document type determined in S414 and the item value determined in S415 to the extraction target character string determination algorithm registered in S406 to determine the extraction target character string.

[0058] In S417, the information processing device 130 outputs the extraction target character string determined in S416 to the user.

[0059] [Generation process of document type estimation model and field value estimation model] Fig. 5 is a flowchart for explaining the details of S402 to S405 in Fig. 4(a) that are executed by the learning device 120. The series of processes shown in the flowchart in Fig. 5 are performed by at least one of the CPU 331 and GPU 339 of the learning device 120 expanding program code stored in the ROM 332 or storage 335 into the RAM 334 and executing it. In addition, some or all of the functions of the steps in Fig. 5 may be realized by hardware such as an ASIC or electronic circuit.

[0060] In S501, the learning data generation unit 122 acquires a plurality of document images input by an engineer in S401 of Fig. 4 as a plurality of document image samples 121. The following description will be given on the assumption that images of documents handled by accounting-related clerical staff are acquired as the document image samples 121.

[0061] A document image 600 in FIG. 6(a) is an example of a document image sample acquired in S501, and shows a document image obtained by scanning an estimate.

[0062] The next steps S502 to S508 are loop processes, and the processes of S502 to S508 are repeatedly executed for the multiple document image samples acquired in S501. That is, in S502, the learning data generation unit 122 selects a document image sample to be processed from unprocessed document image samples among the multiple document image samples acquired in S501. Then, the processes of S503 to S507 are executed for the document image sample to be processed. When the process for the document image sample to be processed is completed, in S508, it is determined whether the process has been executed for all the multiple document image samples. If it is determined that the process has not been executed for all the document image samples, the process returns to S502, and a document image sample to be processed is selected again from the unprocessed document image samples.

[0063] In S503, the learning data generation unit 122 acquires information on the document type of the document image sample to be processed (document type information). The document type information is information that indicates the document type, and includes, for example, a document type name that is the name of the document type, and a document type ID that is a unique value assigned to each document type name. Since the document type information is assigned to each of the multiple document image samples by the engineer 400 in S401 of Fig. 4, the learning data generation unit 122 can acquire document type information corresponding to the document image sample to be processed.

[0064] Fig. 6(b) is a diagram showing an example of document type information. Fig. 6(b) is an example of document type information corresponding to the document image 600 shown in Fig. 6(a). As shown in Fig. 6(b), for example, "2" is acquired as the document type ID and "quotation" is acquired as the document type name.

[0065] In S504, the learning data generating unit 122 acquires data of character strings included in the document image sample to be processed, and acquires the names of items (item names) represented by each character string. This information is acquired based on the information that the engineer 400 has assigned to the document image sample to be processed.

[0066] 6(c) is an enlarged view of a dotted-line region 610 in the document image 600. The region 610 includes character string regions 611 to 614. In S504, a recognized character string is obtained from each character string region in the document image sample to be processed. A unique value for identifying each character string region is assigned to each character string region as a region ID.

[0067] FIG. 6(d) is a diagram summarizing the information obtained in S503 in a table format. Column 641 in FIG. 6(d) holds values ​​representing area IDs. Column 642 holds strings included in the string area indicated by the area IDs. Column 643 holds item names corresponding to the strings held in column 642. Column 644 holds strings to be extracted from the strings held in column 642. The item names in column 643 and the strings to be extracted in column 644 are referred to as "item value information." In this way, the strings recognized from the document image sample and the item value information corresponding to the recognized strings are obtained.

[0068] For example, in FIG. 6(d), column 642 of the row (record) in which column 641 holds area ID "613" holds the character string "Ms. Jane Smith." Furthermore, column 643 of the same record holds the item name "Name of billing contact," and column 644 of the same record holds "Jane Smith," which is the character string data to be extracted. In this way, information is obtained that associates the character string "Ms. Jane Smith" in the document image sample with the item name "Name of billing contact."

[0069] Similarly, as shown by the record holding area ID "621," the character string "AAA Inc." is associated with the field name "Supplier Company Name." In the record holding area ID "631," the character string "ABC Bank" is associated with the field name "Transfer Bank Name." In the record holding area ID "632," the character string "AAA Inc." is associated with the field name "Transfer Account Name." Column 543 of the record holding area ID "611" does not contain a field name string. In such a case, the character string "Bill To" is obtained as the field value information.

[0070] In S505, the learning data generating unit 122 generates a document image token sequence corresponding to the document image sample to be processed. The document image token sequence generating process executed in S505 may be executed by the information processing unit 131 of the information processing device 130.

[0071] 7 is a flowchart for explaining the details of the process for generating a document image token string. The process of S505 will be explained in detail with reference to FIG.

[0072] In S701, the learning data generation unit 122 acquires image data of a document image to be processed and data of character strings included in the document image. In the document image token sequence generation process in S505, image data of a document image sample to be processed and data of character strings recognized from the document image sample to be processed are acquired.

[0073] In S702, the learning data generation unit 122 analyzes the layout of the document image to be processed acquired in S701. As a result of the analysis, the learning data generation unit 122 identifies document constituent regions in the document image and extracts the identified document constituent regions. A method for extracting document constituent regions may be to identify blank regions, ruled lines, etc. in the document image and identify regions surrounded by the identified blank regions and ruled lines, etc., as document constituent regions.

[0074] Fig. 8 is a diagram for explaining the process of generating a document image token string from the document image 600 in Fig. 6(a). The dotted rectangular area in Fig. 8(a) indicates document constituent areas 801 to 807 extracted by performing layout analysis on the document image 600 to be processed.

[0075] In S703, the learning data generation unit 122 determines the order (reading order) for reading the document constituent regions extracted in S702. For example, the learning data generation unit 122 may determine the reading order such that the document constituent regions 801 to 807 are read in order starting from the upper left corner of the document image 600.

[0076] The next steps S704 to S709 are loop processes, and in S704, the learning data generation unit 122 selects a processing target from the document constituent areas extracted in S702 according to the reading order determined in S703. For example, when document constituent areas 801 to 807 are identified as shown in FIG. 8(a), document constituent areas are selected as processing targets in order starting from document constituent area 801. Then, the processes of S705 to S708 are executed for the document constituent areas to be processed. When the process for the document constituent area to be processed is completed, in S709, it is determined whether the process has been performed for all document constituent areas. If it is determined that the process has not been performed for all document constituent areas, the process returns to S704, and the next document constituent area is selected as the processing target.

[0077] In S705, the learning data generation unit 122 converts the information of the document constituent region to be processed into <area> This makes it possible to indicate the boundary of a document-constituting region in the generated document image token sequence.

[0078] In S706, when there are multiple character strings in the document composition area to be processed, the learning data generation unit 122 determines the reading order of the character strings. For example, when the composition area to be processed includes multiple character strings like the document composition area 803, the reading order of the character strings may be determined so that the character strings are read in order from the top left of the document composition area 803.

[0079] In S707, the learning data generation unit 122 arranges the character strings included in the document constituent area to be processed in the reading order determined in S706, and converts the arranged character strings into character string tokens. The conversion to character string tokens may be performed by extracting morphemes from the character strings using a morphological analysis method, and converting each morpheme into a character string token.

[0080] Character string tokens 822 to 832 in FIG. 8C represent character string tokens converted from character strings in the document configuration area 802.

[0081] In S708, the learning data generation unit 122 generates a document image token string for the document constituent area to be processed by concatenating the character string tokens obtained in S707 with the region information token obtained in S705 at the beginning. Then, if a document image token string has already been generated, the document image token string generated from the document constituent area to be processed is concatenated to the already generated document image token string.

[0082] 8(b) shows a document image token sequence 810 obtained as a result of performing the processes of S705 to S708 on document constituent area 801, which is the first to be read. Since document constituent area 801 contains the character string "Quote", document image token sequence 810 is a token sequence in which area information token 811 is followed by character string token 812 generated from "Quote".

[0083] 8(c) shows a document image token string 820 obtained as a result of performing the processes of S705 to S708 on document constituting area 802, which is second in the reading order. A document image token string obtained by concatenating area information token 821 and character string tokens 822 to 832 converted from character strings in document constituting area 802 is concatenated to the end of document image token string 810 that has already been generated, thereby generating document image token string 820.

[0084] If it is determined in S709 that there is no unprocessed document constituent area, the flowchart in FIG. 7 ends, and the process of generating a document image token sequence for the document image to be processed ends.

[0085] Fig. 8(d) shows a document image token sequence 840 generated as a result of performing the processes of S705 to S708 on all of the document constituent areas 801 to 807 in Fig. 8(a) The rectangles that make up the document image token sequence are called tokens.

[0086] Returning to Fig. 5, the description of the flowchart will continue. In S506, the learning data generation unit 122 generates an item value token sequence corresponding to the document image sample to be processed. The item value token sequence is generated from the document image token sequence 840 generated in S505.

[0087] Fig. 9 is a diagram for explaining the process of generating an item value token sequence. Fig. 9(a) is a diagram showing an example of an item name ID list used for generating an item value token sequence. An item name ID list 900 stores character strings of item names and item name IDs, which are unique values ​​for identifying item names, in association with each other. An item value token sequence is generated by converting character strings included in each token constituting a document image token sequence into the value of an item name ID using the item name ID list 900.

[0088] FIG. 9B is a diagram showing an item value token sequence 910 generated from the document image token sequence 840 in FIG. 8D. For example, “ <area> A search is performed to see if the item name represented by the character string corresponding to " is held as an item name in the item name ID list 900. <area> " is not a string representing an item name, so " <area> " is replaced with the item name ID "0" associated with "not applicable" in the item name ID list 900. A token 911 indicates the token replaced from the area information token 811.

[0089] Similarly, "Quote" included in string token 812 in document image token sequence 840 is a string indicating the item name "Document Name", as shown in column 643 of the record in which area ID "601" is held in FIG. 6(d). In item name ID list 900, "1" is held as the item name ID corresponding to the item name "Document Name". Therefore, string token 812 in document image token sequence 840 is replaced with token 912 containing item name ID "1". A similar process is performed on all tokens constituting document image token sequence 840, thereby generating item value token sequence 910 shown in FIG. 9(b).

[0090] Note that the item name ID is not limited to the values ​​held in the item name ID list 900 in Fig. 9(a). The item names included in the item name ID list 900 are the item names of the item name IDs estimated by the item value estimation model. However, in this embodiment, the item names "issued to" and "issuer" are not included in the item name ID list 900. This is because, as described above, the burden of generation increases when attempting to generate an item value estimation model capable of estimating "issued to" and "issuer".

[0091] Also, the method of assigning item name IDs is not limited to the above-mentioned method. For example, item value tokens may be generated using the IOB format or the BILOU format as item name IDs. In the case of the IOB format, the start item value token may be assigned "B-" (Begin), and the intermediate item value tokens may be assigned "I-" (Inside). In the case of the BILOU format, in addition to the IOB format, the end item value token may be assigned "L-" (Last), and in the case of a single item value token, "U-" (Unit). By processing in this way, the range of the extracted character string can be clarified, enabling learning and inference.

[0092] The description of the process will be continued by returning to Fig. 5. In S507, the learning data generation unit 122 generates a learning data set for training the document type estimation model. Also, a learning data set for training the item value estimation model is generated.

[0093] The learning dataset for training the document type estimation model is, for example, data for supervised learning, and is a dataset in which the document image token sequence 840 generated in S505 is used as input data and the document type ID acquired in S503 is used as supervised data.

[0094] A training data set for training the item value estimation model is, for example, a data set in which the document image token sequence 840 generated in S505 is used as input data and the item value token sequence 910 generated in S505 is used as training data.

[0095] By performing S503 to S507, learning data sets for training a document type estimation model and an item value estimation model are generated from the document image sample to be processed. By repeating the processes of S503 to S507 until there are no unprocessed document image samples, it is possible to obtain multiple learning data sets generated from the multiple document image samples acquired in S501. If the learning data generation unit 122 determines in S508 that all document image samples have been processed, the process proceeds to S509.

[0096] In S509, the learning unit 123 generates a document type estimation model by machine learning using the generated learning dataset. The document type estimation model is a trained model that is trained to estimate and output a document type ID corresponding to the document type of a document represented by a document image when a document image token string generated from the document image is input.

[0097] In S510, the learning unit 123 transmits the document type estimation model generated in S509 to the information processing device 130. Then, the document type estimation model is stored in the storage 365 in the information processing device 130.

[0098] In S511, the learning unit 123 generates an item value estimation model by machine learning using the generated learning dataset. The item value estimation model is a trained model that is trained to estimate and output an item value token sequence having the same structure as that shown in FIG. 9(b) when a document image token sequence generated from a certain document image is input.

[0099] In S512, the learning unit 123 transmits the item value estimation model generated in S511 to the information processing device 130. Then, the item value estimation model is stored in the storage 365 in the information processing device 130.

[0100] In the processes of S509 and S511, the model can learn not only the relationship between the character string token to be extracted and the character string tokens before and after it, but also the relationship between character string tokens in the same area or across areas. In other words, the model can learn the tendency that character strings that serve as clues for finding the character string to be extracted (such as keywords corresponding to item names) tend to appear in the same area and rarely appear across areas.

[0101] The document type estimation model and the item value estimation model can be trained using a known machine learning method, such as RNN, Seq2Seq, Transformer, and BERT, which are used in natural language machine translation, document classification, and named entity extraction.

[0102] In addition, although the document type estimation model and the item value estimation model have been described as being generated as independent trained models, it is also possible to train the models to generate a single trained model that performs document type estimation and item value estimation simultaneously.

[0103] [Process to determine the string to extract] Fig. 10 is a flowchart for explaining the details of the process of determining the extraction target character string performed in S413 to S417 in Fig. 4(b). The series of processes shown in the flowchart in Fig. 10 are performed by the CPU 361 of the information processing device 130 expanding the program code stored in the ROM 362 or the storage 365 into the RAM 364 and executing it. In addition, some or all of the functions of the steps in Fig. 10 may be realized by hardware such as an ASIC or an electronic circuit. As described above, in this embodiment, a process is performed to determine character strings indicating the issue destination and issuer as extraction target character strings from a document image.

[0104] In S1001, the information processing unit 131 acquires the document type estimation model transmitted from the learning device 120 in S403 of Fig. 4 and stored in the storage 365. In addition, the information processing unit 131 acquires the item value estimation model transmitted from the learning device 120 in S405 of Fig. 4 and stored in the storage 365.

[0105] In S1002, the information processing unit 131 acquires the document image transmitted from the image forming apparatus 110 in S412 of FIG.

[0106] In S1003, the information processing unit 131 extracts a character string area included in the document image acquired in S1002, and then executes character recognition processing (OCR processing) on ​​the extracted character string area to acquire data of the character string included in the document image (character string data).

[0107] In S1004, the information processing unit 131 generates a document image token string of the document image acquired in S1002 based on the document image acquired in S1002 and the character string data acquired in S1003. Details of the document image token string generation process are performed according to the flow shown in Fig. 7. Details of S1004 are omitted because they are the same as those in the case where the learning data generation unit 122 in the explanation of Fig. 7 is replaced with the information processing unit 131.

[0108] In S1005, the information processing unit 131 inputs the document image token sequence generated in S1004 to the document type estimation model acquired in S1001, and acquires a document type ID based on the estimation result. In this way, the information processing unit 131 determines the document type of the document image acquired in S1002 by having the document type estimation model perform an inference process.

[0109] Fig. 11 is a diagram for explaining the process of determining the character string to be extracted. Fig. 11(a) is an example of the document image acquired in S1002. Fig. 11(b) shows that the document image token sequence generated from the document image 1100 in Fig. 11(a) is input to the document type estimation model, and as a result, the document type ID is determined to be "3". It also shows that the document type name corresponding to the document type ID "3" is "purchase order".

[0110] The document type estimation model may output a probability value representing the likelihood of a document type corresponding to each of the multiple document type IDs as an inference result. In this case, the information processing unit 131 may determine the document type indicated by the document type ID with the highest probability value as the document type of the document image to be processed.

[0111] In S1006, the information processing unit 131 inputs the document image token sequence to the item value estimation model acquired in S1001. The item value estimation model outputs an item value token sequence having the same structure as that shown in FIG. 9(b) as an inference result.

[0112] The inference process performed by the item value estimation model is to estimate which item name ID in the item name ID list 900 each character string token included in the document image token sequence corresponds to, based on the relationship between the character string token and the area information token used in learning. The item value estimation model may output a probability value representing the likelihood that each item name ID corresponds to an item name, for each character string token. In this case, the information processing unit 131 may determine the item name indicated by the item name ID with the highest probability value as the item name of the character string indicated by the character string token.

[0113] Then, based on the item value token sequence output from the item value estimation model, the information processing unit 131 determines a string (item value) corresponding to the item name from the string recognized in S1003, and further determines the item name corresponding to that item value.

[0114] For example, the information processing unit 131 searches the output item value token string for a token that includes a numeric value other than "0", which indicates no match. Since each token in the item value token string output from the item value estimation model can be associated with a token in the document image token string input to the item value estimation model, a token corresponding to the searched token is identified from the document image token string. Based on the result, the output item value token string is converted into a character string. For example, assume that the item value token string output from the item value estimation model is the item value token string 910 in FIG. 9(b). When "22" is searched for from tokens 913 and 914, the tokens "AAA" and "Inc." at the same position in the input data are obtained, and the tokens are combined to determine the character string "AAA Inc." as the item value. In this way, the item name ID "22" is associated with the character string "AAA Inc.", which is the item value.

[0115] Fig. 11(c) is a diagram summarizing the processing results of S1006 in table format. In Fig. 11(c), the item value, the determined character string, and the item name of the determined character string are associated with each other. For example, it shows that "CCC company" recognized from the area ID "1111" is determined as the character string of the item value corresponding to the item name "orderer company name" with the item name ID "21". The item value determined in S1006 is used as a character string that is a candidate for the extraction target character string in this embodiment.

[0116] In S1007, the information processing unit 131 determines extraction target character strings from among the item values ​​(character strings) corresponding to each item name obtained in S1006. Specifically, the extraction target character strings in this embodiment are character strings indicating the name of the company to which the document is issued, the name of the person in charge of the issue, the name of the company from which the document is issued, and the name of the person in charge of the issue. In this embodiment, the document type determined in S1005 and the item values ​​determined in S1006 are applied to an extraction target character string determination algorithm to determine the extraction target character strings. Details of the processing in S1007 will be described later.

[0117] In S1008, the information processing unit 131 performs processing to output the extraction target character string determined in S1007 and present it to the user.

[0118] In S1009, the information processing unit 131 determines whether or not to end the process, and repeats the processes of S1002 to S1008 until a notification to end the process is received from the user.

[0119] [About the algorithm for determining the string to be extracted] 12 is a diagram for explaining the process of determining the character string to be extracted. In order to determine the character string to be extracted, the relationship between the company name or the person in charge name written in the document is defined.

[0120] As shown in FIG. 12(a), in the case of selling goods, the user company (your own company) may be either the seller or the buyer. When the user is the buyer, after placing an order for a good, the user will receive from the trading partner to whom these documents were issued a quotation showing the results of the quotation, an invoice showing the billing amount and transfer destination, etc., and a delivery note showing information related to the delivery details. On the other hand, when the user is the seller, an order is placed by the trading partner, and the user will receive from the trading partner to whom the purchase order was issued a purchase order showing information about the goods that the trading partner wants to purchase, etc. In this way, depending on the content of the transaction, the user may be either the buyer or the seller.

[0121] FIG. 12(b) is a table in which possible user positions when the user is on the buyer side, the role of each position, and a priority rank according to the importance of the role in each position are associated with each other. For example, possible positions on the buyer side include the purchaser, billing party, and delivery destination. Of these positions, the purchaser who actually performs the processing is considered to be the most important in conducting a transaction. In FIG. 12(b) and FIG. 12(c), the priority is stored with a value such that the smaller the value, the higher the rank. For this reason, when the position is a purchaser, a priority with a higher rank (smaller value) is associated with it than other positions.

[0122] Fig. 12(c) is a table in which possible user positions when the user is a seller are associated with the role of each position and a priority according to the importance of the role in each position. It is considered that the most important person in a seller's position when conducting a transaction is the recipient who actually performs the processing. For this reason, the recipient position is associated with a higher priority than other positions.

[0123] The character strings representing the position (e.g., "billing destination") stored in the tables of Figures 12(b) and 12(c) are used as character strings constituting the item names in Figure 9(a). As in the item name ID list 900 of Figure 9(a), item names (e.g., "billing destination company name") are included that are composed of a combination of a character string representing the seller or buyer's position (e.g., "billing destination") and a character string for the "company name" or "person in charge". The reason for using such item names will be explained later.

[0124] The positions, roles, and priorities shown in Figures 12(b) and 12(c) are examples. Although not shown, the positions may include various other positions such as contact points. In addition, company information that is not to be extracted may be set.

[0125] Fig. 13 is a flowchart for explaining the extraction target character string determination algorithm. The process of S1007 in Fig. 10 will be described in detail with reference to Fig. 13.

[0126] In S1301, the information processing unit 131 acquires the document type determined in S1005, and the item value (character string) and item name determined in S1006. For example, information including the document type shown in Fig. 11(b) and information in which the extracted item value and item name are associated with each other as shown in Fig. 11(c) are acquired.

[0127] The next steps S1302 to S1308 are loop processes, and in S1302, the information processing unit 131 selects either the buyer side or the seller side as a processing target. For example, the buyer side is selected first. Then, the processing of S1303 to S1307 is performed with the buyer side as the processing target.

[0128] In S1303, the information processing unit 131 judges whether an item value corresponding to an item name consisting of the position of the processing target (e.g., the buyer) and the company name has been extracted as a result of the determination process in S1006. If the information processing unit 131 judges that the corresponding item value has been extracted (YES in S1303), the process proceeds to S1304.

[0129] For example, in Fig. 12(b), character strings of "orderer", "billing destination", and "receiver" are stored from the buyer's perspective. Therefore, it is sufficient to determine whether item values ​​corresponding to the item names "orderer company name", "billing destination company name", and "receiver company name", each of which is composed of each character string and "company name", have been extracted from the document image in S1006.

[0130] For example, column 1104 in Fig. 11(b) contains "orderer company name", "recipient company name", and "bill-to company name" as item names corresponding to item values ​​extracted from the document image. Therefore, item values ​​corresponding to the item names "orderer company name", "bill-to company name", and "bill-to company name", which are composed of the position and company name of the buyer side being processed, are extracted. Therefore, when Fig. 11(b) is acquired as a result of the processing in S1006, YES is determined in S1303 and the process proceeds to S1304.

[0131] In this embodiment, the item value estimation model is trained to estimate item values ​​corresponding to item names including each position of the buyer side or each position of the seller side. Therefore, by checking the item names determined using the item value estimation model, the information processing unit 131 can determine whether the document image includes a character string indicating the buyer side and a character string indicating the seller side.

[0132] In S1304, the information processing unit 131 selects, as the output target for "buyer company name", the item value corresponding to the item name consisting of the position and company name with the highest priority rank among the item names consisting of the position and company name of the processing target. For example, when the processing target is the buyer side, in FIG. 12(b), the position with the highest priority is "purchaser". Therefore, if "purchaser company name" is present in column 1104 holding the item names in FIG. 11(c), a character string is selected from column 1102 holding the item value corresponding to that item name. If the item value corresponding to the item name "purchaser company name" has not been extracted, the item value corresponding to the item name of "billing company name" with the next highest priority rank is selected.

[0133] In Figure 11(c), the record with area ID "1121" holds the item value "DDD LLC" for the item name "Orderer Company Name," so "DDD LLC" is selected as the output target for the buyer's company name. Also, when the processing target is the seller, the position with the higher priority is "contractor." In Figure 11(c), the record with area ID "1111" holds the item value "CCC Company" for the item name "Contractor Company Name," so "CCC Company" is selected as the output target for the seller's company name.

[0134] In S1305, the information processing unit 131 selects an item value corresponding to an item name consisting of the position and person in charge name included in the item name selected in S1304 from the item values ​​extracted in S1006. The selected item value is the buyer's person in charge name to be output. For example, if an item value corresponding to "purchasing party company name" is selected in S1304, an item value corresponding to the item name "purchasing party person in charge name" is selected in S1305.

[0135] In Figure 11(c), the record with the domain ID "1122" holds the item value "Dana Morgan" corresponding to the item name "Purchaser's Contact Name", so "Dana Morgan" is selected as the output target for the buyer's contact name. If the name of a contact in the same position is not included in the extracted item value, it is determined that there is no corresponding contact name.

[0136] Also, if the processing target is the seller, the record with area ID "1112" contains the field value "John Smith" corresponding to the field name "Order recipient representative name," so "John Smith" is selected as the output target for the seller's representative name.

[0137] On the other hand, if the information processing unit 131 determines in S1303 that an item value corresponding to an item name consisting of the position of the processing target and the company name has not been extracted (NO in S1303), the process proceeds to S1306.

[0138] In S1306, the process in S1303 is replaced with "person in charge" instead of "company name". That is, the information processing unit 131 judges whether an item value corresponding to an item name consisting of a character string representing the position of the processing target and "person in charge" has been extracted as a result of the process in S1006. For example, when the processing target is the buyer, it may be judged based on the extraction result of the item values ​​in FIG. 11(c) whether item values ​​corresponding to the item names "orderer person in charge", "billing person in charge", and "destination person in charge" have been extracted.

[0139] If it is determined that the corresponding item value has been extracted (YES in S1306), the process proceeds to S1307. If it is determined that the corresponding item value has not been extracted (NO in S1306), the process proceeds to S1308.

[0140] In S1307, the process in S1304 is replaced with "name of person in charge". That is, the information processing unit 131 selects, as the output target, the item value corresponding to the item name including the position with the highest priority rank among the item values ​​corresponding to the item names consisting of the character string of the position of the processing target side and the name of the person in charge. As shown in FIG. 12(b), the position with the highest priority among the positions of the buyer side is "purchaser". Therefore, when the processing target is the buyer side, if the item name "purchaser person in charge name" exists in FIG. 11(c), the item value corresponding to that item name is selected. If the item name "purchaser person in charge name" does not exist, the item value corresponding to the item name with the next highest priority rank "billing person in charge name" is selected.

[0141] In S1308, the information processing unit 131 determines whether the processing has been completed on both the seller side and the buyer side. If it is determined in S1308 that the processing has been completed on both the buyer side and the seller side, the process proceeds to S1309.

[0142] In S1309, the information processing unit 131 determines whether the document type acquired in S1301 is a purchase order. If the information processing unit 131 determines that the document type is a purchase order (YES in S1309), the process proceeds to S1310.

[0143] In S1310, the information processing unit 131 determines that the buyer is the issuer of the document and the seller is the destination of the document, and proceeds to S1312. In the case of a purchase order, the buyer is associated with the issuer in advance based on the relationship in Fig. 12(a), so the information processing unit 131 can determine that the buyer is the issuer of the document.

[0144] When the process moves from S1310 to S1312, the information processing unit 131 sets the character string corresponding to the buyer's company name selected as a result of S1302 to S1308 as the character string corresponding to the issuing company name. Also, it sets the character string corresponding to the buyer's person in charge name as the character string corresponding to the issuing person in charge name. Also, it sets the character string corresponding to the seller's company name as the character string corresponding to the issuing company name, and the character string corresponding to the seller's person in charge name as the character string corresponding to the issuing person in charge name. Then, the process of the flowchart in FIG. 13 ends, and the process moves to S1008 in FIG. 10.

[0145] 14A and 14B are diagrams for explaining the determination results of extraction target character strings. Fig. 14A is a diagram showing a list of extraction target character strings determined from item values ​​in the document image of the purchase order sheet shown in Fig. 11B.

[0146] As shown in FIG. 14(a), when the document image is an image of a purchase order, the character string corresponding to the issuing company name is determined to be "DDD LLC" selected as the output target character string of the buying company name. The character string corresponding to the issuing contact person name is determined to be "Dana Morgan" selected as the buying contact person name. The character string corresponding to the issuing company name is determined to be "CCC company" selected as the selling company name. The character string corresponding to the issuing contact person name is determined to be "John Smith" selected as the selling contact person name.

[0147] On the other hand, if the information processing unit 131 determines that the document type is other than a purchase order (NO in S1309), the process proceeds to S1311.

[0148] In S1311, the information processing unit 131 determines that the seller is the document issuer and the buyer is the document recipient, and proceeds to S1312. In the case of quotations, invoices, and delivery notes other than purchase orders, the seller is associated as the issuer based on the relationship in Fig. 12(a). Therefore, the information processing unit 131 can determine that the seller is the document issuer.

[0149] When the process moves from S1311 to S1312, the information processing unit 131 sets the character string corresponding to the buyer's company name selected as a result of S1302 to S1308 as the character string corresponding to the destination company name. Also, it sets the character string corresponding to the buyer's person in charge name as the character string corresponding to the destination person in charge name. Also, it sets the character string corresponding to the seller's company name as the character string corresponding to the issuing company name, and the character string corresponding to the seller's person in charge as the character string corresponding to the issuing person in charge name. Then, the process of the flowchart in FIG. 13 ends, and the process moves to S1008 in FIG. 10.

[0150] 14B shows an example of a confirmation screen for the extraction target character string, which is displayed on the display device 367 of the information processing device 130 as a result of S1008. The confirmation screen 1400 includes a preview image display area 1401, a result display area 1402, a next button 1403, and an end button 1404.

[0151] The result display area 1402 is an area where character strings 1421 and 1422 determined to be extraction target character strings are displayed. That is, in this embodiment, character strings corresponding to the destination company name, the destination person in charge name, the issuing company name, and the issuing person in charge name are displayed. The user can correct the character strings 1421 and 1422 determined to be extraction target character strings by pressing edit buttons 1431 and 1432, so that the OCR results can be corrected if they are incorrect. The document image is displayed in the preview image display area 1401, and the areas including the character strings 1421 and 1422 displayed in the result display area 1402 are highlighted.

[0152] When the press of the Next button 1403 is accepted, the information processing unit 131 judges in S1009 not to end the process, and the processes of S1002 to S1007 are repeated to execute the process of determining the extraction target items for the next document image. On the other hand, when the press of the End button 1404 is accepted, the information processing unit 131 judges in S1009 to end the process, and the flowchart in FIG. 10 ends.

[0153] As described above, in this embodiment, the document type of the document image and character strings that can be candidates for the extraction target character string are determined using a trained model generated by machine learning. Then, an appropriate extraction target character string can be determined from the candidate character strings using a rule-based algorithm that performs a judgment according to a combination of a predefined document type and description content. In this way, in this embodiment, the extraction target character string is determined by using both the trained model and the rule-based algorithm. By combining the determination of the document type and the candidate character string using a trained model that is good at information extraction and the determination process of the extraction target character string that executes processing based on knowledge, experience, etc. using an algorithm, the accuracy of the determination of the extraction target character string can be improved. In addition, by combining a trained model that is good at information extraction with an algorithm that describes complex judgment conditions as rules, it is possible to facilitate the generation of the trained model and the design of the algorithm.

[0154] Therefore, according to this embodiment, it is possible to determine extraction target character strings that can be extracted by understanding the relationship (context) of character strings in a document image while taking into consideration the layout information of the document image. Therefore, it is possible to determine extraction target character strings from document images created with different layouts, generally called semi-standardized documents or non-standardized documents, while reducing the burden.

[0155] <Second embodiment> In the first embodiment, a method of determining one character string for one extraction target item and presenting it to the user is described. In the present embodiment, a method of assisting the user in making corrections by determining multiple candidate character strings for one extraction target item and presenting them to the user is described.

[0156] Fig. 15 is a diagram for explaining the determination of extraction target character strings in this embodiment. S1501 to S1503 in Fig. 15 are the same processes as S1301 to S1303 in Fig. 13, so their explanation will be omitted. The explanation will focus on steps (S1504, S1505, S1507) that perform processes different from the steps in Fig. 13.

[0157] In S1504, the information processing unit 131 selects an item value corresponding to an item name consisting of the position and company name of the processing target. For example, when the processing target is the buyer, if the item names "orderer company name", "billing company name", and "recipient company name" are present in the column 1104 in which the item names in the document image in Fig. 11(c) are held, the item value corresponding to the item name is selected. Then, the item values ​​of the item names including the positions with the highest priority ranks are selected as the item values ​​with the highest priority to be displayed to the user as candidates.

[0158] For example, in FIG. 11(b), the record with the area ID "1121" has an item name "orderer company name" that includes a position with the highest priority rank of "1", and holds an item value "DDD LLC" corresponding to the item name. Therefore, the information processing unit 131 selects "DDD LLC" as a character string of the first candidate for the buyer's company name. Similarly, the record with the area ID "1141" has an item name "billing company name" that includes a position with the next highest priority rank of "2", and holds an item value "DDD LLC" corresponding to the item name. In addition, the record with the area ID "1131" has an item name "destination company name" that includes a position with a priority rank of "3", and holds an item value "DDD Group" corresponding to the item name. Therefore, the information processing unit 131 selects "DDD LLC" as a second candidate for the buyer's company name, and selects "DDD Group" as a third candidate for the buyer's company name.

[0159] In S1505, the information processing unit 131 selects an item value corresponding to an item name composed of a position and a person in charge name included in the item name corresponding to the item value selected in S1504. Then, the item value of the item name including the position with the highest priority rank is selected as an item value with the highest priority to be displayed to the user as a candidate.

[0160] In FIG. 11(b), the record with the area ID "1122" holds the item value "Dana Morgan" corresponding to the item name "Purchaser Personnel Name" including the position of "Purchaser" set as the first candidate in S1504. Therefore, the information processing unit 131 selects "Dana Morgan" as the character string of the first candidate for the buyer's personnel name. The item name "Billing Personnel Name" including the position of "Billing" set as the second candidate in S1504 is not included in FIG. 11(b). The record with the area ID "1133" holds the item value "James A. Brown" corresponding to the item name "Destination Personnel Name" including the position of "Destination" set as the third candidate in S1504. Therefore, the information processing unit 131 selects "James A. Brown" as the second candidate for the buyer's personnel name.

[0161] In S1507, the process in S1504 is performed in which the "company name" is replaced with the "person in charge" name.

[0162] Since S1506 is the same as S1306, and S1508 to S1512 are the same as S1308 to S1312, the explanation will be omitted. When the processes from S1501 to S1511 are completed, in S1512, extraction target strings and change candidate strings are determined for each extraction target item.

[0163] FIG. 16(a) is a diagram showing a list of extraction target strings and string candidates determined from the item names and item values ​​included in FIG. 11(b) when it is determined that the buyer is the issuer and the seller is the destination.

[0164] In S1512, the character string selected as the first candidate is determined as the extraction target character string, the character string selected as the second candidate as change candidate character string 1, and the character string selected as the third candidate as change candidate character string 2. Therefore, as shown in Fig. 16(a), the first candidate character string for the buyer's company name, "DDD LLC", is determined as the extraction target character string for the issuer's company name, and the second candidate character string for the buyer's company name, "DDD LLC", is determined as change candidate character string 1 for the issuer's company name. In addition, the first candidate character string for the buyer's contact person name, "Dana Morgan", is determined as the extraction target character string for the issuer's contact person name.

[0165] Fig. 16(b) shows an example of a confirmation screen 1600 that displays the extraction target character string displayed by display control by the information processing unit 131. In Fig. 16(b), the same parts as those in the confirmation screen 1400 in Fig. 14(b) are given the same reference numerals. For example, when the user presses an edit button 1431, the user can modify the character string 1421 determined as the issuing company name.

[0166] When the edit button 1431 is pressed, a pop-up 1611 is displayed in the preview image display area 1601 at the position of the character string of the issuing company name that has been determined as the character string to be extracted. When the change candidate character strings are determined as in FIG. 16(a), when the edit button 1431 is pressed, pop-ups 1612 and 1613 are displayed at the positions of the respective change candidate character strings of the issuing company name. Assume that the user presses a change button included in one of the pop-ups 1612 and 1613 corresponding to the change candidate character strings. In this case, the information processing unit 131 corrects the current extraction target character string displayed in the result display area 1402 to the change candidate character string corresponding to the pop-up whose change button was pressed. For example, when the change button of the pop-up 1613 is pressed, the information processing unit 131 corrects the character string "DDD Group" corresponding to the pop-up 1613 as the extraction target character string of the issuing company name.

[0167] As described above, according to this embodiment, it is possible to present the user with lower-ranked candidates for each extraction target character string, thereby reducing the burden on the user when correcting an error in determining an extraction target character string.

[0168] [Variation 1] When determining candidate character strings for the buyer's (seller's) company name and the buyer's (seller's) person in charge name, each candidate character string may be determined from the item name that includes the same position in the company name and the person in charge name. For example, the candidate character strings for the buyer's company name and the candidate character strings for the buyer's person in charge name may be determined together according to the priority defined in Fig. 12(b). That is, S1504 and S1505 may be performed together.

[0169] This example will be described taking the case where the processing target is the buyer. For example, the information processing unit 131 selects the item value corresponding to the item name including "orderer" with a position priority of "1" as the first candidate for the buyer's company name and the buyer's person in charge name. From FIG. 11(b), the item value "DDD LLC" corresponding to the item name "orderer's company name" is selected as the first candidate for the buyer's company name, and the item value "Dana Morgan" corresponding to "orderer's person in charge name" is selected as the first candidate for the buyer's person in charge name.

[0170] Similarly, the information processing unit 131 selects the item value corresponding to the item name including "billing destination" with a position priority of "2" as the second candidate for the buyer's company name and the buyer's person in charge name. As a result, the item value "DDD LLC" corresponding to the item name "billing destination company name" is selected as the second candidate for the buyer's company name. Also, since the item value corresponding to the item name "billing destination person in charge name" is not extracted, the second candidate for the buyer's person in charge name is not selected and is left blank. Similarly, the information processing unit 131 selects the item value of the item name including the character string "destination" with a position priority of "3" as the third candidate for the buyer's company name and the buyer's person in charge name. As a result, the item value "DDD Group" corresponding to the item name "destination destination company name" is selected as the third candidate for the buyer's company name, and the item value "James A. Brown" corresponding to the item name "destination destination person in charge name" is selected as the third candidate for the buyer's person in charge name.

[0171] Fig. 17(a) is a diagram showing a list of extraction target character strings and change candidate character strings determined from Fig. 11(b) when the buyer is determined to be the issuer and the seller is determined to be the issuer. Unlike Fig. 16(a), change candidate character string 1 for the issuer's person in charge name is left blank.

[0172] Fig. 17(b) shows an example of a confirmation screen 1700 that displays an extraction target character string that is displayed by display control by the information processing unit 131. In Fig. 17(b), the same parts as those in the confirmation screen 1600 in Fig. 16(b) are denoted by the same reference numerals.

[0173] 16(b), when it is detected that the user has pressed the issuing company name edit button 1431, a pop-up 1711 is displayed at the position of the currently determined character string to be extracted in the issuing company name on the preview image display area 1701. Furthermore, pop-ups 1712 and 1713 are displayed at the positions of candidate character strings for changing the issuing company name.

[0174] As in the change candidate character string 2 in FIG. 17(a), there are cases where change candidate character strings of the same order are determined for both the company name and the person in charge name. In this case, both a change company name only button 1702 and a change company name and person in charge name button 1703 are displayed in a pop-up 1713 displayed at the position of the change candidate character string 2. For example, assume that the change company name and person in charge name button 1703 is pressed. In this case, the information processing unit 131 corrects the character string of the change candidate character string 2 of the issuing company name to the extraction target character string of the issuing company name. Furthermore, the information processing unit 131 corrects the character string of the change candidate character string 2 of the issuing person in charge name to the extraction target character string of the issuing person in charge name. In this way, the correction using the change candidate character string is performed simultaneously, reducing the user's effort.

[0175] As described above, according to this embodiment, it is possible to determine candidate character strings for extraction target items based on the same criteria, taking into consideration the relevance between the extraction target items. This makes it possible to simultaneously correct related extraction target character strings, thereby reducing the burden on the user.

[0176] [Variation 2] In the above description, the method of correcting the determined extraction target character string based on the user's instruction has been described, but the priority associated with the position may be changed based on the content of the correction. By updating the priority, it is possible to improve the accuracy of determining the extraction target character string from the next time onwards.

[0177] For example, assume that the character strings to be extracted and the change candidate character strings of the issuing company name and the issuing person in charge name have been determined as shown in Fig. 17(a). Then, assume that the user issues an instruction to modify the character strings of the issuing company name and the issuing person in charge name to the character string of change candidate character string 2. In this case, the information processing unit 131 displays on the confirmation screen 1700 a confirmation message including an option of whether to change the priority of the position based on the character string selected by the user from the next time onwards. Then, when the user selects to change the position, the information processing unit 131 updates the priority associated with the position shown in Fig. 12(b).

[0178] FIG. 18 is a diagram showing the priority associated with the buyer's position shown in FIG. 12(b) after updating it by the method of this example. For example, it is assumed that the character string determined as the extraction target character string by the user is modified to a character string corresponding to the item name including the destination determined as the modification candidate character string 2. In this case, it is considered that the position of "destination" included in the item name used for the modification is selected by the user. Therefore, as shown in FIG. 18, the information processing unit 131 updates the priority so that the priority rank of "destination" becomes the highest. As a result, when determining the extraction target character string from the next time onwards, it becomes possible to determine the extraction target character string based on the judgment condition based on the modified priority.

[0179] It is also possible to set a priority associated with a position for each document type, and update the priority for each document type. For example, if the document type is determined to be a purchase order in the extraction target character string determination process, the priority of the buyer's position for a purchase order may be updated as shown in FIG. 18. In that case, for other document types, the initial setting in FIG. 12(b) may be used. Furthermore, it is also possible to update the priority associated with a position for each company name and person in charge name.

[0180] As described above, according to this embodiment, the criteria for determining the extraction target character string are updated based on the content of the user's instruction, so that from the next time onwards, the extraction target character string desired by the user can be determined.

[0181] <Other embodiments> In the above embodiment, the items to be extracted are the recipient company name (person in charge name) and the issuing company name (person in charge name), but the items to be extracted are not limited to this. For example, there are various types of monetary information to be extracted from a document image, such as "unit price," "total excluding tax," "total including tax," and "total unpaid," but the monetary information to be extracted may not be included in the document image.

[0182] FIG. 19 is a diagram for explaining another example of an item to be extracted. For example, the item to be extracted is the amount including tax, but as shown in FIG. 19(a), the amount including tax may not be written in the document. In this case, it is not possible to determine a numeric string corresponding to the amount including tax from the estimation result of the item value estimation model. In such a case, the information processing unit 131 determines a numeric string of other amount information that can be determined from the estimation result of the item value estimation model, for example, a numeric string corresponding to the amount excluding tax. Then, the information processing unit 131 applies the numeric string corresponding to the determined amount excluding tax to a preset rule-based algorithm to calculate the amount including tax. In this way, a numeric string corresponding to the amount including tax may be determined.

[0183] In addition, the item to be extracted is the tax-inclusive price, but as shown in Fig. 19(b), the document may only state the tax-inclusive price including shipping. In such a case, the information processing unit 131 determines a numeric string corresponding to the tax-inclusive price using an item value estimation model. Then, the determined numeric string may be applied to a rule-based algorithm to determine a numeric string corresponding to the tax-inclusive price excluding shipping.

[0184] In addition, the method of the above-mentioned embodiment can be applied to the case where a character string corresponding to a document number belonging to a document is extracted from the document. Various document numbers are included in a document. For example, as shown in FIG. 19(c), an invoice may include an invoice number, an order number when an order is received, and a delivery number in the case of an invoice after delivery. For this reason, in order to extract a document number belonging to a document from the document, it is necessary to understand each document number included in the document and determine a document number that is considered appropriate according to the document type as the document to be extracted. For example, in the case of an invoice, it is necessary to determine the invoice number as the document to be extracted. Also, as shown in FIG. 19(d), in the case of a delivery note, it is necessary to determine the delivery number as the document number to be extracted. Also, if there is no invoice number and only a delivery number is written in the invoice, it must be determined that there is no document number to be extracted.

[0185] Therefore, when the item to be extracted is the document number belonging to the document, an item value estimation model is generated to estimate character strings corresponding to the invoice number, order number, and delivery number. The information processing unit 131 uses the item value estimation model to determine character strings corresponding to the invoice number, order number, and delivery number. Then, the character strings corresponding to the document number belonging to the document may be determined by applying the character strings corresponding to the invoice number, order number, and delivery number and the document type to a rule-based algorithm.

[0186] In addition, it is necessary to determine the document issue date as an appropriate date according to the document type from among the dates included in the document that indicate the invoice date, order date, and delivery date. Therefore, the document issue date may also be determined using a trained model and algorithm in a manner similar to that for the document number.

[0187] The present disclosure can also be realized by a process in which a program for implementing one or more functions of the above-described embodiments is supplied to a system or device via a network or a storage medium, and one or more processors in a computer of the system or device read and execute the program. It can also be realized by a circuit (e.g., ASIC) that implements one or more functions.

[0188] The disclosure of the above-described embodiment includes the following configurations.

[0189] (Configuration 1) An acquisition means for acquiring a token string generated based on a character string included in a document image; a first determination means for determining a document type represented by the document image and a character string corresponding to a first item included in the document image, using a result obtained by inputting the token sequence into a trained model; a second determination means for determining a character string corresponding to a second item included in the document image and subordinate to the first item by applying a rule-based algorithm to the document type and the character string corresponding to the first item; 13. An information processing device comprising:

[0190] (Configuration 2) The second determination means includes: A character string corresponding to the second item is determined from among character strings corresponding to the first item. 2. The information processing device according to configuration 1.

[0191] (Configuration 3) the character string corresponding to the first item and the character string corresponding to the second item are numeric strings, The second determination means includes: A number string obtained by performing a calculation based on the algorithm on the number string corresponding to the first item is determined as a number string corresponding to the second item. 2. The information processing device according to configuration 1.

[0192] (Configuration 4) The trained model includes a first trained model generated by machine learning to estimate a document type represented by a document image. 4. The information processing device according to any one of configurations 1 to 3.

[0193] (Configuration 5) The trained model includes a second trained model generated by machine learning to estimate an item corresponding to a character string included in a document image. 5. The information processing device according to configuration 4.

[0194] (Configuration 6) The first determination means includes: determining a type of document represented by the document image using a result obtained by inputting the token sequence into the first trained model; The first item is selected from among items corresponding to character strings included in the document image obtained by inputting the token sequence into the second trained model, and a character string corresponding to the first item included in the document image is determined. 6. The information processing device according to configuration 5.

[0195] (Configuration 7) The trained model is a single trained model generated by machine learning to estimate a document type represented by a document image and an item corresponding to a character string included in the document image. 4. The information processing device according to any one of configurations 1 to 3.

[0196] (Configuration 8) the algorithm is an algorithm in which a judgment condition is set for determining a character string corresponding to the second item, The second determination means determines the character string corresponding to the second item by applying at least one of the document type and the character string corresponding to the first item to the determination condition. 3. The information processing device according to configuration 1 or 2.

[0197] (Configuration 9) a correcting means for correcting the character string determined by the second determining means to a character string designated by a user; and updating means for updating the information relating to the judgment conditions based on the content modified by the user. 9. The information processing device according to configuration 8.

[0198] (Configuration 10) The display control means further includes a display control means for displaying the character string determined by the second determination means on a display unit. 10. The information processing device according to any one of configurations 1 to 9.

[0199] (Configuration 11) The second determination means further determines a character string that is a lower-rank candidate of the character string corresponding to the second item; The display control means further displays character strings of the lower-ranked candidates, The method further includes a correction means for correcting, when a character string of the lower-level candidate is selected by a user, the character string of the lower-level candidate so as to become a character string corresponding to the second item. 11. The information processing device according to configuration 10.

[0200] (Configuration 12) The second item is made up of a plurality of items, The second determination means further determines character strings that are lower-rank candidates of character strings corresponding to the plurality of items; the display control means further displays character strings of the lower-ranked candidates corresponding to the plurality of items; The method further includes a correction means for correcting the character strings corresponding to each of the plurality of items by using the character strings of the lower-level candidates corresponding to the plurality of items when the user selects a character string of the lower-level candidates corresponding to one of the plurality of items. 11. The information processing device according to configuration 10.

[0201] (Configuration 13) The second item is an item representing a destination of the document represented by the document image and an item representing a source of the document. 13. The information processing device according to any one of configurations 1 to 12.

[0202] (Configuration 14) the document represented by the document image is a document related to the sale of goods, The first item is composed of a plurality of items including an item representing the seller's company name or the name of a person in charge and an item representing the buyer's company name or the name of a person in charge. 14. The information processing device according to configuration 13.

[0203] (Configuration 15) An acquisition step of acquiring a token sequence generated based on a character string included in a document image; a first determination step of determining a document type represented by the document image and a character string corresponding to a first item included in the document image using a result obtained by inputting the token sequence into a trained model; a second determination step of applying a rule-based algorithm to the document type and a character string corresponding to the first item to determine a character string corresponding to a second item included in the document image and subordinate to the first item; 13. An information processing method comprising:

[0204] (Configuration 16) A program for causing a computer to execute each of the means of the information processing device according to any one of configurations 1 to 14. [Explanation of symbols]

[0205] 130 Information processing equipment 131 Information Processing Section

Claims

1. A means for obtaining a token sequence generated based on a string contained in a document image, A first determination means that determines the document type represented by the document image and the string corresponding to the first item contained in the document image, using the results obtained by inputting the token sequence into a trained model, A second determination means for determining a string corresponding to a second item by applying the string corresponding to the document type and the first item to a rule-based algorithm, An information processing device characterized by having the following features.

2. The second determination means is, From among the strings corresponding to the first item, determine the string corresponding to the second item. The information processing apparatus according to feature 1.

3. The string corresponding to the first item and the string corresponding to the second item are sequences of numbers. The second determination means is, The sequence of numbers obtained by calculating based on the algorithm for the sequence of numbers corresponding to the first item is determined to be the sequence of numbers corresponding to the second item. The information processing apparatus according to feature 1.

4. The aforementioned trained model includes a first trained model generated by machine learning to estimate the document type represented by a document image. The information processing apparatus according to feature 1.

5. The aforementioned trained model includes a second trained model generated by machine learning to estimate items corresponding to strings contained in document images. The information processing apparatus according to feature 4.

6. The first determination means is, Using the results obtained by inputting the token sequence into the first trained model described above, the type of document represented by the document image is determined. The first item is selected from among the items corresponding to the strings contained in the document image obtained by inputting the token sequence into the second trained model, and the string corresponding to the first item contained in the document image is determined. The information processing apparatus according to feature 5.

7. The aforementioned trained model is a single trained model generated by machine learning to estimate the document type represented by the document image and the items corresponding to the strings contained in the document image. The information processing apparatus according to feature 1.

8. The aforementioned algorithm is an algorithm in which a determination condition is set for determining the string corresponding to the second item, The second determination means determines the string corresponding to the second item by applying the document type and at least one of the strings corresponding to the first item to the determination condition and making a determination. The information processing apparatus according to feature 1.

9. A modification means for modifying the string determined by the second determination means to a string specified by the user, The system further includes an update means for updating the information relating to the aforementioned determination conditions based on the content modified by the user. The information processing apparatus according to feature 8.

10. The display unit further includes a display control means for displaying the string determined by the second determination means. The information processing apparatus according to feature 1.

11. The second determination means further determines the lower candidate strings of the string corresponding to the second item, The display control means further displays the lower candidate strings, If the user selects one of the lower-ranking candidate strings, the system further includes a modification means for modifying the lower-ranking candidate string so that it corresponds to the string of the second item. The information processing apparatus according to feature 10.

12. The second item consists of multiple items, The second determination means further determines lower candidate strings for the strings corresponding to the plurality of items, The display control means further displays the lower candidate strings corresponding to the plurality of items, If the user selects one of the sub-candidate strings corresponding to one of the multiple items, the system further includes a modification means for modifying the strings corresponding to each of the multiple items using the sub-candidate strings corresponding to the multiple items. The information processing apparatus according to feature 10.

13. The second item is an item representing the recipient of the document represented by the document image, and an item representing the source of the document. The information processing apparatus according to feature 1.

14. The document represented by the aforementioned document image is a document relating to the sale of goods. The first item described above consists of multiple items, including an item representing the seller's company name or contact person's name, and an item representing the buyer's company name or contact person's name. The information processing apparatus according to feature 13.

15. A retrieval step to obtain a token sequence generated based on the string contained in the document image, A first determination step in which the document type represented by the document image and the string corresponding to the first item contained in the document image are determined using the results obtained by inputting the token sequence into a trained model, A second determination step involves applying the strings corresponding to the document type and the first item to a rule-based algorithm to determine the string corresponding to the second item, An information processing method characterized by having the following features.

16. A program for causing a computer to execute each of the means of the information processing apparatus described in any one of claims 1 to 14.

Images