Information processing device, information processing system, information processing method, and program

By generating pseudo document data and retraining the model on failed extractions, the system addresses the inefficiency of existing methods, improving the accuracy and efficiency of named entity extraction.

JP2024163530AInactive Publication Date: 2024-11-22CANON KK
View PDF 1 Cites 0 Cited by

Patent Information

Application Number
JP2023079238
Authority / Receiving Office
JP · JP
Patent Type
Applications
Current Assignee / Owner
Filing Date
2023-05-12
Publication Date
2024-11-22
Estimated Expiration
Not applicable · inactive patent

AI Technical Summary

Technical Problem

Existing technologies for named entity extraction require large amounts of training data and time due to the use of pseudo document image data generated by data augmentation, leading to low training efficiency.

Method used

A system that generates pseudo document data and relearns the learning model based on failed extractions, increasing the probability of selecting the failed extraction target items, and uses this data to improve the model's accuracy.

Benefits of technology

The system efficiently trains the model using pseudo document data, enhancing its ability to accurately extract named entities from document images.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure 2024163530000001_ABST
    Figure 2024163530000001_ABST
Patent Text Reader

Abstract

To efficiently implement training of a pre-trained model using pseudo document data.SOLUTION: A learning device detects a content in which a pre-trained model has failed in extraction of an item value, and analyses a factor of failure of the item value extraction of the pre-trained model. The learning device reflects the analysis result to a database for generating a learning data set, and generates a learning data set on the basis of the database to which the analysis result is reflected. The learning device implements learning of the replicated model produced by replicating the pre-trained model, and generates a candidate of the pre-trained model. The learning device extracts a character string data being an extraction object candidate by using the generated candidate of the pre-trained model. When the extracted character string data is the same as character string data being the extraction objects determined for all of the extraction object items, the learning device updates the pre-trained model with the candidate of the pre-trained model.SELECTED DRAWING: Figure 6
Need to check novelty before this filing date? Find Prior Art

Description

[Technical field]

[0001] The present invention relates to an information processing technique for extracting named entities from document data. [Background technology]

[0002] Conventionally, there is a technology that extracts character strings of item values ​​corresponding to predetermined items to be extracted (e.g., document number, company name, date, amount, title) from document images called semi-standard forms (e.g., invoices, estimates, purchase orders) that are created with different layouts for each issuing company. This technology is generally realized by optical character recognition (OCR) and named entity recognition (NER). That is, first, the character strings resulting from character recognition (OCR) of the document image are input, and morphological analysis is performed based on the feature amount expressed by the embedding vector of the character string. Then, named entity recognition (NER) is performed on the result, and a predetermined label (e.g., company name) is assigned to the character string corresponding to the item value to be extracted, and the result is output. The task of named entity extraction is generally realized by machine learning, and therefore a large amount of learning data with previously labeled character strings is required.

[0003] In Patent Document 1, in order to improve the accuracy of form recognition, the types of new forms and variant forms are registered based on the form recognition results, and a learning model that has learned the registered form types is generated. The document image data to be trained by the learning model uses multiple patterns of pseudo document image data generated by data augmentation of image data obtained by scanning an actual form. This allows the learning model to learn not only the scanned form but also similar forms. [Prior art documents] [Patent documents]

[0004] [Patent Document 1] JP 2021-131659 A Summary of the Invention [Problem to be solved by the invention]

[0005] However, in the technology of Patent Document 1, when pseudo-document image data generated by data augmentation is used for training, the improvement in recognition accuracy is low compared to when document image data obtained by scanning actual forms is used, and more training data and training time are required. [Means for solving the problem]

[0006] The present invention is an information processing device comprising: a generation means for generating pseudo-document data from input document data when a learning model that extracts named entities from input document data fails to extract named entities from the input document data; and a learning means for re-training the learning model based on learning data consisting of pairs of the pseudo-document data generated by the generation means and / or the input document data from which the named entity extraction failed and correct answer data, wherein the generation means increases the probability that the failed extraction target items will be selected when generating the pseudo-document data. Effect of the Invention

[0007] According to the present invention, it is possible to efficiently train a trained model for extracting named entities using pseudo document data. [Brief description of the drawings]

[0008] [Figure 1] FIG. 1 is a diagram illustrating a configuration of an information processing system. [Diagram 2] FIG. 2 is a diagram illustrating the configuration of an information processing device, a learning device, and an information processing server. [Diagram 3] FIG. 2 is a diagram showing a usage sequence of the information processing system in the first embodiment. [Figure 4] 13 is a flowchart showing a process for generating an item value extraction model. [Diagram 5]10 is a flowchart showing a process of estimating item value candidate character strings and accepting corrections thereto from a user in the first embodiment. [Figure 6] 13 is a flowchart showing a process for reflecting user corrections in an item value extraction model in the first embodiment. [Figure 7] 13 is a flowchart showing a process for detecting the details of a failure when an item value extraction fails. [Figure 8] 13 is a flowchart showing a process for analyzing the cause of failure in item value extraction. [Figure 9] 11 is a flowchart for acquiring a result of an analysis of factors when generating a document image. [Figure 10] 13 is a flowchart illustrating an example of a flow for generating a learning dataset. [Figure 11] FIG. 13 is a diagram showing an example of a screen on which a user corrects an extraction result. [Figure 12] FIG. 13 is a diagram showing a specific example of a generated document image. [Figure 13] FIG. 13 is a diagram illustrating an example of a database that stores names of items to be extracted and items not to be extracted. [Figure 14] FIG. 13 is a diagram showing a specific example of a corresponding pattern when analyzed as a layout factor. [Figure 15] FIG. 13 is a diagram showing a specific example of a corresponding pattern when a non-layout factor analysis is performed; [Figure 16] FIG. 11 is a diagram illustrating a method for calculating a matching rate of a layout. [Figure 17] 13 is a diagram showing an example of a screen for notifying a result of reflection of correction contents in an item value extraction model. FIG. [Figure 18] FIG. 11 is a diagram showing a usage sequence of an information processing system in the second embodiment. [Figure 19] 13 is a flowchart showing a process of estimating item value candidate character strings and accepting corrections thereto from a user in the second embodiment. [Figure 20] 13 is a flowchart showing a process for reflecting user corrections in an item value extraction model in the second embodiment. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0009] Hereinafter, an embodiment of the present invention will be described with reference to the drawings. Note that the components described in the present embodiment are merely examples and are not intended to limit the scope of the present invention.

[0010] [Embodiment 1] <Information Processing System> 1 is a diagram showing an example of the configuration of an information processing system. As shown in FIG. 1, the information processing system 100 is composed of, for example, an information processing device 101, a learning device 102, and an information processing server 103, which are connected to each other via a network 104. In the information processing system 100, the information processing device 101, the learning device 102, and the information processing server 103 may be configured to be connected to the network 104 in a multi-connection manner rather than a single connection. For example, the information processing server 103 may be composed of a first server device having high-speed computing resources and a second server device having large-capacity storage, which are connected to each other via the network 104.

[0011] The information processing device 101 is realized by an MFP (Multi-Function Peripheral) having multiple functions such as printing, scanning, and faxing, and also functions as an image acquisition unit. The information processing device 101 can, for example, optically read an original 111 printed on a storage medium such as paper by the image acquisition unit, and transmit document image data 113 generated by performing a predetermined scan image processing to the information processing server 103. The information processing device 101 can also, for example, receive FAX data 112 transmitted from a FAX transmitter (not shown) by the image acquisition unit, and transmit document image data 113 generated by performing a predetermined FAX image processing to the information processing server 103. The information processing device 101 may be realized by a PC (Personal Computer) or the like other than an MFP having the above-mentioned scanning and faxing functions. Specifically, for example, document image data 113 such as PDF or JPEG generated by using a document creation application operating on the PC as the information processing device 101 may be transmitted to the information processing server 103.

[0012] The learning device 102 functions as a learning data generation unit that generates, as learning document data, document data in which a correct answer label is assigned to a character string to be extracted from a group of character strings included in a document image, based on a plurality of document image samples 114 provided by an engineer. Furthermore, the learning device 102 functions as a learning unit that uses the learning document data generated by the learning data generation unit as learning data and trains a trained model 115 that estimates a character string to be extracted included in the training document data.

[0013] The information processing server 103 functions as an information processing unit that extracts predetermined character string data 116 to be extracted after acquiring a plurality of character strings included in the document image data 113 received from the information processing device 101, and as a storage unit that stores the extracted results. First, the information processing unit of the information processing server 103 executes OCR processing on the document image data 113, and acquires a plurality of character string data as a character recognition result obtained by the OCR processing. Furthermore, the information processing unit of the information processing server 103 uses the trained model 115 to classify and extract the predetermined character string data 116 to be extracted from the plurality of character string data acquired from the document image data 113. Here, the character string data 116 to be extracted deals with proper nouns such as personal names and place names that have various expressions for each country and language, which are generally called named entities, date expressions, and monetary expressions. Items to be extracted include, for example, company names, issue dates, total amounts, and document names.

[0014] The network 104 is realized by a LAN, a WAN, or the like, and is a communication unit that connects the information processing device 101, the learning device 102, and the information processing server 103 to one another and transmits and receives data between the devices.

[0015] <Device configuration> FIG. 2 is a diagram showing an example of the hardware configuration of the information processing device 101, the learning device 102, and the image processing server 103 for implementing the information processing system 100 in FIG.

[0016] 2(a) is a diagram showing an example of the hardware configuration of the information processing device 101. As shown in FIG. 2(a), the information processing device 101 is composed of a CPU 201, a ROM 202, a RAM 204, a printer device 206, a scanner device 206, a storage 208, an external interface 211, and the like, which are connected to each other via a data path 203.

[0017] The CPU 201 is a control unit for controlling the overall operation of the information processing device 101. The CPU 201 executes a startup program stored in a ROM 202 to start up the system of the information processing device 101, and executes a control program stored in a storage 208 to realize functions of the information processing device 101, such as printing, scanning, and faxing.

[0018] The ROM 202 is realized by a non-volatile memory, and is a storage unit for storing a startup program for starting up the information processing device 101. The RAM 204 is realized by a volatile memory, and is a storage unit used as a work memory when the CPU 201 executes a control program. The storage 205 is realized by an HDD (Hard Disk Drive) or the like, and is a storage unit for storing the above-mentioned control program and document images. The data bus 203 is a communication unit for transmitting and receiving data between devices constituting the information processing device 101.

[0019] The printer device 206 is an image output device and a processing unit for printing and outputting a document image inside the information processing device 101 on a storage medium such as paper. The scanner device 207 is an image input device and a processing unit for optically reading a storage medium such as paper on which characters, figures, etc. are printed and acquiring the document image. The document transport device 208 is realized by an ADF (Auto Document Feeder) or the like and is a processing unit for detecting a document placed on a document table and transporting the detected document one sheet at a time to the scanner device 207.

[0020] The input device 209 is realized by a touch panel, hard keys, or the like, and is a device for a user to input operations to the information processing device 101. The display device 210 is realized by a liquid crystal display, or the like, and is a device for displaying and outputting a setting screen of the information processing device 101 to the user. The external interface 211 connects the information processing device 101 and the network 104, and is an interface unit for receiving FAX data from a FAX transmitter (not shown) and transmitting document images to the information processing server 103.

[0021] 2(b) is a diagram showing the configuration of the learning device 102. As shown in FIG. 2(b), the learning device 102 is composed of a CPU 231, a ROM 232, a RAM 234, a storage 235, an input device 236, a display device 237, an external interface 238, and a GPU 232, which are connected to each other via a data bus 233.

[0022] The CPU 231 is a control unit for controlling the overall operation of the learning device 102. The CPU 231 executes a boot program stored in the ROM 232 to start up the system of the learning device 102, and executes a learning program stored in the storage 208 to generate a trained model 115 that extracts named entities from character string data. The GPU 232 is a calculation unit configured with an image processing processor. The GPU 232 executes calculations for generating the trained model 115 based on character string data included in a given document image, for example, in accordance with a control command provided from the CPU 231.

[0023] The ROM 233 is realized by a non-volatile memory, and is a storage unit for storing a boot program that starts up the learning device 102. The RAM 234 is realized by a volatile memory, and is a storage unit used as a work memory when the CPU 231 executes the learning program. The storage 235 is realized by an HDD (Hard Disk Drive) or the like, and is a processing unit for accepting operation inputs to the learning device 102 from engineers. The data bus 233 is a communication unit for transmitting and receiving data between devices constituting the learning device 102.

[0024] The input device 236 is realized by a mouse, a keyboard, or the like, and is a device for an engineer to input operations to the learning device 102. The display device 237 is realized by a liquid crystal display, or the like, and is a device for displaying and outputting a setting screen of the learning device 102 to the engineer. The external interface 238 connects the learning device 102 and the network 104, and is an interface unit for receiving document images 114 from a PC (not shown) or the like, and for transmitting the trained model 115 to the information processing server 103.

[0025] 2(c) is a diagram showing the configuration of the information processing server 103. As shown in FIG. 2(c), the information processing server 103 is composed of a CPU 261, a ROM 262, a RAM 264, a storage 265, an input device 266, a display device 267, and an external interface 268, which are connected to each other via a data bus 263.

[0026] The CPU 261 is a control unit for controlling the overall operation of the information processing server 103. The CPU 261 starts up the system of the information processing server 103 by executing a boot program stored in the ROM 262, and executes an information processing program stored in the storage 265 to perform information processing such as character recognition and information extraction.

[0027] The ROM 262 is realized by a non-volatile memory, and is a storage unit for storing a boot program that starts up the information processing server 103. The RAM 264 is realized by a volatile memory, and is a storage unit used as a work memory when the CPU 261 executes the information processing program. The storage 265 is realized by an HDD (Hard Disk Drive) or the like, and is a storage unit for storing the above-mentioned information processing program, the document image data 113, the trained model 115, the character string data 116, and the like. The data bus 263 is a communication unit for transmitting and receiving data between devices constituting the information processing server 103.

[0028] The input device 266 is realized by a mouse, a keyboard, or the like, and is a device for performing operational input to the information processing server 103. The display device 267 is realized by a liquid crystal display, or the like, and is a device for displaying and outputting the item value extraction results by the information processing server 103. The external interface 268 connects the information processing server 103 and the network 104, and is an interface unit for receiving the trained model 115 from the learning device 102 and the document image data 113 from the information processing device 101.

[0029] <Usage sequence> FIG. 3 is a diagram showing a usage sequence of the information processing system 100 of FIG.

[0030] Fig. 3(a) is a diagram for explaining a flow in which the learning device 102 generates the trained model 115. Note that a detailed processing flow of Fig. 3(a) will be described later with reference to Fig. 4.

[0031] In S301, the learning device 102 acquires a plurality of document image samples 114 for generating the trained model 115. The document image samples 114 may be input by an engineer who develops the information processing system 100, or the document image samples 114 may be acquired by manually inputting the document image samples 114 by an engineer each time the information processing system 100 is executed. Alternatively, the document image samples 114 may be acquired from a predetermined storage device such as the storage 265 in the information processing server 103, or may be acquired from a second information processing system that automatically collects the document image samples 114.

[0032] In S302, the learning device 102 performs block selection (BS) processing, optical character recognition (OCR) processing, and named entity extraction (NER) processing on the named entity character strings on the document image sample 114 acquired in S301. Then, the learning device 102 assigns an item name label corresponding to the string data to be extracted as a correct label. Here, the assignment of the correct label may be performed manually by an engineer, or may be automated using a trained model that performs trained item value extraction that has previously learned document data. Then, the learning device 102 generates a learning dataset in which a correct label is assigned to the named entity character string data for each document image.

[0033] In S303, the learning device 102 generates a trained model trained using the training data set generated in S302.

[0034] In S304, the learning device 102 transmits the trained model generated in S303 to the information processing server 103, and the information processing server 103 stores the received trained model in the storage 265 within the information processing server 103.

[0035] Fig. 3(b) is a diagram explaining the following two processes performed by the information processing server 103. In the first process, in response to a request from a user, the information processing server 103 extracts and outputs character string data that is a candidate for extraction from among a plurality of character string data obtained from the document image data 113. In the second process, the information processing server 103 transmits to the learning device 102 character string data that is a candidate for extraction and character string data that is a correct answer data and is determined to be an extraction target based on an operation from the user. The detailed flow of Fig. 3(b) will be described later with reference to Fig. 5.

[0036] In S305, a user who uses the information processing system 100 places the document 111 on the information processing device 101 and instructs the information processing device 101 to scan the document 111 in order to obtain character string data to be extracted from the document 111.

[0037] In S306 , the information processing apparatus 101 transmits the document image data 113 acquired by scanning the original 111 to the information processing server 103 .

[0038] In S307, the information processing server 103 performs OCR processing on the document image data 113 transmitted in S306, and acquires character string data included in the document image data 113. Then, the information processing server 103 uses the trained model 115 generated in S303 to extract character string data that is a candidate for extraction.

[0039] In S308, the information processing server 103 outputs the character string data of the extraction target candidates extracted in S307 to the information processing device 101, and the information processing device 101 displays the character string data of the extraction target candidates output from the information processing server 103 on the display device 210.

[0040] In S309, the user of the information processing system 100 checks whether the character string data of the extraction target candidate displayed in S308 is the same as the character string data of the extraction target, i.e., whether the item value extraction has been successful. If the character string data of the extraction target candidate displayed in S308 is different from the character string data of the extraction target, the user uses the input device 209 of the information processing device 101 to correct the character string data of the extraction target candidate displayed in S308 and confirm the character string data of the extraction target. If the character string data displayed in S308 is the same as the character string data of the extraction target, the user uses the input device 209 of the information processing device 101 to confirm the character string data of the extraction target candidate displayed in S308 as the character string data of the extraction target. Then, the information processing device 101 transmits the confirmed character string data of the extraction target to the information processing server 103.

[0041] Here, the display of the character string data in S308 and the confirmation and correction of the character string data in S309 will be described in more detail with reference to FIG. 11. FIG. 11 is a diagram showing an example of a screen for a user using the information processing system 100 to confirm and correct the character string data output in S308. The confirmation screen 1100 of the display device 210 includes a preview image display screen 1101, a result display screen 1102, and a "Save" button 1132. The result display screen 1102 displays the character string data output to the information processing device 101 in S308, such as an issue date 1121, a recipient company name 1123, and an issuer person in charge name 1128. In the example shown in FIG. 11, the image areas from which the character string extraction results (1121, 1123, 1128) are extracted are highlighted as an issue date 1111, an issuer company name 1113, and an issuer person in charge name 1118 on the preview image display screen 1101 and the result display screen 1102. Highlighting may be performed on only one of the preview image display screen 1101 and the result display screen 1102, or highlighting may not be performed on either screen.

[0042] Here, consider a case where the extraction target of "issuer person in charge name" is corrected to the image area 1117. The user operates the pointer 1131 with the input device 209 to select the character string extraction result 1128 as the source of correction and select the image area 1117 as the destination of correction. Note that instead of selecting the character string extraction result 1128 as the source of correction, the character string extraction result 1118 may be selected as the source of correction. Also, before selecting the source of correction and the destination of correction, the "correct" button may be pressed. After the image area 1117 of the destination of correction is selected, the information processing device 101 acquires "Dana Morgan" obtained by performing character recognition processing on the image area 1117 as the character string extraction result 1128. The user confirms the correction of the character string extraction result 1128 by pressing the "save" button 1132. Note that the operation of confirming the correction may be combined with the operation of selecting the image area 1117 of the destination of correction with the pointer 1131, for example, and is not limited to pressing the "save" button 1132.

[0043] In S310, the information processing server 103 checks whether the character string data extracted in S307 is identical to the character string data to be extracted transmitted in S309 for all items to be extracted. If the character string data differs for one or more items to be extracted, it is determined that the trained model stored in the storage 265 in the information processing server 103 needs to be updated, and the process proceeds to S311. If the character strings are identical for all items to be extracted, there is no need to update the trained model stored in the storage 265 in the information processing server 103, and the information processing server 103 ends the process.

[0044] In S311, the information processing server 103 transmits to the learning device 102 the document image data 113 transmitted in S306, the character string data to be the extraction target candidates extracted in S307, and the character string data to be the confirmed extraction target transmitted in S309. Fig. 3(c) is a diagram for explaining the process in which the learning device 102 trains a trained model based on the correction contents by the user to the character string data of the extraction target candidates. Note that the detailed flow of Fig. 3(c) will be described later with reference to Fig. 6.

[0045] In S312, the learning device 102 detects the details of the failure of the trained model 115 to extract item values ​​from the character string data that is the extraction target candidate transmitted in S311 and the confirmed extraction target character string data. Here, "failure to extract item values" refers to a case where the character string data that is the extraction target candidate does not match the confirmed extraction target character string data for even one of the extraction target items. Also, "failure details" is information indicating which item failed to extract item values ​​and what the correct item value was. Details of this process will be described later with reference to FIG. 7.

[0046] In S313, the learning device 102 analyzes and identifies the cause of the failure to extract the item value based on the details of the failure of the trained model 115 to extract the item value detected in S312. Details of this process will be described later with reference to FIG.

[0047] In S314, the learning device 102 generates a document image sample, which is pseudo document image data, based on a database reflecting the cause of the failure in the item value extraction from the analysis result of the cause of the failure in the item value extraction identified in S313. Details of this process will be described later with reference to Figs. 9 and 10.

[0048] In S315, the learning device 102 uses the pseudo document image sample generated in S314 to acquire character string data and item values ​​to be extracted in the document image, and generates a learning dataset. Details of this process will be described later with reference to FIG.

[0049] In S316, the learning device 102 generates a candidate trained model by re-training a duplicated model, which is a duplicate of the trained model 115, using the training dataset generated in S315.

[0050] In S317, the learning device 102 acquires character string data included in the document image data 113 transmitted in S311, and extracts character string data of the extraction target candidate using the trained model candidate generated in S316. Based on the confirmed string data of the extraction target transmitted in S311, the learning device 102 checks whether the character string data of the extraction target candidate by the trained model candidate generated in S316 is identical to the character string data of the confirmed extraction target. If the character string data of the extraction target candidate extracted by the trained model candidate is identical to the confirmed string data of the extraction target, the learning device 102 proceeds to S318, and if they are different, returns to S314.

[0051] In S318, the learning device 102 transmits the trained model candidate generated in S316 to the information processing server 103, stores it as the trained model 115 in the storage 265 in the information processing server 103, and updates the trained model 115.

[0052] In S319, the information processing server 103 notifies the information processing system 100 that the trained model 115 has been updated in the storage 265 within the information processing server 103, and the information processing system 100 notifies the user using the display device 210 or the like.

[0053] <Generation of trained model for extracting item values> 4 is a flowchart illustrating the flow in which the learning device 102 generates the trained model 115 in FIG. 3(a). Note that the execution program for each step shown in FIG. 4 is stored in the ROM 232, the RAM 234, or the storage 235 of the learning device 102, and is executed by the CPU 231 or the GPU 232 of the learning device 102.

[0054] 3 and stored in the RAM 234 or the storage 235. Specifically, for example, document image samples 114 called semi-standard forms such as invoices, estimates, and purchase orders, which are created with different layouts for each issuing company, are acquired.

[0055] In S402, the CPU 231 executes BS processing and OCR processing on the document image sample 114 read in S401, and acquires a string data group of the character recognition results constituting the document image sample 114. Here, each string data included in the acquired string data group may be handled in units of word segments arranged with spaces or separated by lines in the document image, for example, using a technique such as BS that identifies object units constituting the document image of a semi-standard form. Also, each string data included in the acquired string data group may be handled in units of word segments divided using a morphological analysis technique on a sentence constituting the document image of an unstandard form.

[0056] In S403, the CPU 231 assigns an extraction target item label as a correct answer label to the character string data to be extracted that is included in the character string data group acquired in S402. Specifically, for example, consider a case where the character string data group acquired in S402 includes character string data representing "November 24, 2022" indicating the publication date of the document. In this case, the CPU 231 assigns a label of "Publication Date", which is the extraction target item name, to the character string data representing "November 24, 2022", which is the extraction target.

[0057] In S404, the CPU 231 generates a trained model for extracting item values ​​by machine learning using the string data group acquired in S402 and the extraction target item labels assigned in S403. The input to the trained model may be a feature vector of the string data converted using known techniques such as Word2Vec, fastText, BERT, XLNet, ALBERT, and the like, and the position coordinates of the string data on the document image sample 114. Specifically, for example, a single string data can be converted into a feature vector expressed by a 768-dimensional numerical value by using a language model of BERT that has been trained in advance on a general sentence (e.g., the full text of an article on Wikipedia). In addition, the output of the trained model is trained so that the labels of the extraction target items assigned in S403 can be output, and any of the predetermined extraction target item labels is output as an inference result by the trained model for the input string data. Here, the trained model may be a logistic regression, a decision tree, a random forest, a support vector machine, a neural network, or the like that are generally known as machine learning algorithms. Specifically, the trained model only needs to be able to output an inference result for any of the predetermined labels of items to be extracted, based on the output value of the fully connected layer of a neural network to which a feature vector output by, for example, the BERT language model is input.

[0058] In S405, the CPU 231 stores the trained model generated in S404 in the storage 265 in the information processing server 103.

[0059] <Extraction and correction of character string data that is a candidate for extraction> 5 is an example of a flowchart illustrating a flow in which the information processing server 103 estimates character string data to be extraction candidates from the document image data 113 using the trained model 115 and determines the character string data to be extracted in the step of FIG. 3(b). Note that an execution program for each step shown in FIG. 5 is stored in any one of the ROM 262, the RAM 264, or the storage 265 of the information processing server 103 and is executed by the CPU 261 of the information processing server 103.

[0060] In S501, the CPU 261 acquires the trained model 115 stored in the storage 265 in S304 shown in FIG. 3 and S405 shown in FIG.

[0061] In S502, the CPU 261 acquires the document image data 113 transmitted from the information processing apparatus 101 in S306 shown in FIG. 3, as displayed on a preview image display screen 1101 shown in FIG.

[0062] In S503, the CPU 261 executes the BS process and the OCR process on the document image data 113 read out in S502, and acquires a character string data group included in the document image data 113.

[0063] In S504, the CPU 261 uses the trained model 115 read out in S501 to extract character string data that is a candidate for extraction from a group of character string data included in the document image data 113 acquired in S503. Note that the technology for extracting character string data that is a candidate for extraction is generally known as a classification task called NER (Named Entity Recognition), and can be realized by a machine learning algorithm that uses features of images and natural language.

[0064] In S505, the CPU 261 transmits the character string data extracted in S504 as candidates for extraction to the information processing device 101, and presents the character string data as candidates for extraction to the user by displaying the screen shown in FIG. 11 on the display device 210 of the information processing device 101.

[0065] In S506, the CPU 261 acquires character string data to be extracted that has been confirmed by the user's operation on the screen shown in FIG.

[0066] In S507, the CPU 261 checks whether or not the character string data to be extracted acquired in S506 is identical to the character string data to be extracted in S504 for all items to be extracted. If they are different for one or more items (NO in S507), the trained model 115 stored in the storage 265 in the information processing server 103 needs to be updated, so the CPU 261 proceeds to S508. If they are identical for all items to be extracted (YES in S507), the trained model 115 stored in the storage 265 in the information processing server 103 does not need to be updated, so the CPU 261 ends the process.

[0067] 11 is pressed, the CPU 261 saves the document image data 113 acquired in S502, the character string data that is the extraction target candidate extracted in S504, and the character string data that is the extraction target confirmed in S506. The save destination is the storage 265 in the information processing server 103.

[0068] <Updating trained models> 6 is a flowchart showing a process of reflecting user corrections in a trained model. Note that an execution program for each step shown in FIG. 6 is stored in the ROM 232, the RAM 234, or the storage 235 of the training device 102, and is executed by the CPU 231 or the GPU 232 of the training device 102.

[0069] In S601, the CPU 231 detects the failure of the trained model 115 to extract an item value. Details of this process will be described later with reference to FIG.

[0070] In S602, based on the content detected in S601, the CPU 231 analyzes the cause of the failure of the trained model 115 to extract the item value. Details of this process will be described later with reference to FIG.

[0071] In S603, the CPU 231 reflects the result of the factor analysis performed in S602 in a database for generating a learning data set. Details of this process will be described later with reference to FIG.

[0072] In S604, the CPU 231 generates a learning dataset based on a database reflecting the analysis result of the reason why the trained model 115 failed to extract the item value in S603. Details of this process will be described later with reference to FIG.

[0073] In S605, the CPU 231 generates a trained model candidate by re-training a duplicated model obtained by duplicating the trained model 115 using the training data set generated in S604. The method of generating the trained model candidate is almost the same as that described above with reference to Fig. 4, except that a duplicated model of the trained model 115 is used, and the document image sample 114 acquired in S401 may be replaced with the training data set generated in S604.

[0074] In S606 , the CPU 231 acquires the document image data 113 stored in the storage 265 from the information processing server 103 .

[0075] In S607, the CPU 231 acquires a character string data group contained in the document image data 113 from the document image data 113 acquired in S606. The method of acquiring the character string data group is the same as that described above with respect to S503 shown in FIG. 5, and therefore will not be described here.

[0076] In S608, the CPU 231 uses the trained model candidates generated in S605 to extract character string data that are candidates for extraction targets from the character string data group acquired in S607. Note that the method for extracting character string data that are candidates for extraction targets is the same as described above with respect to S504 shown in FIG. 5, and therefore will not be described here.

[0077] In S609, the CPU 231 checks whether or not the character string data acquired in S608 is identical to the confirmed character string data to be extracted, included in the information transmitted from the information processing server 103 in S311, for all items to be extracted. If the character string data is different for one or more items (NO in S609), the CPU 231 proceeds to S610, and if the character string data is identical for all items to be extracted (YES in S609), the CPU 231 proceeds to S612. Note that the checking method is the same as S507 shown in FIG. 5, and therefore description thereof will be omitted.

[0078] In S610, the CPU 231 judges whether or not to end the trial for updating the model in the learning device 102. The criterion for the end judgment is, for example, whether or not the occurrence probability of the character string data to be extracted reflected in the database in S603 in the learning data set generated in S604 by performing S601 to S603 reaches an upper limit. Here, the occurrence probability indicates the probability that the character string data is selected as an item value for an item such as "document number" or "total amount" when generating the learning data set as shown in FIG. 10, and is defined as a ratio of 0 to 1. The upper limit may be 1, or any predetermined value within the range of 0 to 1 may be used. Another criterion for the end judgment is to specify the number of repetitions. If the trial is to be ended (YES in S610), it indicates that the learning device 102 cannot handle it any more, so proceed to S611. If the trial is to be continued (NO in S610), return to S603.

[0079] In S611, since it is indicated that the learning device 102 cannot handle the problem any further, the CPU 231 transmits feedback contents to the engineer developing the information processing system 100. The feedback contents are the processing history of S601 to S613 excluding S611, and data associated with the processing history, such as the document image data 113 acquired in S606 and the correct answer data acquired in S601 (S702 in FIG. 7). When transmitting the feedback, the CPU 231 displays a transmission confirmation screen 1710 as shown in FIG. 17(b) on the display device 210 of the information processing device 101, etc., and presents it to the user. FIG. 17(b) is an example of a screen for requesting permission to transmit the feedback contents to the engineer developing the information processing system 100. The transmission confirmation screen 1710 is composed of a check box 1712 for automatically determining whether or not transmission is permitted, a "Yes" button 1713, and a "No" button 1714, and accepts designation by an operation input 1711 from the input device 209. Only when the "Yes" button 1713 is pressed by the user on the transmission confirmation screen 1710, the CPU 231 transmits the feedback content to a predetermined destination. At that time, after transmitting the feedback content, the CPU 231 displays a transmission notification screen 1720 as shown in FIG. 17(c) on the display device 210 of the information processing device 101, etc., to notify the user who uses the information processing system 100. On the other hand, when the "No" button 1714 is pressed by the user on the transmission confirmation screen 1710, the CPU 231 does not transmit the feedback content and ends the process. Furthermore, when a check box 1712 displayed on the transmission confirmation screen 1710 is checked, the CPU 231 may store the transmission permission result (whether the "Yes" button 1713 or the "No" button 1714 was pressed) in the storage 235. This allows the CPU 231 to determine whether or not to permit transmission based on the transmission permission result acquired from the storage 235, without presenting the transmission confirmation screen 1710 to the user.

[0080] In S612, the CPU 231 updates the trained model 115 stored in the storage 265 in the information processing server 103 with the trained model candidate generated in S605.

[0081] In S613, the CPU 231 notifies the user of the information processing system 100 that the trained model 115 has been updated in S612 by presenting an update notification screen 1700 such as that shown in FIG. 17(a) to the user on the display device 210 or the like.

[0082] Details of the process of S601 will now be described with reference to Fig. 7. Fig. 7 is a flowchart showing a process for detecting the details of a failure when extraction of an item value from the document image data 113 fails.

[0083] In S701, the CPU 231 acquires character string data that is to be an extraction target candidate from the storage 265 in the information processing server 103.

[0084] In S702, the CPU 231 acquires character string data to be extracted from the storage 265 in the information processing server 103.

[0085] In S703, the CPU 231 executes the processes of S704 to S708 for each item (extraction target item) of the character string data that is the extraction target candidate acquired in S701.

[0086] In S704, the CPU 231 determines whether or not the character string data acquired in S702 is the same as the character string data acquired in S701 for the extraction target item specified in S703. If the character string data acquired in S702 is the same as the character string acquired in S701 (YES in S704), the CPU 231 proceeds to S707, and if it is different from the character string data acquired in S701 (NO in S704), the CPU 231 ends the process.

[0087] In S707, the CPU 231 saves the item names of the extraction target items for which the result of the determination in S704 is YES as failed item names in the storage 235. The failed item names are used for the purpose of managing the failed item names, and are saved in, for example, an array format. Note that any format may be used when saving the failed item names as long as they can be managed.

[0088] In S708, the CPU 231 determines whether the process has been completed for all items of character string data of the extraction target candidates acquired in S701, and repeats the processes of S703 to S708 until it is determined that the process has been completed for all items.

[0089] Next, details of the process performed in S602 based on the failed item name detected in S601 will be described with reference to Fig. 8. Fig. 8 is a flowchart showing the process of analyzing the cause of failure in extracting item values ​​from the document image data 113 based on the failed item name detected in S601.

[0090] In S801, the CPU 231 acquires the failed item name from the storage 235.

[0091] In S802, the CPU 231 judges whether the document layout of the document image data 113 is an unknown layout. If the document layout of the document image data 113 is a known layout (NO in S802), the CPU 231 proceeds to S803, and if the document layout is an unknown layout (YES in S802), the CPU 231 proceeds to S804. The layout information of the document image data 113 holds the arrangement position of the area of ​​each sub-template in the document image data 113, and a plurality of patterns are stored as a database in the storage 235. The sub-template indicates each component included in a document, and for example, refers to the area 1210 in which the issuing destination information is collected in FIG. 12, or a collection of "issuance date" and "document number" shown in 1301 and 1311 in FIG. 13. Here, an example of judging whether the layout is unknown will be described with reference to FIG. 14. It is assumed that the layout information 1410 is stored in the storage 235. At this time, by comparing sub-templates for each area on the document image data 113 with sub-templates for each area defined in the layout information 1410 that correspond in layout position, it is determined whether the layout of the document image data 113 is unknown. In the example shown in Fig. 14, an image area 1401 including an issue date on the document image data 113 and an image area 1402 including a document number are arranged horizontally, whereas an issue date area 1411 and a document number area 1412 on the layout information 1410 are arranged vertically. In other words, since the document image data 113 has a different layout from the layout information 1410, if there is no other layout information, the document layout of the document image data 113 is determined to be an unknown layout. Details of the process of determining whether the layout is unknown or not will be described later with reference to Fig. 16.

[0092] In S803, the CPU 231 saves information that the analysis results regarding the layout are "known" in the storage 235. Specifically, in this embodiment, an associative array for managing the analysis results is prepared, and the character string "known" is added as the value for the "layout" key of the array. However, as long as the analysis results regarding the layout can be managed, any format may be used when saving.

[0093] In S804, the CPU 231 saves information that the layout analysis result is "unknown" in the storage 235. Specifically, in this embodiment, an associative array for managing the analysis results is prepared, and the character string "unknown" is added as the value for the "layout" key in the array. However, as in S803, any format may be used when saving as long as the layout analysis result can be managed.

[0094] In S805, the CPU 231 selects one of the failed items acquired in S801, and executes the processes of S806 to S809 for the selected failed item.

[0095] In S806, the CPU 231 determines whether the character string data to be an extraction target candidate acquired in S701 is null character string data. Note that when the character string data to be an extraction target candidate acquired in S701 is null character string data, it refers to a state in which nothing has been extracted for the character number that is an extraction target item, such as character string data 1501 to be an extraction target candidate for the character number shown in Fig. 15(a). If the character string data to be an extraction target candidate acquired in S701 is null character string data (NO in S806), the CPU 231 proceeds to S807, and if it is not null character string data (YES in S806), the CPU 231 proceeds to S808.

[0096] In S807, the CPU 231 saves information that the character string data extracted as an extraction target candidate for one of the failure items selected in S805 is "mis-extracted" in the storage 235. Specifically, in this embodiment, an associative array for managing the analysis results is prepared, and the item name of the failure item selected in S805 is added as the key of the array, and the character string "mis-extracted" is added as the value. However, any format may be used when saving as long as the item name of the failure item and the value can be managed in association with each other as the analysis result for one of the failure items acquired in S805.

[0097] In S808, the CPU 231 saves information that the extracted character string data candidate for extraction is "not extracted" in the storage 235 as an analysis result for one of the failure items selected in S805. Specifically, in this embodiment, an associative array for managing the analysis results is prepared, and the item name of the failure item acquired in S805 is added as the key of the array, and the character string "not extracted" is added as the value. However, as in S807, any format may be used when saving as long as the item name of the failure item and its value can be associated and managed as the analysis result for one of the failure items acquired in S805.

[0098] In S809, the CPU 231 determines whether or not the processes for all the failure items acquired in S801 have been completed, and repeats the processes of S805 to S809 until it is determined that the processes for all the failure items acquired have been completed.

[0099] In S810, the CPU 231 compiles the analysis results for each failed item saved in S807 and S808 and saves them in the storage 235 as analysis results related to item extraction. Specifically, in this embodiment, an associative array for managing the analysis results is prepared, and an item extraction array, which is the analysis results related to the item values ​​for each item to be extracted, is added as a value for the "item extraction" key of that array. However, as with S803, any format may be used when saving as long as the analysis results related to the item values ​​can be managed.

[0100] Here, the process of determining whether or not the layout is unknown in S802 will be described with reference to Fig. 16. The process of determining whether or not the layout is unknown determines the matching rate of the areas between the arrangement positions of the areas of the character strings of each item included in the document image 1100 and the layout information of the document image 1100. Fig. 16 is a diagram for explaining a method of calculating the matching rate of the layout.

[0101] First, as image areas of character strings of items included in document image 1100 in FIG. 16(a), there are document name image area 1601 and document number image area 1602. The matching rates of these areas with intra-template document name area 1611 and intra-template document number area 1412 in template 1410 are obtained. First, as shown in FIG. 16(b), document name image area 1601 is compared with template document name area 1611. In this case, since document name image area 1601 is included in template document name area 1611, the matching rate is 100%. Next, as shown in FIG. 16(c), document number image area 1602 is compared with template document number area 1412. In this case, the rate of area of ​​document number image area 1602 that overlaps with template document number area 1412 is taken as the matching rate. For example, when the coordinates of each region have the position of the upper left vertex and the length of each side shown in FIG. 16(d), the area of ​​the document number image region 1602 is w2×h2=30000. In addition, the area of ​​the region where each region overlaps is 300 in width and 50 in height, so 300×50=15000. Therefore, the matching rate is 15000÷30000×100=50%. The matching rate of the image regions of the character strings of all items with the corresponding regions in the template is calculated, and if the average of the sum of the calculated matching rates is less than a predetermined threshold, it is determined to be an unknown layout. The threshold is 75% as an example, but any value may be used.

[0102] Details of the process of S603 will now be described with reference to Fig. 9. Fig. 9 is a flowchart showing how the results of factor analysis are acquired when generating a document image.

[0103] In S901, the CPU 231 acquires the analysis result stored in the storage 235 in S602.

[0104] In S902, the CPU 231 determines whether the analysis result regarding the document layout of the document image data 113 is unknown among the analysis results acquired in S901. If the document layout of the document image data 113 is unknown (YES in S902), the CPU 231 proceeds to S903, and if it is known (NO in S902), the CPU 231 proceeds to S904.

[0105] In S903, the CPU 231 adds the document layout of the document image data 113 to the database. The adding process will be described as a specific example with reference to Fig. 14. A layout database in which layout information 1410 is registered exists on the storage 235, and document layout information 1420 of the document image data 113 whose layout has been analyzed as unknown is registered in the layout database.

[0106] In S904, the CPU 231 executes the processes of S905 to S912 for each failure item acquired in S801.

[0107] In S905, the CPU 231 determines what value is associated with the item name of the failed item in the analysis result acquired in S901. If the value associated with the item name as the analysis result is "mis-extracted" ("mis-extracted" in S905), the CPU 231 proceeds to S906, and if the value associated with the item name as the analysis result is "not extracted" ("not extracted" in S905), the CPU 231 proceeds to S907.

[0108] In S906, the CPU 231 determines whether or not the item name of the character string data that is a candidate for extraction obtained in S701 is unknown. If the item name of the character string data that is a candidate for extraction is unknown (YES in S906), the CPU 231 proceeds to S909, and if it is known (NO in S906), the CPU 231 proceeds to S908. The process of determining whether or not the item name of the character string data that is a candidate for extraction obtained in S701 is unknown is performed by determining whether or not the character string data is stored in a non-extraction item name database 1310 as shown in Fig. 13(b). A specific example will be described with reference to Fig. 15(b).

[0109] FIG. 15B is a diagram showing a specific example of a corresponding pattern when the value associated with the item name as the analysis result from the analysis result acquired in S901 is "mis-extracted". The non-extraction item name database 1310 as shown in FIG. 13B is stored in advance on the storage 235. The non-extraction item name database 1310 includes detailed data 1571 and 1573 describing the occurrence probability 1561 for each non-extraction item name 1313 as shown in FIG. 15B. For one of the non-extraction item names, if the occurrence probability is 1, the item name is always selected, and if it is 0, the item name is never selected. Here, consider the case where the item name of the character string data that is a candidate for extraction acquired in S701 is "Total Before VAT". In the detailed data 1571, since the item name 1313 does not include "Total Before VAT", it is determined that the item name of the character string data that is a candidate for extraction acquired in S701 is unknown (YES in S906). On the other hand, in the case of the detailed data 1573, since the item name 1313 includes "Total Before VAT", it is determined that the item name of the character string data that is the extraction target candidate acquired in S701 is known (NO in S906).

[0110] In S907, the CPU 231 determines whether or not the item name of the character string data to be extracted acquired in S702 is unknown. If the item name of the character string data to be extracted acquired in S702 is unknown (YES in S907), the CPU 231 proceeds to S911, and if it is known (NO in S906), the CPU 231 proceeds to S910. The process of determining whether or not the item name of the character string data to be extracted acquired in S702 is unknown is performed by determining whether or not the item name is stored in an extraction target item name database 1300 as shown in Fig. 13(a). A specific example will be described with reference to Fig. 15(a). A specific example will be described with reference to Fig. 15(a).

[0111] FIG. 15(a) is a diagram showing a specific example of a corresponding pattern when the value associated with the item name as the analysis result from the analysis result acquired in S901 is "not extracted." An extraction target item name database 1300 as shown in FIG. 13(a) is stored in advance on the storage 235. As shown in FIG. 15(a), the extraction target item name database 1300 includes detailed data 1521, 1523 describing occurrence probability 1511 for each extraction target item name 1303. Here, consider the case where the item name of the character string data to be extracted acquired in S702 is "Invoice#." In the case of detailed data 1521, "Invoice#" is not included in the item name 1303, so it is determined that the item name of the character string data to be extracted acquired in S702 is unknown (YES in S907). On the other hand, in the case of the detailed data 1523, since the item name 1303 includes "Invoice#", it is determined that the item name of the character string to be extracted acquired in S702 is known (NO in S907).

[0112] In S908, the CPU 231 increases the probability of occurrence as an item name not subject to extraction at the time of generation. A specific example will be described with reference to FIG. 15(b). S908 corresponds to a case where processing is performed based on the detailed data 1573, and the occurrence probability 1575 of "Total Before VAT" in the detailed data 1573 is 0.2, so it is increased to 0.4 (occurrence probability 1585). In this example, the occurrence probability is increased by 0.2, but the amount of increase in the occurrence probability may be set by the developer in advance, for example, or may be determined via a second information processing system that automatically adjusts the occurrence probability. Then, since the detailed data 1573 before the change in the occurrence probability is stored on the storage 235, the detailed data 1573 is replaced with the detailed data 1583 after the change in the occurrence probability.

[0113] In S909, the CPU 231 adds the item name to the non-extraction item name database as shown in FIG. 13(b). Specifically, the example shown in FIG. 15(b) will be used for explanation. S909 corresponds to a case where processing is performed based on the detailed data 1571. Since "Total Before VAT" is not included in the item name 1303 in the detailed data 1571 (1572), the item name "Total Before VAT" is added to the detailed data 1571 (1582). The occurrence probability of the detailed data 1581 after the addition is set to 0.1, but the increase range of the occurrence probability set at the time of addition may be set by the developer in advance to an arbitrary increase range, or multiple document image data 113 may be automatically or manually analyzed. Since the detailed data 1571 before the addition is stored on the storage 235, the detailed data 1571 is replaced with the detailed data 1581 after the addition.

[0114] In S910, the CPU 231 increases the probability of occurrence as an item name to be extracted at the time of generation. A specific example will be described with reference to FIG. 15(a). S910 corresponds to a case where processing is performed based on the detailed data 1523, and the occurrence probability 1525 of "Invoice#" in the detailed data 1523 is 0.2, so it is increased to 0.4 (occurrence probability 1535). In this example, the occurrence probability is increased by 0.2, but the increase in the occurrence probability may be set by the developer in advance to an arbitrary increase amount, or may be determined via a second information processing system that automatically adjusts the occurrence probability. Then, since the detailed data 1523 before the change in the occurrence probability is stored on the storage 235, the detailed data 1523 is replaced with the detailed data 1533 after the change in the occurrence probability.

[0115] In S911, the CPU 231 adds the extracted item name to the extraction target item name database as shown in FIG. 13(a). Specifically, an example shown in FIG. 15(a) will be used for explanation. S911 corresponds to a case where processing is performed based on the detailed data 1521, and since "Invoice#" is not included in the item name 1303 in the detailed data 1521 (1522), the item name "Invoice#" is added to the detailed data 1521 (1532). The occurrence probability of the detailed data 1531 after the addition is set to 0.1, but the increase range of the occurrence probability set at the time of addition may be set by the developer in advance to an arbitrary increase range, or multiple document image data 113 may be automatically or manually analyzed. Since the detailed data 1521 before the addition is stored on the storage 235, the detailed data 1521 is replaced with the detailed data 1531 after the addition.

[0116] In S912, the CPU 231 determines whether or not the processing has been completed for all the failed items acquired in S801, and repeats the processing of S904 to S912 until it is determined that the processing has been completed for all the item names.

[0117] Details of the process of S604 will now be described with reference to Fig. 10. Fig. 10 is a flowchart showing the process of generating a learning dataset.

[0118] In step S1001, the CPU 231 acquires the number of sheets for generating pseudo document image samples. For example, the CPU 231 acquires a value stored in the storage 235 in advance.

[0119] In S1002, the CPU 231 executes generation of pseudo document image samples for the number of sheets to be generated.

[0120] In S1003, the CPU 231 randomly selects one layout from a plurality of layouts stored in advance in the storage 235.

[0121] In S1004, the CPU 231 executes the processes of S1005 to S1011 for each sub-template present in the layout selected in S1003.

[0122] In S1005, the CPU 231 executes the processes of S1006 to S1010 for each item present in the sub-template selected in S1004.

[0123] In S1006, the CPU 231 determines the item name of the item selected in S1005. The character string of the item name is selected at random from candidates for item names 1303 and 1313 in the extraction target item name database 1300 and the non-extraction target item name database 1310 shown in Fig. 13. Note that the candidates for item names 1303 and 1313 may include a null character. When a null character is selected, it means that the item name is not displayed.

[0124] In S1007, the CPU 231 determines the value of the item selected in S1005. The character string that is the value of the item can be one randomly selected from the candidates listed as corresponding item value examples in the extraction target item name database 1300 in Fig. 13, or a numeric string or character string that is randomly generated without referring to the database.

[0125] In S1008, the CPU 231 places the character strings determined in S1006 and S1007 in the sub-template corresponding to the item specified in S1005. As a specific example, Fig. 12 is used. The placement in the sub-template is realized by placing character strings 1211 and 1212 in an area 1210 of the sub-template indicating the issuing destination information in a pseudo document image sample 1200. Note that the character string 1211 is a character string that is the item name determined in S1006, and the character string 1212 is character string data that is the value of the item determined in S1007.

[0126] In S1009, only when the item name determined in S1006 is the name of the item to be extracted, the CPU 231 associates the item name to be extracted with the value and position information of the item determined in S1007 and registers them as correct answer data. Note that the item name to be extracted here refers to the item name registered in the extraction target item name database 1300 as shown in Fig. 13, which is used when determining the item name in S1006.

[0127] In S1010, CPU 231 determines whether or not processing has been completed for all items present in the sub-template selected in S1004, and repeats the processing of S1005 to S1010 until it is determined that processing has been completed for all items.

[0128] In S1011, the CPU 231 determines whether or not the processing has been completed for all sub-templates present in the layout selected in S1003, and repeats the processing of S1004 to S1011 until it is determined that the processing has been completed for all sub-templates. When the processing has been completed for all sub-templates, a pseudo document image sample 1200 as shown in FIG. 12(a) is obtained.

[0129] In S1012, the CPU 231 returns to S1002 until the number of sheets to be generated obtained in S1001 is reached, and ends the process when the number of sheets to be generated obtained in S1001 is reached.

[0130] As described above, by changing the database used for data augmentation according to the cause of the recognition failure, a learning data set with a high degree of improvement in recognition accuracy can be generated, and learning can be achieved in a shorter time with a smaller learning data set. Note that, in this embodiment, the document image data 113 and the pseudo document data generated by the data augmentation process are form images, but the type of document is not limited to forms, and the type of data is not limited to images. For example, the document image data 113 and the pseudo document data may be PDL data including character string data and position information.

[0131] In addition, in this embodiment, the pseudo document data is used for re-learning the duplicated model, but the document image data 113 from which item value extraction has failed may be used for re-learning. When the document image data 113 is used for re-learning, pseudo document data that is not used for re-learning is used for evaluating the trained model candidates in S317.

[0132] [Embodiment 2] In the first embodiment, the candidate trained model is evaluated using the document image data 113 from which the user failed to extract item values ​​(S317). In the second embodiment, information on the document image data 113 previously input by the user is stored, and this is also used to evaluate the trained model 115 (S1802). The second embodiment can be realized by replacing Figs. 3, 5, and 6 in the first embodiment with Figs. 18 to 20. Other than this, the second embodiment is similar to the first embodiment, so a description thereof will be omitted.

[0133] <Usage sequence> Fig. 18 is a diagram showing a usage sequence of an information processing system in embodiment 2. In Fig. 18, the processes having the same reference numerals as those in the usage sequence in Fig. 3 are the same as those in embodiment 1, and therefore description thereof will be omitted.

[0134] In S1801, the information processing server 103 transmits the document image data 113 transmitted in S306 and the character string data to be extracted that was confirmed by the user in S309 to the learning device 102 and stores them in the storage 235 within the learning device 102.

[0135] In S1802, the learning device 102 acquires the document image data 113 stored in S1801 and the confirmed character string data of the extraction target from the storage 235 in the learning device 102. The data acquired here includes document image data previously input by the user. Character string data is acquired from each of the acquired document image data 113, and the character string data of the extraction target candidate is extracted using the trained model candidate generated in S316. The learning device 102 checks whether the character string data of the extraction target candidate based on the trained model candidate generated in S316 is identical to the acquired character string data of the confirmed extraction target. If the character string data of the extraction target candidate is identical to the character string data of the confirmed extraction target, the learning device 102 proceeds to S318, and if it is different from the character string data of the confirmed extraction target, the learning device 102 proceeds to S314.

[0136] <Extraction and correction of item value candidate strings> Fig. 19 is a flowchart showing a process of estimating character string data to be item value candidates and accepting corrections from a user in embodiment 2. Note that an execution program for each step shown in Fig. 19 is stored in any one of the ROM 262, RAM 264, or storage 265 of the information processing server 103, and is executed by the CPU 261 of the information processing server 103. In Fig. 19, the processes having the same reference numerals as those in the flowchart of Fig. 5 are the same as those in embodiment 1, and therefore description thereof will be omitted.

[0137] In S1901, the CPU 261 transmits the document image data 113 acquired in S502 and the character string data to be extracted that was confirmed by the user in S506 to the learning device 102, and stores them in the storage 235 in the learning device 102.

[0138] <Item value extraction model update process> Fig. 20 is a flowchart showing a process of reflecting corrections made by a user in a trained model in embodiment 2. Note that an execution program for each step shown in Fig. 20 is stored in any one of the ROM 232, the RAM 234, or the storage 235 of the learning device 102, and is executed by any one of the CPU 231 and the GPU 232 of the learning device 102. In addition, in Fig. 20, the processes having the same reference numerals as those in the flowchart of Fig. 6 are similar to those in embodiment 1, and therefore description thereof will be omitted.

[0139] In S2001, the CPU 231 acquires document image data from the storage 235. Note that the document image data acquired here also includes document image data that the user has previously input using the information processing apparatus 101.

[0140] In S2002, the CPU 231 executes the processes of S607, S608, and S2003 for each document image data acquired in S2001.

[0141] In S2003, the CPU 231 determines whether or not the processing has been completed for all the document image data acquired in S2001, and repeats the processing of S2002, S607, S608, and S2003 until it is determined that the processing has been completed for all the document image data.

[0142] In S2004, the CPU 231 checks whether the character string data acquired in S2003 is identical to the character string data acquired from the storage 235 for all extraction target items. If the character string data acquired in S2003 is identical to all extraction target items (YES in S2004), the CPU 231 proceeds to S612. If different character string data exists in one or more target items among the character string data acquired in S2003 (NO in S2004), the CPU 231 proceeds to S610. Note that the condition for YES in S2004 does not have to be that all the character string data acquired in S2003 is identical to all extraction target items. For example, the condition for YES in S2004 may be that the ratio of character string data that is identical to all extraction target items among the entire character string data acquired in S2003 is equal to or greater than a preset ratio. Alternatively, for each character string data acquired in S2003, if the data is identical in all extraction target items, the average of the values ​​may be set to 1, and if the data is different in one or more target items, the average may be set to 0, and the condition may be that the average is equal to or greater than a preset average value. When taking the average, an arithmetic average may be used, or a weighted average in which the priority of each character string data acquired in S2003 is weighted. As an example of weighting the priority, the older the document image data entered by the user, the lower the priority, so a smaller weight may be used. As another example of weighting the priority, the number of identical forms may be calculated based on the layout matching rate shown in FIG. 16, and the document image data with a larger number of identical forms may have a higher priority, so a larger weight may be used.

[0143] As described above, by evaluating and analyzing the trained model for extracting item values ​​using document image data previously input by the user, it is possible to update the trained model to one that can handle document image data previously input by the user. In other words, the updated trained model will be able to extract item values ​​with high accuracy even for document image data similar to document image data previously input by the user (pseudo document data). Since document image data similar to actual manuscripts is likely to be input by users in the future, training the trained model with a large amount of data related to actual manuscripts leads to improved inference accuracy of the trained model.

[0144] (Other Examples) The present invention can also be realized by a process in which a program for implementing one or more of the functions of the above-described embodiments is supplied to a system or device via a network or a storage medium, and one or more processors in a computer of the system or device read and execute the program. The present invention can also be realized by a circuit (e.g., ASIC) that implements one or more of the functions.

[0145] The present disclosure includes the following configurations and methods. [Configuration 1] a generating means for generating pseudo-document data of input document data when a learning model for extracting named entities from input document data fails to extract named entities from the input document data; a learning means for re-learning the learning model based on learning data consisting of pairs of the pseudo-document data generated by the generating means and / or the input document data from which the named entity extraction has failed and correct answer data; Equipped with the generating means increases a probability that the failed extraction target item will be selected when generating the pseudo-document data. 23. An information processing apparatus comprising: [Configuration 2] the learning means performs the re-learning on a duplicate model of the learning model, and when the duplicate model that has been re-learned is successful in extracting named entities from the pseudo-document data that is not included in the learning data and / or the input document data from which the named entity extraction has failed, updates the learning model with the duplicate model that has been re-learned. 2. The information processing device according to configuration 1. [Configuration 3] when the named entity extraction by the replicated model that has been retrained on the pseudo document data not included in the training data or the input document data fails, the generating means generates another pseudo document data based on the input document data from which the named entity extraction has failed; the learning means re-learns the learning model based on learning data consisting of a pair of the different pseudo-document data and the correct answer data; 3. The information processing device according to configuration 1 or 2. [Configuration 4] when the replicated model that has been re-trained successfully extracts named entities from input document data and previously input document data from which the learning model has failed to extract named entities, the learning means updates the learning model with the replicated model that has been re-trained. 4. The information processing device according to any one of configurations 1 to 3. [Configuration 5] when the learning means succeeds in extracting named entities using the re-trained replicated model for input document data from which the learning model has failed to extract named entities and for pseudo-document data not included in the learning data, the learning means updates the learning model with the re-trained replicated model. 3. The information processing device according to configuration 1 or 2. [Configuration 6] when the named entity extraction using the re-trained duplicated model for the pseudo-document data not included in the training data is successful, the training means updates the training model with the re-trained duplicated model. 4. The information processing device according to any one of configurations 1 to 3. [Configuration 7] The learning model is a learning model that extracts character string data that is a candidate for an item value corresponding to a predetermined item from a plurality of character strings included in the target document image data. 7. The information processing device according to any one of configurations 1 to 6. [Configuration 8] the generating means increases the occurrence probability of an item to be extracted when an item value corresponding to the item to be extracted has not been extracted or has been erroneously extracted; 8. The information processing device according to configuration 7. [Configuration 9] the generating means, when an item value corresponding to an item to be extracted is not extracted or is erroneously extracted and the occurrence probability of the item to be extracted is not registered, registers the name of the item to be extracted and the occurrence probability of the item to be extracted in association with each other; 9. The information processing device according to configuration 8. [Configuration 10] an information processing server that extracts named entities from input document data using a learning model; A learning device that learns the learning model; Equipped with The learning device includes: a generating means for generating pseudo-document data of input document data when a learning model for extracting named entities from input document data fails to extract named entities from the input document data; a learning means for re-learning the learning model based on learning data consisting of pairs of the pseudo-document data generated by the generating means and / or the input document data from which the named entity extraction has failed and correct answer data; Including, the generating means increases a probability that the failed extraction target item is selected for generating the pseudo-document data. An information processing system comprising: [Configuration 11] generating pseudo-document data of the input document data when a learning model for extracting named entities from input document data fails to extract named entities from the input document data; a step of re-learning the learning model based on learning data consisting of pairs of the pseudo-document data generated by the generating step and / or the input document data from which the named entity extraction has failed and correct answer data; having In the generating step, a probability that the failed extraction target item is selected when generating the pseudo-document data is increased. 23. An information processing method comprising: [Configuration 12] A program for causing a computer to function as the information processing device according to any one of configurations 1 to 9.

Claims

1. a generating means for generating pseudo-document data of input document data when a learning model for extracting named entities from input document data fails to extract named entities from the input document data; a learning means for re-learning the learning model based on learning data consisting of pairs of the pseudo-document data generated by the generating means and / or the input document data from which the named entity extraction has failed and correct answer data; Equipped with the generating means increases a probability that the failed extraction target item will be selected when generating the pseudo-document data.

23. An information processing apparatus comprising:

2. the learning means performs the re-learning on a duplicate model of the learning model, and when the duplicate model that has been re-learned is successful in extracting named entities from the pseudo-document data that is not included in the learning data and / or the input document data from which the named entity extraction has failed, updates the learning model with the duplicate model that has been re-learned.

2. The information processing apparatus according to claim 1,

3. when the named entity extraction by the replicated model that has been retrained on the pseudo document data not included in the training data or the input document data fails, the generating means generates another pseudo document data based on the input document data from which the named entity extraction has failed; the learning means re-learns the learning model based on learning data consisting of a pair of the different pseudo-document data and the correct answer data; 2. The information processing apparatus according to claim 1,

4. when the replicated model that has been re-trained successfully extracts named entities from input document data and previously input document data from which the learning model has failed to extract named entities, the learning means updates the learning model with the replicated model that has been re-trained.

2. The information processing apparatus according to claim 1,

5. when the learning means succeeds in extracting named entities using the re-trained replicated model for input document data from which the learning model has failed to extract named entities and for pseudo-document data not included in the learning data, the learning means updates the learning model with the re-trained replicated model.

2. The information processing apparatus according to claim 1,

6. when the named entity extraction using the re-trained duplicated model for the pseudo-document data not included in the training data is successful, the training means updates the training model with the re-trained duplicated model.

2. The information processing apparatus according to claim 1,

7. The learning model is a learning model that extracts character string data that is a candidate for an item value corresponding to a predetermined item from a plurality of character strings included in the target document image data.

2. The information processing apparatus according to claim 1,

8. the generating means increases the occurrence probability of an item to be extracted when an item value corresponding to the item to be extracted has not been extracted or has been erroneously extracted; 8. The information processing apparatus according to claim 7,

9. the generating means, when an item value corresponding to an item to be extracted is not extracted or is erroneously extracted and the occurrence probability of the item to be extracted is not registered, registers the name of the item to be extracted and the occurrence probability of the item to be extracted in association with each other; 9. The information processing apparatus according to claim 8,

10. an information processing server that extracts named entities from input document data using a learning model; A learning device that learns the learning model; Equipped with The learning device includes: a generating means for generating pseudo-document data of input document data when a learning model for extracting named entities from input document data fails to extract named entities from the input document data; a learning means for re-learning the learning model based on learning data consisting of pairs of the pseudo-document data generated by the generating means and / or the input document data from which the named entity extraction has failed and correct answer data; Including, the generating means increases a probability that the failed extraction target item will be selected when generating the pseudo-document data. An information processing system comprising:

11. generating pseudo-document data of the input document data when a learning model for extracting named entities from input document data fails to extract named entities from the input document data; a step of re-learning the learning model based on learning data consisting of pairs of the pseudo-document data generated by the generating step and / or the input document data from which the named entity extraction has failed and correct answer data; having In the generating step, a probability that the failed extraction target item is selected when generating the pseudo-document data is increased.

23. An information processing method comprising:

12. A program for causing a computer to function as the information processing device according to any one of claims 1 to 9.

Citation Information

Patent Citations

  • Information processor, control method and program

    JP2021131659A