Image processing apparatus, program, and image processing method

Abstract

To solve the problem that performing entity extraction based on a character recognition result sometimes results in failure to extract character strings corresponding to predetermined items.SOLUTION: When a character string corresponding to a given item about a document image cannot be extracted by entity extraction processing, a character string corresponding to the given item about the document image is obtained based on position information of a region where a character string corresponding to the given item was extracted before in a document of the same format as the document image.SELECTED DRAWING: Figure 5

Description

Technical Field

[0001] Relates to a method for extracting a character string corresponding to a predetermined item from a paper document.

Background Art

[0002] As a technique for extracting a character string that becomes a value corresponding to a desired item from the character strings included in a document image, there is a method of finding the character string of the item from the document image and extracting the corresponding character string using a search rule according to the item (see, for example, Patent Document 1).

Prior Art Documents

Patent Documents

[0003]

Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0004] However, there are cases where a character string corresponding to a predetermined item cannot be extracted from a document image.

Means for Solving the Problems

[0005] In order to solve the above problems, an image processing apparatus according to the present invention includes character recognition means for obtaining a character recognition result by performing character recognition processing on a document image to be processed, entity extraction means for extracting a character string corresponding to a predetermined item by performing entity extraction based on the character recognition result, and acquisition means for acquiring a character string corresponding to the predetermined item for the document image based on position information of an area where the character string corresponding to the predetermined item was extracted in a document having the same format as the document image before when the character string corresponding to the predetermined item for the document image is not extracted by the entity extraction means. It is characterized by comprising.

Effects of the Invention

[0006] According to this proposal, even when a character string corresponding to a predetermined item cannot be extracted by the entity extraction process, the extraction result can be corrected.

Brief Description of Drawings

[0007]

Figure 1

Figure 2

Figure 3

Figure 4

Figure 5

Figure 6

Figure 7

Figure 8

Figure 9

Figure 10

Figure 11

Figure 12

Figure 13

Figure 14

Figure 15

Modes for Carrying Out the Invention

[0008] Hereinafter, embodiments for carrying out the present invention will be described with reference to the drawings.

Example

[0009] FIG. 1 is a diagram showing an example of the configuration of the image processing system in this embodiment. The image processing system includes an image forming apparatus 101, an image processing server (image processing apparatus) 102, an information processing terminal 103, and a storage server 104. These apparatuses and servers are interconnected by a network 105 and can communicate with each other.

[0010] In this embodiment, as an example of the image forming apparatus 101, a multifunction printer having a scan function, a print function, a copy function, etc. will be described, but it is not limited to a multifunction printer. For example, any apparatus with a scan function can execute the processing of the present invention described later. Note that the scan function is a function of transmitting externally the image data generated by reading a document (such as a form) with a scanner provided in the image forming apparatus 101. Also, the print function is a function of printing print data received from the information processing terminal 103 or the like. Further, the copy function is a function of obtaining a copy of a document by printing the image data of the document read by the scanner.

[0011] As will be described later, the image forming apparatus 101 of this embodiment can request the storage server 104 to transmit the image data of the scanned document via the image processing server 102.

[0012] The image processing server 102 can perform image processing on the image data received from the image forming apparatus 101 and transmit the image data after the image processing to the storage server 104 for storage. Also, the image processing server 102 may be configured to transmit to the information processing terminal 103 that the image data has been received from the image forming apparatus 101, and upon receiving a request for storing the image data from the information processing terminal 103, transmit the image data to the storage server 104.

[0013] FIG. 2 is a diagram showing an example of the hardware configuration of the image processing system according to the present embodiment. The image forming apparatus 101 includes a printer 202, a scanner 203, an operation unit 204, a CPU 211, a RAM 212, an HDD 213, a network I / F 214, a printer I / F 215, a scanner I / F 216, an operation unit I / F 217, and an expansion I / F 218. The CPU 211 can exchange data with the RAM 212, the HDD 213, the network I / F 214, the printer I / F 215, the scanner I / F 216, the operation unit I / F 217, and the expansion I / F 218. Further, the CPU 211 expands the instructions (computer programs) read from the HDD 213 in the RAM 212 and executes the instructions expanded in the RAM 212, thereby controlling the execution of each process described later. That is, the processor of the image forming apparatus 101 functions as a processing unit that executes each process described later by reading and executing a program from a computer-readable storage medium (such as an HDD).

[0014] The HDD 213 can store instructions (programs) executable by the CPU 211, setting values used in the image forming apparatus 101, data related to processes requested by the user, and the like. The RAM 212 is an area for temporarily storing the instructions read by the CPU 211 from the HDD 213. The RAM 212 can also store various types of data necessary for the execution of the instructions. For example, in image processing, it is possible to perform processing by expanding the input data in the RAM 212.

[0015] The network I / F 214 is an interface for performing network communication with devices in the image forming system. The network I / F 214 can transmit to the CPU 211 that data reception has been performed, and can transmit the data on the RAM 212 to the network 105 in accordance with an instruction from the CPU 211.

[0016] The printer I / F 215 can transmit print data to be printed to the printer 202 according to an instruction from the CPU 211, and can transmit the status of the printer received from the printer 202 to the CPU 211.

[0017] The scanner I / F 216 can transmit an image reading instruction instructed by the CPU 211 to the scanner 203, or transmit the image data received from the scanner 203 to the CPU 211. Further, the scanner I / F 216 can also transmit the status information of the scanner received from the scanner 203 to the CPU 211.

[0018] The operation unit I / F 217 can transmit an instruction from the user made through the operation unit 204 to the CPU 211, or display screen information for the user to operate on the operation unit 204. The expansion I / F 218 is an interface that enables an external device to be connected to the image forming apparatus 101. The expansion I / F 218 includes, for example, an interface in the USB (Universal Serial Bus) format. When an external storage device such as a USB memory is connected to the expansion I / F 218, the image forming apparatus 101 can read data stored in the external storage device and write data to the external storage device.

[0019] The printer 202 can print the image data received via the printer I / F 215 on paper, and can also transmit the status of the printer 202 to the printer I / F 215. The scanner 203 can transmit the image data obtained by reading a document (such as a form) placed on the scanner in accordance with the image reading instruction received via the scanner I / F 216 to the scanner I / F 216. Further, the scanner 203 can also transmit the status of the scanner to the scanner I / F 216. The operation unit 204 is an interface for giving various instructions to the image forming apparatus 101 based on the operation of the user. For example, the operation unit 204 includes a touch panel liquid crystal screen, displays an operation screen, and receives an operation from the user.

[0020] The image processing server 102 is composed of a CPU 221, a RAM 222, an HDD 223, and a network I / F 224. The CPU 221 controls the entire device and can control the transfer of data among the RAM 222, the HDD 223, and the network I / F 224. Also, the CPU 221 expands and executes the control program (instructions) read from the HDD 223 in the RAM 222. That is, the processor of the image processing server 102 functions as a processing unit that executes each of the processes described later by reading and executing a program from a computer-readable storage medium (such as the HDD 223). Further, the HDD 223 of the image processing server 102 can manage information related to scanning and information for identifying entities to be extracted.

[0021] The information processing terminal 103 is composed of a CPU 231, a RAM 232, an HDD 233, a network I / F 234, an operation unit I / F 235, and an operation unit 236. The CPU 231 controls the entire device and can control the transfer of data among the RAM 232, the HDD 233, the network I / F 234, the operation unit I / F 235, and the operation unit 236. Also, the CPU 231 expands and executes the control program (instructions) read from the HDD 233 in the RAM 232. The operation unit I / F 235 is an interface that transmits an instruction from the user input from the operation unit 236 to the CPU 231 and transmits information regarding the operation screen to be displayed to the operation unit 236 based on the display control by the CPU 231. An application for checking image data is installed in the information processing terminal 103, and by executing the application, the image data of the image processing server can be displayed and the function of requesting the storage of the image data can be utilized. When the application for checking image data is provided as a web application, the information processing terminal 103 may also execute the web application via a web browser to display the image data and request the storage of the image data. That is, the processor of the information processing terminal 103 can check the image data by reading and executing a program from a computer-readable storage medium (such as the HDD 233) or by executing a program received via a network.

[0022] The storage server 104 is composed of a CPU 241, a RAM 242, an HDD 243, and a network I / F 244. The CPU 241 controls the entire device and can control the transfer of data among the RAM 242, the HDD 243, and the network I / F 244. Also, the CPU 241 expands and executes the control program (instructions) read from the HDD 243 in the RAM 242. The HDD 243 of the storage server 104 can hold the image data received from the image processing server 102.

[0023] FIG. 3 is a conceptual diagram showing an example of the software configuration of the image processing system in this embodiment. The CPU of each device executes a software program and functions as a processing unit that performs the following respective processes by cooperating with hardware such as a scanner and an HDD.

[0024] The controller 310 of the image forming apparatus 101 includes a control unit 311, a storage unit 312, a communication unit 313, a display unit 314, and a scan unit 315. The control unit 311 has functions related to all processes regarding the functions of the controller 310. The storage unit 312 has a function of holding settings related to scanning and a function of holding image data received by scanning. The communication unit 313 has a function of transmitting image data scanned via the network 105 and setting information related to scanning to the image processing server 102. The display unit 314 has a function of receiving a scan request from the user via the operation unit 204 and notifying the control unit 311. The scan unit 315 receives a scan request from the control unit 311 and executes a scan process on the scanner 203 via the scanner I / F 216. The scan unit 315 transmits the received image data obtained by performing the scan to the control unit 311.

[0025] The image processing service 320 provided by the image processing server 102 is realized by a control unit 321, a storage unit 322, a communication unit 320, and an image processing unit 324. The control unit 321 has functions related to all processes regarding the functions of the image processing service 320. The storage unit 322 has a function of holding settings related to scanning, a function of holding image data received from the image forming apparatus 101, and a function of holding a learned form for performing matching of the image data.

[0026] The function of the storage unit 322 for holding settings related to scanning will be described. The scan settings can be different for each button operated by the user and are held in a scan setting table as shown in FIG. 10, for example.

[0027] The scan setting table in FIG. 10 is composed of nine columns: button number, name, entity setting, color, resolution, duplex / single-sided, paper size, file type, and destination.

[0028] The button number is a unique number for identifying the button, and the name is the button name that is displayed together with the button on the display unit of the image forming apparatus 101. The entity setting describes the type of entity that the image processing unit 324 of the image processing server 102 extracts from the OCR result of the scanned image. The acquisition of the entity will be described later.

[0029] Color, resolution, duplex / single-sided, and paper size indicate the operation settings during scanning. That is, they indicate whether the image generated by scanning is a color image or a black-and-white image, the resolution of the image, whether it is duplex scanning or single-sided scanning only, and the size of the paper to be scanned, respectively.

[0030] File type and destination indicate the file format and destination when outputting the scanned image.

[0031] When the user logs in (not shown), the image forming apparatus 101 acquires the scan setting table and displays the buttons on the display unit. When the user selects a displayed button, the image forming apparatus 101 detects the selected button and executes scanning of a document such as a form according to the settings of the selected button to generate a scanned image (document image). The image forming apparatus 101 transmits the scanned image (document image) generated by scanning and the number of the button selected by the user.

[0032] The communication unit 323 has functions of receiving document image data from the image forming apparatus 101, transmitting the processing result to the information processing terminal 103, receiving a document image data storage request from the information processing terminal 103, and transmitting image data to the storage server 104.

[0033] The image processing unit 324 has functions of processing the image data received from the image forming apparatus 101, such as image binarization, a process of dividing into regions for each attribute such as characters, tables, and figures (BS process), and character recognition process (OCR process). Further, it has a function of determining whether forms are of the same format by comparing the positions and sizes of the regions obtained by the BS process, a function of learning a form as a learned form, and a function of extracting an entity from the processing results of BS / OCR.

[0034] Note that image binarization is a process of converting a color document image into binary, for example, white and black. The conversion method is not particularly limited.

[0035] The BS / OCR executed by the image processing unit 324 will be described. The BS (Block Selection) process is a process of analyzing a scanned image to identify regions indicating positions such as character strings and figures. The OCR (Optical Character Recognition) process is a process of recognizing a character string (character code) from a character image. In this embodiment, when the image processing unit 324 performs BS / OCR, a BS / OCR result table as shown in FIG. 11 is generated.

[0036] The BS / OCR result table in FIG. 11 is composed of three columns: number, BS region, and character string, and one row of data is generated for each character string. The number is a unique number for identifying each character string region. In this embodiment, they are BS1 to BS9. The BS region (information on the region determined by the BS process) is coordinates indicating the position of the character string region, and is composed of, for example, four numbers: the X coordinate and Y coordinate of the upper left of each region, and the width and height of each region. The BS region of number BS1 is 874, 333, 511, 73, where the X coordinate is 874, the Y coordinate is 333, the width is 511, and the height is 73. The character string included in the BS region of number BS1 is a quotation.

[0037] Next, the learning executed by the image processing unit 324 will be described.

[0038] Execute BS / OCR processing on the document image (scan image data) received from the image forming apparatus, generate a learned form based on the BS / OCR result table and the entity result table obtained from the information processing terminal 103, and add it to the learned form table as shown in Fig. 12.

[0039] The learned form table holds the information of the BS area of the image and the entity extraction results in association, so that when processing images with the same position of the BS area (i.e., images with the same form format), the previous entity extraction results can be retrieved. The learned form table in Fig. 12 is composed of five columns: button number, form number, number, BS area, and type. The button number is a unique number for identifying the learned form for each button. The form number is a unique number for identifying the learned form. The number is uniquely identified by a number with "BS" at the beginning indicating the BS area and a number with "ET" at the beginning indicating the result of entity extraction.

[0040] For example, assume that scanning is executed with the button number BTN1, and as a result of performing entity extraction processing on the BS / OCR result table, the area with the title BS1, the organization name on the document issuing side in the area BS7, and the personal name BS8 on the document issuing side are obtained. In this case, a learned form recognized with the button number BTN1 and the form number 1 is added to the learned form table. In this case, the same number and BS area as in the BS / OCR table are added to the learned form table, and further, the number, BS area, and type obtained as a result of the entity extraction processing are added.

[0041] Next, the form matching executed by the image processing unit 324 will be described.

[0042] Form matching is a process of comparing the coordinate positions of the BS areas to determine whether they are the same. For the BS areas to be the same means that the X coordinate, Y coordinate, width, and height are approximately the same. The image processing unit 324 acquires all the BS areas of the learned form table and compares the BS areas for each form number in the acquired learned form table. For example, if the BS area as a result of performing BS processing on the scanned image data is the BS / OCR result table in FIG. 11, the BS of BS1 in the BS / OCR result table is compared with BS1 to BS9 of form number 1 in the learned form table. When each of BS1 to BS9 in the BS / OCR result table and BS1 to BS9 of form number 1 in the learned form table approximately matches, it is determined that they are the same. If it is determined that it is different from the learned form with form number 1, next, it is compared with the learned form with form number 2. If, as a result of comparing with all the learned forms, it is determined that they are all different, it is determined that there is no matching form.

[0043] Next, the entity extraction process executed by the image processing unit 324 will be described.

[0044] The entity in this embodiment refers to a value (string) extracted by natural language processing and means a value (string) corresponding to a predetermined item. In the entity extraction of this embodiment, it is assumed that values corresponding to item types such as "organization name on the document issuing side", "organization name on the document receiving side", "personal name on the document issuing side", "personal name on the document receiving side", "document creation date", "document number", and "title" are extracted. Note that the types of entities described above are just examples, and other types of character strings may be configured to be extracted as entities. As a method for extracting these entities, the character string is word-segmented by morphological analysis, and a Transformer-based machine learning model is generated using the segmented words. By inputting the character string into the generated model and making inferences, entities corresponding to the desired item types are extracted from the character string, and an entity result table is generated. For the generation of a general model, pre-trained models such as BERT (Bidirectional Encoder Representations from Transformers) and GPT (Generative Pre-trained Transformers) are used. The model is generated by fine-tuning using the word group segmented by morphological analysis from the character string, which is a feature of the entity to be extracted based on the pre-trained model.

[0045] Values (strings) corresponding to each item type extracted by the entity extraction process and the coordinate information of the BS area containing the string are stored in the entity result table. For example, the entity result table in FIG. 13 is composed of five columns: number, type, BS area, string, and to-be-confirmed. One row of data is generated for each string. The number is a number for uniquely identifying the string in the table. In this embodiment, they are ET1, ET2, and ET3. The type indicates the entity to be extracted. The BS area is the coordinates indicating the area of the string determined as each entity to be extracted, and is composed of four numbers: X coordinate, Y coordinate, width, and height. The BS area of number 1 is 874, 333, 511, 73, where X is 874, Y is 333, width is 511, and height is 73. The string is the string determined as each entity to be extracted. To-be-confirmed indicates whether the extraction result requires special confirmation by the user. If special confirmation is required, 1 is entered; otherwise, 0 is entered. Note that for the type for which the string that becomes the entity cannot be extracted, a row with empty columns for the BS area and the string is generated (see ET3 in FIG. 14 for example).

[0046] The image data confirmation application 330 of the information processing terminal 103 is composed of a control unit 331, a storage unit 332, a communication unit 333, and a display unit 334.

[0047] The control unit 331 has functions for all processes related to the functions of the image data confirmation application 330. The storage unit 332 has a function of holding the image data received from the image processing server 102 and the entity result table. The communication unit 333 has functions of receiving the image data and the entity extraction result table from the image processing server 102 and transmitting a scan image data storage request. The display unit 334 has a function of displaying the image data received from the image processing server 102 and the entity extraction result and prompting the user for confirmation.

[0048] The data storage service 340 of the storage server 104 is composed of a control unit 341, a storage unit 342, and a communication unit 343.

[0049] The control unit 341 has functions for all processes related to the functions of the data storage service 340. The storage unit 342 has a function of storing the image data received from the image processing server 102 in association with the entity extraction result. The communication unit 343 has a function of receiving the image data and the entity extraction result table from the image processing server 102.

[0050] FIG. 4 is a sequence diagram showing the flow of processing for extracting an entity from an image scanned by the image forming apparatus (MFP) 110 by the image processing server 102 and transmitting it to the storage server 104. Here, the description will focus on the communication between each device.

[0051] First, a button for transmitting the scanned image to the cloud storage is displayed on the touch panel of the MFP 110 according to the information stored in the scan setting table. When this button is pressed by the user, the processing shown in the sequence of FIG. 4 for transmitting the scanned image to the cloud storage is started. In this embodiment, an example in which the user executes BTN1 in the scan setting table will be described.

[0052] The control unit 311 of the image forming apparatus 101 executes document scanning processing via the scanning unit 315 according to the scan settings set for the button BTN1 pressed by the user on the display unit 314 (S401).

[0053] The control unit 311 transmits the scanned image data generated in S401 and the button number BTN1 for which scanning was executed in S401 to the image processing server 102 via the communication unit 313 (S402). In this embodiment, the scanned image data generated by scanning a document is transmitted to the image processing server as the image to be processed, but it is not limited to this. For example, a document image received by facsimile communication may be transmitted to the image processing server as the image to be processed.

[0054] The control unit 321 of the image processing server 102 receives the scanned image data transmitted in S402 and the button number BTN1 via the communication unit 323, and stores them via the storage unit 322 (S403).

[0055] The control unit 321 performs image processing on the scanned image data received in S403 using the image processing unit 324 (S404). Details will be described later.

[0056] The control unit 321 stores the result processed in S404 via the storage unit 322 (S405).

[0057] The control unit 321 transmits the result processed in S404 (BS / OCR result table, entity result table, scanned image data) to the information processing terminal (client) 103 via the communication unit 323 (S406).

[0058] The control unit 331 of the information processing terminal 103 receives the processing result transmitted in S406 via the communication unit 333 (S407).

[0059] The control unit 331 uses the display unit 334 to display the processing result and prompt the user for confirmation (S408). Details will be described later.

[0060] The control unit 331 reflects the entity extraction result confirmed and corrected by the user in the entity result table, and transmits the information of the corrected entity result table together with the storage request of the scanned image data to the image processing server 102 via the communication unit 333 (S409).

[0061] The control unit 321 of the image processing server 102 receives the scanned image data storage request via the communication unit 323 (S410).

[0062] The control unit 321 acquires the entity result table included in the scan image data storage request received in S410, acquires the scan image data via the storage unit 322, and further transmits the acquired entity result table and the scan image data to the storage server 104 via the communication unit 323.

[0063] The control unit 341 of the storage server 104 receives the entity result table and the scan image data via the communication unit 342 (S412).

[0064] The control unit 341 associates the entity result table received in S412 with the metadata of the received scan image data and stores it via the storage unit 342 (S413).

[0065] The control unit 321 of the image processing server 102 acquires the entity result table included in the scan image data storage request received in S410 and performs learning via the image processing unit 324 (S414). Details will be described later.

[0066] Figure 5 is a flowchart showing the details of the image processing in S404.

[0067] The control unit 321 of the image processing server 102 causes the image processing unit 324 to read the scan image data received in S403 (S501).

[0068] The control unit 321 performs binarization processing on the scan image data read in S501 via the image processing unit 324 to generate binarized image data (S502).

[0069] The control unit 321 executes BS / OCR on the generated binarized image data via the image processing unit 324, and acquires the BS / OCR results (the position information of the character string regions in the document image and the character recognition processing results of each character string). The acquired BS regions and the character recognition results of the character strings are stored as a BS / OCR result table via the storage unit 322 (S503). In this embodiment, it is assumed that the BS / OCR table shown in FIG. 11 is created.

[0070] The control unit 321 acquires the entity setting corresponding to the button number BTN1 stored in S403 from the scan setting table, extracts the type described in the acquired entity setting as the extraction target, and performs entity extraction processing on the character string acquired in S503. The result of the entity extraction is stored as an entity result table via the storage unit 322 (S504). In this embodiment, it is assumed that at the time of S504, an entity result table as shown in FIG. 14 is created. In the example of FIG. 14, since no entity corresponding to "Personal name on the document issuing side" of ET3 was extracted, the BS region and the character string are blank.

[0071] The control unit 321 acquires information on the learned form associated with the same button number as the button number BTN1 stored in S403 from the learned form table (S505). In this embodiment, information on the learned form (learned document) with the same button number is acquired from the learned form table shown in FIG. 12 and used as a comparison target for form matching.

[0072] The control unit 321 compares the BS region of the learned form acquired in S505 with the BS region of the scan image data acquired in S503 to perform form matching, and determines whether there is a learned form with the same layout (i.e., the same format) as the scan image data (S506). In this embodiment, it is assumed that the information on the learned form shown in FIG. 15 is determined to be a form with the same layout (same format).

[0073] If the control unit 321 determines that there is a learned form (document) in the same format as a result of the form matching in S506 (Yes in S507), it proceeds to S508. On the other hand, if it determines that there is no learned form in the same format (No in S507), the process ends.

[0074] The control unit 321 determines whether there is a row in the entity result table for which the processing after S509 has not been performed (S508). If there is a row for which the processing after S509 has not been performed (Yes in S508), it proceeds to S509. If the processing after S509 has been performed for all rows (No in S508), the process ends.

[0075] The control unit 321 selects one row from the rows included in the entity result table for which the processing after S510 has not been performed (S509). For example, assume that the row numbered ET3 in FIG. 14 is selected.

[0076] The control unit 321 determines whether the BS area and the character string column are not empty for the row in the entity result table selected in S509 (S510). If the BS area and the character string column are not empty (i.e., the entity of the relevant type has been extracted) (Yes in S510), it returns to S508. If the BS area and the character string column are empty (i.e., the entity of the relevant type has not been extracted) (No in S510), it proceeds to S511. For example, since the row numbered ET3 in FIG. 14 has an empty BS area and a character string column, it proceeds to S511.

[0077] The control unit 321 determines whether the BS area of the row in the learned form obtained in S506, which has the same type as the one selected in S509, is not empty (S511). If the BS area is not empty, i.e., the entity of the relevant type could be extracted in the learned form (Yes in S511), it proceeds to S512. Otherwise (No in S511), it returns to S508. For example, the row in FIG. 15 that has the same type as the type "personal name on the document issuing side" of the row numbered ET3 in FIG. 14 is the row numbered ET3, and since the column of the BS area is not empty, it proceeds to S512.

[0078] The control unit 321 acquires the position information of the BS area in the same row as the one selected in S509 among the learned forms acquired in S506 (S512). For example, it acquires "197, 833, 309, 35", which is the position information of the BS area numbered ET3 in FIG. 15.

[0079] The control unit 321 acquires the character string of the row that matches the position information of the BS area acquired in S512 from the BS / OCR result table stored in S503 (S513). For example, it acquires "Toshishiro Miki", which is the character string of the BS area existing at the position corresponding to the position information "197, 833, 309, 35" acquired in S512.

[0080] The control unit 321 reflects the BS area acquired in S512 and the character string acquired in S513 in the row of the entity result table selected in S509 (S514).

[0081] The control unit 321 enters "1" in the "To be confirmed" column of the row of the entity result table selected in S509 and records that it is an entity that requires confirmation (S515). This is to prompt confirmation as to whether the entity result has been corrected in a BS area different from what the user desires by the process of S514 in the case where a form in a format different from the scanned image has been erroneously matched in S506.

[0082] As a result of the processes of S514 and S515, the entity result table shown in FIG. 14 is updated as shown in FIG. 13.

[0083] FIG. 6 is a flowchart showing the details of S408.

[0084] Based on the processing result received in S407, the control unit 331 of the information processing terminal 103 causes the display unit 334 to display a screen based on the scanned image data and the content of the entity result table (S601). An example of this display is shown in FIG. 7. In FIG. 7, the area 701 is an area for displaying a preview of the scanned image data. The area 702 is an area for displaying entity extraction results, and displays the BS area and character strings in the entity result table for each type. The user can correct the content of the entity result table via the operation unit 236 by selecting the text boxes of the displayed BS area and character strings and performing a correction operation. The save button 703 is a button for instructing the saving of the image after the user's confirmation and correction are completed. The warning display 704 indicates an entity that requires confirmation by the user and is displayed beside the type. The warning display 704 is displayed by the process of S603 described later.

[0085] The control unit 331 determines whether there is a row in the entity result table received in S407 where the column to be confirmed is "1" (S602). If there is a row where the column to be confirmed is "1" (Yes in S602), the process proceeds to S603. Otherwise (No in S602), the process ends.

[0086] The control unit 331 displays the warning display 704 beside the type corresponding to the row where the column to be confirmed is "1" and ends the process (S603).

[0087] FIG. 8 is a flowchart showing the details of S414.

[0088] The control unit 321 of the image processing server 102 acquires the entity result table that has been confirmed and corrected by the user and is included in the scanned image data storage request received in S410 (S801).

[0089] The control unit 321 acquires the OCR results of the BS area and character strings stored in S503 (S802).

[0090] The control unit 321 acquires the learned form in the same manner as S505 via the storage unit 322 (S803).

[0091] The control unit 321 performs form matching by comparing the BS area of the learned form acquired in S803 with the BS area of the processing result acquired in S802 via the image processing unit 324, and acquires the learned form determined to have the same layout (S804).

[0092] The control unit 321 determines whether there is a learned form (the learned form determined to have the same layout) acquired in S804 (S805).

[0093] If the control unit 321 determines that there is a learned form with the same layout as a result of form matching (Yes in S805), it updates the BS area of the learned form acquired in S804 with the BS area acquired in S802 and the BS area of the entity result table acquired in S801, and ends the process (S806).

[0094] On the other hand, if the control unit 321 determines that there is no learned form with the same layout (i.e., it is unlearned) as a result of form matching (No in S805), it generates a learned form from the BS area acquired in S802 and the entity result table acquired in S801, adds it to the learned form table, and ends the process (S807).

[0095] In this embodiment, the network 105 may be a wired LAN (Local Area Network), a wireless LAN, the Internet, or the like, as long as the devices in the image processing system can communicate with each other.

[0096] The image processing server 102 and the cloud storage server 104 may operate on a server PC on a local network, or may be virtual servers or services realized on a cloud service.

[0097] Settings related to scanning are held in the scan settings table, but may be arbitrarily set from the outside.

[0098] In addition, in the comparison of the BS areas in form matching, in this embodiment, when the position, width, and height of the coordinates are the same, it is determined that they are the same. However, a method may be adopted in which the width is given to the coordinate values for the same determination, and if it is within a certain range, it is determined to be the same. Also, in form matching, when all BS areas are the same, it is determined to match. However, for example, if the BS areas in the upper part (e.g., the upper half) of the form are the same, it may be determined to match, or if a predetermined number or more of the BS areas are the same, it may be determined to match.

[0099] As described above, according to this embodiment, when the character string of the entity corresponding to the desired item cannot be extracted from the document (e.g., ET3 in FIG. 14), the entity extraction result can be corrected (e.g., FIG. 13) by using the position information (e.g., FIG. 15) at the time when the entity character string was previously extracted in the learned form of the same format.

Example

[0100] In the second embodiment, in addition to the first embodiment, even when the result of entity extraction from the image to be processed is not empty, a method for prompting the user to confirm the extraction results determined to be of low validity by comparing with the entity extraction results of the learned form will be described. In this embodiment, the same explanations as those in the first embodiment regarding the system configuration, software configuration, etc. are omitted.

[0101] FIG. 9 is a flowchart showing the details of S404.

[0102] For the row in the entity result table selected in S509, when the BS area and the character string are not empty (Yes in S510), the control unit 321 determines whether the BS area of the row of the learned form obtained in S506, the type of which is the same as the one selected in S509, is not empty (S901).

[0103] When the BS area of the corresponding row of the learned form is not empty (No in S901), the control unit 321 enters "1" in the "To be confirmed" column of the row of the entity result table selected in S509, and records that the entity requires confirmation (S902). This is because it is natural for the same entity to be extracted at the same coordinates in the same format, but a state where the extraction success or failure is different between the previous extraction result and the extraction result of the current scanned image is considered to have low validity of the extraction result, and this is to prompt the user for confirmation.

[0104] The control unit 321 compares the position coordinates of the BS area of the row in the entity result table obtained in S509 with the position coordinates of the BS area of the row in the learned form obtained in S505, where the type is the same as that obtained in S509, and determines whether the distance between the two BS areas is greater than or equal to a predetermined threshold (S903). The distance between the two BS areas is measured as the distance between the coordinates of the upper left vertices of each BS area. If the distance between the two BS areas is less than the threshold (No in S903), the process returns to S508.

[0105] When the distance between the two BS areas is greater than or equal to the threshold (Yes in S903), the control unit 321 enters "1" in the "To be confirmed" column of the row of the entity result table selected in S509, and records that the entity requires confirmation (S904). This is because it is natural for the same entity to be extracted at the same coordinates in the same format, but a state where the coordinates are separated between the previous extraction result and the extraction result of the current scanned image is considered to have low validity of the extraction result, and this is to prompt the user for confirmation.

[0106] In this embodiment, the distance between the BS areas uses the distance between the coordinates of the upper left vertices, but other methods such as the distance between the centroids of the BS areas may be used.

[0107] As described above, according to the present embodiment, even when the result of entity extraction is not empty, by comparing with the entity extraction result of the learned form, it is possible to prompt the user for confirmation of an extraction result with low validity.

[0108] (Other embodiments) Further, the present invention can also be realized by executing the following process. The process is to supply software (program) for realizing the functions of the above-described embodiments to a system or device via a network or a computer-readable storage medium, and to cause a computer (such as a CPU) of the system or device to execute the program.

Claims

1. Character recognition means for obtaining a character recognition result by performing character recognition processing on a document image to be processed; Entity extraction means for extracting a character string corresponding to a predetermined item by performing entity extraction based on the character recognition result; When a character string corresponding to the predetermined item for the document image is not extracted by the entity extraction means, based on the position information of the area where the character string corresponding to the predetermined item was extracted in a document having the same format as the document image previously, acquisition means for acquiring a character string corresponding to the predetermined item for the document image; An image processing apparatus, characterized by comprising the above.

2. The image processing apparatus according to claim 1, wherein the character recognition means obtains the character recognition result by performing character recognition processing on a character string area in the document image to be processed.

3. The image processing apparatus according to claim 1 or 2, wherein the entity extraction includes natural language processing.

4. The image processing apparatus according to any one of claims 1 to 3, wherein the document image to be processed is at least one of a document image generated by scanning or a document image received by facsimile.

5. The image processing apparatus according to any one of claims 1 to 4, wherein the document having the same format is determined based on the position of the character string area in the document image.

6. The image processing apparatus according to any one of claims 1 to 5, wherein when the acquisition means acquires a character string corresponding to the predetermined item, the user is prompted to confirm the character string.

7. The image processing apparatus according to any one of claims 1 to 6, wherein when a character string corresponding to the predetermined item for the document image is extracted by the entity extraction means and the character string corresponding to the predetermined item was not extracted in a document having the same format as the document image previously, the user is prompted to confirm the character string corresponding to the predetermined item extracted by the entity extraction means.

8. When a character string corresponding to the predetermined item for the document image is extracted by the entity extraction means, and a character string corresponding to the predetermined item is extracted in a document having the same format as the document image previously, and the distance between the regions where those character strings are extracted is separated by a predetermined threshold or more, the user is prompted to confirm the character string corresponding to the predetermined item extracted by the entity extraction means. The image processing apparatus according to any one of claims 1 to 7, characterized in that.

9. A program for causing a computer to function as each means of the image processing apparatus according to any one of claims 1 to 8.

10. A character recognition step of obtaining a character recognition result by performing character recognition processing on a document image to be processed, An entity extraction step of extracting a character string corresponding to a predetermined item by performing entity extraction based on the character recognition result, When the character string corresponding to the predetermined item for the document image is not extracted in the entity extraction step, based on the position information of the region where the character string corresponding to the predetermined item was extracted in a document having the same format as the document image previously, an acquisition step of acquiring the character string corresponding to the predetermined item for the document image, An image processing method characterized by comprising:

Images