Information processing device, control method of information processing device, and program

Abstract

To improve usability in a confirmation work of a character recognition result.SOLUTION: An information processing device comprises: extraction means of extracting specific character strings having specific attributes and unit characters constituting the specific character strings on the basis of a result of character recognition processing to a document image; generation means of generating attribute information representing the attributes of the extracted specific character strings and context information regarding the extracted unit characters; and display control means of displaying the extracted unit characters and the context information corresponding to the unit characters on a UI picture.SELECTED DRAWING: Figure 11

Description

[Technical field]

[0001] The present disclosure relates to a technique for efficiently checking recognition results for each character. [Background technology]

[0002] Character recognition technology that extracts characters from an image that contains characters is known and is applied in various technical fields. Characters are misrecognized in character recognition processing, so the extracted results are manually checked and proofread. Such work generally takes time and places a burden on the worker. To solve this problem, a UI that efficiently performs the checking work has been proposed. Patent Document 1 discloses a method of displaying a list of multiple character images that have the same character recognition result (same cluster) so that the user can visually check them. That is, only the character image portion that has the same character recognition result is cut out, and multiple images of one character that have been cut out are lined up and displayed in a list, so that the user can check the characters that have been processed by character recognition processing. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] JP 2013-073608 A Summary of the Invention [Problem to be solved by the invention]

[0004] For example, in the case of handwritten characters, when checking the character recognition results, the character recognition results may look like other characters that are different from the original characters. Such confusion with other characters is not limited to handwritten characters, but may also occur in the case of type characters with similar character shapes, such as the number "1" and the alphabet "l". In this regard, the technology of Patent Document 1, which cuts out images of multiple characters with the same character recognition results and displays them in a list, does not show information other than the characters, so it takes a long time for the user to judge whether the character recognition results are correct. The present disclosure has been made in consideration of the above problems, and aims to improve usability in the work of checking character recognition results. [Means for solving the problem]

[0005] The information processing device according to the present disclosure is characterized by comprising: an extraction means for extracting a specific character string having a specific attribute and unit characters constituting the specific character string based on the result of character recognition processing on a document image; a generation means for generating attribute information representing attributes of the extracted specific character string and context information related to the extracted unit characters; and a display control means for displaying the extracted unit characters and the context information corresponding to the unit characters on a UI screen. Effect of the Invention

[0006] According to the technology of the present disclosure, usability in the task of checking character recognition results is improved. [Brief description of the drawings]

[0007] [Figure 1] FIG. 1 is a diagram showing an example of the configuration of an image processing system. [Diagram 2] FIG. 2 is a diagram showing an example of the hardware configuration of an image processing server constituting the present system. [Diagram 3] FIG. 2 is a diagram showing an example of the software configuration of an image processing server that constitutes the present system. [Figure 4] 4 is a flowchart illustrating an overall process executed by the image processing server. [Diagram 5] 6 is a flowchart illustrating an image processing job generation process executed by the image processing server. [Figure 6] FIG. 4 is a diagram showing a configuration of character string area information. [Figure 7] FIG. 13 is a diagram showing the configuration of context information. [Figure 8] 11 is a flowchart illustrating a screen generation process executed by the image processing server. [Figure 9] FIG. 4 is a diagram showing an example of a screen configuration for displaying a list of character recognition results. [Figure 10] 6 is a flowchart for explaining various screen generation processes executed by the image processing server. [Figure 11] FIG. 13 is a diagram showing an example of a screen configuration in which context information is superimposed on an image. [Figure 12] FIG. 13 is a diagram showing an example of a screen configuration in which an identity / similarity recognition result is superimposed on an image. [Figure 13] FIG. 13 is a diagram showing an example of a screen configuration for displaying a list of similar recognition results. [Figure 14] 5 is a flowchart for explaining an image processing job generation process and a screen generation process executed by the image processing server. [Figure 15] FIG. 13 is a diagram showing an example of a screen configuration in which context information and a confidence level are superimposed on an image. [Figure 16] 11 is a flowchart illustrating a screen generation process executed by the image processing server. [Figure 17] 4 is a flowchart for explaining an image forming process, an image processing job generation process, and a screen generation process executed in the image processing system. [Figure 18] FIG. 13 is a diagram showing an example of a screen configuration for displaying an identical / similar recognition result. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0008] Hereinafter, embodiments of the present disclosure will be described with reference to the accompanying drawings. Note that the following embodiments do not limit the present disclosure, and not all of the combinations of features described in the embodiments are necessarily essential to the solution of the present disclosure. In addition, each process (step) in the flow chart is indicated by a reference number beginning with "S".

[0009] [First embodiment] <Overall composition> FIG. 1 is a diagram showing an example of the configuration of an image processing system 100 in this embodiment. The image processing system 100 includes an image forming apparatus 101, an image processing server 102 (information processing apparatus 102), and an information processing terminal 103. These apparatuses and servers are connected to each other via a network 105 and are capable of communicating with each other. The image forming apparatus 101 has a function of optically reading a paper document form to obtain form image data (document image) and transmitting the form image data to the image processing server 102 via the network 105. Examples of paper document forms include various forms such as invoices or receipts that include multiple items such as amounts and dates and times. These forms have handwritten characters or printed characters or the like written for each of the multiple items described above.

[0010] The image processing server 102 is communicably connected to the image forming apparatus 101 and the information processing terminal 103 via a network 105. Examples of the network 105 include the Internet, a local area network (LAN), and Wi-Fi (registered trademark). As an example, the image processing server 102 is a general-purpose computer such as a server computer or a personal computer.

[0011] The information processing terminal 103 is a device operated by a worker who performs the confirmation work. An application for the worker to operate is installed in the information processing terminal 103, and displays a UI (User Interface) screen for the confirmation work. The confirmation work includes the work of checking the recognition results of characters and the like contained in the form image, and the work of correcting the character recognition results. Examples of the information processing terminal 103 include a general-purpose computer such as a personal computer, a smartphone, or a portable terminal device such as a tablet terminal.

[0012] <Hardware configuration> Fig. 2 is a diagram showing an example of a hardware configuration of the image processing system 100 in this embodiment. As shown in Fig. 2, the image processing server 102 includes a control unit 220, a communication unit 225, a display unit 226, an operation unit 227, and a storage unit 228. The control unit 220 includes a CPU 221, a ROM 222, a RAM 223, and an input / output interface (I / O) 224, and these units are connected via a bus.

[0013] The CPU 221 reads out programs and data stored in the ROM 222 and the storage unit 228 into the RAM 223 and executes various processes described below. The ROM 222 stores fixed data such as programs and data executed by the CPU 221. The RAM 223 is used as a main memory for the CPU 221 and a temporary storage area such as a work area. The I / O 224 is connected to a communication unit 225, a display unit 226, an operation unit 227, and a storage unit 228. Each of these functional units can communicate with the CPU 221 via the I / O 224.

[0014] The display unit 226 may be, for example, a liquid crystal display (LCD) or an organic electroluminescence (EL) display. The operation unit 227 may be, for example, a device for inputting an operation such as a keyword or a mouse. In the above description, the display unit 226 and the operation unit 227 are separate, but this is not limited to this. The display unit 226 and the operation unit 227 may be integrated by using, for example, a touch panel. The CPU 221 also receives various instructions from a user to the image processing server 102 via the operation unit 227. The display unit 226 displays various information such as the results of processing executed in response to the instructions received from the user and notifications regarding the processing. The storage unit 228 may be, for example, a hard disk drive (HDD), a solid state drive (SSD), a flash memory, or the like. The storage unit 228 stores an information processing program 229 and the like.

[0015] <Software configuration> Fig. 3 is a diagram showing an example of the software configuration of the image processing system 100 in this embodiment. As shown in Fig. 3, the information processing program 229 includes a main module 300 and a rendering module 310. The main module 300 includes a main processing unit 301, a request transmission / reception unit 302, an image processing unit 303, an extraction unit 304, an information generation unit 305, a screen generation unit 306, and a job list generation unit 307. The rendering module 310 includes a data transmission / reception unit 311, a UI display control unit 312, and an input acquisition unit 313. The information processing program 229 is loaded into the RAM 223 and executed by the CPU 221.

[0016] The main module 300 is executed by the CPU 221, and executes, in response to a request, a process of processing an image to extract information, and a process of generating a screen to be displayed on a UI from the extracted information. Here, the main module 300 stores the generated information in the storage unit 228, but may store the information in another information processing device, for example, in the information processing terminal 103 via the network 105. The main module 300 accepts events from the operation unit 207 and the rendering module 310, but may also accept events from another information processing device, for example, in the information processing terminal 103 via the network 105.

[0017] The main processing unit 301 controls the overall processing of the main module 300 and requests each unit included in the main module 300 to perform processing. The request transmission / reception unit 302 transmits a request from the main module 300 to the rendering module 310 and receives a request from the rendering module 310. The image processing unit 303 performs image processing such as binarization of a document image. The extraction unit 304 extracts a character string (specific character string) and a rectangular area (specific character string area) containing the character string by performing BS and OCR in the document image. The extraction unit 304 also extracts each character and a rectangular area (unit character area) containing each character by cutting out each character (unit character) in the character string area. The extraction unit 304 also extracts the confidence of character recognition for each character. Furthermore, if the extracted character string contains information having a person's name or an identification number that identifies a person, the extraction unit 304 also extracts the information. The information generation unit 305 generates attribute information representing the extracted character string and context information for each extracted character. A screen generation unit 306 generates a screen for confirming the character recognition result from the extracted character string and the generated context information. A job list generation unit 307 generates a processing unit of the character recognition result confirmed by the user on the UI screen as a job, and generates a job list that stores all the jobs.

[0018] The rendering module 310 is executed by the CPU 221, and executes a process of displaying a received screen, and a process of receiving an input event from a user and transmitting it as a request. Here, the rendering module 310 displays the received screen on the display unit 226, but may also display it on a display unit of another information processing device, for example, on the display unit of the information processing terminal 103 via the network 105.

[0019] The data transmission / reception unit 311 receives the screen generated by the screen generation unit 306 and transmits a processing request to the main module. The UI display control unit 312 receives the screen generated by the screen generation unit 306 via the data transmission / reception unit 311 and displays the received UI screen on the display unit 226. The input acquisition unit 313 accepts user input on the UI screen. In addition, the input acquisition unit 313 transmits to the main module 300 via the data transmission / reception unit 311 a processing request for an event that occurs when the user presses the UI screen.

[0020] <Overall process flow> 4 is a flowchart showing details of a display process of a confirmation UI for a character recognition result executed by the information processing program 229. This display process is executed by mutual communication between the main module 300 and the rendering module 310. In addition, the main module 300 and the rendering module are executed by the CPU 221 of the image processing server 102.

[0021] In S401, the CPU 221 generates an event for creating a main screen of a confirmation UI for the character recognition result, and the process proceeds to S402. In S402, the CPU 221 waits for an event to occur. Here, the event is, for example, an input from a user via the operation unit 227. If an event occurs (if S402 is YES), the process proceeds to S403. If an event does not occur (if S402 is NO), the process returns to S402 and waits for an event to occur. In S403, the CPU 221 creates a processing request according to the event that occurred in S402, and transmits the request to the main module 300 via the data transmission / reception unit 311. When the request is transmitted to the main module 300, the process proceeds to S404. Here, the processing request is, for example, a request for processing to generate a screen that displays the character recognition result in a filter. When transmitting the request to the main module 300, an image may be attached to the request. In this embodiment, the image to be attached is obtained from the image forming apparatus 101, but is not limited to this. Other acquisition methods such as uploading via the rendering module 310 or acquisition from online storage may also be used. Also, information indicating the storage location of the image, such as the URL of the image, may be attached to the request, and the side receiving the request may acquire the image from the location of the image.

[0022] In S404, the CPU 221 determines whether the type of the request received via the request transmission / reception unit 302 is a request to perform image processing, generate a job that is a processing unit that requires confirmation of the recognition result, and register it in the job list. The request is, for example, a request for screen generation created in S403 or a request for performing image processing and generating a job. If the type of the request is job registration (if S404 is YES), the process proceeds to S405. If the type of the request is not job registration (if S404 is NO), the process proceeds to S406. In S405, the CPU 221 performs character recognition processing on the image, generates a job that is a processing unit that requires confirmation of the character recognition result, and registers it in the job list. When the job is registered in the job list, the process returns to S401.

[0023] FIG. 5 is a flowchart for explaining the detailed procedure of the process of S405 in this embodiment. This flowchart is executed by the main module 300. First, in S501, the CPU 221 acquires image data included in the request, and the process proceeds to S502. In S502, the CPU 221 binarizes the image acquired in S501 to generate a binary image, and the process proceeds to S503. Here, the binarization of an image is a process of converting a color image into binary, for example, black and white, and a known method is used as the conversion method. In S503, the CPU 221 executes Block Selection (BS) and Optical Character Recognition (OCR) on the binary image generated in S502, and the process proceeds to S504. Here, Block Selection refers to a process of extracting a rectangular area indicating the location of a character string from an image. OCR refers to a process of extracting a character string from an image. In this embodiment, when the CPU 221 executes BS and OCR, character string area information 600 shown in FIG. 6 is generated.

[0024] 6 shows an example of character string region information, which is configured in JSON (JavaScript Object Notation) format. Explanation of the JSON format will be omitted. In this embodiment, the character string region information is configured in JSON format, but other formats such as XML (Extensible Markup Language) and CSV (Comma-Separated Values) may also be used.

[0025] Character string area information 600 holds information on one or more character string areas. Character string area information 600 includes rectangle information 601, a type of character string area, a character string 602, one-character area information, attribute information 603, and an identification number 604. Rectangle information 601 includes the top left coordinate, width, and height of the rectangle of the character string area. Character string 602 is a character string extracted by OCR. Attribute information 603 is additional information that explains the character string, which will be described later. Identification number 604 is a unique number assigned to the character string area. One-character area information 605 includes one-character rectangle information 606 and a character 607. Although this embodiment has the above configuration, other information may also be included.

[0026] In S504, the CPU 221 executes attribute information recognition using the character string region information 600 generated in S503, and generates attribute information 603 that describes the character string region. Once the attribute information 603 is generated, the process proceeds to S505. In S504, attribute information indicated by the character string is extracted by natural language processing using AI (Artificial Intelligence). For example, if the character string region is a character string region that indicates the amount of money "¥330,000" shown in the character string 602, the CPU 221 assigns attribute information of "item type: amount" shown in the attribute information 603.

[0027] Furthermore, when the character string regions are in a pair relationship of an item name and an item value, the character strings are associated with each other as a key-value relationship. Here, the key-value relationship is extracted as attribute information of the character string by a method using the layout relationship of the image and a rule base. For example, when the character string "¥330,000" is present to the right of the character string "Bill amount" on the image, attribute information is assigned with the two character strings as key values. Furthermore, when the extracted character string includes a character string indicating the type of image, the CPU221 assigns attribute information indicating the type of image. Here, too, the attribute information indicated by the character string is extracted by natural language processing using AI. For example, when the character string "invoice" is present, the CPU221 assigns attribute information "document type: invoice" as the type of image. Moreover, the attribute information may be provided in advance by input from the user. Moreover, the attribute information is not limited to the above.

[0028] In S505, the CPU 221 generates context information 710 using the character string region information 600 generated in S503 and the attribute information 603 generated in S504. After the context information 710 is generated and saved, the process proceeds to S506. FIG. 7 is a diagram showing an example of the context information 710. First, the CPU 221 acquires single character region information from the character string region information 600, groups the information by character, and stores the information by character group in a list. Here, a known grouping method is used as the grouping method. Each character group, which is an element of the list, is composed of a list of the recognized character type and single character region of the group. For example, the recognized character 711 indicates that the recognition result of the single character region of the character group is "0". The single character region recognized as "0" shown in information 712 is stored in the list. Each single character region includes rectangle information 713, an identification number 714, and context information 715. The rectangle information 713 includes the coordinates of the top left corner, width, and height of the rectangle of the one-character region as shown in FIG. 7, and is information acquired from the rectangle information 606. The identification number 714 is a number that is uniquely assigned to the one-character region as shown in FIG. 7. In this embodiment, the context information 715 includes the following information. The first information is attribute information of a character string in which the one-character region is included and information of an image region corresponding to the character string in which the one-character region is included. The second information is attribute information of an adjacent character string that has a key-value relationship with the character string and information of an image region corresponding to the adjacent character string. The third information is attribute information of a character string that indicates the type of document image and information of an image region of a character string that indicates the type of document image. However, the context information 715 is not limited to the above information.

[0029] 7, in this embodiment, the context information 715 holds the identification number of the character string region in "valueRegionId", and the character string region and attribute information can be obtained by reverse lookup using the identification number 604. Furthermore, the CPU 221 stores the generated character string region information 600 and context information 715 in the storage unit 228.

[0030] In S506, CPU 221 generates a job, which is a processing unit that requires the user to confirm the character recognition result. For each job, one image acquired in S501, one piece of character string area information generated in S503, and one piece of context information generated in S505 are linked. Furthermore, the generated job is inserted into a job list. Table 1 is an example of a job list. The image information, character string area information, and context information are assigned identification numbers and stored in storage unit 228. Each identification number is registered in the job list, and the job is associated with the image, character string area information, and context information. When the job is generated, CPU 221 registers the generated job in the job list, and the processing of the flowchart shown in FIG. 5 ends.

[0031] [Table 1] Returning to the description of Fig. 4, in S406, the CPU 221 determines whether the type of the received request is a UI request. If the type of the request is a UI request (if S406 is YES), the process proceeds to S407. If the type of the request is not a UI request (if S406 is NO), the process proceeds to S409. In S407, the CPU 221 (main module 300) generates a screen for the user to operate according to the contents of the request.

[0032] 8 is a flowchart for explaining the detailed procedure of the process of S407 in this embodiment. This flowchart is executed by the main module 300. First, in S801, the CPU 221 determines whether the type of the received request is a main screen generation request. The main screen generation request is a request generated in S401, and is a request to generate a main screen that is displayed by default when the UI of this embodiment is started.

[0033] If the request type is main screen generation (YES in S801), the process proceeds to S802. If the request type is not main screen generation (NO in S801), the process proceeds to S803. In S802, the CPU 221 (main module 300) executes generation of a collective confirmation screen. An example of the collective confirmation screen generated is shown in FIG. 9(a). In the example of FIG. 9(a), the collective confirmation screen 900 is composed of a recognition result filter screen 901. When a filter item is selected by the user while the collective confirmation screen 900 is displayed, the screen shown in FIG. 9(b) is displayed. On this screen, the OCR recognition results can be displayed in a list for each character, and the accuracy of the recognition results can be confirmed and the recognition results can be proofread.

[0034] FIG. 10A is a flowchart for explaining the detailed procedure of the process of S802 in this embodiment. This flowchart is executed by the main module 300. First, in S1001, the CPU 221 acquires the job list generated in S506. Furthermore, the CPU 221 acquires a job from the job list and acquires context information 710 associated with the job. Here, all jobs may be acquired from the job list, or only jobs associated with the user or the group to which the user belongs may be acquired. When the acquisition of the job and context information 710 is completed, the process proceeds to S1002. In S1002, the CPU 221 acquires the character types of all the recognition result characters from the acquired job from the context information 710 acquired in S1001. More specifically, the CPU 221 acquires the recognized characters 711 of the character group for each element of the list. When the character types of all the recognition result characters are acquired, the process proceeds to S1003. In S1003, the CPU 221 generates a collective confirmation screen 900 including the recognition result filter screen 801. Furthermore, the CPU 221 generates a recognition result filter 910 on the recognition result filter screen 901 using the character types of all the recognition result characters acquired in S1002. An example of the screen generated here is shown in the recognition result filter screen 901 of FIG. 9(a). The recognition result filter screen 901 allows the user to filter and display the character recognition result using any character by the recognition result filter 910. The recognition result filter 910 also sorts and displays the character types of all the recognition result characters acquired in S1002. Here, a known method is used as the sorting method. In this embodiment, the sorting is performed in numerical order, but is not limited to this. For example, the sorting may be performed in ascending order of the character code. When the collective confirmation screen 900 and the recognition result filter screen 901 are generated, the process of the flowchart shown in FIG. 10(a) ends.

[0035] Returning to the description of Fig. 8, in S803, the CPU 221 determines whether the received request is a selection of a filter item by the user. Here, the filter item selection request is a request transmitted in S403 in response to an event that occurs when the user presses any item of the recognition result filter 910. If the request is a selection of a filter item by the user (if S803 is YES), the process proceeds to S804. If the request is not a selection of a filter item by the user (if S803 is NO), the process proceeds to S805. In S804, the CPU 221 executes generation of a list display screen of the recognition results.

[0036] FIG. 10B is a flowchart for explaining the detailed procedure of the process of S804 in this embodiment. This flowchart is executed by the main module 300. First, in S1011, the CPU 221 acquires the filter item selected by the user. Furthermore, the CPU 221 acquires all one-character information whose recognition result is the same as the character of the filter item selected by the user from the acquired job. More specifically, all character groups corresponding to the character of the filter item selected by the user are acquired from the context information 710 of all the acquired jobs. When the acquisition of the character recognition result is completed, the process proceeds to S1012. In S1012, the CPU 221 generates a screen that displays the one-character information acquired in S1011.

[0037] A screen 902 in FIG. 9B is an example of a screen to be displayed. The screen 902 displays an area image 911, a recognition result 912, context information 913, a confirmation check box 914, a collective confirmation button 915, and a confirmation OK button 916, and accepts confirmation and correction of the recognition result from the user. The area image 911 displays the one-character area acquired in S1011 in a list. More specifically, a rectangular area on the image is cut out and displayed using rectangular information of the one-character area. The recognition result 912 displays the character recognition result of the area image 911, and accepts correction from the user. The context information 913 displays context information associated with the area image 911. The context information is hidden by default, and is displayed when the user presses an accordion section 917 described later. The confirmation check box 914 accepts whether the user has checked the recognition result for the area image 911 and corrected it, and whether the user has finished checking the item. When the user presses the All Confirm button 915, all items in the confirmation check boxes 914 are checked. When the user presses the Confirm OK button 916, the corrected recognition result is saved and the user's confirmation work ends. When a screen displaying one-character information is generated, the process of the flowchart shown in FIG. 10(b) ends.

[0038] Returning to the description of FIG. 8, in S805, the CPU 221 determines whether or not the received request is a selection of an accordion by the user. Here, an accordion selection request is a request sent in S403 in response to an event that occurs when the user presses the accordion section 917 of any item of the context information 913. If the request is a selection of an accordion by the user (if S805 is YES), the process proceeds to S806. If the request is not a selection of an accordion by the user (if S805 is NO), the process proceeds to S807. In S806, the CPU 221 executes generation of a context information display screen.

[0039] FIG. 10C is a flowchart for explaining the detailed procedure of the process of S806 in this embodiment. This flowchart is executed by the main module 300. First, in S1021, the CPU 221 acquires context information of the one-character region selected by the user, and the process proceeds to S1022. In S1022, the CPU 221 generates a screen that displays the context information acquired in S1021. A screen 903 in FIG. 9C is an example of a screen that is displayed. When the screen 903 receives a press on the accordion section 917 from the user, the screen is updated. The screen 903 displays context information 918 associated with the one-character region selected by the user. When an accordion is already open and a press on the accordion section 917 from the user is received, the accordion is closed and the corresponding context information is hidden.

[0040] Returning to the description of Fig. 8, in S807, the CPU 221 judges whether or not the request received in the main processing unit 301 is a selection of an area image by the user. Here, the area image selection request is a request transmitted in S403 in response to an event that occurs when the user presses an image of any item in the area image 911. If the request is a selection of an area image by the user (if S807 is YES), the process proceeds to S808. If the request is not a selection of an area image by the user (if S807 is NO), the process of the flowchart shown in Fig. 8 ends. In S808, the CPU 221 executes generation of a side screen.

[0041] Fig. 10(d) is a flowchart for explaining the detailed procedure of the processing of S808 in this embodiment. In this flowchart, a form context information screen, an identical / similar character comparison screen, or a similar character list screen is generated as a side screen (detailed screen). The form context information screen, the identical / similar character comparison screen, and the similar character list screen can be switched between each other by receiving an input from the user. Here, for example, the screens are switched by tabs, but other switching methods may also be used. In this embodiment, the form context information screen is initially displayed.

[0042] In S1031, the CPU 221 acquires an image including a single character region selected by the user and context information associated with the single character region, or information on identical / similar characters. When acquiring information on identical characters, information on a single character region of a character group corresponding to the same character of a character selected by the user is acquired from a list of character groups included in the context information. When acquiring information on similar characters, information on similar characters of the character selected by the user is acquired by referring to a similar character dictionary. Table 2 is an example of a similar character dictionary used to acquire similar characters. A similar character is a character that is likely to be erroneously recognized as a target character in OCR processing. The CPU 221 stores a dictionary in which a target character and a similar character are paired in advance. For example, in Table 2, for the character "0", "D", "O", and "Q" are similar characters. Furthermore, the similar character dictionary may collect statistical information on characters proofread by the user and add pairs of the target character and similar characters. The CPU 221 acquires information on a single character region of a character group corresponding to a similar character from a list of character groups included in the context information. When the context information or information on identical / similar characters is acquired, the process proceeds to S1032. In S1032, the CPU 221 generates a side screen using the image, context information, or identical / similar character information acquired in S1031.

[0043] [Table 2] FIG. 11 is a diagram showing an example of a collective confirmation screen generated by the CPU 221 in S1032. In this screen, the context information of the one-character region selected by the user can be displayed superimposed on the image, and the user can confirm the accuracy of the recognition result or proofread the recognition result. The collective confirmation screen 900 in FIG. 11 is composed of a recognition result filter screen 904 and a form context information screen 1100. In the display of the recognition result filter screen 904, the collective confirmation screen 900 accepts the selection of a one-character region image that the user is unsure of, deletes the display of the context information 913, reduces it, and displays it on the left side of the screen. At this time, the collective confirmation screen 900 displays the form context information screen 1100 on the right side of the screen.

[0044] The form context information screen 1100 displays an image including a single character region selected by the user and a character string region of context information associated with the single character region. Furthermore, the single character region and the character string region on the image are highlighted with a colored frame, and attribute information of the single character region and the character string region are displayed in a balloon. Highlighting 1110 highlights the single character region selected by the user and the character string region of context information associated with the single character region on the image with a colored frame. Here, the frame colors of the single character region selected by the user and the character string region of the context information are set to be different. A balloon 1111 displays attribute information of the character string region indicated by the highlighting 1110. In this embodiment, the context information is the character string region and attribute information including the single character region, the character string region and attribute information having a key-value relationship with the character string region, and the character string region and attribute information indicating the type of image. Therefore, in FIG. 11, the character strings "¥330,000-", "billing amount", and "invoice" are highlighted, and attribute information of item type, item name, and form type are displayed in a balloon. This screen clearly shows the user the context in which the target character appears, and makes it possible to prompt the user to determine the target character.

[0045] 12(a) and (b) are diagrams showing an example of a batch confirmation screen generated by the CPU 221 in S1032. On this screen, the same character and similar characters in a single character area image selected by the user can be highlighted on the scanned image, and the recognition result can be checked for accuracy or proofread. The batch confirmation screen 900 in FIG. 12(a) includes a recognition result filter screen 904 and an identical / similar character comparison screen 1200.

[0046] The comparison screen 1200 for identical / similar characters highlights the identical / similar characters on an image including a single character area selected by the user. The notation 1210 displays a legend of a colored frame in which the single character area selected by the user and the single character area of ​​the identical / similar character are highlighted. The display image 1211 is an image including the single character area selected by the user, and displays an image including the single character area and the single character area of ​​the identical / similar character. The highlighting 1212 highlights the single character area selected by the user and the single character area of ​​the identical / similar character on the display image 1211 with a colored frame according to the notation of the notation 1210. By default, the single character area selected by the user and the single character area of ​​the same character are highlighted with a colored frame. Here, the color of the frame may be different between the single character area selected by the user and the single character area of ​​the same character, or the transparency may be changed in the same color system. The check box 1213 allows the user to select the similar characters to be highlighted. When the check box 1213 receives a selection from the user, it is updated to the screen 1201 (see FIG. 12(b)). Screen 1201 highlights the single character area of ​​similar characters selected by the user with a colored frame on the image. Here, the color of the frame of the single character area of ​​similar characters is different from the single character area selected by the user and the single character area of ​​the same character. FIG. 12(b) shows an example in which the user selects "1" in the similar characters check box 1213. In FIG. 12(b), "1" such as in "No. 71" is highlighted as a similar character. This screen makes it possible for the user to compare the same character and similar characters on the same image and determine the character to be recognized.

[0047] 13(a) and (b) are diagrams showing an example of a collective confirmation screen generated by the CPU 221 in S1032. On this screen, the user can display a list of the one-character region image and similar characters selected by the user, and can check the accuracy of the recognition result or proofread the recognition result. In the example of FIG. 13(a), the collective confirmation screen 900 includes a recognition result filter screen 904 and a similar character list screen 1300.

[0048] The similar character list screen 1300 displays a list of similar characters in the single character area selected by the user. The similar characters are displayed in a check box 1310, and the user can select a similar character. When the similar character list screen 1300 accepts a selection from the user in the check box 1310, it displays a similar character list 1311 (see FIG. 13(b)). The similar character list 1311 shown in FIG. 13(b) displays a list of single character areas recognized as similar characters selected by the user. Here, since character shapes resulting from fonts or writing habits are similar within the same image, the job including the single character area selected by the user and the single character area of ​​similar characters in the same job are preferentially displayed at the top of the list. Here, a line may be displayed at the boundary between the single character area of ​​similar characters in the same job and the single character area of ​​similar characters in another job to clearly indicate the boundary, or similar characters other than those in the same job may not be displayed. This screen allows the user to compare the target character with similar characters in a list display, thereby urging the user to determine the recognition result for the target character.

[0049] In this embodiment, the character type of a character string, a character, or a single character is not limited to numbers, and may include kanji, hiragana, katakana, the alphabet, symbols, and other character types. For example, the item type "address" may be linked as context information for the single character region "Higashi" of the character string whose recognition result is "Tokyo" and displayed so that it can be confirmed. In addition, similar characters may include other character types, or the character types used may be set to be limited. For example, for a single character whose recognition result is the alphabet "l," the hiragana "shi" or the symbol "| (vertical bar)" may be displayed as similar characters.

[0050] Return to the description of FIG. 4. In S408, the CPU 221 displays on the display unit 226 the screen generated in S407 and received via the data transmission / reception unit 311. When the screen is displayed on the display unit 226, the process returns to S402. In S409, the CPU 221 determines whether the type of the request is program termination. Here, the program termination request is a request transmitted in S403 in response to an event that occurs when the user presses the confirmation OK button 916. The program termination request also occurs by other means, such as when the UI is closed or when transmitted from another information processing device. If the type of the request is program termination (if S409 is YES), the process of the flowchart shown in FIG. 4 ends. If the type of the request is not program termination (if S409 is NO), the process returns to S404 and waits for a request from the rendering module 310.

[0051] By implementing the present disclosure according to the above processing procedure, when a single character region image that is difficult for a user to judge is displayed during the work of checking the character recognition result, it is possible to display information that helps the user make a judgment. This reduces the time required for the user to make a judgment and improves the usability in the work of checking the character recognition result.

[0052] [Second embodiment] In this embodiment, the efficiency of the user's confirmation work is further improved by reducing the operation required for the user to display context information based on the confidence level of the character recognition result, which will be specifically described with reference to figures and flowcharts. In the description of this embodiment, the description of the same configuration and processing procedure as in the first embodiment will be omitted, and only the differences will be described. FIG. 14(a) is a flowchart for describing the detailed procedure of image processing job generation in this embodiment. In this embodiment, in S1411, when performing character recognition in the BS / OCR process, the CPU 221 calculates the confidence level of character recognition for each character. Furthermore, in S1412, in the context information generation process, the CPU 221 adds the confidence level of each character to the character string area information 600 and the context information 710 and stores them in the storage unit 228.

[0053] FIG. 14(b) is a flowchart for explaining a detailed procedure of screen generation in this embodiment. In S1413, in the generation process of the list display screen of the recognition results, the CPU 221 generates a list display screen of the recognition results with a confidence level display. FIG. 15 is a diagram showing a collective confirmation screen 1500 with a confidence level display displayed by this embodiment. Here, the CPU 221 generates the list display screen of the recognition results described in the first embodiment by adding a display of the confidence level 1501. Furthermore, the confidence level of each character is obtained from the context information 710, and a colored bar according to the confidence level is displayed in the confidence level item. In this embodiment, the confidence level is displayed as a colored bar to express the difference in the confidence level, but the confidence level may be displayed by numbers, letters, or other expression methods. If the confidence level is less than a threshold value, the accordion may be opened in advance to disclose and display the context information.

[0054] According to the present embodiment, items for which the recognition results are likely to be erroneous are displayed in a manner that makes them easy for the user to recognize, and it is possible to reduce the user's operations for displaying context information for the items, thereby further improving the usability in the task of checking character recognition results.

[0055] [Third embodiment] In this embodiment, the confirmation work can be made more efficient by switching the screen to be displayed based on the user's operation history, which will be specifically described with reference to a flowchart. Note that in describing this embodiment, descriptions of the same configurations and processing procedures as in the first and second embodiments will be omitted, and only differences will be described.

[0056] In this embodiment, CPU 221 saves the operation history of the user while the side screen is displayed in RAM 223 or storage unit 228. Here, the operation history refers to the time during which each side screen is displayed or the number of times the user presses the check button while each side screen is displayed, but other information may be saved as the operation history. FIG. 16 is a flowchart for explaining the detailed procedure of screen generation in this embodiment. In S1601, CPU 221 executes a side screen generation process. At this time, the initial display screen to be generated is switched according to the operation history. For example, if the time during which the similar character list screen is displayed is longer than other screens, or if the number of times the check button is pressed is greater than other screens, the initial display screen is displayed as the similar character list screen. Also, the initial display screen may be switched by setting.

[0057] According to the present embodiment, it is possible to reduce the number of operations to display the screen that the user most frequently uses for confirmation or judgment in checking the character recognition result, thereby further improving the usability in the work of checking the character recognition result.

[0058] [Fourth embodiment] In this embodiment, when displaying context information and the same or similar characters, multiple images in which the same person has written characters are preferentially displayed, and how the checking work can be made more efficient will be specifically described with reference to figures and flow charts. Note that in describing this embodiment, descriptions of the same configurations and processing procedures as the first, second, and third embodiments will be omitted, and only differences will be described.

[0059] FIG. 17(a) is a flowchart for explaining the detailed procedure of image generation in this embodiment. In S1711, the image forming apparatus 101 optically reads a paper document form to generate form image data, and the process proceeds to S1712. In S1712, the image forming apparatus 101 acquires information such as the time when the image was generated, the account and identification number of the user who generated the image, and the terminal number of the image forming apparatus 101, and the process proceeds to S1713. Here, the above information is referred to as scan information. In S1713, the image forming apparatus 101 transmits the generated image and scan information to the image processing server 102. The image processing server 102 receives the generated image and scan information, and associates the generated image with the scan information and stores them in the storage unit 228. When the storage of the generated image and scan information is completed, the process of the flowchart shown in FIG. 17(a) ends.

[0060] FIG. 17B is a flowchart for explaining the detailed procedure of image processing job generation in this embodiment. In S1714, in addition to the process of the first embodiment, if the extracted character string contains information having a person's name or an identification number that identifies a person, the CPU 221 extracts the information. Here, the above information is referred to as person information. For example, if there is a character string "Yamada Taro", it is extracted as a person's name. For example, if there is a character string "person's name code: 000001", it is extracted as a person's name code. Furthermore, the CPU 221 determines the writer of multiple images based on the scan information and person information associated with the images. For example, if the images have the same generation time and the same terminal number of the image forming device, the CPU 221 determines that the images are forms created by the same writer. For example, if the images have the same character string as a person's name and a number that identifies a person, the CPU 221 determines that the images are forms created by the same writer. The method of determining the writer is not limited to the above means, and other means may be used. Furthermore, in S1715, the CPU 221 associates the determined writer information with the job and registers it in the job list. Fig. 17(c) is a flowchart for explaining a detailed procedure for generating a screen in this embodiment. In S1716 for generating a side screen, the CPU 221 generates a screen that displays a plurality of images with the same writer.

[0061] 18(a) shows a comparison screen for identical / similar characters that displays multiple images according to this embodiment. The CPU 221 acquires all images included in the job list. Furthermore, the screen is generated by arranging multiple images to be displayed in an image list. Here, the image list displays an image 1801 containing a single character area selected by the user, an image 1802 written by the same writer as that image, and others in that order from the top. Then, all displayed images are highlighted with a colored frame according to the notation on the screen.

[0062] 18B is a similar character list screen for displaying a plurality of images displayed by this embodiment. The CPU 221 acquires a character group of similar characters of the single character area selected by the user from all jobs included in the job list. Then, in response to the selection of similar characters by the user, the single character area included in the acquired character group is displayed. Here, the similar character list displays the single character area 1811 of similar characters included in the image including the single character area selected by the user, the single character area 1812 of similar characters included in the image written by the same writer as the image, or other orders from the top. Also, lines may be displayed so that the boundaries of the single character area 1811 of similar characters included in the image including the single character area selected by the user, the single character area 1812 of similar characters included in the image written by the same writer as the image, and others can be clearly seen.

[0063] According to this embodiment, multiple images of the same writer with similar fonts or writing styles are preferentially displayed, which supports the user's judgment and reduces the time required for the judgment. This further improves the usability in the confirmation work of character recognition results.

[0064] [Other embodiments] The present disclosure can also be realized by a process in which a program for implementing one or more functions of the above-described embodiments is supplied to a system or device via a network or a storage medium, and one or more processors in a computer of the system or device read and execute the program. It can also be realized by a circuit (e.g., ASIC) that implements one or more functions.

[0065] The disclosure of the above-described embodiment includes the following configurations.

[0066] (Configuration 1) An information processing device comprising: an extraction means for extracting a specific character string having a specific attribute and unit characters constituting the specific character string based on the result of character recognition processing on a document image; a generation means for generating attribute information representing attributes of the extracted specific character string and context information regarding the extracted unit characters; and a display control means for displaying the extracted unit characters and the context information corresponding to the unit characters on a UI screen.

[0067] (Configuration 2) The information processing device according to configuration 1, further comprising an acquisition means for acquiring one unit character from a character list composed of the unit characters, wherein the display control means highlights on a UI screen a unit character area corresponding to the unit character acquired by the acquisition means and the context information.

[0068] (Configuration 3) The information processing device according to configuration 1, wherein the context information includes attribute information of the specific string, information of an image area in the document image corresponding to the specific string, attribute information of an adjacent string having a specific relationship with the specific string, information of an image area in the document image corresponding to the adjacent string, attribute information of a string representing a type of the document image, and information of an image area in the document image of a string representing the type.

[0069] (Configuration 4) The information processing device according to configuration 3, wherein the specific relationship is a key-value relationship.

[0070] (Configuration 5) The information processing device according to configuration 2, wherein the display control means has a button for switching between displaying and not displaying the acquired context information of the unit characters.

[0071] (Configuration 6) The information processing device according to configuration 2, wherein the display control means highlights on the UI screen a unit character area corresponding to the acquired unit character and a character string area including the acquired unit character.

[0072] (Configuration 7) The information processing device according to configuration 6, wherein the display control means highlights a unit character area corresponding to the acquired unit character and the character string area including the acquired unit character in different colors on the UI screen.

[0073] (Configuration 8) The information processing device according to any one of configurations 2 to 7, characterized in that the extraction means further extracts a degree of confidence in character recognition of the unit characters, and the display control means displays the obtained degree of confidence in character recognition of the unit characters on the UI screen.

[0074] (Configuration 9) The information processing device according to configuration 8, wherein when the degree of certainty of character recognition of the acquired unit character is less than a threshold, the display control means displays context information of the acquired unit character.

[0075] (Configuration 10) The information processing device according to configuration 7, further comprising a storage means for storing the acquired unit character and a similar character similar to the acquired unit character as a pair, wherein the display control means highlights a character area of ​​a similar character similar to the acquired unit character on the UI screen based on the data stored in the storage means.

[0076] (Configuration 11) The information processing device described in Configuration 10, wherein the display control means highlights on the UI screen in different colors a unit character area corresponding to the acquired unit character, the string area including the acquired unit character, and a character area of ​​a similar character that is similar to the acquired unit character.

[0077] (Configuration 12) The information processing device described in Configuration 10, characterized in that the UI screen further includes an area for accepting input as to whether to highlight the character area of ​​the similar characters, and when the acquisition means accepts input indicating whether to highlight the character area of ​​the similar characters, the display control means highlights the character area of ​​the similar characters on the UI screen.

[0078] (Configuration 13) The information processing device according to configuration 10, wherein the display control means displays a screen showing the recognition result on a main screen of the UI screen, and switches between multiple detailed screens of the recognition result and displays them on a side screen of the UI screen.

[0079] (Configuration 14) The information processing device described in Configuration 13, wherein the multiple detail screens are a context information screen that highlights a unit character area and context information corresponding to the acquired unit character, a comparison screen that compares the acquired unit character with similar characters that are similar to the acquired unit character, and a list screen of similar characters that are similar to the acquired unit character.

[0080] (Configuration 15) The information processing device according to configuration 14, wherein the storage means further stores a user's operation history of the side screen.

[0081] (Configuration 16) The information processing device described in Configuration 15, characterized in that the user's operation history is the time during which each of the multiple detail screens is displayed or the number of times the user presses the check button while each of the multiple detail screens is displayed.

[0082] (Configuration 17) The information processing device according to configuration 16, wherein the display control means switches and displays each of the plurality of detailed screens in descending order of the number of operation histories of the user.

[0083] (Configuration 18) The information processing device described in Configuration 14, characterized in that the storage means further stores results of character recognition processing on a plurality of document images, the extraction means further extracts a person's name or an identification number identifying a person from the results of character recognition processing on the document images, and the display control means displays a context information screen, a comparison screen, and a list screen of similar characters for the document image and the plurality of document images on the plurality of detail screens based on the extracted person's name or identification number identifying the person.

[0084] (Configuration 19) A method for controlling an information processing device, comprising the steps of: extracting a specific character string having a specific attribute and unit characters constituting the specific character string based on the results of character recognition processing on a document image; generating attribute information representing the attributes of the extracted specific character string and context information regarding the extracted unit characters; and displaying the extracted unit characters and the context information corresponding to the unit characters on a UI screen.

[0085] (Configuration 20) A program for causing a computer to function as the information processing device according to any one of configurations 1 to 18.

Claims

1. An extraction means for extracting a string and the unit characters that make up the string based on the results of character recognition processing on a document image, A generation means that generates context information relating to the extracted unit character based on the extracted string and attribute information representing the attributes of the extracted string, Display control means for displaying the extracted unit character and the context information corresponding to the unit character on a UI screen, An information processing device characterized by comprising:

2. The system further includes an acquisition means for obtaining one unit character from a character list composed of the extracted unit characters, The information processing apparatus according to claim 1, characterized in that the display control means highlights the unit character area corresponding to the acquired unit character and the context information on the UI screen.

3. The information processing apparatus according to claim 1, wherein the context information includes attribute information of the string, information of an image region corresponding to the string in the document image, attribute information of adjacent strings having a specific relationship with the string, information of an image region corresponding to the adjacent string in the document image, attribute information of a string representing the type of the document image, and information of an image region of a string representing the type in the document image.

4. The information processing apparatus according to claim 3, characterized in that the aforementioned specific relationship is a key-value relationship.

5. The information processing apparatus according to claim 2, characterized in that the UI screen includes a button for switching the display or hiding of the context information of the acquired unit character.

6. The information processing apparatus according to claim 2, characterized in that the display control means highlights the unit character area corresponding to the acquired unit character and the string area containing the acquired unit character on the UI screen.

7. The information processing apparatus according to claim 6, characterized in that the display control means highlights the unit character area corresponding to the acquired unit character and the string area containing the acquired unit character in different colors on the UI screen.

8. The extraction means further extracts the confidence level of character recognition for the acquired unit character, The information processing apparatus according to claim 2, characterized in that the display control means displays the confidence level of character recognition of the acquired unit character on the UI screen.

9. The information processing apparatus according to claim 8, wherein if the confidence level of character recognition of the acquired unit character is less than a threshold, the display control means displays context information of the acquired unit character.

10. The storage means further stores the acquired unit character and a similar character similar to the acquired unit character as a pair in the storage unit, The information processing apparatus according to claim 7, characterized in that the display control means highlights the character area of ​​a similar character to the acquired unit character on the UI screen based on the data stored by the storage means.

11. The information processing apparatus according to claim 10, characterized in that the display control means highlights the unit character area corresponding to the acquired unit character, the string area containing the acquired unit character, and the character area of ​​similar characters similar to the acquired unit character on the UI screen in different colors.

12. The UI screen further includes an area for receiving input on whether to highlight the character area of ​​the similar characters, The information processing apparatus according to claim 10, characterized in that when the acquisition means receives an input indicating the highlighting of the character area of ​​the similar character, the display control means highlights the character area of ​​the similar character on the UI screen.

13. The information processing apparatus according to claim 10, characterized in that the display control means displays a screen showing the recognition result on the main screen of the UI screen, and switches between displaying a plurality of detailed screens of the recognition result on the side screen of the UI screen.

14. The information processing apparatus according to claim 13, wherein the plurality of detail screens are a context information screen that highlights the unit character area and context information corresponding to the acquired unit character, a comparison screen that compares the acquired unit character with similar characters similar to the acquired unit character, and a list screen of similar characters similar to the acquired unit character.

15. The information processing apparatus according to claim 14, characterized in that the storage means stores the user's operation history of the side screen in the storage unit.

16. The information processing device according to claim 15, characterized in that the user's operation history is the time each of the multiple detail screens is displayed or the number of times the user presses the check button while each of the multiple detail screens is displayed.

17. The information processing apparatus according to claim 16, characterized in that the display control means switches and displays each of the plurality of detail screens in order of the user's operation history.

18. The storage means stores the results of character recognition processing on multiple document images in the storage unit. The extraction means further extracts a person's name or identification number that identifies a person from the results of character recognition processing on the plurality of document images. The information processing apparatus according to claim 14, wherein the display control means displays the plurality of document images, the context information screen of the plurality of document images, the comparison screen, and the list screen of similar characters on the plurality of detail screens based on the extracted person name or identification number that identifies the person.

19. A step of extracting a string and the unit characters that make up the string based on the results of character recognition processing on a document image, A step of generating context information relating to the extracted unit character based on the extracted string and attribute information representing the attributes of the extracted string, The steps include displaying the extracted unit character and the context information corresponding to the unit character on a UI screen, A control method for an information processing device, characterized by comprising the following:

20. A program for causing a computer to function as an information processing device according to any one of claims 1 to 18.

Images