Information processing device, program, and information processing method

Abstract

To reduce the risk that the occurence of false recognition is overlooked in character recognition processing.SOLUTION: A character recognition device 1 includes: a described content recognition unit 113 which recognizes one character shown in a character image GC by using each of first character recognition processing and second character recognition processing; a recognition result determination unit 114 which, when a candidate recognized character R[m1][1] which is a recognition result of the one character by the first character recognition processing and a candidate recognized character R[m2][1] which is a recognition result of the one character by the second character recognition processing are an identical character, determines that the candidate recognized character R[m1][1] and the candidate recognized character R[m2][1] are a decided recognized character RX and, when the candidate recognized character R[m1][1] and the candidate recognized character R[m2][1] are different characters, determines that one of the candidate recognized character R[m1][1] and the candidate recognized character R[m2][1] is a likely recognized character RY; and a display information generation unit 115 which generates display information JH for displaying the likely recognized character RY on a display unit 31 in an aspect different from the decided recognized character RX.SELECTED DRAWING: Figure 11

Description

[Technical field]

[0001] The present invention relates to an information processing device and a program. [Background technology]

[0002] A technique has been proposed for character recognition processing that recognizes characters included in an image (see, for example, Patent Document 1). [Prior art documents] [Patent documents]

[0003] [Patent Document 1] JP 2015-069256 A Summary of the Invention [Problem to be solved by the invention]

[0004] However, in the conventional technology, no matter how much one tries to improve the accuracy of character recognition in the character recognition process, it is difficult to completely prevent the occurrence of recognition errors in the character recognition process. Furthermore, when a recognition error occurs in the character recognition process, an operator who checks the results of the character recognition process may overlook the occurrence of the recognition error in the character recognition process.

[0005] The present invention has been made in consideration of the above-mentioned circumstances, and one of the problems to be solved is to provide a technology that makes it possible to reduce the risk of overlooking misrecognition in character recognition processing, compared to conventional technologies. [Means for solving the problem]

[0006] In order to solve the above problems, the information processing device of the present invention includes a recognition unit that recognizes a character represented in a character image by each of a first character recognition process that recognizes characters included in an image and a second character recognition process that recognizes characters included in an image; a determination unit that, when a first character that is the recognition result of the one character by the first character recognition process and a second character that is the recognition result of the one character by the second character recognition process are the same character, determines the first character and the second character as a confirmed character corresponding to the recognition result of the one character, and, when the first character and the second character are different characters, determines one of the first character and the second character as a candidate character corresponding to the recognition result of the one character; and a display control unit that displays the confirmed character or the candidate character determined by the determination unit on a display unit, and is characterized in that the display control unit displays the candidate character on the display unit in a manner different from the confirmed character.

[0007] In addition, the program of the present invention causes a processor to function as a recognition unit that recognizes a character represented in a character image by each of a first character recognition process that recognizes characters included in an image and a second character recognition process that recognizes characters included in an image; a determination unit that, when a first character that is the recognition result of the one character by the first character recognition process and a second character that is the recognition result of the one character by the second character recognition process are the same character, determines the first character and the second character as a confirmed character corresponding to the recognition result of the one character, and, when the first character and the second character are different characters, determines one of the first character and the second character as a candidate character corresponding to the recognition result of the one character; and a display control unit that displays the confirmed character or the candidate character determined by the determination unit on a display unit, and the display control unit displays the candidate character on the display unit in a manner different from the confirmed character. Effect of the Invention

[0008] According to the present invention, it is possible to reduce the risk that occurrence of erroneous recognition in character recognition processing will be overlooked, as compared with the conventional techniques. [Brief description of the drawings]

[0009] [Figure 1] FIG. 1 is an explanatory diagram showing an example of an overview of a character recognition system Sys according to an embodiment of the present invention. [Diagram 2] FIG. 13 is an explanatory diagram showing an example of an outline of an application form image GM. [Diagram 3] FIG. 13 is an explanatory diagram showing an example of an outline of an application content display image GH. [Figure 4] FIG. 13 is an explanatory diagram showing an example of an outline of an application content display image GH. [Diagram 5] 1 is a block diagram showing an example of a configuration of a character recognition device 1. FIG. [Figure 6] 1 is a block diagram showing an example of a hardware configuration of a character recognition device 1. FIG. [Figure 7] FIG. 13 is a diagram showing an example of the data configuration of application form information JM. [Figure 8] FIG. 13 is a diagram showing an example of the data configuration of algorithm property information JR. [Figure 9] 4 is a flowchart showing an example of the operation of the character recognition device 1. [Figure 10] 4 is a flowchart showing an example of the operation of the character recognition device 1. [Figure 11] 11A and 11B are explanatory diagrams for explaining an example of a character recognition process and a display character determination process; [Figure 12] 4 is a flowchart showing an example of the operation of the character recognition device 1. [Figure 13] FIG. 13 is an explanatory diagram showing an example of an overview of a character recognition system Sys-B according to a fourth modified example. [Figure 14] FIG. 2 is a block diagram showing an example of the configuration of a character recognition device 1B. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0010] Hereinafter, the embodiments of the present invention will be described with reference to the drawings. In each drawing, the dimensions and scale of each part are appropriately different from the actual ones. In addition, the embodiments described below are preferred examples of the present invention, and therefore various technically preferable limitations are attached, but the scope of the present invention is not limited to these embodiments unless otherwise specified in the following description to the effect that the present invention is limited.

[0011] [A. Embodiment] Hereinafter, an embodiment of the present invention will be described.

[0012] <1. Overview of the character recognition system> The character recognition system Sys will be outlined below with reference to FIGS.

[0013] FIG. 1 is an explanatory diagram for explaining an example of the configuration of a character recognition system Sys according to this embodiment.

[0014] As illustrated in FIG. 1, the character recognition system Sys includes a character recognition device 1, a terminal device 3, and an imaging device 5.

[0015] The character recognition device 1 (an example of an "information processing device") executes character recognition processing to recognize characters contained in an image, for example, using optical character recognition (OCR) technology. In this embodiment, it is assumed, as an example, that the image to be subjected to the character recognition processing by the character recognition device 1 is an application form image GM obtained by capturing an application form for joining a predetermined service. The application form image GM will be described later with reference to FIG. 2.

[0016] The imaging device 5 is, for example, an optical device such as a camera or an image scanner, which captures an image of the membership application form and supplies the terminal device 3 with image information JG indicating an application form image GM obtained as a result of the imaging.

[0017] The terminal device 3 includes a display unit 31 capable of displaying images, and an operation unit 32 that accepts operations by an operator of the terminal device 3. The terminal device 3 may be a stationary information device such as a personal computer, or may be a portable information terminal such as a tablet terminal or a smartphone.

[0018] In this embodiment, the terminal device 3 can communicate with the character recognition device 1 via the network NW, and transmits image information JG supplied from the imaging device 5 to the character recognition device 1 via the network NW. When the image information JG is supplied from the terminal device 3, the character recognition device 1 performs character recognition processing on the application form image GM indicated by the image information JG supplied from the terminal device 3. Then, based on the recognition result of the character recognition processing on the application form image GM, the character recognition device 1 generates display information JH for displaying an application content display image GH on the display unit 31, and transmits the generated display information JH to the terminal device 3. When the display information JH is supplied from the character recognition device 1, the terminal device 3 displays the application content display image GH on the display unit 31 based on the display information JH supplied from the character recognition device 1. The application content display image GH will be described later with reference to FIG. 3 and FIG. 4.

[0019] FIG. 2 is an explanatory diagram for explaining an example of the application form image GM.

[0020] As exemplified in FIG. 2, the application form image GM includes a plurality of fields. Each of the plurality of fields includes an item name writing area AM in which an item name, which is the name of an item of the application content to be written in the field, is written, and an application content writing area AK in which the application content related to the field is written. In the following, an image of the application form image GM showing a character string related to the application content written in the application content writing area AK is referred to as a character string image GCL. In the following, an image of the character string image GCL showing each of a plurality of characters included in the character string shown by the character string image GCL is referred to as a character image GC. In other words, the character string image GCL is an image including a plurality of character images GC.

[0021] In this embodiment, as an example, it is assumed that the membership application form has a field for an "application number," a field for "kana name," a field for "name," a field for "postal code," a field for "address," a field for "email," and a field for "telephone number," as shown in Figure 2. In the following, in the application form image GM, the character string image GCL showing the application contents entered in the entry field related to "application number" may be referred to as character string image GCL1, the character string image GCL showing the application contents entered in the entry field related to "kana name" may be referred to as character string image GCL2, the character string image GCL showing the application contents entered in the entry field related to "name" may be referred to as character string image GCL3, the character string image GCL showing the application contents entered in the entry field related to "postal code" may be referred to as character string image GCL4, the character string image GCL showing the application contents entered in the entry field related to "address" may be referred to as character string image GCL5, the character string image GCL showing the application contents entered in the entry field related to "email" may be referred to as character string image GCL6, and the character string image GCL showing the application contents entered in the entry field related to "telephone number" may be referred to as character string image GCL7.

[0022] In the following, it is assumed that the character recognition device 1 performs character recognition processing on an application form image GM having a format exemplified in FIG. 2, but the present invention is not limited to this aspect. In this embodiment, the character recognition device 1 can perform character recognition processing on documents having any format. In the following, it is assumed that the application contents written in the entry field are written in handwritten characters, which are handwritten characters, but the present invention is not limited to this aspect. In this embodiment, some or all of the multiple application contents written in the entry field may be written in type.

[0023] FIG. 3 is an explanatory diagram for explaining an example of the application content display image GH.

[0024] As illustrated in FIG. 3, the application content display image GH includes a plurality of item names that correspond one-to-one to a plurality of description columns in the application form image GM, a plurality of character string images GCL (character string images GCL1 to GCL7) that correspond one-to-one to the plurality of description columns in the application form image GM, and a plurality of recognized character strings LMK (recognized character strings LMK1 to LMK7) that correspond one-to-one to the plurality of description columns in the application form image GM.

[0025] Here, the recognized character string LMK is a character string obtained based on the result of the character recognition process executed on the character string image GCL included in the application form image GM by the character recognition device 1. Hereinafter, each of the plurality of characters included in the recognized character string LMK is referred to as a display character RH. That is, the display character RH is a character obtained based on the result of the character recognition process executed on the character image GC by the character recognition device 1.

[0026] Also, as illustrated in FIG. 3, the application content display image GH may include a caution mark MK. Here, the caution mark MK is an image displayed in the application content display image GH when the character recognition device 1 executes a character recognition process on the character image GC and determines the display character RH based on the result of the character recognition process, and the character recognition device 1 determines that there is a possibility that the character indicated by the character image GC is different from the display character RH. For example, in the example shown in FIG. 3, since the character recognition device 1 determines that there is a possibility that the character indicated by the character image GC1 included in the character string image GCL5 is different from the display character RH1, which is the character "height" included in the recognized character string LMK5, a caution mark MK1 is displayed at the position corresponding to the character image GC1 and the display character RH1. Also, in the example shown in FIG. 3, since the character recognition device 1 determines that there is a possibility that the character indicated by the character image GC2 included in the character string image GCL7 is different from the display character RH2, which is the character "7" included in the recognized character string LMK7, a caution mark MK2 is displayed at the position corresponding to the character image GC2 and the display character RH2.

[0027] As shown in FIG. 3, the application content display image GH includes a correction button SSB. Here, the correction button SSB is a button for making the display character RH modifiable when the operator operating the terminal device 3 compares the character image GC with the display character RH corresponding to the character image GC in the application content display image GH and determines that the display character RH is a character different from the character indicated by the character image GC.

[0028] FIG. 4 is an explanatory diagram for explaining an example of the application content display image GH when the correction button SSB is pressed.

[0029] 4, when the correction button SSB is pressed in the application content display image GH, the operator operating the terminal device 3 can edit each recognition character string LMK. Specifically, when the correction button SSB is pressed in the application content display image GH, the operator operating the terminal device 3 places the cursor CSR at a position corresponding to the display character RH selected by the operator operating the terminal device 3, thereby making the display character RH editable.

[0030] Also, as illustrated in FIG. 4, when the correction button SSB is pressed in the application content display image GH, and the operator who operates the terminal device 3 places the cursor CSR at a position corresponding to the display character RH and makes the display character RH editable, a pop-up image PRZ in which a correction candidate character RZ corresponding to the display character RH is displayed is displayed at the position corresponding to the display character RH. Here, the correction candidate character RZ is a character that is a correction candidate for the display character RH. More specifically, the correction candidate character RZ is a recognition candidate character R different from the display character RH among a plurality of recognition candidate characters R obtained by performing a character recognition process on the character image GC corresponding to the display character RH. Here, the recognition candidate character R is a character that is a candidate for the character indicated by the character image GC corresponding to the display character RH and is obtained by performing a character recognition process on the character image GC corresponding to the display character RH. Then, the operator who operates the terminal device 3 can replace the displayed character RH with the correction candidate character RZ by selecting the correction candidate character RZ displayed in the pop-up image PRZ.

[0031] In addition, in FIG. 4, an example is shown in which the operator who operates the terminal device 3 places the cursor CSR on the displayed character RH1 which is "zhang" included in the recognized character string LMK5 and makes the displayed character RH1 editable. And in FIG. 4, corresponding to the displayed character RH1 which is "zhang", there are the correction candidate character RZ-1 which is "tai", the correction candidate character RZ-2 which is "da", and the correction candidate character RZ-3 which is "quan", and an example is shown in which a pop-up image PRZ in which these correction candidate characters RZ1 to RZ3 are displayed is displayed near the displayed character RH1. And in FIG. 4, the operator who operates the terminal device 3 can correct the displayed character RH1 to one correction candidate character RZ by selecting one correction candidate character RZ from among the correction candidate characters RZ1 to RZ3 displayed in the pop-up image PRZ. For example, in FIG. 4, when the operator who operates the terminal device 3 selects "da" which is the correction candidate character RZ-2 from among the correction candidate character RZ-1 (tai), the correction candidate character RZ-2 (da), and the correction candidate character RZ-3 (quan) displayed in the pop-up image PRZ, the displayed character RH1 will be corrected (changed) from "zhang" to "da".

[0032] Also, as illustrated in FIGS. 3 and 4, the application content display image GH includes an OK button KTB. Here, the OK button KTB is a button for finalizing a plurality of recognized character strings LMK included in the application content display image GH. When the OK button KTB is pressed, each displayed character RH included in each recognized character string LMK is finalized, and the operator who operates the terminal device 3 cannot change each displayed character RH. Thereby, the character recognition system Sys finalizes the recognition result of the characters included in the application form image GM as a plurality of displayed characters RH displayed in the application content display image GH.

[0033] In this embodiment, an example is shown in which the operator operating the terminal device 3 can edit each recognition character string LMK only when the correction button SSB is pressed while the application content display image GH is displayed on the display unit 31, but the present invention is not limited to such an example. For example, the operator operating the terminal device 3 may be able to edit the recognition character string LMK at the time when the application content display image GH is displayed on the display unit 31. In this case, the application content display image GH does not need to include the correction button SSB.

[0034] <2. Functions of character recognition device> The functions of the character recognition device 1 will be described below with reference to FIGS.

[0035] FIG. 5 is a functional block diagram showing an example of the configuration of the character recognition device 1. As shown in FIG.

[0036] As illustrated in FIG. 5, the character recognition device 1 includes a control unit 11 that controls each part of the character recognition device 1, a memory unit 12 that stores various information, and a communication unit 13 that performs communication with an external device located outside the character recognition device 1.

[0037] As illustrated in FIG. 5, the control unit 11 includes an image information acquisition unit 111, a character area identification unit 112, a content recognition unit 113, a recognition result determination unit 114, a display information generation unit 115, an operation reception unit 116, and a learning execution unit 117.

[0038] The image information acquisition unit 111 acquires the image information JG from the terminal device 3 . The character area identification unit 112 identifies a plurality of application content description areas AK in the application form image GM indicated by the image information JG acquired by the image information acquisition unit 111. Specifically, the character area identification unit 112 identifies a plurality of application content description areas AK included in the application form image GM based on, for example, information (not shown) relating to the layout of the application form image GM stored in the storage unit 12.

[0039] The content recognition unit 113 (an example of a "recognition unit") performs character recognition processing on each character image GC contained in the character string image GCL displayed in the application content entry area AK, and identifies one or more recognition candidate characters R as the recognition result of the character indicated by the character image GC.

[0040] Specifically, the written content recognition unit 113 includes M character recognition units RG[1] to RG[M]. Here, the value M is a natural number that satisfies M≧2. In the following, the m-th character recognition unit RG among the M character recognition units RG[1] to RG[M] may be referred to as the character recognition unit RG[m]. Here, the variable m is a natural number that satisfies 1≦m≦M. The character recognition unit RG[m] performs character recognition processing on the character image GC, and can identify one or more recognition candidate characters R as the recognition result of the character indicated by the character image GC. Note that various methods can be applied to the character recognition process by the character recognition unit RG[m]. In this embodiment, as an example, it is assumed that a method using optical character recognition (OCR) is applied.

[0041] In addition, in this embodiment, the written content recognition unit 113 selects W character recognition units RG from among M character recognition units RG[1] to RG[M], and causes each of the selected W character recognition units RG to execute character recognition processing. Here, W is a natural number that satisfies 2≦W≦M. Note that in this embodiment, as an example, a case where "W=2" is assumed. In the following, the selected W (=2) character recognition units RG are referred to as character recognition unit RG[m1] and character recognition unit RG[m2]. Here, variable m1 is a natural number that satisfies 1≦m1≦M, and variable m2 is a natural number that satisfies 1≦m2≦M.

[0042] The recognition result determination unit 114 (an example of a "determination unit") determines one recognition candidate character R from among one or more recognition candidate characters R identified by the content recognition unit 113 as the recognition result of the character indicated by the character image GC, as the display character RH. The display information generation unit 115 (an example of a “display control unit”) generates display information JH for displaying, on the display unit 31, the application content display image GH representing the display characters RH determined by the recognition result determination unit 114.

[0043] The operation reception unit 116 (an example of a “reception unit”) receives operation information JS indicating the operation content of the operation unit 32 by the operator who operates the terminal device 3. The learning execution unit 117 (an example of a "learning unit") causes a learning model NN[m], which will be described later, to learn the relationship between the character image GC and the character indicated by the character image GC.

[0044] As illustrated in Fig. 5, the storage unit 12 stores application form information JM, algorithm property information JR, M learning models NN[1] to NN[M] in one-to-one correspondence with the M character recognition units RG[1] to RG[M], and a control program PG1 for controlling each unit of the character recognition device 1. If there are multiple types of documents, such as membership applications, that can be subject to character recognition processing by the character recognition device 1, the storage unit 12 stores multiple pieces of application form information JM in one-to-one correspondence with the multiple types of documents. The application form information JM will be described later in Fig. 7. The algorithm property information JR will be described later in Fig. 8.

[0045] The learning model NN[m] is, for example, a multi-layer neural network, which is a learning model that has learned, for one or more types of characters, the relationship between images containing those types of characters and the characters contained in those images. For convenience of explanation, in this embodiment, it is assumed that the types of characters shown in the image are six types: kanji, hiragana, katakana, Arabic numerals (hereinafter simply referred to as "numerals"), alphabets, and symbols. Here, symbols are a concept including, for example, dots, commas, periods, commas, hyphens, and arithmetic symbols.

[0046] In this embodiment, the character recognition unit RG[m] recognizes characters indicated by the character image GC by using the learning model NN[m]. That is, in this embodiment, the character recognition unit RG[m] executes character recognition processing by AI (Artificial Intelligence)-OCR by using the learning model NN[m] stored in the storage unit 12. In addition, in this embodiment, the M character recognition units RG execute the character recognition process using different algorithms. That is, the algorithm used by the character recognition unit RG[m1] in the character recognition process is different from the algorithm used by the character recognition unit RG[m2] in the character recognition process. Here, "different algorithms" may mean, for example, that the types of learning models NN used in the character recognition process (for example, the types of neural networks) are different, or that the data sets used when training the learning models NN used in the character recognition process are different, or may include both of these concepts. Specifically, in this embodiment, as an example, it is assumed that the characters included in the image used when the character recognition unit RG[m1] trains the learning model NN[m1] used in the character recognition process include characters of a different type from the characters included in the image used when the character recognition unit RG[m2] trains the learning model NN[m2] used in the character recognition process. In this case, the characters included in the image used when the character recognition unit RG[m1] trains the learning model NN[m1] used in the character recognition process may include the same type of characters as the characters included in the image used when the character recognition unit RG[m2] trains the learning model NN[m2] used in the character recognition process. In addition, in this embodiment, as an example, it is assumed that the neural network constituting the learning model NN[m1] used in the character recognition process by the character recognition unit RG[m1] and the neural network constituting the learning model NN[m2] used in the character recognition process by the character recognition unit RG[m2] are different types of neural networks. However, the present invention is not limited to such an aspect. The neural network constituting the learning model NN[m1] used by the character recognition unit RG[m1] in the character recognition processing and the neural network constituting the learning model NN[m2] used by the character recognition unit RG[m2] in the character recognition processing may be the same type of neural network.

[0047] In this embodiment, as an example, it is assumed that the storage unit 12 stores M learning models NN[1] to NN[M] that correspond one-to-one to the M character recognition units RG[1] to RG[M], but the present invention is not limited to this. For example, some or all of the M character recognition units RG[1] to RG[M] may be components that perform character recognition processing without using a neural network.

[0048] FIG. 6 is a diagram showing an example of a hardware configuration of the character recognition device 1. As shown in FIG.

[0049] As illustrated in FIG. 6, the character recognition device 1 includes a processor 1001 that controls each part of the character recognition device 1, a storage device 1002 that stores various information, and a communication device 1003 for communicating with an external device located outside the character recognition device 1.

[0050] Of these, the storage device 1002 includes one or both of a volatile memory such as a RAM (Random Access Memory) that functions as a working area for the processor 1001, and a non-volatile memory such as an EEPROM (Electrically Erasable Programmable Read-Only Memory) that stores various information such as the control program PG1, and provides the function of the storage unit 12. The processor 1001 is configured to include, for example, one or more central processing units (CPUs). The processor 1001 executes a control program PG1 stored in the storage device 1002 and operates according to the control program PG1 to function as the control unit 11. The processor 1001 may be configured to include hardware such as a graphics processing unit (GPU), a digital signal processor (DSP), or a field programmable gate array (FPGA) in addition to the one or more CPUs, or in place of a part or all of the one or more CPUs. In this case, a part or all of the control unit 11 realized by the processor 1001 may be realized by hardware such as a DSP. The communication device 1003 is hardware for communicating with an external device that exists outside the character recognition device 1 via one or both of a wired network and a wireless network, and provides the function of the communication unit 13 .

[0051] 7 is an explanatory diagram showing an example of the data configuration of application form information JM. Here, application form information JM is information regarding a plurality of fields included in application form image GM.

[0052] 7, the application form information JM has a number of records that correspond one-to-one to the number of fields in the application form image GM. Each record in the application form information JM has an item ID, an item name, and character type information JMM.

[0053] Among these, the item ID is information for identifying each entry field from among a plurality of entry fields that the application form image GM has. Moreover, the item name is the name written in the item name writing area AM of each entry field of the application form image GM.

[0054] Furthermore, the character type information JMM is information regarding the type of characters that may be entered in the application content entry area AK of each entry field of the application form image GM. Specifically, in this embodiment, the character type information JMM includes kanji use or not information JMM1 indicating whether kanji may be entered in the application content entry area AK, hiragana use or not information JMM2 indicating whether hiragana may be entered in the application content entry area AK, katakana use or not information JMM3 indicating whether katakana may be entered in the application content entry area AK, number use or not information JMM4 indicating whether numbers may be entered in the application content entry area AK, alphabet use or not information JMM5 indicating whether alphabets may be entered in the application content entry area AK, and symbol use or not information JMM6 indicating whether symbols may be entered in the application content entry area AK. 7, when the kanji usage information JMM1 indicates "1", it indicates that there is a possibility that a kanji will be written in the entry field corresponding to the kanji usage information JMM1, and when the kanji usage information JMM1 indicates "0", it indicates that there is no possibility that a kanji will be written in the entry field corresponding to the kanji usage information JMM1. However, the kanji usage information JMM1 is not limited to binary information indicating either "1" or "0", and may be information indicating the possibility that a kanji will be written in the entry field corresponding to the kanji usage information JMM1, for example, with a real number between 0 and 1. Note that the hiragana usage information JMM2, the katakana usage information JMM3, the number usage information JMM4, the alphabet usage information JMM5, and the symbol usage information JMM6 are similar to the kanji usage information JMM1, except that the types of characters used are different.

[0055] 8 is an explanatory diagram showing an example of the data configuration of the algorithm property information JR. Here, the algorithm property information JR is information indicating the properties of the algorithm used by the character recognition unit RG in the character recognition process.

[0056] 8, the algorithm property information JR has M records in one-to-one correspondence with the M character recognition units RG[1] to RG[M]. Each record of the algorithm property information JR has an algorithm ID, learning presence / absence information, learning model information, and valid character type information JRM.

[0057] Of these, the algorithm ID is information for identifying the algorithm that the character recognition unit RG[m] uses when performing character recognition processing from among multiple algorithms that may be used in the content recognition unit 113. The learning information is information indicating whether or not the algorithm used by the character recognition unit RG[m] when performing character recognition processing is an algorithm that uses a neural network. Moreover, the learning model information is information indicating a learning model NN[m] configured by a neural network when the algorithm used by the character recognition unit RG[m] when performing character recognition processing is an algorithm using a neural network. In other words, the learning model information is information for identifying the learning model NN[m] to be used by the character recognition unit RG[m] from among the multiple learning models NN stored in the storage unit 12.

[0058] The valid character type information JRM is information that indicates the types of characters that can be recognized by the algorithm used when the character recognition unit RG[m] performs character recognition processing. Specifically, in this embodiment, the valid character type information JRM includes kanji compatibility information JRM1 indicating whether the algorithm used by the character recognition unit RG[m] when performing character recognition processing is an algorithm suitable for recognizing kanji, hiragana compatibility information JRM2 indicating whether the algorithm used by the character recognition unit RG[m] when performing character recognition processing is an algorithm suitable for recognizing hiragana, katakana compatibility information JRM3 indicating whether the algorithm used by the character recognition unit RG[m] when performing character recognition processing is an algorithm suitable for recognizing katakana, number compatibility information JRM4 indicating whether the algorithm used by the character recognition unit RG[m] when performing character recognition processing is an algorithm suitable for recognizing numbers, alphabet compatibility information JRM5 indicating whether the algorithm used by the character recognition unit RG[m] when performing character recognition processing is an algorithm suitable for recognizing the alphabet, and symbol compatibility information JRM6 indicating whether the algorithm used by the character recognition unit RG[m] when performing character recognition processing is an algorithm suitable for recognizing symbols. 8, when the kanji support information JRM1 indicates "1", it indicates that the algorithm used in the character recognition process by the character recognition unit RG corresponding to the kanji support information JRM1 is an algorithm suitable for recognizing kanji, and when the kanji support information JRM1 indicates "0", it indicates that the algorithm used in the character recognition process by the character recognition unit RG corresponding to the kanji support information JRM1 is an algorithm not suitable for recognizing kanji. However, the kanji support information JRM1 is not binary information indicating either "1" or "0", but may be information indicating the degree of suitability of the algorithm used in the character recognition process by the character recognition unit RG corresponding to the kanji support information JRM1 for use in recognizing kanji, using a real number between 0 and 1. Hiragana support information JRM2, katakana support information JRM3, number support information JRM4, alphabet support information JRM5, and symbol support information JRM6 are similar to kanji support information JRM1, except for the types of characters.

[0059] <3. Operation of the character recognition device> Hereinafter, the operation of the character recognition device 1 will be described with reference to FIGS.

[0060] <3.1. Application content display process> Figures 9 and 10 are flowcharts showing an example of the operation of the character recognition device 1 when the character recognition device 1 executes a process (hereinafter referred to as the "application content display process") of generating display information JH indicating an application content display image GH based on image information JG indicating an application form image GM.

[0061] As illustrated in FIG. 9, when the application content display process is started, the image information acquisition unit 111 acquires image information JG supplied from the terminal device 3 (S101). Next, the character area identifying unit 112 identifies a plurality of application content writing areas AK in the application form image GM indicated by the image information JG acquired by the image information acquiring unit 111 in step S101 (S103). Then, the character area specification unit 112 selects one application content description area AK from among the multiple application content description areas AK specified in step S103 (S105). Note that in step S105, the character area specification unit 112 selects, from among the multiple application content description areas AK, an application content description area AK that has not been selected since the application content display process was started.

[0062] As illustrated in FIG. 9, the description content recognition unit 113 selects W character recognition units RG for performing character recognition processing on the character string image GCL displayed in the application content entry area AK selected in step S105 (S107). In this embodiment, in step S107, the description content recognition unit 113 selects character recognition unit RG[m1] and character recognition unit RG[m2] from among the M character recognition units RG[1] to RG[M] based on the character type information JMM corresponding to the application content entry area AK and the algorithm property information JR. Specifically, in step S107, the description content recognition unit 113 selects character recognition unit RG[m1] and character recognition unit RG[m2] from among the M character recognition units RG[1] to RG[M] based on the similarity between the character type information JMM corresponding to the application content entry area AK and the valid character type information JRM corresponding to each character recognition unit RG. More specifically, in step S107, the content recognition unit 113 selects two character recognition units RG from the character recognition units RG[1] to RG[M] that have a higher similarity between the character type information JMM corresponding to the application content entry area AK and the valid character type information JRM corresponding to each character recognition unit RG than the others, as character recognition unit RG[m1] and character recognition unit RG[m2]. Here, as the similarity between the character type information JMM and the valid character type information JRM, for example, the inner product of a vector VM having each of the pieces of information (JMM1 to JMM6) included in the character type information JMM as its elements and a vector VR having each of the pieces of information (JRM1 to JRM6) included in the valid character type information JRM as its elements may be used. More specifically, when the vector VM is set as VM=(JMM1, JMM2, JMM3, JMM4, JMM5, JMM6) and the vector VR is set as VR=(JRM1, JRM2, JRM3, JRM4, JRM5, JRM6), the inner product of the vector VM and the vector VR (JMM1*JRM1+JMM2*JRM2+JMM3*JRM3+JMM4*JRM4+JMM5*JRM5+JMM6*JRM6) may be used as the similarity between the character type information JMM and the valid character type information JRM.

[0063] As illustrated in FIG. 9, each of the character recognition units RG[m1] and RG[m2] divides the character string image GCL displayed in the application content entry area AK selected in step S105 into a plurality of character images GC (S109). Then, each of the character recognition units RG[m1] and RG[m2] selects one character image GC from among the multiple character images GC classified in step S109 (S111). Next, each of the character recognition units RG[m1] and RG[m2] executes character recognition processing on the character image GC selected in step S111 (S113). 10, the recognition result determination unit 114 determines the display character RH based on the result of the character recognition process in step S113 (S115). Note that, hereinafter, the process of determining the display character RH by the recognition result determination unit 114 is referred to as a display character determination process.

[0064] FIG. 11 is an explanatory diagram for explaining the character recognition process performed by the character recognition units RG[m1] and RG[m2] in step S113, and the display character determination process performed by the recognition result determination unit 114 in step S115.

[0065] As illustrated in Fig. 11, the character recognition unit RG[m1] performs character recognition processing on the character image GC to identify Q1 recognition candidate characters R[m1][1] to R[m1][Q1]. Here, the value Q1 is a natural number that satisfies Q1 ≥ 1. Hereinafter, the q1-th recognition candidate character R among the Q1 recognition candidate characters R[m1][1] to R[m1][Q1] is referred to as recognition candidate character R[m1][q1]. Here, the variable q1 is a natural number that satisfies 1 ≤ q1 ≤ Q1. The character recognition unit RG[m1] also calculates a recognition accuracy D[m1][q1] corresponding to the recognition candidate character R[m1][q1]. Here, the recognition accuracy D[m1][q1] is an index value indicating the likelihood of the recognition result of the character represented in the character image GC when the character recognition unit RG[m1] performs character recognition processing on the character image GC. In this embodiment, the higher the recognition accuracy, the more likely the recognition result is. In this embodiment, the Q1 recognition candidate characters R[m1][1] to R[m1][Q1] are determined such that the Q1 recognition accuracy D[m1][1] to D[m1][Q1] corresponding one-to-one to the Q1 recognition candidate characters R[m1][1] to R[m1][Q1] satisfy the relationship "D[m1][1]≧D[m1][2]≧...≧D[m1][Q1]". That is, the recognition candidate character R[m1][1] is the recognition candidate character R with the highest recognition accuracy among the Q1 recognition candidate characters R[m1][1] to R[m1][Q1] identified by the character recognition unit RG[m1]. In this embodiment, the character recognition process performed by the character recognition unit RG[m1] is an example of a "first character recognition process", the recognition candidate character R[m1][1] with the highest recognition accuracy identified by the character recognition unit RG[m1] is an example of a "first character", and the learning model NN[m1] used in the character recognition process by the character recognition unit RG[m1] is an example of a "first learning model".

[0066] As illustrated in Fig. 11, the character recognition unit RG[m2] performs character recognition processing on the character image GC to identify Q2 recognition candidate characters R[m2][1] to R[m2][Q2]. Here, the value Q2 is a natural number that satisfies Q2 ≥ 1. Hereinafter, the q2th recognition candidate character R among the Q2 recognition candidate characters R[m2][1] to R[m2][Q2] is referred to as recognition candidate character R[m2][q2]. Here, the variable q2 is a natural number that satisfies 1 ≤ q2 ≤ Q2. The character recognition unit RG[m2] also calculates the recognition accuracy D[m2][q2] corresponding to the recognition candidate character R[m2][q2]. Here, the recognition accuracy D[m2][q2] is an index value indicating the likelihood of the recognition result of the character represented in the character image GC when the character recognition unit RG[m2] performs character recognition processing on the character image GC. In this embodiment, the Q2 recognition candidate characters R[m2][1] to R[m2][Q2] are determined so that the Q2 recognition accuracy D[m2][1] to D[m2][Q2] that correspond one-to-one to the Q2 recognition candidate characters R[m2][1] to R[m2][Q2] satisfy the relationship "D[m2][1]≧D[m2][2]≧...≧D[m2][Q2]". That is, the recognition candidate character R[m2][1] is the recognition candidate character R with the highest recognition accuracy among the Q2 recognition candidate characters R[m2][1] to R[m2][Q2] identified by the character recognition unit RG[m2]. In this embodiment, the character recognition process performed by the character recognition unit RG[m2] is an example of a "second character recognition process," and the recognition candidate character R[m2][1] with the highest recognition accuracy identified by the character recognition unit RG[m2] is an example of a "second character."

[0067] As illustrated in FIG. 11, the recognition result determination unit 114 executes a display character determination process which determines a display character RH based on the recognition candidate character R[m1][1] having the highest recognition accuracy among the Q1 recognition candidate characters R[m1][1] to R[m1][Q1] identified as a result of the character recognition process by the character recognition unit RG[m1], and the recognition candidate character R[m2][1] having the highest recognition accuracy among the Q2 recognition candidate characters R[m2][1] to R[m2][Q2] identified as a result of the character recognition process by the character recognition unit RG[m2]. Specifically, in the display character determination process, if the recognition candidate character R[m1][1] and the recognition candidate character R[m2][1] are the same character, the recognition result determination unit 114 determines the recognition candidate character R[m1][1] and the recognition candidate character R[m2][1] as the confirmed recognition character RX (an example of the "confirmed character"). In addition, if the recognition candidate character R[m1][1] and the recognition candidate character R[m2][1] are different characters and the recognition accuracy D[m1][1] and the recognition accuracy D[m2][1] satisfy D[m1][1]>D[m2][1], the recognition result determination unit 114 determines the recognition candidate character R[m1][1] as the likely recognition character RY (an example of the "candidate character"). Furthermore, when the recognition candidate character R[m1][1] and the recognition candidate character R[m2][1] are different characters and the recognition accuracy D[m1][1] and the recognition accuracy D[m2][1] satisfy D[m2][1]≧D[m1][1], the recognition result determination unit 114 determines the recognition candidate character R[m2][1] as a likely recognized character RY (another example of a "candidate character").Then, the recognition result determination unit 114 determines the confirmed recognition character RX or the likely recognized character RY as the display character RH.

[0068] In FIG. 11, among the Q1 recognized candidate characters R[m1][1] to R[m1][Q1] specified as the result of the character recognition process by the character recognition unit RG[m1], the recognized candidate character R[m1][1] is "丈", the recognition accuracy D[m1][1] is "98%", the recognized candidate character R[m1][2] is "大", the recognition accuracy D[m1][2] is "85%", the recognized candidate character R[m1][Q1] is "炎", and the recognition accuracy D[m1][1] is "25%". This is an example case. Also, in FIG. 11, among the Q2 recognized candidate characters R[m2][1] to R[m2][Q2] specified as the result of the character recognition process by the character recognition unit RG[m2], the recognized candidate character R[m2][1] is "太", the recognition accuracy D[m2][1] is "92%", the recognized candidate character R[m2][2] is "犬", the recognition accuracy D[m2][2] is "80%", the recognized candidate character R[m2][Q2] is "人", and the recognition accuracy D[m2][Q2] is "45%". This is an example case. As a result, in the example shown in FIG. 11, since the "丈" which is the recognized candidate character R[m1][1] and the "太" which is the recognized candidate character R[m2][1] are different characters, the recognition result determination unit 114 determines in the display character determination process that there is no definite recognized character RX. Also, in the example shown in FIG. 11, since the recognition accuracy D[m1][1] corresponding to the "丈" which is the recognized candidate character R[m1][1] is "98%", which is greater than the "92%" of the recognition accuracy D[m2][1] corresponding to the "太" which is the recognized candidate character R[m2][1], the recognition result determination unit 114 determines the "丈" which is the recognized candidate character R[m1][1] as the likely recognized character RY in the display character determination process. And in the example shown in FIG. 11, since there is no definite recognized character RX and the "丈" which is the recognized candidate character R[m1][1] is determined as the likely recognized character RY, the recognition result determination unit 114 determines the "丈" which is the recognized candidate character R[m1][1] determined as the likely recognized character RY as the display character RH in the display character determination process.

[0069] Return to FIG. 10 for the explanation. As illustrated in FIG. 10, the content recognition unit 113 determines whether or not the character recognition process in step S113 and the display character determination process in step S115 have been completed for all of the multiple character images GC separated from the character string image GCL in step S109 (S117). If the result of the determination in step S117 is negative, the description content recognition unit 113 advances the process to step S111.

[0070] As illustrated in FIG. 10, if the result of the judgment in step S117 is positive, the character area identification unit 112 determines whether or not the character recognition process in step S113 and the display character determination process in step S115 have been completed for all of the multiple application content description areas AK identified in step S103 (S119). If the result of the determination in step S119 is negative, the description content recognition unit 113 advances the process to step S105.

[0071] As illustrated in FIG. 10, if the result of the determination in step S119 is positive, the display information generating unit 115 generates display information JH showing an application content display image GH including display characters RH (S121). Specifically, in step S121, the display information generating unit 115 generates the display information JH so that the display character RH corresponding to the recognition candidate character R determined to be the likely recognition character RY in the display character determination process and the display character RH corresponding to the recognition candidate character R determined to be the confirmed recognition character RX in the display character determination process are displayed in different manners in the application content display image GH. More specifically, in step S121, when the display character RH is the recognition candidate character R determined to be the likely recognition character RY in the display character determination process, the display information generating unit 115 generates the display information JH so that an attention drawing mark MK is placed at a position corresponding to the display character RH in the application content display image GH. In the application content display image GH illustrated in FIG. 3, among the plurality of display characters RH included in the recognition character strings LMK1 to LMK7, the display character RH1 "height" included in the recognition character string LMK5 and the display character RH2 "7" included in the recognition character string LMK7 correspond to the recognition candidate character R determined as the strong recognition character RY, and the display characters RH other than the display characters RH1 and RH2 correspond to the recognition candidate character R determined as the confirmed recognition character RX. Therefore, in the application content display image GH illustrated in FIG. 3, an attention call mark MK1 is displayed at the position corresponding to the display character RH1, and an attention call mark MK2 is displayed at the position corresponding to the display character RH2.

[0072] Note that when the process of step S121 is completed, the control unit 11 ends the application content display process shown in FIGS. 9 and 10.

[0073] As described above, according to the present embodiment, the display information generation unit 115 generates display information JH indicating an application content display image GH in which the display character RH corresponding to the strong recognition character RY and the display character RH corresponding to the confirmed recognition character RX are displayed in different manners. Specifically, according to the present embodiment, the display information generation unit 115 generates display information JH indicating an application content display image GH in which the display character RH corresponding to the strong recognition character RY is emphasized by the attention call mark MK as compared with the display character RH corresponding to the confirmed recognition character RX. Therefore, according to the present embodiment, when the possibility that the character indicated by the character image GC and the display character RH are different characters is high, the display character RH can be emphasized in the application content display image GH as compared with the case where the possibility is low. Thus, according to the present embodiment, the possibility that the operator operating the terminal device 3 overlooks that the character indicated by the character image GC and the display character RH are different characters can be reduced as compared with the case where they are displayed in the same manner.

[0074] According to the present embodiment, the written content recognition unit 113 selects the character recognition unit RG[m1] and the character recognition unit RG[m2] suitable for the character recognition process on the character string image GCL from among the M character recognition units RG[1] to RG[M], and executes the character recognition process using the two character recognition units RG[m1] and RG[m2]. As described above, the learning model NN[m1] used in the character recognition process by the character recognition unit RG[m1] is different from the learning model NN[m2] used in the character recognition process by the character recognition unit RG[m2]. Therefore, in the present embodiment, the accuracy of the character recognition of the character image GC by the character recognition device 1 when the result of the character recognition process by the character recognition unit RG[m1] and the result of the character recognition process by the character recognition unit RG[m2] match is higher than the accuracy of the character recognition of the character image GC by the character recognition device 1 when the result of the character recognition process by the character recognition unit RG[m1] and the result of the character recognition process by the character recognition unit RG[m2] differ. That is, in this embodiment, when the result of the character recognition process by the character recognition unit RG[m1] and the result of the character recognition process by the character recognition unit RG[m2] match, it is estimated that the accuracy of the character recognition of the character image GC by the character recognition device 1 is relatively high, whereas when the result of the character recognition process by the character recognition unit RG[m1] and the result of the character recognition process by the character recognition unit RG[m2] differ, it is estimated that the accuracy of the character recognition of the character image GC by the character recognition device 1 is relatively low. In this embodiment, when the result of the character recognition process by the character recognition unit RG[m1] and the result of the character recognition process by the character recognition unit RG[m2] differ, the display character RH related to the result of the character recognition process is highlighted by the attention mark MK. Therefore, according to this embodiment, the operator who operates the terminal device 3 can easily visually recognize the display character RH with a relatively low accuracy of the character recognition by the character recognition device 1. In addition, in this embodiment, when character recognition processing is performed using two character recognition units RG, ie, character recognition unit RG[m1] and character recognition unit RG[m2], and when the result of the character recognition processing by character recognition unit RG[m1] matches the result of the character recognition processing by character recognition unit RG[m2], the accuracy of character recognition of the character image GC is at least equal to or higher than the accuracy of character recognition of the character image GC when character recognition processing is performed using one of character recognition units RG of character recognition unit RG[m1] and character recognition unit RG[m2]. That is, in this embodiment, when character recognition processing is performed using two character recognition units RG, ie, character recognition unit RG[m1] and character recognition unit RG[m2], the accuracy of character recognition of the character image GC recognized as the confirmed recognized character RX is equal to or higher than the accuracy of character recognition of the character image GC when character recognition processing is performed using one of character recognition units RG of character recognition unit RG[m1] and character recognition unit RG[m2]. Therefore, according to this embodiment, compared to the case where the character recognition unit RG is executed by a single character recognition unit RG, the possibility that the character indicated by the character image GC and the display character RH determined as the confirmed recognized character RX without the warning mark MK will be different characters can be reduced.

[0075] <3.2. Character correction processing> FIG. 12 is a flowchart showing an example of the operation of the character recognition device 1 when the character recognition device 1 executes a process of correcting the display characters RH displayed in the application content display image GH based on an operation by an operator of the terminal device 3 (hereinafter referred to as the "character correction process"). In this embodiment, the character correction process is started when the operator operating the terminal device 3 presses the correction button SSB in the application content display image GH. As described above, the character correction process may be started regardless of whether the correction button SSB is pressed. Specifically, the character correction process may be started, for example, when the application content display image GH is displayed on the display unit 31.

[0076] As illustrated in FIG. 12, when the character correction process is started, the operation reception unit 116 specifies the display character RH at the position corresponding to the cursor CSR in the application content display image GH based on the operation information JS supplied from the terminal device 3 (S201).

[0077] Next, the description content recognition unit 113 specifies a plurality of recognition candidate characters R corresponding to the display character RH specified in step S201 (S203). Specifically, in step S203, the description content recognition unit 113, when performing character recognition processing on the character image GC corresponding to the display character RH specified in step S201, specifies Q1 recognition candidate characters R[m1][1] to R[m1][Q1] and Q2 recognition candidate characters R[m2][1] to R[m2][Q2] identified as the recognition results of the character image GC.

[0078] Then, the description content recognition unit 113 specifies a correction candidate character RZ from among the plurality of recognition candidate characters R specified in step S203 (S205). Specifically, the description content recognition unit 113 specifies, as the correction candidate character RZ, a recognition candidate character R among the plurality of recognition candidate characters R specified in step S203, where the recognition accuracy D is equal to or higher than a predetermined threshold value Dth and the recognition candidate character R is other than the display character RH. In the present embodiment, as an example, a case where the threshold value Dth is "80%" is assumed. For this reason, for example, in the example shown in FIG. 11, the recognition candidate character R[m2][1] which is "太" with a recognition accuracy D[m2][1] of "92%", the recognition candidate character R[m1][2] which is "大" with a recognition accuracy D[m1][2] of "85%", and the recognition candidate character R[m2][2] which is "犬" with a recognition accuracy D[m2][2] of "80%" are specified as the correction candidate character RZ. Note that in the example shown in FIG. 11, since the recognition candidate character R[m1][1] which is "丈" with a recognition accuracy D[m1][1] of "98%" is determined as the display character RH, the recognition candidate character R[m1][1] is not specified as the correction candidate character RZ.

[0079] 12, the display information generating unit 115 updates the display information JH indicating the application content display image GH so that a pop-up image PRZ is displayed in the application content display image GH and the correction candidate character RZ identified in step S205 is displayed in the pop-up image PRZ, and controls each unit of the character recognition device 1 so that the updated display information JH is transmitted to the terminal device 3 (S207). For this reason, for example, as shown in the example of FIG. 4, in the application content display image GH displayed on the display unit 31, a pop-up image PRZ displaying the correction candidate character RZ-1 "Futoshi", the correction candidate character RZ-2 "Large", and the correction candidate character RZ-3 "Dog" is displayed near the display character RH1 "Length".

[0080] As illustrated in FIG. 12, the operation receiving unit 116 determines whether or not operation information JS indicating an instruction to correct the display character RH has been supplied from the terminal device 3 (S209). If the result of the determination in step S209 is negative, the operation receiving unit 116 advances the process to step S213. If the result of the determination in step S209 is positive, the operation receiving unit 116 receives operation information JS indicating the contents of an instruction to correct the displayed character RH.

[0081] If the result of the determination in step S209 is positive, the learning execution unit 117 generates or updates the correction history information JSH based on the operation information JS accepted by the operation acceptance unit 116 in step S209 (S211), and the process proceeds to step S213. Here, the correction history information JSH is information including the correction content in step S209 (display character RH after correction), the display character RH corrected in step S209 (display character RH before correction), a character image GC corresponding to the display character RH before correction, and the character recognition unit RG[m1] and the character recognition unit RG[m2] that have identified the recognition candidate character R corresponding to the display character RH before correction.

[0082] Thereafter, the operation receiving unit 116 determines whether or not operation information JS indicating that the confirmation button KTB has been pressed has been supplied from the terminal device 3 (S213). If the result of the determination in step S213 is negative, the operation receiving unit 116 advances the process to step S201.

[0083] If the result of the judgment in step S213 is positive, the learning execution unit 117 trains the learning model NN[m1] corresponding to the character recognition unit RG[m1] and the learning model NN[m2] corresponding to the character recognition unit RG[m2] to learn that the character image GC represents the corrected display character RH based on the correction history information JSH (S215). In addition, in step S215, the learning execution unit 117 causes the learning model NN[m1] corresponding to the character recognition unit RG[m1] to learn that the character image GC represents the corrected display character RH only when the corrected display character RH and the recognition candidate character R[m1][1] are different characters. However, in step S215, even when the corrected display character RH and the recognition candidate character R[m1][1] are the same character, the learning execution unit 117 may cause the learning model NN[m1] corresponding to the character recognition unit RG[m1] to learn that the character image GC represents the corrected display character RH. Also, in step S215, the learning execution unit 117 causes the learning model NN[m2] corresponding to the character recognition unit RG[m2] to learn that the character image GC represents the corrected display character RH only when the corrected display character RH and the recognition candidate character R[m2][1] are different characters. However, in step S215, even when the corrected display character RH and the recognition candidate character R[m2][1] are the same character, the learning execution unit 117 may cause the learning model NN[m2] corresponding to the character recognition unit RG[m2] to learn that the character image GC represents the corrected display character RH.

[0084] As described above, according to this embodiment, the learning model NN[m1] corresponding to the character recognition unit RG[m1] and the learning model NN[m2] corresponding to the character recognition unit RG[m2] are made to learn that the character image GC represents the corrected display character RH based on the correction history information JSH. Therefore, according to this embodiment, the accuracy of recognition of the character represented by the character image GC by the character recognition unit RG[m] can be improved compared to a case where the learning model NN[m] is not subjected to learning based on the result of the character correction process.

[0085] <4. Conclusion of the embodiment> As described above, in this embodiment, the character recognition device 1 (an example of an "information processing device") recognizes a character (an example of a "first character") displayed in a character image GC by each of the character recognition process performed by the character recognition unit RG[m1] (an example of a "first character") and the character recognition process performed by the character recognition unit RG[m2] (an example of a "second character"), when a recognized candidate character R[m1][1] (an example of a "first character") which is a recognition result of the character displayed in the character image GC by the character recognition process performed by the character recognition unit RG[m1], and a recognized candidate character R[m2][1] (an example of a "second character") which is a recognition result of the character displayed in the character image GC by the character recognition process performed by the character recognition unit RG[m2] are the same character, The system includes a recognition result determination unit 114 (an example of a "determining unit") that determines R[m1][1] and the recognition candidate character R[m2][1] as a confirmed recognition character RX (an example of a "confirmed character") corresponding to the recognition result of the character image GC, and, if the recognition candidate character R[m1][1] and the recognition candidate character R[m2][1] are different characters, determines one of the recognition candidate character R[m1][1] and the recognition candidate character R[m2][1] as a likely recognition character RY (an example of a "candidate character") corresponding to the recognition result of the character image GC, and a display information generation unit 115 (an example of a "display control unit") that generates display information JH for displaying the confirmed recognition character RX or the likely recognition character RY on the display unit 31, and the display information generation unit 115 causes the display unit 31 to display the likely recognition character RY in a manner different from the confirmed recognition character RX using the display information JH.

[0086] That is, according to this embodiment, the display information generating unit 115 differentiates the display mode of the likely recognized character RY when the recognition result determining unit 114 determines the recognition result of the character image GC as the likely recognized character RY and there is a high possibility that the character represented in the character image GC differs from the display mode of the confirmed recognized character RX when the recognition result determining unit 114 determines the recognition result of the character image GC as the confirmed recognized character RX and there is a low possibility that the character represented in the character image GC differs from the display mode of the confirmed recognized character RX. Thus, according to this embodiment, it is possible to reduce the possibility of overlooking that the recognition result of the character represented in the character image GC is not accurate, compared to the case where the likely recognized character RY and the confirmed recognized character RX are displayed in the same manner. Furthermore, according to this embodiment, the character recognition process is performed by the character recognition unit RG[m1] and the character recognition unit RG[m2]. Therefore, according to this embodiment, the recognition accuracy in the character recognition process can be improved compared to the case where the character recognition process is performed by one character recognition unit RG.

[0087] In addition, in this embodiment, the character recognition process executed by the character recognition unit RG[m1] is a process of recognizing characters shown in a character image GC using a learning model NN[m1] (an example of a "first learning model") that has learned the relationship between an image including characters and the characters shown in the image.

[0088] Therefore, according to this embodiment, it is possible to improve the recognition accuracy in the character recognition process compared to the case where the character recognition process is performed without using a learning model.

[0089] Furthermore, in this embodiment, the character recognition device 1 is characterized by comprising an operation accepting unit 116 (an example of a "receiving unit") that accepts corrections to the confirmed recognized character RX or the likely recognized character RY displayed on the display unit 31, and a learning execution unit 117 (an example of a "learning unit") that, when the operation accepting unit 116 accepts a correction to change the confirmed recognized character RX or the likely recognized character RY to a character represented in the character image GC, causes the learning model NN[m1] to learn that the character image GC indicates the character represented in the character image GC.

[0090] Therefore, according to this embodiment, it is possible to improve the recognition accuracy in the character recognition process compared to the case where the character recognition process is performed without using a learning model.

[0091] Furthermore, in this embodiment, the character recognition device 1 is characterized by comprising an operation accepting unit 116 that accepts a correction of the confirmed recognition character RX or the likely recognition character RY displayed on the display unit 31, and a learning execution unit 117 that, when the operation accepting unit 116 accepts a correction to change the confirmed recognition character RX or the likely recognition character RY to a character represented in the character image GC and the recognition candidate character R[m1][1] differs from the character represented in the character image GC, causes the learning model NN[m1] to learn that the character image GC indicates the character represented in the character image GC.

[0092] Therefore, according to this embodiment, it is possible to improve the recognition accuracy in the character recognition process compared to the case where the character recognition process is performed without using a learning model.

[0093] Furthermore, in this embodiment, the character recognition device 1 includes an operation accepting unit 116 that accepts corrections to the confirmed recognition character RX or likely recognition character RY displayed on the display unit 31, and when the recognition result determination unit 114 has determined one of the recognition candidate character R[m1][1] and the recognition candidate character R[m2][1] as the likely recognition character RY and the operation accepting unit 116 accepts correction of the likely recognition character RY, the display information generating unit 115 causes the display unit 31 to display the other of the recognition candidate character R[m1][1] and the recognition candidate character R[m2][1] as a correction candidate character RZ.

[0094] Therefore, according to this embodiment, it is possible to reduce the burden on the operator who corrects the confirmed recognized character RX or the likely recognized character RY.

[0095] Further, in this embodiment, the display information generating unit 115 causes the display unit 31 to display the character image GC and the confirmed recognized character RX or the likely recognized character RY based on the display information JH.

[0096] Therefore, according to this embodiment, it is possible to reduce the possibility of overlooking the fact that the recognition result of the characters shown in the character image GC is inaccurate.

[0097] Furthermore, in this embodiment, the content recognition unit 113 is characterized in that it selects character recognition unit RG[m1] and character recognition unit RG[m2] from among M character recognition units RG[1] to RG[M] that perform character recognition processing using mutually different algorithms depending on the type of character represented in the character image GC.

[0098] Therefore, according to this embodiment, it is possible to improve the recognition accuracy in the character recognition process, compared to the case where the character recognition process is performed by one character recognition unit RG.

[0099] [B. Modifications] Each of the above embodiments may be modified in various ways. Specific modified embodiments are exemplified below. Two or more embodiments arbitrarily selected from the following examples may be appropriately combined within a range that does not contradict each other. In the modified examples exemplified below, the elements whose actions and functions are equivalent to those of the embodiment will be appropriately omitted by using the symbols referenced in the above explanation.

[0100] <Variation 1> In the above embodiment, the display information generating unit 115 displays the attention mark MK in the application content display image GH at a position corresponding to the display character RH determined as the likely recognition character RY, but the present invention is not limited to such an embodiment. The display information generating unit 115 may display the display character RH determined as the likely recognition character RY and the display character RH determined as the confirmed recognition character RX in the application content display image GH in a distinguishable manner. For example, the display information generation unit 115 may display the display character RH determined as the likely recognized character RY and the display character RH determined as the confirmed recognized character RX in the application content display image GH as characters of different sizes, different colors, or different fonts. Also, for example, the display information generation unit 115 may display the display character RH determined as the likely recognized character RY in the application content display image GH in an emphasized manner compared to the display character RH determined as the confirmed recognized character RX.

[0101] <Variation 2> In the above embodiment, the recognition result determination unit 114 determines whether the recognition candidate character R[m1][1] and the recognition candidate character R[m2][1] are the same character in the display character determination process to determine the confirmed recognition character RX or the likely recognition character RY, and determines the confirmed recognition character RX or the likely recognition character RY as the display character RH, but the present invention is not limited to such an embodiment. For example, the recognition result determination unit 114 may determine the recognition candidate character R[m1][1] as the display character RH when the recognition accuracy D[m1][1] and the recognition accuracy D[m2][1] satisfy D[m1][1]>D[m2][1], and may determine the recognition candidate character R[m2][1] as the display character RH when the recognition accuracy D[m1][1] and the recognition accuracy D[m2][1] satisfy D[m2][1]≧D[m1][1]. In this case, the display information generation unit 115 may determine whether or not to display an attention warning mark MK at a position based on the display character RH based on the difference value between the recognition accuracy D[m1][1] and the recognition accuracy D[m2][1].

[0102] <Variation 3> In the above-described embodiment and variants 1 and 2, the character recognition unit RG[m] divides the character string image GCL into a plurality of character images GC and performs character recognition processing on each of the divided character images GC, but the present invention is not limited to such an embodiment. For example, the M character recognition units RG[1] to RG[M] included in the written content recognition unit 113 may include a word recognition unit RGT that can recognize an image representing a character string included in the character string image GCL as a character string (word). Note that, hereinafter, a character recognition unit RG that performs character recognition processing on each character image GC, like the character recognition unit RG described in the above embodiment, will be referred to as an individual character recognition unit RGk to be distinguished from the word recognition unit RGT.

[0103] For example, in this modification, the written content recognition unit 113 may include M word recognition units RGT as the M character recognition units RG[1] to RG[M]. That is, in this modification, all of the M character recognition units RG[1] to RG[M] included in the written content recognition unit 113 may be word recognition units RGT. In this case, when a character string (word) recognized by the character recognition unit RG[m1] matches a character string (word) recognized by the character recognition unit RG[m2], the recognition result determination unit 114 may determine each of the characters constituting the character string (word) as a confirmed recognized character RX. In addition, in this case, when a character string recognized by the character recognition unit RG[m1] does not match a character string recognized by the character recognition unit RG[m2], the recognition result determination unit 114 may determine each of the characters constituting one of the character strings recognized by the character recognition unit RG[m1] and the character string recognized by the character recognition unit RG[m2] as a likely recognized character RY. In this case, the other character string may be selected based on the recognition accuracy D of each of the characters constituting the character string.

[0104] Also, for example, in this modified example, the written content recognition unit 113 may include one or more individual character recognition units RGk and one or more word recognition units RGT as the M character recognition units RG[1] to RG[M]. In this case, the written content recognition unit 113 may select the word recognition unit RGT as the character recognition unit RG[m1], and select the individual character recognition unit RGk as the character recognition unit RG[m2]. In this case, when a character constituting a character string (word) recognized by the character recognition unit RG[m1] which is the word recognition unit RGT matches a character recognized by the character recognition unit RG[m2] which is the individual character recognition unit RGk, the recognition result determination unit 114 may determine the character as a confirmed recognized character RX. In this case, if the characters constituting the character string (word) recognized by the character recognition unit RG[m1] which is the word recognition unit RGT do not match the characters recognized by the character recognition unit RG[m2] which is the individual character recognition unit RGk, the recognition result determination unit 114 may determine one of the characters constituting the character string recognized by the character recognition unit RG[m1] and the characters recognized by the character recognition unit RG[m2] as a likely recognized character RY. In this case, the one character may be selected based on the recognition accuracy D.

[0105] <Variation 4> In the above-described embodiment and modifications 1 to 3, the character recognition device 1 and the terminal device 3 are separate entities. However, the character recognition device 1 and the terminal device 3 may be integrated into one device.

[0106] FIG. 13 is an explanatory diagram for explaining an example of the configuration of a character recognition system Sys-B according to this modified example.

[0107] As illustrated in FIG. 13, character recognition system Sys-B is configured in the same manner as character recognition system Sys of the embodiment, except that it has character recognition device 1B instead of character recognition device 1 and does not have terminal device 3.

[0108] FIG. 14 is a functional block diagram showing an example of the configuration of the character recognition device 1B.

[0109] As illustrated in FIG. 14, the character recognition device 1B has the same configuration as the character recognition device 1 according to the embodiment, except that it includes a display unit 31 and an operation unit 32. [Explanation of symbols]

[0110] 1...character recognition device, 3...terminal device, 5...imaging device, 11...control unit, 12...memory unit, 13...communication unit, 31...display unit, 32...operation unit, 111...image information acquisition unit, 112...character area identification unit, 113...written content recognition unit, 114...recognition result determination unit, 115...display information generation unit, 116...operation reception unit, 117...learning execution unit, RG...character recognition unit, NN...learning model.

Claims

1. A first character recognition process for recognizing characters included in an image; and Each of the second character recognition processes for recognizing characters included in the image a recognition unit that recognizes one character of a character string represented for each item in a character image; a first character that is a recognition result of the one character by the first character recognition process; and When a second character that is a recognition result of the first character by the second character recognition process is the same character, determining the first character and the second character as confirmed characters corresponding to a recognition result of the one character; When the first character and the second character are different characters, a determining unit that determines one of the first character and the second character as a candidate character corresponding to a recognition result of the one character; a display control unit that causes the determined character or the candidate character determined by the determination unit to be displayed on a display unit; Equipped with In the recognition unit, the first character recognition process and the second character recognition process are selected based on a similarity between effective character type information in each of a plurality of character recognition processes prepared in advance and character type information in the item, The display control unit is displaying the candidate characters on the display unit in a manner different from that of the finalized characters; 23. An information processing apparatus comprising:

2. The information processing device described in Claim 1, characterized in that the similarity is calculated by the dot product of a vector whose elements are each of the effective character type information and a vector whose elements are each of the character type information in the item.

3. The first character recognition process includes: Using a first learning model that has learned a relationship between an image including characters and the characters shown in the image, A process of recognizing the one character represented in the character image.

3. The information processing device according to claim 1, wherein the information processing device further comprises:

4. a reception unit that receives a correction to the candidate character or the final character displayed on the display unit; When the reception unit receives a correction to change the candidate character or the confirmed character to the one character, a learning unit that causes the first learning model to learn that the character image represents the one character; Equipped with 4. The information processing device according to claim 3.

5. a reception unit that receives a correction to the candidate character or the final character displayed on the display unit; When the reception unit receives a correction to change the candidate character or the confirmed character to the one character, When the first character and the one character are different, a learning unit that causes the first learning model to learn that the character image represents the one character; Equipped with 4. The information processing device according to claim 3.

6. a reception unit that receives a correction to the candidate character or the final character displayed on the display unit, The display control unit is When the determination unit determines one of the first character and the second character as the candidate character, When the reception unit receives a correction to the candidate character, displaying the other of the first character and the second character on the display unit as a correction candidate for the candidate character; 6. The information processing device according to claim 1, wherein the information processing device further comprises:

7. The display control unit is displaying the character image and the confirmed character or the candidate character on the display unit; 7. The information processing device according to claim 1, wherein the information processing device is a data processing device.

8. The recognition unit is According to the type of the one character represented in the character image, Among the multiple character recognition processes that recognize characters contained in images, selecting the first character recognition process and the second character recognition process; 8. The information processing device according to claim 1, wherein the information processing device is a data processing device.

9. The processor, A first character recognition process for recognizing characters included in an image; and Each of the second character recognition processes for recognizing characters included in the image a recognition unit that recognizes one character of a character string represented for each item in a character image; a first character that is a recognition result of the one character by the first character recognition process; and When a second character that is a recognition result of the first character by the second character recognition process is the same character, determining the first character and the second character as confirmed characters corresponding to a recognition result of the one character; When the first character and the second character are different characters, a determining unit that determines one of the first character and the second character as a candidate character corresponding to a recognition result of the one character; a display control unit that causes the determined character or the candidate character determined by the determination unit to be displayed on a display unit; and make it work. In the recognition unit, the first character recognition process and the second character recognition process are selected based on a similarity between effective character type information in each of a plurality of character recognition processes prepared in advance and character type information in the item, The display control unit is displaying the candidate characters on the display unit in a manner different from that of the finalized characters; A program characterized by:

10. A method for recognizing a character of a character string represented by each item in a character image by a first character recognition process and a second character recognition process, when a first character that is a recognition result of the one character by the first character recognition process and a second character that is a recognition result of the one character by the second character recognition process are the same character, determining the first character and the second character as confirmed characters corresponding to the recognition result of the one character; When the first character and the second character are different characters, determining one of the first character and the second character as a candidate character corresponding to a recognition result of the one character; An information processing method for displaying the determined confirmed character or the candidate character on a display unit, the first character recognition process and the second character recognition process are selected based on a similarity between effective character type information in each of a plurality of character recognition processes prepared in advance and character type information in the item; The information processing method according to claim 1, wherein the candidate characters are displayed on the display unit in a manner different from that of the confirmed characters.

Images