Information processing device, program, and information processing method

Abstract

To enhance the accuracy of recognition of characters in character recognition processing.SOLUTION: A character recognition device 10 comprises: a described content recognition unit 124 that recognizes one character contained in an application form image GM through each of a plurality of character recognition processes that recognize characters contained in an image and calculate a degree of certainty of the recognized characters; and a recognition result determination unit 126 that determines one of a plurality of candidate characters obtained as a result of recognizing one character by each of the plurality of character recognition processes as a confirmed character corresponding to the recognition result of the one character. The application form image GM is an image of a document containing one or more items. Priorities of the character recognition processes used for the recognition of a character is determined for each item. The recognition result determination unit 126 determines one of the plurality of candidate characters as a confirmed character on the basis of the degree of certainty of each of the plurality of candidate characters and the priority of each of the plurality of character recognition process.SELECTED DRAWING: Figure 4

Description

[Technical field]

[0001] The present invention relates to an information processing device. 、 program and information processing method Regarding. [Background technology]

[0002] In information processing devices such as character recognition devices, optical character recognition (OCR) technology is generally adopted (for example, see Patent Document 1). OCR technology is a technology that captures characters written on a medium such as a form as an image by optical means such as a camera or an image scanner, and converts the characters in the captured image into character information (for example, character code) that can be used by a computer or the like. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] JP 2010-073201 A Summary of the Invention [Problem to be solved by the invention]

[0004] However, the character recognition accuracy of this type of OCR technology is not 100%, so there is a demand for an information processing device that can improve the accuracy of character recognition in character recognition processing. [Means for solving the problem]

[0005] An information processing device according to a preferred embodiment of the present invention includes a recognition unit that recognizes a character included in a character image, which is an image of a document including one or more items, by each of a plurality of character recognition processes that recognize characters included in an image and calculate an accuracy of the recognized character, and a determination unit that determines one of a plurality of candidate characters obtained as a recognition result of the one character by each of the plurality of character recognition processes as a confirmed character corresponding to the recognition result of the one character, wherein a priority of the character recognition processes used for character recognition is determined for each item, and the determination unit determines one of the plurality of candidate characters as the confirmed character based on the accuracy of each of the plurality of candidate characters and the priority of each of the plurality of character recognition processes.

[0006] In addition, an information processing device according to another preferred embodiment of the present invention includes a recognition unit that recognizes a character included in a character image by each of a first character recognition process that recognizes a character included in an image and calculates an accuracy of the recognized character, and a second character recognition process that recognizes a character included in an image and calculates an accuracy of the recognized character, and a determination unit that, when a common character exists that is included in both one or more first candidate characters obtained as a recognition result of the one character by the first character recognition process and one or more second candidate characters obtained as a recognition result of the one character by the second character recognition process, determines whether or not to set the common character as a confirmed character corresponding to the recognition result of the one character.

[0007] In addition, an information processing device according to another preferred aspect of the present invention includes a recognition unit that executes a first character recognition process to recognize a plurality of characters constituting a character string included in an image and a second character recognition process to recognize one of the plurality of characters included in the image, and a determination unit that, when a first candidate character obtained as a recognition result of the one character by the first character recognition process and a second candidate character obtained as a recognition result of the one character by the second character recognition process are the same character, determines the same character as a confirmed character corresponding to the recognition result of the one character.

[0008] A program according to a preferred embodiment of the present invention causes a processor to function as: a recognition unit that recognizes a single character included in a character image, which is an image of a document including one or more items, using each of a plurality of character recognition processes that recognize characters included in an image and calculate the accuracy of the recognized character; and a determination unit that determines one of a plurality of candidate characters obtained as a recognition result of the single character by each of the plurality of character recognition processes as a confirmed character corresponding to the recognition result of the single character, wherein a priority of the character recognition processes used to recognize characters is determined for each item, and the determination unit determines one of the plurality of candidate characters as the confirmed character based on the accuracy of each of the plurality of candidate characters and the priority of each of the plurality of character recognition processes.

[0009] In addition, a program according to another preferred embodiment of the present invention causes a processor to function as a recognition unit that recognizes a character contained in a character image by each of a first character recognition process that recognizes a character contained in an image and calculates an accuracy of the recognized character, and a second character recognition process that recognizes a character contained in an image and calculates an accuracy of the recognized character, and a determination unit that, when a common character exists that is included in both one or more first candidate characters obtained as a recognition result of the one character by the first character recognition process and one or more second candidate characters obtained as a recognition result of the one character by the second character recognition process, determines whether or not to set the common character as a confirmed character corresponding to the recognition result of the one character.

[0010] In addition, a program according to another preferred aspect of the present invention causes a processor to function as a recognition unit that performs a first character recognition process to recognize a plurality of characters that constitute a character string included in an image, and a second character recognition process to recognize one of the plurality of characters included in the image, and a determination unit that, when a first candidate character obtained as a recognition result of the one character by the first character recognition process and a second candidate character obtained as a recognition result of the one character by the second character recognition process are the same character, determines the same character as a confirmed character corresponding to the recognition result of the one character. Effect of the Invention

[0011] According to the present invention, it is possible to improve the accuracy of character recognition in character recognition processing. [Brief description of the drawings]

[0012] [Figure 1] FIG. 1 is an explanatory diagram for explaining an overview of a character recognition system according to a first embodiment. [Diagram 2] FIG. 13 is an explanatory diagram showing an example of an application form image. [Diagram 3] 2 is a diagram illustrating an example of a hardware configuration of the character recognition device illustrated in FIG. 1. [Figure 4] 2 is a functional block diagram showing an example of the configuration of the character recognition device shown in FIG. 1. [Diagram 5] FIG. 4 is an explanatory diagram illustrating an example of the priority table illustrated in FIG. [Figure 6] FIG. 11 is an explanatory diagram for explaining an example of a confirmed character determination process. [Figure 7] FIG. 13 is an explanatory diagram showing an example of an outline of a content confirmation screen. [Figure 8] 2 is a flowchart showing an example of the operation of the character recognition device shown in FIG. [Figure 9] 9 is a flowchart showing an example of the confirmed character determination process shown in FIG. 8. [Figure 10] 10 is a flowchart showing another example of the confirmed character determination process shown in FIG. 8. [Figure 11] FIG. 11 is an explanatory diagram for explaining an overview of a character recognition device according to a second embodiment. [Figure 12] 12 is an explanatory diagram for explaining an example of the operation of a recognition result determination unit shown in FIG. 11; [Figure 13] 12 is a flowchart showing an example of an operation of a recognition result determination unit shown in FIG. 11. [Figure 14] FIG. 11 is an explanatory diagram for explaining an overview of a character recognition device according to a third embodiment. [Figure 15] 15 is an explanatory diagram for explaining an example of the operation of the recognition result determination unit shown in FIG. 14; [Figure 16]FIG. 11 is an explanatory diagram for explaining an example of a weighting table according to a first modified example. [Figure 17] FIG. 11 is an explanatory diagram for explaining an example of a confirmed character determination process according to a first modified example. [Figure 18] FIG. 13 is a diagram illustrating an example of a character recognition system according to a second modified example. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0013] Hereinafter, the embodiments of the present invention will be described with reference to the drawings. In each drawing, the dimensions and scale of each part are appropriately different from the actual ones. In addition, the embodiments described below are preferred examples of the present invention, and therefore various technically preferable limitations are attached, but the scope of the present invention is not limited to these embodiments unless otherwise specified in the following description to the effect that the present invention is limited.

[0014] [1. First embodiment] First, an example of an overview of a character recognition system 1 according to a first embodiment will be described with reference to FIG.

[0015] FIG. 1 is an explanatory diagram for explaining an overview of a character recognition system 1 according to a first embodiment.

[0016] The character recognition system 1 shown in FIG. 1 includes a character recognition device 10, a terminal device 20, and an image input device 30. The character recognition device 10 is an example of an "information processing device." The character recognition device 10 and the terminal device 20 are communicatively connected to each other, for example, via a network NW. The network NW may include the Internet and a local area network. For example, the network NW includes one or both of a wired network and a wireless network. In addition, the connection between the network NW and the character recognition device 10 may be, for example, a connection that allows communication between multiple elements, and may be a connection using either a wired or wireless connection, or a connection using both a wired and wireless connection.

[0017] The character recognition device 10 may be any information processing device connectable to a network NW. The character recognition device 10 executes character recognition processing to recognize characters included in an image, for example, by using optical character recognition (OCR) technology. An example of an image to be subjected to the character recognition processing is described in FIG. 2, which will be described later. The configuration of the character recognition device 10 is described in FIG. 3 and FIG. 4, which will be described later. In this embodiment, it is assumed that the character recognition device 10 recognizes characters written in a membership application form for joining a predetermined service. For example, in this embodiment, the character recognition device 10 converts information written in the membership application form into electronic data by using optical character recognition technology.

[0018] As the terminal device 20, any information processing device connectable to the network NW can be adopted. Specifically, the terminal device 20 may be, for example, a stationary information device such as a personal computer, or may be a portable information terminal such as a notebook personal computer, a tablet terminal, or a smartphone. The terminal device 20 shown in FIG. 1 has an operation device 27 that accepts an operation by an operator of the terminal device 20, and a display device 28 that displays images, etc. For example, the terminal device 20 displays the results of the character recognition process, etc. on the display device 28. The results of the character recognition process are transmitted to the terminal device 20 from the character recognition device 10 via the network NW, for example. The display device 28 is an example of a "display unit."

[0019] Moreover, the terminal device 20 is communicably connected to the image input device 30. The connection between the terminal device 20 and the image input device 30 may be a wired or wireless connection, or may be a wired and wireless connection. For example, the terminal device 20 acquires image information indicating an image to be subjected to character recognition processing from the image input device 30, and transmits the image information acquired from the image input device 30 to the character recognition device 10 via the network NW.

[0020] The terminal device 20 may not include the display device 28. When the terminal device 20 does not include the display device 28, the terminal device 20 is communicatively connected to the display device 28. In this case, the connection between the terminal device 20 and the display device 28 may be a wired or wireless connection, or may be both a wired and wireless connection.

[0021] The image input device 30 is, for example, an optical device such as a camera or an image scanner. For example, the image input device 30 captures an image of a document such as a membership application form, thereby generating image information showing an image of the document. The image information generated by the image input device 30 is transmitted to the terminal device 20. The image input device 30 may be included in the terminal device 20.

[0022] FIG. 2 is an explanatory diagram showing an example of an application form image GM.

[0023] The application form image GM is, for example, an image of a membership application form including multiple items. A membership application form including multiple items is an example of a "document including one or more items." In other words, the application form image GM is an example of a "text image."

[0024] For example, the application form image GM is an image that is the subject of character recognition processing. Therefore, the application form image GM includes multiple fields corresponding to multiple items. In the example shown in Fig. 2, the multiple items written on the application form are the application number, kana name, name, postal code, address, e-mail, and phone number.

[0025] Each of the multiple entry fields includes an item name entry area AM in which an item name, which is the name of an item of the application content to be entered in the entry field, is entered, and an application content entry area AK in which the application content related to the entry field is entered. Hereinafter, among the application form image GM, an image showing a character string related to the application content entered in each of the multiple application content entry areas AK is also referred to as a character string image GCL. In addition, in FIG. 2, in order to distinguish the multiple character string images GCL from one another, a number (1, 2, 3, 4, 5, 6, or 7) is added to the end of each of the symbols of the multiple character string images GCL.

[0026] For example, character string image GCL1 shows a character string related to the application content written in the application content entry area AK related to "application number". Character string image GCL2 shows a character string related to the application content written in the application content entry area AK related to "kana name", and character string image GCL3 shows a character string related to the application content written in the application content entry area AK related to "name". Character string image GCL4 shows a character string related to the application content written in the application content entry area AK related to "zip code", and character string image GCL5 shows a character string related to the application content written in the application content entry area AK related to "address". Character string image GCL6 shows a character string related to the application content written in the application content entry area AK related to "email", and character string image GCL7 shows a character string related to the application content written in the application content entry area AK related to "telephone number".

[0027] In the following description, an image showing each of the characters included in the character string represented by each of the character string images GCL is also referred to as an individual character image GC. That is, each of the character string images GCL is an image including a plurality of individual character images GC. For example, the character string image GCL5 includes an individual character image GC1 of "sea" and an individual character image GC2 of "big".

[0028] In the following, it is assumed that the character recognition device 10 performs character recognition processing on an application form image GM having the format exemplified in FIG. 2, but the present invention is not limited to this aspect. In this embodiment, the character recognition device 10 can perform character recognition processing on documents having any format or text with no set format (e.g., blank notes, etc.). In the following, it is assumed that the application content written in the application content writing area AK is written in handwritten characters, which are handwritten characters, but the present invention is not limited to this aspect. For example, in this embodiment, some or all of the multiple characters shown in one or more character string images GCL may be written in type.

[0029] Next, the hardware configuration of the character recognition device 10 will be described with reference to FIG.

[0030] FIG. 3 is a diagram showing an example of a hardware configuration of the character recognition device 10 shown in FIG.

[0031] The character recognition device 10 includes a processing device 12 that controls each unit of the character recognition device 10 , a storage device 14 that stores various information, and a communication device 16 .

[0032] The storage device 14 includes, for example, one or both of a volatile memory such as a random access memory (RAM) that functions as a working area for the processing device 12 and a non-volatile memory such as an electrically erasable programmable read-only memory (EEPROM) that stores various information such as the control program PG1. The storage device 14 may be detachable from the character recognition device 10. Specifically, the storage device 14 may be a storage medium such as a memory card that is detachable from the character recognition device 10. The storage device 14 may also be, for example, a storage device (e.g., online storage) that is communicatively connected to the character recognition device 10 via a network NW or the like.

[0033] The storage device 14 shown in Fig. 3 stores a control program PG1, a priority table PTBL, and a plurality of learning models LM. In Fig. 3, in order to distinguish the plurality of learning models LM from one another, a number (e.g., 1) or a lowercase alphabet (e.g., m) is added to the end of each of the symbols of the plurality of learning models LM. For example, m indicates a natural number of 2 or more.

[0034] The control program PG1 is an example of a "program." In this embodiment, the control program PG1 includes, for example, an application program for the character recognition device 10 to execute the character recognition process. However, the control program PG1 may also include, for example, an operating system program for the processing device 12 to control each part of the character recognition device 10.

[0035] The priority table PTBL stores information indicating a plurality of priorities associated with a plurality of character recognition units CR shown in FIG. 4, which will be described later. The priorities are the priorities of the character recognition units CR used for character recognition, and are determined for each item. Details of the priority table PTBL will be described in FIG. 5, which will be described later.

[0036] Each of the multiple learning models LM is, for example, a multi-layer neural network, and is a learning model that learns the relationship between an image containing one or more types of characters and the characters contained in the image. For convenience of explanation, in this embodiment, it is assumed that the types of characters shown in the image are six types: kanji, hiragana, katakana, Arabic numerals (hereinafter also simply referred to as "numbers"), alphabets, and symbols. Here, symbols are a concept that includes, for example, dots, commas, periods, commas, hyphens, and arithmetic symbols.

[0037] The processing device 12 is a processor that controls the entire character recognition device 10, and is configured to include, for example, one or more CPUs (Central Processing Units). The processing device 12 executes, for example, a control program PG1 stored in the storage device 14, and operates according to the control program PG1, thereby functioning as a control unit 120 shown in Fig. 4, which will be described later. Note that the control program PG1 may be transmitted from another device via the network NW.

[0038] Furthermore, for example, when the processing device 12 is configured to include multiple CPUs, some or all of the functions of the processing device 12 may be realized by the multiple CPUs operating in cooperation with each other according to a program such as the control program PG1. Furthermore, the processing device 12 may be configured to include hardware such as a GPU (Graphics Processing Unit), a DSP (Digital Signal Processor), or an FPGA (Field Programmable Gate Array) in addition to one or more CPUs, or instead of some or all of the one or more CPUs. In this case, some or all of the functions of the processing device 12 may be realized by hardware such as a DSP.

[0039] The communication device 16 is hardware for communicating with an external device that exists outside the character recognition device 10 via one or both of a wired network and a wireless network.

[0040] Next, the function of the character recognition device 10 will be described with reference to FIG.

[0041] FIG. 4 is a functional block diagram showing an example of the configuration of the character recognition device 10 shown in FIG.

[0042] The control unit 120 is realized by the processing device 12 as described in Fig. 3. When the processing device 12 is configured to include multiple CPUs, some or all of the functions of the control unit 120 may be realized by the multiple CPUs operating in cooperation with each other according to the control program PG1. When the processing device 12 is configured to include hardware such as a DSP, some or all of the functions of the control unit 120 may be realized by the hardware such as a DSP.

[0043] The control unit 120 has, for example, an image information acquisition unit 122, a description content recognition unit 124, a recognition result determination unit 126, and a display information generation unit 128. The description content recognition unit 124 is an example of a "recognition unit," and the recognition result determination unit 126 is an example of a "determination unit." Also, the display information generation unit 128 is an example of a "display control unit."

[0044] The image information acquisition unit 122 acquires, for example, image information (image information generated by the image input device 30) indicating the application form image GM to be subjected to character recognition processing from the terminal device 20 via the communication device 16.

[0045] The written content recognition unit 124 has a plurality of character recognition units CR. In this embodiment, it is assumed that the plurality of character recognition units CR correspond one-to-one to the plurality of learning models LM. In Fig. 4, in order to distinguish the plurality of character recognition units CR from one another, a number (e.g., 1) or a lowercase alphabet (e.g., m) is added to the end of each of the reference symbols of the plurality of character recognition units CR. For example, m indicates a natural number equal to or greater than 2.

[0046] Each of the multiple character recognition units CR executes, as a character recognition process, a process of recognizing characters included in an image and calculating a degree of accuracy indicating the likelihood of the recognized characters. The written content recognition unit 124, for example, causes two or more of the multiple character recognition units CR to execute character recognition processing on the application form image GM indicated by the image information acquired by the image information acquisition unit 122. For example, the written content recognition unit 124 selects two or more character recognition units CR from the multiple character recognition units CR based on information such as priority stored in the priority table PTBL. Then, the written content recognition unit 124 recognizes characters included in the application form image GM by causing each of the two or more character recognition units CR selected from the multiple character recognition units CR to execute character recognition processing on the application form image GM. Hereinafter, the degree of accuracy indicating the likelihood of the characters recognized by the character recognition unit CR is also simply referred to as the degree of accuracy of the characters.

[0047] Each of the multiple character recognition units CR (CR1 to CRm) recognizes a character indicated by an individual character image GC using a corresponding learning model LM among the multiple learning models LM (LM1 to LMm). That is, in this embodiment, each of the multiple character recognition units CR uses a corresponding learning model LM among the multiple learning models LM stored in the storage device 14 to perform character recognition processing by AI (Artificial Intelligence)-OCR.

[0048] In this embodiment, it is assumed that the character recognition units CR use different algorithms to perform the character recognition process. For example, the algorithm used by the character recognition unit CR1 in the character recognition process is different from the algorithm used by the character recognition unit CRm in the character recognition process. Here, "different algorithms" may mean, for example, that the types of learning models LM used in the character recognition process are different, or that the data sets used when training the learning models LM used in the character recognition process are different. Alternatively, "different algorithms" may include, for example, both the concepts of the types of learning models LM used in the character recognition process being different and the data sets used when training the learning models LM used in the character recognition process being different. The type of learning model LM is, for example, a type of neural network.

[0049] Furthermore, "different data sets" means, for example, that the characters included in the images used in one of the two data sets include characters of a different type from the characters included in the images used in the other of the two data sets. Note that when the two data sets are different from each other, the images used in one of the two data sets may include both characters of a different type and characters of the same type as the characters included in the images used in the other of the two data sets.

[0050] In addition, when the two data sets used in the two learning models LM are different from each other, the two neural networks constituting the two learning models LM may be of different types or may be of the same type. In other words, the fact that the two data sets used in the two learning models LM are different from each other corresponds to "different algorithms" regardless of whether the types of the two neural networks are different from each other. In addition, for example, the fact that the types of the two neural networks constituting the two learning models LM are different from each other corresponds to "different algorithms" regardless of whether the two data sets used in the two learning models LM are different from each other.

[0051] In this embodiment, it is assumed that the multiple character recognition units CR correspond one-to-one to the multiple learning models LM, but the present invention is not limited to this embodiment. For example, some or all of the multiple character recognition units CR may be components that perform character recognition processing without using a neural network. If all of the multiple character recognition units CR are components that perform character recognition processing without using a neural network, the storage device 14 does not need to store the learning models LM.

[0052] The multiple character recognition unit CR may include a character recognition unit CR that recognizes multiple characters that make up a character string included in an image. This type of character recognition unit CR may recognize multiple characters that make up a character string included in an image, for example, by using a predetermined dictionary in which multiple character strings are registered. Alternatively, the character recognition unit CR may recognize multiple characters that make up a character string included in an image by using a learning model LM that has learned to recognize character strings. The learning model LM that has learned to recognize character strings may be, for example, a learning model LM that has learned one or both of a relationship between one or more characters and a character that follows the one or more characters, and a relationship between a character and one or more characters that follows the one character.

[0053] The recognition result determination unit 126 acquires recognition information indicating the result of character recognition processing for one character by the written content recognition unit 124. For example, the recognition information indicates the recognition result of one character by the character recognition processing executed by each of two or more character recognition units CR among the multiple character recognition units CR. The recognition result of one character by the character recognition processing executed by each of the two or more character recognition units CR includes, for example, multiple candidate characters that are candidates for the recognition result of one character and the accuracy of the multiple candidate characters.

[0054] For example, the recognition result determination unit 126 determines one of a plurality of candidate characters obtained as the recognition results of two or more character recognition units CR that have executed character recognition processing on one character as a confirmed character corresponding to the recognition result of the one character. Specifically, the recognition result determination unit 126 determines one of the plurality of candidate characters as a confirmed character based on the accuracy of each of the plurality of candidate characters and the priority of each of the plurality of character recognition units CR.

[0055] For example, when a weight value based on priority is determined for each of the multiple character recognition units CR, the recognition result determination unit 126 calculates an evaluation value for each of the multiple candidate characters by weighting the accuracy of each of the multiple candidate characters with the weight value of each of the multiple character recognition units CR. The weight value is, for example, a real number equal to or greater than 0. The recognition result determination unit 126 determines the candidate character with the highest evaluation value among the multiple candidate characters as the confirmed character corresponding to the recognition result of one character. Note that, when there are multiple candidate characters with the same evaluation value, the recognition result determination unit 126 may determine the candidate character with the highest accuracy among the multiple candidate characters as the confirmed character, or may determine the candidate character with the highest priority (or weight value) as the confirmed character. Alternatively, the recognition result determination unit 126 may accumulate, for each character recognition unit CR, whether or not a candidate character previously determined as a confirmed character has been corrected, and may determine the candidate character with the highest accuracy rate among multiple candidate characters with the same evaluation value as the confirmed character.

[0056] The display information generation unit 128 causes the display device 28 to display the confirmed characters determined by the recognition result determination unit 126. For example, the display information generation unit 128 generates display information for displaying an image for confirming the confirmed characters determined by the recognition result determination unit 126 (for example, a content confirmation screen CCH shown in FIG. 7 described later) on the display device 28. Then, the display information generation unit 128 outputs the generated display information to the terminal device 20 via the communication device 16.

[0057] The configuration of the character recognition device 10 is not limited to the examples shown in Fig. 3 and Fig. 4. For example, the character recognition device 10 may have an operation device that accepts operations by an administrator of the character recognition device 10 or the like.

[0058] Next, the priority table PTBL will be described with reference to FIG.

[0059] Fig. 5 is an explanatory diagram showing an example of the priority table PTBL shown in Fig. 3. The priority table PTBL is referred to by the written content recognition unit 124 when the written content recognition unit 124 selects two or more character recognition units CR from the multiple character recognition units CR, and is referred to by the recognition result determination unit 126 when the recognition result determination unit 126 calculates an evaluation value.

[0060] The priority table PTBL has, for example, a number of records that correspond one-to-one to a number of fields in the application form image GM (the image to be subjected to character recognition processing). Each record in the priority table PTBL has an item ID, an item name, and a priority.

[0061] The item ID is information for identifying each entry field among the multiple entry fields in the application form image GM. The item name is the name written in the item name writing area AM of each entry field in the application form image GM.

[0062] The priority is, for example, the priority of the character recognition unit CR used to recognize characters written in the entry field indicated by the item ID. Note that a hyphen ("-") in FIG. 5 means that a priority is not associated with the character recognition unit CR. For example, in each entry field (each item), a character recognition unit CR to which a priority is not associated is not used to recognize characters written in the entry field. Also, in this embodiment, a smaller priority number has a higher priority than a larger priority number. In each item, a character recognition unit CR with a higher priority is more suitable for recognizing characters written in the entry field of that item than a character recognition unit CR with a lower priority.

[0063] For example, the priority is determined for each item according to the learning degree of each of the multiple character recognition units CR. The "learning degree" may be a concept including, for example, the number of samples (images) included in the data set used for learning, the number of learnings, and some or all of the accuracy rate. In the priority table PTBL shown in FIG. 5, it is assumed that the written content recognition unit 124 has six character recognition units CR1, CR2, CR3, CR4, CR5, and CR6.

[0064] For example, the character recognition units CR1 and CR2 are character recognition units CR for alphanumeric characters specialized in recognizing alphabets, numbers, and symbols. Note that one or both of the character recognition units CR1 and CR2 may be character recognition units CR with higher recognition accuracy for alphabets, numbers, and symbols than other character types (e.g., kanji, etc.). For example, the character recognition units CR3 and CR4 are character recognition units CR for kanji characters specialized in recognizing kanji and hiragana. Note that one or both of the character recognition units CR3 and CR4 may be character recognition units CR with higher recognition accuracy for kanji and hiragana than other character types (e.g., numbers, etc.). For example, the character recognition units CR5 and CR6 are character recognition units CR for kana characters specialized in recognizing katakana. Note that one or both of the character recognition units CR5 and CR6 may be character recognition units CR with higher recognition accuracy for katakana than other character types (e.g., numbers, etc.).

[0065] Also, for example, in one item among a plurality of items, the priority of the character recognition unit CR having a higher learning level for character strings related to the one item among the two character recognition units CR is set to be higher than the priority of the other character recognition unit CR. In the example shown in FIG. 5, it is assumed that, among the character recognition units CR3 and CR4 for kanji, the character recognition unit CR4 has a higher learning level for character strings related to "address" than the character recognition unit CR3. Therefore, in the item related to "address", the priority of the character recognition unit CR4 is higher than the priority of the character recognition unit CR3. Therefore, in the item related to "address", for example, the priority of the character recognition unit CR4 is set to "1", and the priority of the character recognition unit CR3 is set to "2".

[0066] As shown in Fig. 5, in this embodiment, the priority of the character recognition unit CR used to recognize characters is determined for each item. For example, when recognizing a character string written in the application content writing area AK of one of the multiple items, the description content recognition unit 124 refers to a record corresponding to the one item among the multiple records held in the priority table PTBL. Then, the description content recognition unit 124 selects, from among the multiple character recognition units CR, a character recognition unit CR to which a priority is associated in the record corresponding to the one item as the character recognition unit CR to recognize the character string written in the application content writing area AK of the one item.

[0067] Specifically, for example, when recognizing a character string written in the application content entry area AK related to "zip code", the content recognition unit 124 refers to a record with an item ID of "k004" and selects character recognition units CR1 and CR2 for alphanumeric characters. Also, for example, when recognizing a character string written in the application content entry area AK related to "address", the content recognition unit 124 refers to a record with an item ID of "k005". Then, the content recognition unit 124 selects character recognition units CR3 and CR4 for kanji, character recognition unit CR2 for alphanumeric characters, and character recognition unit CR5 for kana.

[0068] Furthermore, when determining a confirmed character corresponding to a recognition result for a character included in a character string written in the application content description area AK of one item, the recognition result determination unit 126 refers to a record corresponding to the one item among a plurality of records held in the priority table PTBL. Then, the recognition result determination unit 126 determines one of a plurality of candidate characters obtained as a recognition result of two or more character recognition units CR that have recognized the one character as a confirmed character based on the accuracy of each of the plurality of candidate characters and each of a plurality of priorities stored in the record corresponding to the one item. Details of the process of determining the confirmed character will be described later in FIG. 6. Hereinafter, the process of determining the confirmed character will also be referred to as a confirmed character determination process.

[0069] The priority table PTBL is not limited to the example shown in FIG. 5. For example, some of the items among the multiple items may have only one character recognition unit CR associated with a priority. Also, for example, the priority table PTBL may store a weight value in addition to or instead of the priority. A weight value based on the priority may be determined for each item. For example, the weight value of the character recognition unit CR with a priority of "1" may be different or the same value for the "address" item and the "zip code" item. Also, when a weight value is stored in the priority table PTBL instead of a priority, the weight value corresponds to a "priority". In this embodiment, a large weight value has a higher priority than a small weight value.

[0070] Also, the priority may be determined using indicators other than the learning degree. The indicators other than the learning degree may be, for example, the record of whether an operator has made a correction to a candidate character determined as a definite character in the past (e.g., the correct rate or the correction rate). For example, the priority may be determined for each item according to the correct rate or the correction rate of each of a plurality of character recognition units CR. In this case, for example, in one item, the priority of a character recognition unit CR with a high correct rate (or a character recognition unit CR with a low correction rate) is set higher than the priority of a character recognition unit CR with a low correct rate (or a character recognition unit CR with a high correction rate). Note that the correction rate of each character recognition unit CR in each item is represented, for example, by the ratio of the total number of times of correction by the operator to the candidate characters determined as definite characters in the past to the total number of candidate characters determined as definite characters in the past in that item. Also, the correct rate of each character recognition unit CR in each item is represented, for example, by a value obtained by subtracting the correction rate of each character recognition unit CR in that item from 1. Note that the correct rate or the correction rate of each of the plurality of character recognition units CR may be calculated for each character type.

[0071] Next, with reference to FIG. 6, the outline of the definite character determination process will be described.

[0072] FIG. 6 is an explanatory diagram for explaining an example of the definite character determination process. In FIG. 6, as an example, when character recognition processing is performed on a character string image GCL5 showing the character string described in the application content description area AK related to "address", an example of a method for determining a definite character corresponding to the recognition result of the individual character image GC2 of "big" is described.

[0073] Note that in FIG. 6, it is assumed that the description content recognition unit 124 and the recognition result determination unit 126 refer to the priority table PTBL shown in FIG. 5. Therefore, in the item of "address", as shown in FIG. 5, the character recognition unit CR4 is the character recognition unit CR with the highest priority among the plurality of character recognition units CR2 to CR5. In FIG. 6, it is assumed that a weight value based on the priority is determined for each of the plurality of character recognition units CR. For example, the weight value of the character recognition unit CR4 with a priority of "1" is set to 1.20, the weight value of the character recognition unit CR3 with a priority of "2" is set to 1.05, and the weight values of each of the character recognition units CR2 and CR4 with a priority of "3" are set to 1.00. Further, the evaluation value is represented by, for example, the product of the accuracy of the candidate character and the weight value.

[0074] In the example shown in FIG. 6, the recognition result determination unit 126 acquires recognition information indicating a plurality of candidate characters and the accuracy of each of the plurality of candidate characters as a result of the character recognition process for the individual character image GC2 of "big" by each of the character recognition units CR2 to CR5.

[0075] For example, the candidate character obtained from the character recognition unit CR3 is the character "zhang" with an accuracy of 90%, and the candidate character obtained from the character recognition unit CR4 is the character "big" with an accuracy of 85%. Note that the accuracy (85%) of the candidate character ("big") obtained from the character recognition unit CR4 is lower than the accuracy (90%) of the candidate character ("zhang") obtained from the character recognition unit CR3. Further, for example, the candidate character obtained from the character recognition unit CR2 is the character "K" with an accuracy of 30%, and the candidate character obtained from the character recognition unit CR5 is the character "su" with an accuracy of 60%. The accuracies of the candidate characters obtained from each of the character recognition units CR2 and CR5 are lower than the accuracy of the candidate character obtained from the character recognition unit CR3.

[0076] The recognition result determination unit 126 calculates an evaluation value by weighting the accuracy of each of the plurality of candidate characters with the weight value of each of the plurality of character recognition units CR. Specifically, the recognition result determination unit 126 calculates the evaluation value of the candidate character obtained from the character recognition unit CR3 by weighting the accuracy (90%) of the candidate character obtained from the character recognition unit CR3 with the weight value (1.05) of the character recognition unit CR3. Similarly, the recognition result determination unit 126 calculates the evaluation value of the candidate character obtained from the character recognition unit CR4 by weighting the accuracy (85%) of the candidate character obtained from the character recognition unit CR4 with the weight value (1.20) of the character recognition unit CR4.

[0077] Then, the recognition result determination unit 126 determines, as the confirmed character corresponding to the recognition result of the individual character image GC2 of "big", the candidate character with the highest evaluation value among the plurality of candidate characters. For example, the evaluation value of the candidate character ("zhang") obtained from the character recognition unit CR3 is 95 (= 90 × 1.05), and the evaluation value of the candidate character ("big") obtained from the character recognition unit CR4 is 102 (= 85 × 1.20). Also, the evaluation value of the candidate character ("K") obtained from the character recognition unit CR2 is 30 (= 30 × 1.00), and the evaluation value of the candidate character ("su") obtained from the character recognition unit CR5 is 60 (= 60 × 1.00). Therefore, in the example shown in FIG. 6, among the plurality of candidate characters ("zhang", "big", "K", and "su"), the candidate character with the highest evaluation value is the character "big" with an evaluation value of 102, as indicated by the star mark in FIG. 6. For this reason, the recognition result determination unit 126 determines, as the confirmed character corresponding to the recognition result of the individual character image GC2 of "big", the character "big" with the highest evaluation value among the plurality of candidate characters.

[0078] As described above, in the example shown in FIG. 6, the candidate character ("big") obtained from the character recognition unit CR4 with the highest priority among the plurality of character recognition units CR2 to CR5 is determined as the confirmed character corresponding to the recognition result of the individual character image GC2 of "big". The character recognition unit CR4 with the highest priority among the plurality of character recognition units CR2 to CR5 is more suitable for recognizing the character string related to "address" compared to the other character recognition units CR2, CR3, and CR5. For example, the character recognition unit CR4 is the character recognition unit CR with the highest learning degree regarding the character string related to "address" among the plurality of character recognition units CR2 to CR5. Therefore, for example, the probability that the candidate character ("big") obtained from the character recognition unit CR4 is incorrect is considered to be lower than the probability that the candidate character obtained from another character recognition unit CR (for example, the character recognition unit CR3) is incorrect.

[0079] Here, in the example shown in FIG. 6, among the plurality of candidate characters, the candidate character with the highest accuracy is the character "zhang" with an accuracy of 90% as shown by the double circle in FIG. 6. For example, in the form in which the confirmed character is determined without using the evaluation value (hereinafter, also referred to as the first proportional case), among the plurality of candidate characters, the character "zhang" with the highest accuracy is determined as the confirmed character corresponding to the recognition result of the individual character image GC2 of "big". That is, in the first proportional case, for example, the candidate character ("zhang") obtained from the character recognition unit CR3 with a lower learning degree regarding the character string related to "address" than the character recognition unit CR4 is determined as the confirmed character corresponding to the recognition result of the individual character image GC2 of "big". Therefore, in the first proportional case, the probability that an incorrect character is determined as the confirmed character is higher compared to this embodiment. In other words, in this embodiment, it is possible to suppress an incorrect character from being determined as the confirmed character compared to the first proportional case. That is, in this embodiment, the accuracy of character recognition in the character recognition process can be improved.

[0080] Note that the confirmed character determination process is not limited to the example shown in FIG. 6. For example, the evaluation value may be the sum of the accuracy of the candidate character and the weight value. In this case, the weight value of each character recognition unit CR may be set to a value different from the weight value assumed in FIG. 6.

[0081] Also, for example, whether to determine the candidate character obtained as the recognition result of one character by the character recognition unit CR4 with the highest priority among the plurality of character recognition units CR2 to CR5 as the determined character may be determined without using a weight value as shown in FIG. 10 described later. Hereinafter, among the plurality of character recognition units CR that have executed the character recognition process for one character, the character recognition unit CR with the highest priority is also referred to as the character recognition unit CR with the first priority. Also, hereinafter, the candidate character obtained as the recognition result of one character by the character recognition unit CR with the first priority is also referred to as the candidate character with the first priority. The candidate character with the first priority is an example of the "first candidate character".

[0082] Hereinafter, the outline of the method for determining whether to determine the candidate character with the first priority as the determined character without using a weight value will be described using the recognition result shown in FIG. 6 as an example. For example, the recognition result determination unit 126 may determine whether to prioritize the character recognition unit CR4 with the first priority over other character recognition units CR based on whether the accuracy of the candidate character with the first priority is equal to or higher than a predetermined value.

[0083] For example, when the predetermined value is 80%, the accuracy (85%) of the candidate character with the first priority obtained as the recognition result of one character by the character recognition unit CR4 with the first priority is equal to or higher than the predetermined value. In this case, the recognition result determination unit 126 may determine the candidate character with the first priority ("big") as the determined character. That is, when the accuracy of the candidate character with the first priority obtained as the recognition result of one character by the character recognition unit CR with the highest priority among the plurality of character recognition units CR is equal to or higher than the predetermined value, the recognition result determination unit 126 may determine the candidate character with the first priority among the plurality of candidate characters as the determined character.

[0084] The first-priority character recognition unit CR4 is, for example, the character recognition unit CR that has the highest degree of learning about character strings related to "address" among the multiple character recognition units CR2 to CR5. Therefore, when the accuracy of the first-priority candidate character is equal to or greater than a predetermined value, the probability that the first-priority candidate character is incorrect is considered to be lower than the probability that the candidate character obtained as a recognition result of one character by the other character recognition units CR is incorrect. Therefore, the recognition result determination unit 126 determines the first-priority candidate character as the determined character when the accuracy of the first-priority candidate character is equal to or greater than a predetermined value, thereby preventing an incorrect character from being determined as the determined character.

[0085] Furthermore, when the accuracy of the first-priority candidate character obtained as a recognition result of one character by the first-priority character recognition unit CR is less than a predetermined value, the recognition result determination unit 126 may determine the candidate character with the highest accuracy among the multiple candidate characters as the confirmed character. In this case, the candidate character with accuracy less than the predetermined value is prevented from being determined as the confirmed character.

[0086] Here, for example, in a form in which a candidate character of the first priority is unconditionally selected as a confirmed character (hereinafter also referred to as the second comparison), even if the accuracy of the candidate character of the first priority is less than a predetermined value, the candidate character of the first priority is determined as a confirmed character. That is, in the second comparison, when the accuracy of the candidate character of the first priority is less than a predetermined value, the candidate character of the first priority is determined as a confirmed character regardless of whether or not there is a candidate character whose accuracy is equal to or greater than a predetermined value. When the accuracy of the recognition result of a character is less than a predetermined value, the probability that the character is erroneously recognized is higher than when the accuracy of the recognition result of a character is equal to or greater than a predetermined value. Therefore, in the second comparison, the probability that an incorrect character is determined as a confirmed character is higher than in a form in which a candidate character with the highest accuracy among multiple candidate characters is determined as a confirmed character when the accuracy of the candidate character of the first priority is less than a predetermined value.

[0087] That is, even in a configuration in which whether or not to prioritize the character recognition unit CR of the first priority over other character recognition units CR is determined depending on whether the accuracy of the candidate character of the first priority is equal to or greater than a predetermined value, it is possible to prevent an erroneous character from being determined as a confirmed character. Therefore, in this embodiment, even in a case in which whether or not to set the candidate character of the first priority as a confirmed character is determined without using a weight value, it is possible to improve the accuracy of character recognition in the character recognition process.

[0088] In addition, the recognition result determination unit 126 may suspend the determination of the confirmed character when the accuracy of the first-priority candidate character obtained as the recognition result of one character by the first-priority character recognition unit CR is less than a predetermined value. In this case, for example, the confirmed character may not be displayed on the content confirmation screen CCH shown in Fig. 7 described later, or information indicating that the determination of the confirmed character has been suspended (for example, a "?" mark, etc.) may be displayed. When the determination of the confirmed character by the recognition result determination unit 126 is suspended, the confirmed character may be determined, for example, by having the user input or select the confirmed character.

[0089] Next, with reference to FIG. 7, an overview of the content confirmation screen CCH for confirming the confirmed characters determined by the recognition result determination unit 126 will be described.

[0090] FIG. 7 is an explanatory diagram showing an example of the outline of the content confirmation screen CCH.

[0091] The content confirmation screen CCH is displayed, for example, on the display device 28. For example, the content confirmation screen CCH displays a plurality of item names, a plurality of character string images GCL corresponding one-to-one to the plurality of item names, a plurality of recognition character strings LMK corresponding one-to-one to the plurality of character string images GCL, a correction button SSB, and a confirmation button KTB. The plurality of item names correspond one-to-one to a plurality of entry fields in the application form image GM. In FIG. 7, in order to distinguish the plurality of recognition character strings LMK from one another, the same number (1, 2, 3, 4, 5, 6, or 7) is added to the end of each code of the plurality of recognition character strings LMK as the number added to the end of the code of the corresponding character string image GCL among the plurality of character string images GCL.

[0092] The recognized character string LMK is a character string obtained based on the result of character recognition processing executed on the character string image GCL included in the application form image GM in the character recognition device 10. Therefore, each of the multiple characters included in the recognized character string LMK is a confirmed character determined by the recognition result determination unit 126 of the character recognition device 10.

[0093] Also, a caution mark MK may be displayed on the content confirmation screen CCH. Here, the caution mark MK is an image displayed on the content confirmation screen CCH when the character recognition device 10 determines that the character shown by the individual character image GC and the confirmed character may be different characters. In FIG. 7, it is assumed that the confirmed character ("sea") corresponding to the recognition result of the individual character image GC1 included in the character string image GCL5 is not recognized by the first priority character recognition unit CR4 but is recognized by the character recognition unit CR3. Therefore, in FIG. 7, it is assumed that the character recognition device 10 determines that the character shown by the individual character image GC1 and the confirmed character ("sea") corresponding to the recognition result of the individual character image GC1 may be different characters. In this case, for example, the caution mark MK is displayed at a position corresponding to the display position of the confirmed character corresponding to the recognition result of the individual character image GC1.

[0094] 7, it is assumed that the candidate character ("umi") correctly recognized by the character recognition unit CR3 is determined as the confirmed character by the recognition result determination unit 126. That is, the character recognition device 10 judges that the character ("umi") indicated by the individual character image GC1 and the confirmed character ("umi") corresponding to the recognition result of the individual character image GC1 may be different characters, but ultimately, the correct candidate character ("umi") is determined as the confirmed character.

[0095] 6, the confirmed character ("Large") corresponding to the recognition result of the individual character image GC2 included in the character string image GCL5 is recognized by the character recognition unit CR4 of the first priority. In this case, a warning mark MK indicating that the character ("Large") indicated by the individual character image GC2 included in the character string image GCL5 and the confirmed character ("Large") corresponding to the recognition result of the individual character image GC2 may be different characters is not displayed on the content confirmation screen CCH. In other words, the display mode of the confirmed character when the confirmed character is not recognized by the character recognition unit CR of the first priority is different from the display mode of the confirmed character when the confirmed character is recognized by the character recognition unit CR of the first priority.

[0096] By visually checking the attention mark MK, the operator who operates the terminal device 20 can recognize that the character indicated by the individual character image GC and the finalized character may be different characters. As a result, in this embodiment, it is possible to reduce the possibility of overlooking that the finalized character is incorrect, compared to a mode in which the attention mark MK is not displayed.

[0097] The correction button SSB is a GUI (Graphical User Interface) for making the confirmed character correctable. For example, an operator who operates the terminal device 20 compares an individual character image GC with a confirmed character corresponding to the individual character image GC on the content confirmation screen CCH. Then, when the operator who operates the terminal device 20 determines that the confirmed character is a character different from the character shown by the individual character image GC, he or she selects (e.g., presses) the correction button SSB.

[0098] The Confirm button KTB is a GUI for confirming the multiple recognized character strings LMK displayed on the content confirmation screen CCH. For example, when the Confirm button KTB is selected, each confirmed character included in each recognized character string LMK is confirmed, and the state of the content confirmation screen CCH becomes a state in which the operator operating the terminal device 20 cannot change each confirmed character. As a result, the character recognition system 1 confirms the recognition result of the characters included in the application form image GM as the multiple confirmed characters displayed on the content confirmation screen CCH.

[0099] Note that the example of the content confirmation screen CCH is not limited to the example shown in FIG. 7. For example, when a pointer is placed over a caution mark MK displayed on the content confirmation screen CCH, a pop-up image including one or more correction candidate characters may be displayed. The one or more correction candidate characters may be, for example, a predetermined number of candidate characters selected from a plurality of candidate characters in descending order of evaluation value. Alternatively, when two or more candidate characters exist as a recognition result of one character by the first-priority character recognition unit CR, the correction candidate characters may be two or more candidate characters obtained from the first-priority character recognition unit CR.

[0100] Next, an overview of the operation of the character recognition device 10 will be described with reference to FIG.

[0101] Fig. 8 is a flowchart showing an example of the operation of the character recognition device 10 shown in Fig. 1. The operation shown in Fig. 8 is executed by the processing device 12 functioning as the control unit 120 including the image information acquisition unit 122, the written content recognition unit 124, the recognition result determination unit 126, and the display information generation unit 128. For example, in step S100, the processing device 12 functions as the image information acquisition unit 122, and in steps S200, S220, S240, S260, and S400, the processing device 12 functions as the written content recognition unit 124. In addition, in step S300, the processing device 12 functions as the recognition result determination unit 126, and in step S500, the processing device 12 functions as the display information generation unit 128.

[0102] First, in step S100, the image information acquisition unit 122 acquires image information indicating an application form image GM from the terminal device 20.

[0103] Next, in step S200, the description content recognition unit 124 identifies each of the multiple application content description areas AK included in the application form image GM indicated by the image information acquired in step S100.

[0104] Next, in step S220, the description content recognition unit 124 selects one application content description area AK from the multiple application content description areas AK identified in step S200. For example, the description content recognition unit 124 selects one application content description area AK for which character recognition processing has not been performed from the multiple application content description areas AK.

[0105] Next, in step S240, the description content recognition unit 124 selects two or more character recognition units CR that perform character recognition processing on the character string image GCL displayed in the application content description area AK selected in step S220. For example, the description content recognition unit 124 refers to a record of an item corresponding to the application content description area AK selected in step S220 among multiple records in the priority table PTBL, and selects a character recognition unit CR based on the priority stored in the referred record. In the explanation from step S240 onwards, unless otherwise specified, the "character string image GCL" means the character string image GCL displayed in the application content description area AK selected in step S220.

[0106] Next, in step S260, the description recognition unit 124 causes the two or more character recognition units CR selected in step S240 to perform character recognition processing on the character string image GCL. That is, each of the two or more character recognition units CR selected in step S240 performs character recognition processing on the character string image GCL.

[0107] Next, in step S300, the recognition result determination unit 126 executes a confirmed character determination process for determining a confirmed character corresponding to each of the recognition results of the individual character images GC included in the character string image GCL, based on the result of the character recognition process in step S260. For example, as described in FIG. 6, the recognition result determination unit 126 determines a confirmed character corresponding to each of the recognition results of the individual character images GC, based on the accuracy of each of the candidate characters and the priority of each of the character recognition units CR.

[0108] Next, in step S400, the description content recognition unit 124 judges whether the character recognition process in step S260 has been performed on all of the application content description areas AK identified in step S200. If the result of the judgment in step S400 is positive, the processing device 12 advances the process to step S500. On the other hand, if the result of the judgment in step S400 is negative, the processing device 12 returns the process to step S220.

[0109] In step S500, the display information generation unit 128 generates display information for displaying the finalized characters determined by the recognition result determination unit 126 on the display device 28. For example, the display information generation unit 128 generates display information for displaying a content confirmation screen CCH on the display device 28, and outputs the generated display information to the terminal device 20. Execution of the process of step S500 ends the operation shown in FIG.

[0110] The operation of the character recognition device 10 is not limited to the example shown in Fig. 8. For example, the process of step S300 may be executed after the process of step S400, or may be executed in parallel with the series of processes of steps S200, S240, S260, and S400. Also, for example, the process of step S400 may be executed by a functional block (e.g., recognition result determination unit 126, etc.) other than the written content recognition unit 124.

[0111] Also, for example, in a part of step S240 that is repeated multiple times, the written content recognition unit 124 may select only one character recognition unit CR that performs character recognition processing on the character string image GCL. Also, for example, an upper limit on the number of character recognition units CR selected in step S240 may be determined in advance. In this case, the upper limit on the number of character recognition units CR selected in step S240 may be, for example, two. When the upper limit on the number of character recognition units CR selected in step S240 is two, for example, it is preferable that one or both of the character recognition units CR3 and CR4 can recognize multiple types of characters (six types in the embodiment).

[0112] Also, all of the character recognition units CR may be used for character recognition in all of the items. In this case, a priority is determined for all of the character recognition units CR in each item. Also, when all of the character recognition units CR are used for character recognition in all of the items, for example, the process of step S240 is omitted.

[0113] Next, the flow of the confirmed character determination process will be described with reference to FIG.

[0114] FIG. 9 is a flow chart showing an example of the confirmed character determination process shown in FIG. 8. The process shown in FIG. 9 is an example of the confirmed character determination process of step S300 shown in FIG. 8. In the confirmed character determination process shown in FIG. 9, it is assumed that the confirmed characters are determined based on an evaluation value calculated using a weight value based on priority. The process of step S320 is executed, for example, after the process of step S260 in FIG. 8 is executed. Note that, in FIG. 9 as well, unless otherwise specified, "character string image GCL" refers to the character string image GCL displayed in the application content entry area AK selected in step S220 in FIG. 8.

[0115] First, in step S320, the recognition result determination unit 126 identifies one individual character image GC for which a confirmed character has not been determined, from among the multiple individual character images GC included in the character string image GCL.

[0116] Next, in step S340, the recognition result determination unit 126 calculates an evaluation value for each of a plurality of candidate characters corresponding to one individual character image GC identified in step S320. Note that the plurality of candidate characters corresponding to one individual character image GC are, for example, a plurality of candidate characters obtained as a recognition result of one individual character image GC by two or more character recognition units CR selected in step S240 of Fig. 8. Also, the evaluation value for each of the plurality of candidate characters is expressed, for example, as a product of the accuracy of the candidate character and a weight value of the character recognition unit CR that recognized the candidate character.

[0117] Next, in step S360, the recognition result determination unit 126 determines the candidate character having the highest evaluation value calculated in step S340 among the multiple candidate characters corresponding to one individual character image GC as the confirmed character corresponding to the recognition result of the one individual character image GC.

[0118] Next, in step S380, the recognition result determination unit 126 determines whether or not the finalized characters have been determined for all of the individual character images GC included in the character string image GCL. If the result of the determination in step S380 is negative, the processing device 12 returns the process to step S320. On the other hand, if the result of the determination in step S380 is positive, the processing device 12 ends the finalized character determination process and proceeds to step S400 in FIG. 8.

[0119] Next, with reference to FIG. 10, another example of the confirmed character determination process will be described.

[0120] FIG. 10 is a flowchart showing another example of the confirmed character determination process shown in FIG. 8. The process shown in FIG. 10 is another example of the confirmed character determination process of step S300 shown in FIG. 8. The same elements as those in the operation described in FIG. 9 are given the same reference numerals, and detailed description is omitted. In the confirmed character determination process shown in FIG. 10, it is assumed that a weight value based on priority is not used. For example, in the confirmed character determination process shown in FIG. 10, whether or not the character recognition unit CR of the first priority is prioritized over other character recognition units CR is determined depending on whether or not the accuracy of the candidate character of the first priority obtained as a recognition result of one character by the character recognition unit CR of the first priority is a predetermined value or more. The process of step S320 is executed, for example, after the process of step S260 in FIG. 8 is executed. In FIG. 10, unless otherwise specified, the "character string image GCL" means the character string image GCL displayed in the application content description area AK selected in step S220 in FIG. 8.

[0121] First, in step S320, the recognition result determination unit 126 identifies one individual character image GC for which a confirmed character has not been determined, from among the multiple individual character images GC included in the character string image GCL.

[0122] Next, in step S342, the recognition result determination unit 126 determines whether the accuracy of the first-priority candidate character obtained as a recognition result of one character by the first-priority character recognition unit CR is equal to or greater than a predetermined value. If the result of the determination in step S342 is positive, the processing device 12 advances the process to step S362. On the other hand, if the result of the determination in step S342 is negative, the processing device 12 advances the process to step S364.

[0123] In step S362, the recognition result determination unit 126 determines the candidate character with the first priority as the confirmed character corresponding to the recognition result of one individual character image GC. After executing the process of step S362, the processing device 12 advances the process to step S380.

[0124] In step S364, the recognition result determination unit 126 determines the candidate character with the highest probability among the multiple candidate characters corresponding to one individual character image GC as the confirmed character corresponding to the recognition result of one individual character image GC. After executing the process of step S364, the processing device 12 proceeds to the process of step S380.

[0125] In step S380, the recognition result determination unit 126 determines whether or not a finalized character has been determined for each of the individual character images GC included in the character string image GCL. If the result of the determination in step S380 is negative, the processing device 12 returns the process to step S320. On the other hand, if the result of the determination in step S380 is positive, the processing device 12 ends the finalized character determination process and proceeds to step S400 in FIG. 8.

[0126] As described above, in this embodiment, the character recognition device 10 has the written content recognition unit 124 and the recognition result determination unit 126. The written content recognition unit 124 recognizes a character included in an image and recognizes a character included in an application form image GM, which is an image of a document including one or more items, by each of a plurality of character recognition processes (character recognition processes executed by each of a plurality of character recognition units CR) that recognize characters included in the image and calculate the accuracy of the recognized character. The priority of the character recognition process used for character recognition is determined for each item. The recognition result determination unit 126 determines one of a plurality of candidate characters obtained as a recognition result of a single character by each of the plurality of character recognition processes as a confirmed character corresponding to the recognition result of the single character. Specifically, the recognition result determination unit 126 determines one of a plurality of candidate characters as a confirmed character based on the accuracy of each of the plurality of candidate characters and the priority of each of the plurality of character recognition processes.

[0127] In this manner, in this embodiment, a confirmed character corresponding to one character is determined based on the results of multiple character recognition processes. Therefore, in this embodiment, the accuracy of character recognition can be improved compared to an aspect in which a confirmed character corresponding to one character is determined based on the result of one character recognition process. Furthermore, in this embodiment, a confirmed character is determined based on the accuracy of each of multiple candidate characters and the priority of each of multiple character recognition processes, so the accuracy of character recognition can be improved compared to an aspect in which a confirmed character is determined based only on the accuracy of each of multiple candidate characters.

[0128] In addition, in this embodiment, a weight value based on priority may be determined for each of the multiple character recognition processes. In this case, the recognition result determination unit 126 calculates an evaluation value for each of the multiple candidate characters by weighting the accuracy of each of the multiple candidate characters with the weight value of each of the multiple character recognition processes. Then, the recognition result determination unit 126 determines the candidate character with the highest evaluation value among the multiple candidate characters as the confirmed character. In this case, by appropriately setting the weight value, it is possible to prevent an incorrect character from being determined as the confirmed character. For example, in this embodiment, the accuracy of character recognition can be improved by determining the weight value in each item according to the learning degree of each of the multiple character recognition units CR.

[0129] In addition, in this embodiment, a weight value based on priority may not be determined for each of the multiple character recognition processes. In this case, when the accuracy of a candidate character with a first priority obtained as a recognition result of one character by the character recognition process with the highest priority among the multiple character recognition processes is equal to or greater than a predetermined value, the recognition result determination unit 126 may determine the candidate character with the first priority among the multiple candidate characters as a confirmed character. In this way, in this embodiment, it is possible to prevent an incorrect character from being determined as a confirmed character, and the accuracy of character recognition can be improved.

[0130] Furthermore, when the accuracy of a first-priority candidate character obtained as a recognition result of a character by a character recognition process having the highest priority among the plurality of character recognition processes is less than a predetermined value, the recognition result determination unit 126 may determine the candidate character having the highest accuracy among the plurality of candidate characters as a confirmed character. Even in this case, it is possible to prevent an incorrect character from being determined as a confirmed character, and to improve the accuracy of character recognition.

[0131] In this embodiment, the display information generating unit 128 further displays the confirmed character determined by the recognition result determining unit 126 on the display device 28. The display information generating unit 128 changes the display mode of the confirmed character when the confirmed character is not recognized in the character recognition process having the highest priority among the multiple character recognition processes, from the display mode of the confirmed character when the confirmed character is recognized in the character recognition process having the highest priority among the multiple character recognition processes. Therefore, for example, when the display mode of one confirmed character is different from the display mode of other confirmed characters, the operator can recognize that one confirmed character may be incorrect. As a result, in this embodiment, the possibility of overlooking that the confirmed character is incorrect can be reduced compared to when the display mode of the confirmed character does not change even when the confirmed character may be incorrect.

[0132] [2. Second embodiment] Next, an example of an overview of the character recognition device 10 according to the second embodiment will be described with reference to FIG.

[0133] Fig. 11 is an explanatory diagram for explaining an overview of a character recognition device 10 according to a second embodiment. Elements similar to those explained in Fig. 1 to Fig. 10 are given the same reference numerals and detailed explanations are omitted.

[0134] The character recognition device 10 shown in FIG. 11 has the same configuration as the character recognition device 10 shown in FIG. 3. For example, the character recognition device 10 has a processing device 12, a storage device 14, and a communication device 16. However, the control program PG2 stored in the storage device 14 is different from the control program PG1 shown in FIG. 3. The control program PG2 is an example of a "program". In this embodiment, the control program PG2 includes, for example, an application program for the character recognition device 10 to execute the character recognition process. However, the control program PG2 may include, for example, an operating system program for the processing device 12 to control each part of the character recognition device 10.

[0135] In addition, in this embodiment, it is assumed that the priority is stored in the priority table PTBL as information indicating the character recognition unit CR used to recognize the characters written in the entry field of each item (more specifically, the application content entry area AK). Furthermore, in this embodiment, it is assumed that the number of character recognition units CR used to recognize the characters written in the entry field of each item is one or more and two or less. In at least one item among the multiple items, two character recognition units CR are used to recognize the characters. The priorities of the two character recognition units CR may be the same as each other or may be different. In this embodiment, it is assumed that the priorities of the two character recognition units CR used to recognize the characters are the same as each other. In each item, the character recognition process performed by one of the two character recognition units CR used to recognize the characters is an example of a "first character recognition process", and the character recognition process performed by the other of the two character recognition units CR is an example of a "second character recognition process".

[0136] The processing device 12 shown in Fig. 11 is similar to the processing device 12 shown in Fig. 3, except that the processing device 12 executes a control program PG2 instead of the control program PG1 shown in Fig. 3. For example, the processing device 12 functions as a control unit 120A by executing the control program PG2 read from the storage device 14.

[0137] 4, except that the control unit 120A has a recognition result determination unit 126A instead of the recognition result determination unit 126. For example, the control unit 120A has an image information acquisition unit 122, a description content recognition unit 124, a recognition result determination unit 126A, and a display information generation unit 128. The recognition result determination unit 126A is an example of a "determination unit."

[0138] The image information acquisition unit 122 and the written content recognition unit 124 are similar to the image information acquisition unit 122 and the written content recognition unit 124 shown in Fig. 4. However, each of the multiple character recognition units CR included in the written content recognition unit 124 outputs one or more candidate characters and the accuracy of each of the one or more candidate characters as a recognition result for one character.

[0139] The recognition result determination unit 126A acquires recognition information indicating the result of character recognition processing on one character by the written content recognition unit 124. For example, the recognition information indicates the recognition result of one character by the character recognition processing executed by each of two character recognition units CR among the multiple character recognition units CR, or the recognition result of one character by the character recognition processing executed by one character recognition unit CR among the multiple character recognition units CR. The recognition result of one character by the character recognition processing executed by one character recognition unit CR among the two character recognition units CR or one character recognition unit CR includes, for example, one or more candidate characters that are candidates for the recognition result of one character, and the accuracy of the one or more candidate characters.

[0140] Note that one or more candidate characters obtained as a recognition result of a character by the character recognition process of one of the two character recognition units CR are an example of the “first candidate characters,” and one or more candidate characters obtained as a recognition result of a character by the character recognition process of the other of the two character recognition units CR are an example of the “second candidate characters.” In the following, a candidate character obtained as a recognition result of one of the two character recognition units CR that performed character recognition processing on a character is also referred to as a first candidate character, and a candidate character obtained as a recognition result of the other of the two character recognition units CR is also referred to as a second candidate character.

[0141] For example, when a common character included in both one or more first candidate characters and one or more second candidate characters exists, the recognition result determination unit 126A determines whether or not to make the common character a confirmed character corresponding to the recognition result of a character. The confirmed character determination process including the process of determining whether or not to make the common character a confirmed character corresponding to the recognition result of a character will be described later with reference to FIG.

[0142] The display information generating unit 128 is similar to the display information generating unit 128 shown in FIG. 4 except for the condition for changing the display mode of the confirmed character. For example, the display information generating unit 128 changes the display mode of the confirmed character when there is no common character from the display mode of the confirmed character when there is a common character. Specifically, for example, when there is no common character among the multiple candidate characters obtained as the recognition result of one character, the display information generating unit 128 generates display information for displaying a caution mark MK in the content confirmation screen CCH and outputs the generated display information to the terminal device 20. As a result, for example, the caution mark MK is displayed in a position on the content confirmation screen CCH corresponding to the display position of the confirmed character corresponding to the recognition result of one character.

[0143] For example, when the accuracy of character recognition for one character by the two character recognition units CR is low, it is considered that there is a higher probability that no common character exists compared to when the accuracy of character recognition for one character by the two character recognition units CR is high. Therefore, when there is no common character among the multiple candidate characters obtained as the recognition result of one character by the two character recognition units CR, it is considered that there is a higher probability that an incorrect character is determined as the confirmed character compared to when a common character exists. Therefore, by visually checking the attention mark MK, the operator who operates the terminal device 20 can recognize that the character indicated by the individual character image GC and the confirmed character may be different characters. As a result, in this embodiment, it is possible to reduce the possibility of overlooking that the confirmed character is incorrect compared to an aspect in which the attention mark MK is not displayed.

[0144] Note that the operation of control unit 120A is similar to the operation of control unit 120 shown in Fig. 4, except for the confirmed character determination process of step S300 shown in Fig. 8. For example, the operation of control unit 120A is similar to the operation shown in Fig. 8, except for the confirmed character determination process of step S300 shown in Fig. 8.

[0145] Next, an example of a confirmed character determination process performed by the recognition result determination unit 126A will be described with reference to FIG.

[0146] Fig. 12 is an explanatory diagram for explaining an example of the operation of the recognition result determination unit 126A shown in Fig. 11. In Fig. 12, as in Fig. 6 described above, an example of a method for determining a confirmed character corresponding to the recognition result of the individual character image GC2 of "large" is explained using an example in which character recognition processing is performed on a character string image GCL5 showing a character string written in the application content description area AK related to "address". However, in Fig. 12, in order to make the explanation easier to understand, there are some parts where the accuracy and the like are different from the example shown in Fig. 6.

[0147] In FIG. 12, it is assumed that the character recognition units CR3 and CR4 are selected as the character recognition unit CR used for recognizing the characters of the character string image GCL5. In this case, the character recognition process executed by one of the character recognition units CR3 and CR4 corresponds to the "first character recognition process", and the character recognition process executed by the other of the character recognition units CR3 and CR4 corresponds to the "second character recognition process". Hereinafter, the candidate characters obtained as the recognition result of the character recognition unit CR3 are also referred to as first candidate characters, and the candidate characters obtained as the recognition result of the character recognition unit CR4 are also referred to as second candidate characters.

[0148] In the example shown in FIG. 12, the recognition result determination unit 126A acquires, as the result of the character recognition process for the individual character image GC2 of "big" by the character recognition unit CR3, a plurality of first candidate characters including the characters "zhang", "tai", "da", "huo", and "mu", and the accuracy of each of the plurality of first candidate characters. Further, the recognition result determination unit 126A acquires, as the result of the character recognition process for the individual character image GC2 of "big" by the character recognition unit CR4, a plurality of second candidate characters including the characters "dog", "big", "tai", "mu", and "zhang", and the accuracy of each of the plurality of second candidate characters.

[0149] The recognition result determination unit 126A searches for common characters existing in both the plurality of first candidate characters and the plurality of second candidate characters from the plurality of candidate characters including the plurality of first candidate characters and the plurality of second candidate characters. In FIG. 12, it is assumed that the characters "zhang", "tai", "da", and "mu" exist in both the plurality of first candidate characters and the plurality of second candidate characters. Therefore, the recognition result determination unit 126A recognizes the characters "zhang", "tai", "da", and "mu" as common characters.

[0150] Then, based on whether there is one or more common characters that satisfy both the first condition and the second condition, the recognition result determination unit 126A determines whether to set the common character as the definitive character corresponding to the recognition result of one character. The first condition is a condition regarding the higher of the first accuracy, which is the accuracy of the common character included in one or more first candidate characters, and the second accuracy, which is the accuracy of the common character included in one or more second candidate characters. The second condition is a condition regarding the degree of difference between the first accuracy and the second accuracy. Specifically, the first condition is, for example, the condition that the higher of the first accuracy and the second accuracy is equal to or higher than a predetermined accuracy, and the second condition is the condition that the absolute value of the difference between the first accuracy and the second accuracy is equal to or lower than a predetermined value.

[0151] In FIG. 12, assume that the first condition is the condition that the higher of the first accuracy and the second accuracy is 80% or higher, and the second condition is the condition that the absolute value of the difference between the first accuracy and the second accuracy is 20% or lower. In this case, among the plurality of common characters ("zhang", "tai", "da", and "mu"), the character "zhang" does not satisfy the second condition because the difference between the first accuracy and the second accuracy is 40%. The character "tai" satisfies both the first condition and the second condition because the higher of the first accuracy and the second accuracy is 85% and the difference between the first accuracy and the second accuracy is 5%. The character "da" satisfies both the first condition and the second condition because the higher of the first accuracy and the second accuracy is 87% and the difference between the first accuracy and the second accuracy is 5%. Also, the character "mu" does not satisfy the first condition because the higher of the first accuracy and the second accuracy is 60%.

[0152] Therefore, the recognition result determination unit 126A identifies the characters "thick" and "big" among the plurality of common characters ("length", "thick", "big", and "tree") as the common characters that satisfy both the first condition and the second condition. For example, when the common character satisfies both the first condition and the second condition, the recognition result determination unit 126A determines the common character that satisfies both the first condition and the second condition as the candidate for the definite character corresponding to the recognition result of one character, that is, the definite candidate character. That is, the recognition result determination unit 126A determines the characters "thick" and "big" as the definite candidate characters. When there are a plurality of definite candidate characters, the recognition result determination unit 126A determines one of the plurality of definite candidate characters as the definite character.

[0153] When there is only one definite candidate character, the recognition result determination unit 126A determines the definite candidate character as the definite character. Hereinafter, as an example of the method for determining the definite character when there are a plurality of definite candidate characters, the first determination method, the second determination method, and the third determination method will be described. However, the method for determining the definite character when there are a plurality of definite candidate characters is not limited to the following first determination method, second determination method, and third determination method.

[0154] In the first determination method, the recognition result determination unit 126A determines the definite candidate character with the highest certainty among the plurality of definite candidate characters as the definite character. In the recognition result shown in FIG. 12, among the two definite candidate characters ("thick" and "big"), the certainty of the character "big" is the highest. Therefore, in the first determination method, the recognition result determination unit 126A determines the definite candidate character with the highest certainty ("big") among the two definite candidate characters ("thick" and "big") as the definite character. In the first determination method, since the definite character can be determined without performing operations such as calculating the average value of the first certainty and the second certainty, it is possible to suppress the complication of the definite character determination process.

[0155] In the second determination method, the recognition result determination unit 126A determines, as the determined character, the determined candidate character having the highest average value of the first accuracy and the second accuracy among the multiple determined candidate characters. The average value of the accuracy of the character "bold" is 82.5% (=(85%+80%) / 2), and the average value of the accuracy of "large" is 84.5% (=(82%+87%) / 2). Therefore, in the second determination method, the recognition result determination unit 126A determines, as the determined character, the determined candidate character ("large") having the highest average accuracy among the two determined candidate characters ("bold" and "large"). In the second determination method, the determined character can be determined by a simple calculation (calculation of the average value of the first accuracy and the second accuracy) without using a complex calculation, so that the determined character determination process can be prevented from becoming complicated.

[0156] In the third determination method, the recognition result determination unit 126A determines the confirmed candidate character with the highest evaluation value among multiple confirmed candidate characters as the confirmed character. For example, if the evaluation value is Ev, the higher of the first and second accuracies is ACmax, and the absolute value of the difference between the first and second accuracies is ACdiff, the evaluation value is expressed by the following formula (1) using coefficients α and β. Note that "*" in formula (1) indicates multiplication. The coefficients α and β are real numbers greater than 0. Ev=ACmax*α-ACdiff*β (1)

[0157] In this way, in the third determination method, a determined character is determined using an evaluation value (Ev) based on the higher of the first and second accuracies (ACmax) and the absolute value of the difference between the first and second accuracies (ACdiff). Therefore, in the third determination method, a determined character can be determined with higher accuracy than when a determined character is determined without using the evaluation value (Ev) expressed by the above-mentioned formula (1).

[0158] Note that the relationship between the evaluation value (Ev), the first accuracy, and the second accuracy is not limited to the above formula (1). For example, in the above formula (1), the evaluation value (Ev) may be calculated using the average value of the first accuracy and the second accuracy instead of the higher accuracy (Acmax) of the first accuracy and the second accuracy. Further, the evaluation value (Ev) may be calculated using the ratio of the first accuracy to the second accuracy instead of the absolute value (ACdiff) of the difference between the first accuracy and the second accuracy. The ratio of the first accuracy to the second accuracy is, for example, a value obtained by dividing the higher accuracy of the first accuracy and the second accuracy by the lower accuracy of the first accuracy and the second accuracy. When the value obtained by dividing the lower accuracy of the first accuracy and the second accuracy by the higher accuracy of the first accuracy and the second accuracy is used as the ratio of the first accuracy to the second accuracy, in the above formula (1), the coefficient β becomes a negative real number.

[0159] Here, in the example shown in FIG. 12, among the plurality of common characters, the common character with the highest accuracy is the character "length" with an accuracy of 90%. Although the accuracy of the character "length" is 90%, in the character recognition unit CR4, the accuracy of the character "length" is 50%. Therefore, the character "length" satisfies the first condition regarding the higher accuracy of the first accuracy and the second accuracy, but does not satisfy the second condition regarding the degree of difference between the first accuracy and the second accuracy. When a common character that satisfies only one of the first condition and the second condition is determined as the determined character, it is considered that the probability of an incorrect character being determined as the determined character is higher than when a common character that satisfies both the first condition and the second condition is determined as the determined character.

[0160] In the present embodiment, by determining a common character that satisfies both the first condition and the second condition as the determined character, it is possible to suppress the determination of an incorrect character as the determined character as compared with the case where a common character that satisfies one or both of the first condition and the second condition is determined as the determined character.

[0161] The method of determining the confirmed character is not limited to the above example. For example, the recognition result determination unit 126A may determine the common character with the highest accuracy among the multiple common characters as the confirmed character without considering either the first condition or the second condition. Alternatively, the recognition result determination unit 126A may determine the common character with the highest average value of the first accuracy and the second accuracy among the multiple common characters as the confirmed character without considering either the first condition or the second condition. Alternatively, the recognition result determination unit 126A may determine the common character with the highest evaluation value expressed by the above formula (1) among the multiple common characters as the confirmed character without considering either the first condition or the second condition.

[0162] Even when a common character is determined as a confirmed character without considering either the first or second condition, the probability of an incorrect character being determined as a confirmed character is considered to be lower than in a case where a candidate character with the highest probability among multiple candidate characters is unconditionally determined as a confirmed character. Hereinafter, the case where a candidate character with the highest probability among multiple candidate characters is unconditionally determined as a confirmed character is also referred to as the third comparison.

[0163] For example, in the recognition result shown in FIG. 12, the candidate character with the highest accuracy among the multiple candidate characters is the character "inu" with an accuracy of 92%. The character "inu" is included in the multiple second candidate characters obtained as the recognition result of one character by the character recognition unit CR4, but is not included in the multiple first candidate characters obtained as the recognition result of one character by the character recognition unit CR3. Therefore, in the third comparison example, an incorrect character ("inu") is determined as the confirmed character corresponding to the recognition result of the individual character image GC2 of "big". Note that the candidate character ("big") that is correct as the confirmed character corresponding to the recognition result of the individual character image GC2 of "big" is included in the multiple common characters. Therefore, even if one of the multiple common characters is determined as the confirmed character without considering either the first or second condition, it is considered that the probability of an incorrect character being determined as the confirmed character is lower than in the third comparison example. In other words, in this embodiment, even if neither the first nor the second condition is taken into consideration in determining the confirmed character, by determining one of one or more common characters as the confirmed character, it is possible to prevent an incorrect character from being determined as the confirmed character, compared to the third comparison example.

[0164] Next, the flow of the confirmed character determination process performed by the recognition result determination unit 126A will be described with reference to FIG.

[0165] Fig. 13 is a flowchart showing an example of the operation of the recognition result determination unit 126A shown in Fig. 11. In the confirmed character determination process shown in Fig. 13, it is assumed that a first condition related to the higher of the first and second accuracy and a second condition related to the degree of difference between the first and second accuracy are used to determine a confirmed character.

[0166] The confirmed character determination process shown in Fig. 13 is executed, for example, as the confirmed character determination process of step S300 in the operation shown in Fig. 8. For example, the process of step S320 is executed after the process of step S260 in Fig. 8 is executed. Elements similar to those in the operation described in Fig. 9 and Fig. 10 are given the same reference numerals and detailed description will be omitted.

[0167] In Fig. 13, unless otherwise specified, "character string image GCL" refers to the character string image GCL displayed in the application content entry area AK selected in step S220 in Fig. 8. Also, in Fig. 13, it is assumed that two character recognition units CR are selected in step S240 in Fig. 8. In addition, when the number of character recognition units CR selected in step S240 in Fig. 8 is one, the series of processes from step S350 to step S372 is omitted, and the series of processes in steps S320, S374, and S380 are executed.

[0168] First, in step S320, the recognition result determination unit 126A identifies one individual character image GC for which a confirmed character has not been determined, from among the multiple individual character images GC included in the character string image GCL.

[0169] Next, in step S350, the recognition result determination unit 126A searches for a common character from among a plurality of candidate characters corresponding to one individual character image GC.

[0170] Next, in step S352, the recognition result determination unit 126A determines whether or not a common character exists. That is, the recognition result determination unit 126A determines whether or not a common character has been searched for among a plurality of candidate characters. If the result of the determination in step S352 is positive, the processing device 12 advances the process to step S354. On the other hand, if the result of the determination in step S352 is negative, the processing device 12 advances the process to step S374.

[0171] In step S354, the recognition result determination unit 126A determines whether or not a common character that satisfies the first condition is present. If the result of the determination in step S354 is positive, the processing device 12 advances the process to step S356. On the other hand, if the result of the determination in step S354 is negative, the processing device 12 advances the process to step S374.

[0172] In step S356, the recognition result determination unit 126A determines whether or not a common character that satisfies the second condition is present. If the result of the determination in step S356 is positive, the processing device 12 advances the process to step S370. On the other hand, if the result of the determination in step S356 is negative, the processing device 12 advances the process to step S372.

[0173] In step S370, the recognition result determination unit 126A determines one of the common characters (i.e., the determined candidate characters) that satisfy both the first condition and the second condition as a determined character. Note that, for example, when there are multiple determined candidate characters, the recognition result determination unit 126A may determine one of the multiple determined candidate characters as a determined character by any of the first determination method, the second determination method, and the third determination method described in Fig. 12 above. After executing the process of step S370, the processing device 12 advances the process to step S380.

[0174] In step S372, the recognition result determination unit 126A determines one of the common characters that satisfy the first condition as a confirmed character. For example, when there are multiple common characters that satisfy the first condition, the recognition result determination unit 126A determines the common character with the highest accuracy among the multiple common characters that satisfy the first condition as a confirmed character. Alternatively, the recognition result determination unit 126A may determine the common character with the highest evaluation value represented by the above-mentioned formula (1) among the multiple common characters that satisfy the first condition as a confirmed character. After executing the process of step S372, the processing device 12 advances the process to step S380.

[0175] In step S374, the recognition result determination unit 126A determines one of the multiple candidate characters as a confirmed character. For example, the recognition result determination unit 126A determines the candidate character with the highest probability among the multiple candidate characters as a confirmed character. After executing the process of step S374, the processing device 12 proceeds to the process of step S380.

[0176] In step S380, the recognition result determination unit 126A determines whether or not a finalized character has been determined for each of the individual character images GC included in the character string image GCL. If the result of the determination in step S380 is negative, the processing device 12 returns the process to step S320. On the other hand, if the result of the determination in step S380 is positive, the processing device 12 ends the finalized character determination process and proceeds to step S400 in FIG. 8.

[0177] Note that the operation of the recognition result determination unit 126A is not limited to the example shown in Fig. 13. For example, instead of the determination in step S356, the recognition result determination unit 126A may determine whether or not a common character (i.e., a confirmed candidate character) that satisfies both the first and second conditions is present before the determination in step S354. In this case, the determination in step S354 is performed when a confirmed candidate character is not present, and the process in step S372 is performed when the result of the determination in step S354 is positive.

[0178] Also, for example, if the result of the determination in step S356 is negative, the recognition result determination unit 126A may execute the process of step S374. That is, the process of step S372 may be omitted. Also, for example, if the result of the determination in step S354 is negative, the recognition result determination unit 126A may determine one of the common characters as the confirmed character in step S374. For example, if the result of the determination in step S354 is negative, the recognition result determination unit 126A may determine the common character having the highest evaluation value represented by the above-mentioned formula (1) among the multiple common characters as the confirmed character.

[0179] As described above, in this embodiment, the character recognition device 10 has the written content recognition unit 124 and the recognition result determination unit 126A. The written content recognition unit 124 recognizes a character included in the application form image GM by each of a first character recognition process that recognizes a character included in an image and calculates the accuracy of the recognized character, and a second character recognition process that recognizes a character included in an image and calculates the accuracy of the recognized character (character recognition process executed by each of the two character recognition units CR). When there is a common character included in both one or more first candidate characters obtained as a recognition result of the one character by the first character recognition process and one or more second candidate characters obtained as a recognition result of the one character by the second character recognition process, the recognition result determination unit 126A determines whether or not to set the common character as a confirmed character corresponding to the recognition result of the one character.

[0180] Among a plurality of candidate characters including the first candidate character and the second candidate character, a common character included in both the first candidate character and the second candidate character is a candidate character recognized in both the first character recognition process and the second character recognition process as a recognition result of one character. In other words, a candidate character other than the common character is a candidate character recognized in only one of the first character recognition process and the second character recognition process as a recognition result of one character. Therefore, it is considered that the probability that a correct candidate character is included in the common characters as a recognition result of one character is higher than the probability that a correct candidate character is included in the candidate characters other than the common characters as a recognition result of one character. In this embodiment, when a common character exists, it is determined whether or not the common character is to be a confirmed character, so that it is possible to suppress an incorrect character from being determined as a confirmed character. In other words, in this embodiment, it is possible to improve the accuracy of character recognition in the character recognition process.

[0181] In addition, in this embodiment, the recognition result determination unit 126A may determine a common character as a confirmed candidate character that is a candidate for a confirmed character when the common character satisfies both the first condition and the second condition. The first condition is a condition related to the higher of the first accuracy, which is the accuracy of a common character included in one or more first candidate characters, and the second accuracy, which is the accuracy of a common character included in one or more second candidate characters. The second condition is a condition related to the degree of difference between the first accuracy and the second accuracy. It is considered that the probability that a correct candidate character is included in common characters that satisfy both the first condition and the second condition is higher than the probability that a correct candidate character is included in common characters that do not satisfy one or both of the first condition and the second condition. Therefore, in this embodiment, by determining a common character that satisfies both the first condition and the second condition as a confirmed candidate character, it is possible to suppress an incorrect character from being determined as a confirmed character compared to a form in which a common character is unconditionally determined as a confirmed character.

[0182] For example, in this embodiment, the first condition may be a condition that the higher of the first and second accuracies is equal to or higher than a predetermined accuracy, and the second condition may be a condition that the difference between the first and second accuracies is equal to or lower than a predetermined value. A common character that satisfies both the first and second conditions has a higher of the first and second accuracies equal to or higher than a predetermined accuracy, and a difference between the first and second accuracies is equal to or lower than a predetermined value. That is, a common character that satisfies both the first and second conditions has a higher of the higher of the first and second accuracies compared to a common character that does not satisfy the first condition, and has a smaller difference between the first and second accuracies compared to a common character that does not satisfy the second condition. For this reason, a common character that satisfies both the first and second conditions is considered to have a higher accuracy of character recognition compared to a common character that does not satisfy one or both of the first and second conditions. Therefore, in this embodiment, the accuracy of character recognition in the character recognition process can be improved by determining a common character that satisfies both the first and second conditions as a confirmed candidate character.

[0183] In addition, in this embodiment, when there are multiple confirmed candidate characters, the recognition result determination unit 126A may determine the confirmed candidate character with the highest accuracy among the multiple confirmed candidate characters as the confirmed character. Even in this case, it is possible to prevent an incorrect character from being determined as the confirmed character, and it is possible to improve the accuracy of character recognition in the character recognition process.

[0184] In addition, in this embodiment, when there are multiple confirmed candidate characters, the recognition result determination unit 126A may determine the confirmed candidate character having the highest average value of the first probability and the second probability among the multiple confirmed candidate characters as the confirmed character. Even in this case, it is possible to prevent an erroneous character from being determined as the confirmed character, and it is possible to improve the accuracy of character recognition in the character recognition process.

[0185] In addition, in this embodiment, the above-mentioned first and second conditions may not be set. In a form in which the first and second conditions are not set, when there are multiple common characters, the recognition result determination unit 126A may determine the common character with the highest probability among the multiple common characters as the confirmed character. Even in this case, it is possible to suppress the determination of an incorrect character as the confirmed character compared to a case in which the candidate character with the highest probability among multiple candidate characters is unconditionally determined as the confirmed character, and it is possible to improve the accuracy of character recognition in the character recognition process.

[0186] Alternatively, in a form in which the first condition and the second condition are not set, when there are multiple common characters, the recognition result determination unit 126A may determine the common character having the highest average value of the first probability and the second probability among the multiple common characters as the confirmed character. Even in this case, it is possible to prevent an erroneous character from being determined as the confirmed character, compared to a case in which the candidate character having the highest probability among multiple candidate characters is unconditionally determined as the confirmed character, and it is possible to improve the accuracy of character recognition in the character recognition process.

[0187] In addition, in this embodiment, the display information generating unit 128 is further provided, which displays the confirmed character determined by the recognition result determining unit 126A on the display device 28. The display information generating unit 128 makes the display mode of the confirmed character when there is no common character different from the display mode of the confirmed character when there is a common character. When there is no common character, the accuracy of character recognition for one character by the two character recognition units CR is considered to be lower than when there is a common character. That is, when there is no common character, the probability that an incorrect character is determined as the confirmed character is considered to be higher than when there is a common character. For this reason, for example, when the display mode of one confirmed character is different from the display mode of the other confirmed characters, the operator can recognize that one confirmed character may be incorrect. As a result, in this embodiment, the possibility of overlooking that the confirmed character is incorrect can be reduced compared to when the display mode of the confirmed character does not change even when there is a possibility that the confirmed character is incorrect.

[0188] In this embodiment, similarly to the above-described first embodiment, a weight value may be associated with the character recognition unit CR for each item.

[0189] [3. Third embodiment] Next, an example of an overview of the character recognition device 10 according to the third embodiment will be described with reference to FIG.

[0190] Fig. 14 is an explanatory diagram for explaining an overview of a character recognition device 10 according to a third embodiment. Elements similar to those explained in Fig. 1 to Fig. 13 are given the same reference numerals and detailed explanations are omitted.

[0191] The character recognition device 10 shown in FIG. 14 has the same configuration as the character recognition device 10 shown in FIG. 11. For example, the character recognition device 10 has a processing device 12, a storage device 14, and a communication device 16. However, the control program PG3 stored in the storage device 14 is different from the control program PG2 shown in FIG. 11. The control program PG3 is an example of a "program". In this embodiment, the control program PG3 includes, for example, an application program for the character recognition device 10 to execute the character recognition process. However, the control program PG3 may include, for example, an operating system program for the processing device 12 to control each part of the character recognition device 10.

[0192] In this embodiment, the priority table PTBL and the upper limit of the number of character recognition units CR used to recognize characters written in the description column of each item are the same as those in the above-mentioned second embodiment. For example, in this embodiment, one to two character recognition units CR are used to recognize characters in each item, and two character recognition units CR are used to recognize characters in at least one item of the multiple items. Hereinafter, one of the two character recognition units CR used to recognize characters is also referred to as a first character recognition unit CR, and the other of the two character recognition units CR is also referred to as a second character recognition unit CR.

[0193] In this embodiment, it is assumed that the first character recognition unit CR recognizes multiple characters that make up a character string included in the application form image GM, and the second character recognition unit CR recognizes one character among the multiple characters that make up the character string included in the application form image GM. The character recognition process performed by the first character recognition unit CR is an example of a "first character recognition process," and the character recognition process performed by the second character recognition unit CR is an example of a "second character recognition process." Additionally, the application form image GM is an example of an "image." The character string indicated by the character string image GCL included in the application form image GM corresponds to the character string included in the application form image GM.

[0194] The processing device 12 shown in Fig. 14 is similar to the processing device 12 shown in Fig. 11, except that the processing device 12 executes a control program PG3 instead of the control program PG2 shown in Fig. 11. For example, the processing device 12 functions as a control unit 120B by executing the control program PG3 read from the storage device 14.

[0195] 11 except that the control unit 120B has a recognition result determination unit 126B instead of the recognition result determination unit 126A. For example, the control unit 120B has an image information acquisition unit 122, a description content recognition unit 124, a recognition result determination unit 126B, and a display information generation unit 128. The recognition result determination unit 126B is an example of a "determination unit."

[0196] The image information acquisition unit 122 and the written content recognition unit 124 are the same as the image information acquisition unit 122 and the written content recognition unit 124 shown in Fig. 11. However, the multiple character recognition units CR of the written content recognition unit 124 include a first character recognition unit CR and a second character recognition unit CR. The first character recognition unit CR outputs candidate characters and the accuracy of each candidate character for each of the multiple characters constituting the character string as a recognition result of the character string included in the application form image GM. The second character recognition unit CR outputs candidate characters and the accuracy of the candidate characters as a recognition result of one character out of the multiple characters.

[0197] A candidate character obtained as a result of recognizing a character by the first character recognition unit CR is an example of a “first candidate character,” and a candidate character obtained as a result of recognizing a character by the second character recognition unit CR is an example of a “second candidate character.” Hereinafter, a candidate character obtained as a result of recognizing a character by the first character recognition unit CR is also referred to as a first candidate character, and a candidate character obtained as a result of recognizing a character by the second character recognition unit CR is also referred to as a second candidate character.

[0198] The recognition result determination unit 126B acquires recognition information indicating the result of character recognition processing on one character by the written content recognition unit 124. For example, the recognition information indicates the recognition result of a character string (a character string including the one character) by the character recognition processing executed by the first character recognition unit CR, and the recognition result of the one character by the character recognition processing executed by the second character recognition unit CR.

[0199] For example, when the first candidate character and the second candidate character are the same character, the recognition result determination unit 126B determines the same character as a confirmed character corresponding to the recognition result of one character.

[0200] The display information generating unit 128 is similar to the display information generating unit 128 shown in FIG. 11 except for the condition for making the display form of the confirmed character different. For example, the display information generating unit 128 makes the display form of the confirmed character when the first candidate character and the second candidate character are different from each other different from the display form of the confirmed character when the first candidate character and the second candidate character are the same as each other. Specifically, for example, when the first candidate character and the second candidate character are different from each other, the display information generating unit 128 generates display information for displaying a caution mark MK in the content confirmation screen CCH and outputs the generated display information to the terminal device 20. As a result, for example, the caution mark MK is displayed in a position on the content confirmation screen CCH corresponding to the display position of the confirmed character corresponding to the recognition result of one character.

[0201] For example, when the first candidate character and the second candidate character are different from each other, it is considered that the probability that an incorrect character is determined as the confirmed character is higher than when the first candidate character and the second candidate character are the same from each other. Therefore, by visually checking the attention mark MK, the operator who operates the terminal device 20 can recognize that the character indicated by the individual character image GC and the confirmed character may be different characters. As a result, in this embodiment, it is possible to reduce the possibility of overlooking that the confirmed character is incorrect, compared to an aspect in which the attention mark MK is not displayed.

[0202] The display information generating unit 128 may change the display mode of the confirmed characters when the first candidate character and the second candidate character are different from each other and the accuracy of the first candidate character is lower than the accuracy of the second candidate character by a predetermined value or more from the display mode of the confirmed characters when the first candidate character and the second candidate character are the same. For example, the display information generating unit 128 may generate display information that displays a caution mark MK in the content confirmation screen CCH when the first candidate character and the second candidate character are different from each other and the accuracy of the first candidate character is lower than the accuracy of the second candidate character by a predetermined value or more.

[0203] For example, even if the first candidate character and the second candidate character are different from each other, if the difference between the accuracy of the first candidate character and the accuracy of the second candidate character is less than a predetermined value, the confirmed character is considered to have a higher probability of being correct than when the accuracy of the first candidate character is lower than that of the second candidate character by a predetermined value or more. Therefore, for example, when the first candidate character and the second candidate character are different from each other and the accuracy of the first candidate character is lower than that of the second candidate character by a predetermined value or more, in a form in which the attention mark MK is displayed in the content confirmation screen CCH, the attention mark MK can be displayed effectively. Displaying the attention mark MK effectively means, for example, suppressing the frequent display of the attention mark MK when the confirmed character is correct.

[0204] Furthermore, the display information generating unit 128 may change the display mode of the confirmed character when the first candidate character and the second candidate character are different between a case where the accuracy of the first candidate character is lower than that of the second candidate character by a predetermined value or more and a case where the difference between the accuracy of the first candidate character and the accuracy of the second candidate character is less than a predetermined value. For example, the display information generating unit 128 may change the color of the attention mark MK between a case where the accuracy of the first candidate character is lower than that of the second candidate character by a predetermined value or more and a case where the difference between the accuracy of the first candidate character and the accuracy of the second candidate character is less than a predetermined value. In this case, for example, an operator operating the terminal device 20 can recognize the degree of possibility that the character indicated by the individual character image GC and the confirmed character are different characters by visually checking the color of the attention mark MK.

[0205] Note that the operation of the control unit 120B is the same as that of the control unit 120A shown in FIG. 11, except for the confirmed character determination process. For example, the operation of the control unit 120B is the same as the operation shown in FIG. 8, except that the confirmed character determination process described in FIG. 15 to be described later is executed in step S300 shown in FIG. 8.

[0206] Next, with reference to FIG. 15, an example of the confirmed character determination process by the recognition result determination unit 126B will be described.

[0207] FIG. 15 is an explanatory diagram for explaining an example of the operation of the recognition result determination unit 126B shown in FIG. 14. In FIG. 15, taking as an example the case where character recognition processing is executed on a character string image GCL5 showing the character string described in the application content description area AK related to "address", an example of a method for determining confirmed characters corresponding to the recognition results of individual character images GC of "east", "blue", and "sea" is described.

[0208] Note that in FIG. 15, it is assumed that the character recognition units CR3 and CR4 are selected as the character recognition unit CR used for recognizing the characters of the character string image GCL5. Further, in FIG. 15, it is assumed that the character recognition unit CR4 is the first character recognition unit CR, and the character recognition unit CR3 is the second character recognition unit CR. Also, in FIG. 15, it is assumed that the character recognition unit CR4 uses a dictionary related to addresses (hereinafter also referred to as an address dictionary) to specify a candidate character string that is the recognition result of a character string including one character.

[0209] In FIG. 15, it is assumed that the two first candidate characters corresponding to the recognition results of the individual character images GC of "east" and "blue" by the character recognition unit CR4 are the characters "east" and "blue", respectively. Also, in FIG. 15, it is assumed that the candidates for the first candidate character corresponding to the recognition result of the individual character image GC1 of "sea" by the character recognition unit CR4 are the characters "sea", "plum", and "insult".

[0210] The character recognition unit CR4 determines whether a character string formed by combining two first candidate characters corresponding to the recognition results of the individual character images GC of "Dong" and "Qing" and a candidate for the first candidate character corresponding to the recognition result of the individual character image GC1 of "Hai" is registered in the address dictionary. The cross mark in Fig. 15 indicates that the character string is not registered in the address dictionary, and the round mark in Fig. 15 indicates that the character string is registered in the address dictionary. For example, since the character strings "Dongqinghai" and "Dongqingwu" are not registered in the address dictionary, the character recognition unit CR4 determines that the character strings "Dongqinghai" and "Dongqingwu" are not candidate character strings. Also, since the character string "Dongqingmei" is registered in the address dictionary, the character recognition unit CR4 determines the character string "Dongqingmei" as a candidate character string. That is, the character recognition unit CR4 determines the character "Mei" as the first candidate character corresponding to the recognition result of the individual character image GC1 of "Hai".

[0211] On the other hand, among the candidates for the second candidate characters corresponding to the recognition result of the individual character image GC1 of "Hai" ( "Hai", "Mei", and "Wu"), the character recognition unit CR3 determines the character "Hai" with the highest accuracy as the second candidate character corresponding to the recognition result of the individual character image GC1 of "Hai". In Fig. 15, it is assumed that the two second candidate characters corresponding to the recognition results of the individual character images GC of "Dong" and "Qing" by the character recognition unit CR3 are the characters "Dong" and "Qing", respectively.

[0212] The recognition result determination unit 126B obtains the first candidate character ("Higashi") and the accuracy (90%) of the first candidate character as a result of the character recognition process performed by the character recognition unit CR4 on the individual character image GC of "Higashi". Furthermore, the recognition result determination unit 126B obtains the second candidate character ("Higashi") and the accuracy (88%) of the second candidate character as a result of the character recognition process performed by the character recognition unit CR3 on the individual character image GC of "Higashi". Since the first candidate character ("Higashi") and the second candidate character ("Higashi") are the same, the recognition result determination unit 126B determines the character "Higashi" as the confirmed character corresponding to the recognition result of the individual character image GC of "Higashi". Similarly, since the first candidate character ("blue") and the second candidate character ("blue") obtained as a result of the character recognition process on the individual character image GC of "blue" are identical to each other, the recognition result determination unit 126B determines the character "blue" as the confirmed character corresponding to the recognition result of the individual character image GC of "blue".

[0213] The recognition result determination unit 126B also obtains the first candidate character ("Ume") and the accuracy (30%) of the first candidate character as a result of the character recognition process performed by the character recognition unit CR4 on the individual character image GC1 of "Umi". The recognition result determination unit 126B also obtains the second candidate character ("Umi") and the accuracy (90%) of the second candidate character as a result of the character recognition process performed by the character recognition unit CR3 on the individual character image GC1 of "Umi".

[0214] Although the first candidate character ("ume") and the second candidate character ("umi") are different from each other, the recognition result determination unit 126B determines the first candidate character ("ume") as the confirmed character corresponding to the recognition result of the individual character image GC1 of "umi". However, since the first candidate character ("ume") and the second candidate character ("umi") are different from each other, the recognition result determination unit 126B notifies the display information generation unit 128 that the first candidate character and the second candidate character are different from each other. Then, the display information generation unit 128 generates display information for displaying a caution mark MK in the content confirmation screen CCH, for example, and outputs the generated display information to the terminal device 20. As a result, the caution mark MK is displayed in a position on the content confirmation screen CCH corresponding to the display position of the confirmed character corresponding to the recognition result of the individual character image GC1 of "umi".

[0215] The operation of the recognition result determination unit 126B is not limited to the above example. For example, when the first candidate character ("ume") and the second candidate character ("umi") are different from each other, the recognition result determination unit 126B may determine the second candidate character ("umi"), which has a higher probability among the first and second candidate characters, as the confirmed character corresponding to the recognition result of the individual character image GC1 of "umi". In other words, when the first candidate character and the second candidate character are different from each other, the recognition result determination unit 126B may determine the candidate character with a higher probability among the first and second candidate characters as the confirmed character. Even in this case, the recognition result determination unit 126B notifies the display information generation unit 128 that the first candidate character and the second candidate character are different from each other.

[0216] Alternatively, when the first candidate character and the second candidate character are different from each other, the recognition result determination unit 126B may determine one of the first candidate character and the second candidate character as a confirmed character based on a judgment value that is a value obtained by subtracting the accuracy of the first candidate character from the accuracy of the second candidate character. Specifically, for example, the recognition result determination unit 126B may determine the first candidate character as a confirmed character when the judgment value is less than a predetermined value, and may determine the second candidate character as a confirmed character when the judgment value is equal to or greater than the predetermined value.

[0217] In addition, when the first candidate character and the second candidate character are different from each other and the accuracy of the first candidate character is lower than the accuracy of the second candidate character by a predetermined value or more, the recognition result determination unit 126B may notify the display information generation unit 128 that the first candidate character and the second candidate character are different from each other.

[0218] As described above, in this embodiment, character recognition device 10 has written content recognition unit 124 and recognition result determination unit 126B. Written content recognition unit 124 executes a first character recognition process that recognizes a plurality of characters constituting a character string included in application form image GM, and a second character recognition process that recognizes one of the plurality of characters included in application form image GM. When a first candidate character obtained as a recognition result of one character by the first character recognition process and a second candidate character obtained as a recognition result of one character by the second character recognition process are the same character, recognition result determination unit 126B determines the same character as a confirmed character corresponding to the recognition result of the one character.

[0219] When the first candidate character and the second candidate character are the same, the probability that an incorrect character is determined as a confirmed character is considered to be lower than when the first candidate character and the second candidate character are different from each other. In this embodiment, when the first candidate character and the second candidate character are the same character, the same character is determined as a confirmed character, so that it is possible to prevent an incorrect character from being determined as a confirmed character, and it is possible to improve the accuracy of character recognition in the character recognition process.

[0220] In this embodiment, the display information generating unit 128 is further provided, which displays the confirmed characters determined by the recognition result determining unit 126B on the display device 28. The display information generating unit 128 makes the display mode of the confirmed characters when the first candidate character and the second candidate character are different from each other different from the display mode of the confirmed characters when the first candidate character and the second candidate character are the same. Therefore, for example, when the display mode of one confirmed character is different from the display mode of other confirmed characters, the operator can recognize that one confirmed character may be incorrect. As a result, in this embodiment, the possibility of overlooking that the confirmed character is incorrect can be reduced compared to a case where the display mode of the confirmed character does not change even when the confirmed character may be incorrect.

[0221] Furthermore, in this embodiment, the display information generating unit 128 may change the display mode of the confirmed character when the first candidate character and the second candidate character are different from each other and the accuracy of the first candidate character is lower than the accuracy of the second candidate character by a predetermined value or more from the display mode of the confirmed character when the first candidate character and the second candidate character are the same. Even in this case, it is possible to reduce the possibility of overlooking that the confirmed character is incorrect, compared to a case in which the display mode of the confirmed character does not change even when there is a possibility that the confirmed character is incorrect.

[0222] In this embodiment, similarly to the above-described first embodiment, a weight value may be associated with the character recognition unit CR for each item.

[0223] [4. Modifications] The present invention is not limited to the above-described embodiments. Specific modified embodiments are exemplified below. Two or more embodiments selected from the following examples may be combined.

[0224] [First Modification] In the above-described first embodiment, a weight value is associated with each item in the character recognition unit CR, but the present invention is not limited to this. For example, a weight value may be associated with each character in each character recognition unit CR.

[0225] FIG. 16 is an explanatory diagram for explaining an example of a weighting table WTNL according to the first modified example. In this modified example, it is assumed that the weighting table WTNL is provided separately from the priority table PTBL. For example, the storage device 14 stores the control program PG1, the priority table PTBL, the weighting table WTNL, and a plurality of learning models LM. For example, the weighting table WTNL is referred to by the recognition result determination unit 126 when the recognition result determination unit 126 calculates an evaluation value. Note that, when the written content recognition unit 124 selects a character recognition unit CR to be used for character recognition from a plurality of character recognition units CR, the written content recognition unit 124 refers to the priority table PTBL.

[0226] The weight table WTNL is managed for each character recognition unit CR, for example. The weight table WTNL of each character recognition unit CR is updated, for example, at a predetermined timing. For example, the weight table WTNL of each character recognition unit CR has a plurality of records that correspond one-to-one to one or more candidate characters that have been previously determined as confirmed characters and have been corrected to correct characters. That is, in the example shown in FIG. 16, candidate characters that have not previously been determined as confirmed characters and candidate characters that have not been corrected are not registered in the weight table WTNL. In this modified example, the weight value of characters that are not registered in the weight table WTNL (for example, candidate characters that have not previously been determined as confirmed characters and candidate characters that have not been corrected, etc.) is treated as 1.

[0227] In the weight table WTNL of each character recognition unit CR, each record has a character ID and a weight value. The character ID is information for identifying each character from among a plurality of characters recognized by the character recognition unit CR, and is information indicating each character. For example, the character ID may be a character code assigned to each character. Note that, in FIG. 16, for ease of understanding, the characters indicated by the character IDs are also illustrated.

[0228] The weight value is, for example, a weight value associated with a character indicated by a character ID. The weight value is a real number between 0 and 1. However, as described above, in the example shown in FIG. 16, a weight value of 1 is associated with a character that is not registered in the weight table WTNL, so the weight values ​​associated with characters registered in the weight table WTNL are real numbers between 0 and 1.

[0229] For example, the weight value of a character with a large number of corrections is set to a value smaller than the weight value of a character with a small number of corrections. For example, when the character recognition unit CR3 misrecognizes the individual character image GC of "tree" as the character "big", and the incorrect candidate character ("big") obtained from the character recognition unit CR3 is determined as the confirmed character, the character "big" is corrected to the character "tree", so it is counted in the number of corrections of the character "big". Also, even if the character recognition unit CR3 has misrecognized the individual character image GC of "tree" as an incorrect character, if the individual character image GC other than "tree" has never been misrecognized as the character "tree", the number of corrections of the character "tree" is 0.

[0230] Note that the weight table WTNL is not limited to the example shown in FIG. 16. For example, the weight table WTNL may store, for each character registered in the weight table WTNL, the number of times the character has been determined as a confirmed character since it was first registered in the weight table WTNL. In this case, the weight value may be set in consideration of the ratio of the number of corrections to the number of times determined as a confirmed character (hereinafter also referred to as the correction rate). For example, the weight value of a character with a high correction rate may be set to a value smaller than the weight value of a character with a low correction rate.

[0231] Alternatively, the weight table WTNL of each character recognition unit CR may have a plurality of records that correspond one-to-one with all the characters recognizable by the character recognition unit CR. In this case, the weight table WTNL of each character recognition unit CR may store, for each character, the number of times determined as a confirmed character. Even in this case, the weight value may be set in consideration of the correction rate.

[0232] Next, an example of the confirmed character determination process using the weight table WTNL will be described with reference to FIG. 17.

[0233] FIG. 17 is an explanatory diagram for explaining an example of the confirmed character determination process according to the first modification. In FIG. 17, similar to FIG. 6 described above, an example of a method for determining a confirmed character corresponding to the recognition result of the individual character image GC2 of "big" is described by taking as an example the case where character recognition processing is performed on the character string image GCL5 showing the character string described in the application content description area AK related to "address". However, in FIG. 17, there are some parts where the accuracy etc. are made different from the example shown in FIG. 6 for easier explanation.

[0234] Note that in FIG. 17, it is assumed that the recognition result determination unit 126 refers to the weight table WTNL shown in FIG. 16. Further, in FIG. 17, for easier explanation, it is assumed that the character recognition units CR3 and CR4 are selected as the character recognition unit CR used for recognizing the characters of the character string image GCL5.

[0235] In the example shown in FIG. 17, the recognition result determination unit 126 obtains, as the result of the character recognition processing on the individual character image GC of "big" by the character recognition unit CR3, a plurality of candidate characters including the characters "zhang", "tai", "big", "fire", and "wood", and the accuracy of each of the plurality of candidate characters. Further, the recognition result determination unit 126 obtains, as the result of the character recognition processing on the individual character image GC of "big" by the character recognition unit CR4, a plurality of candidate characters including the characters "dog", "big", "tai", "wood", and "zhang", and the accuracy of each of the plurality of candidate characters.

[0236] The recognition result determination unit 126 calculates an evaluation value by weighting the accuracy of each of the plurality of candidate characters with the accuracy weight value of each of the plurality of candidate characters. For example, the evaluation value of each candidate character is represented by the product of the accuracy of each candidate character and the weight value of each candidate character. Specifically, the recognition result determination unit 126 calculates, for example, the product (90 * 0.8 = 72) of the accuracy (90%) of the candidate character "zhang" obtained from the character recognition unit CR3 and the weight value (0.8) of the candidate character "zhang" as the evaluation value of the candidate character "zhang". For each of the candidate characters other than "zhang" among the plurality of candidate characters obtained from the character recognition unit CR3, and for each of the plurality of candidate characters obtained from the character recognition unit CR4, the evaluation value is calculated in the same manner as the candidate character "zhang".

[0237] Then, the recognition result determination unit 126 determines, among the plurality of candidate characters obtained as the recognition results of the individual character image GC2 of "big" by each of the character recognition units CR3 and CR4, the candidate character with the highest evaluation value as the determined character corresponding to the recognition result of the individual character image GC of "big". When there are a plurality of candidate characters with the highest evaluation value, the recognition result determination unit 126 may determine, among the plurality of candidate characters, the candidate character with the highest priority as the determined character. That is, the recognition result determination unit 126 may determine one of the plurality of candidate characters as the determined character based on the accuracy of each of the plurality of candidate characters, the weight value of each of the plurality of candidate characters, and the priority of each of the plurality of character recognition units CR. Alternatively, when there are a plurality of candidate characters with the highest evaluation value, the recognition result determination unit 126 may determine, among the plurality of candidate characters, the candidate character with the highest accuracy as the determined character.

[0238] Here, in the example shown in FIG. 17, among the plurality of candidate characters, the candidate character with the highest accuracy is the character "dog" with an accuracy of 92% as shown by the double circle in FIG. 17. For example, in a form where no weight value is set for each character (hereinafter also referred to as the fourth proportional example), among the plurality of candidate characters, the character "dog" with the highest accuracy is determined as the determined character corresponding to the recognition result of the individual character image GC of "big". That is, in the fourth proportional example, among the plurality of candidate characters obtained from the character recognition unit CR4, the character "dog" with more correction times than the character "big" is determined as the determined character corresponding to the recognition result of the individual character image GC of "big". Therefore, in the fourth proportional example, compared with this modified example, the probability that an incorrect character is determined as the determined character is increased. In other words, in this modified example, it is possible to suppress the determination of an incorrect character as the determined character compared with the fourth proportional example.

[0239] As described above, in this modified example, the same effects as those of the first embodiment can be obtained. Note that the priority of the character recognition unit CR used to recognize a character may be determined for each character according to the number of times or the correction rate of the character. Also, in the second and third embodiments described above, a weight value may be associated with each character recognized by each character recognition unit CR, as in this modified example.

[0240] [Second modified example] In the above-described embodiment and modified example, the character recognition device 10 and the terminal device 20 are separate from each other, but the present invention is not limited to such an embodiment. For example, the character recognition device 10 and the terminal device 20 may be integrated into one body.

[0241] Fig. 18 is a diagram showing an example of a character recognition system 1A according to a second modified example. Elements similar to those described in Fig. 1 to Fig. 17 are given the same reference numerals, and detailed description thereof will be omitted.

[0242] The character recognition system 1A includes a character recognition device 10A and an image input device 30. For example, the character recognition device 10A is communicably connected to the image input device 30. Note that the connection between the character recognition device 10A and the image input device 30 may be a wired or wireless connection, or may be both a wired and wireless connection.

[0243] Character recognition device 10A has a processing device 12 that controls each part of character recognition device 10A, a storage device 14 that stores various information, a communication device 16, an operation device 27 that accepts operations by an operator of character recognition device 10A, and a display device 28 that displays images and the like. Character recognition device 10A is similar to character recognition device 10 shown in FIG. 3 except that it has operation device 27 and display device 28. However, character recognition device 10A differs from character recognition device 10 in that processing device 12 executes control program PG4 instead of control program PG1. Therefore, storage device 14 stores control program PG4 instead of control program PG1. Character recognition device 10A is another example of an "information processing device", and control program PG4 is another example of a "program".

[0244] In this modification, the control program PG4 includes, for example, an application program for the character recognition device 10A to execute the character recognition process. However, the control program PG4 may also include, for example, an operating system program for the processing device 12 to control each part of the character recognition device 10A. The control program PG4 may be transmitted from another device.

[0245] The processing device 12, which operates according to an application program included in the control program PG4, functions as a control unit 120 including, for example, an image information acquisition unit 122, a description content recognition unit 124, a recognition result determination unit 126, and a display information generation unit 128. The control unit 120 realized by the processing device 12 shown in FIG. 18 is similar to the control unit 120 shown in FIG. 4, except that the control unit 120 executes the process executed via the terminal device 20 without the terminal device 20. For example, the image information acquisition unit 122 acquires image information indicating an image to be subjected to character recognition processing from the image input device 30. Also, for example, the display information generation unit 128 outputs display information to the display device 28 for displaying an image such as a content confirmation screen CCH on the display device 28.

[0246] The configuration of the character recognition system 1A is not limited to the example shown in FIG. 18. For example, the image input device 30 may be included in the character recognition device 10A. Also, for example, the character recognition device 10A may not include the display device 28. When the character recognition device 10A does not include the display device 28, it is connected to the display device 28 so as to be able to communicate with the display device 28. The connection between the character recognition device 10A and the display device 28 may be a wired or wireless connection, or may be a wired and wireless connection. Also, the processing device 12 that operates according to an application program included in the control program PG4 may function as, for example, the control unit 120A or may function as the control unit 120B. When the processing device 12 functions as the control unit 120B, for example, a weight table WTBL is stored in the storage device 14.

[0247] As described above, in this modified example as well, the same effects as those of the above-described embodiment and modified example can be obtained.

[0248] [Third Modification] In the above-described embodiment and modification, the character recognition device 10 or 10A may learn the content of corrections to the result of the character recognition process. As described above, the present modification can also provide the same effects as the above-described embodiment and modification.

[0249] [Fourth Modification] In the above-mentioned embodiment and modified example, the image to be subjected to the character recognition process is an application form image GM, but the present invention is not limited to such an embodiment. For example, the image to be subjected to the character recognition process may be an image of a document such as a prescription, an accounting document (e.g., an invoice, a financial statement, a transfer form, an estimate, etc.), and a recording sheet (e.g., a sheet on which measured values ​​in an experiment, etc. are recorded, etc.). [Explanation of symbols]

[0250] 1, 1A...character recognition system, 10, 10A...character recognition device, 12...processing device, 14...storage device, 16...communication device, 20...terminal device, 27...operation device, 28...display device, 30...image input device, 120, 120A, 120B...control unit, 122...image information acquisition unit, 124...written content recognition unit, 126...recognition result determination unit, 128...display information generation unit, CCH...content confirmation screen, CR...character recognition unit, GM...application form image, LM...learning model, NW...network, MK...warning mark, PTBL...priority table, WTBL...weight table.

Claims

1. a recognition unit that recognizes, in a character image that is an image of a medium, one character of a character string represented for each item of contents to be written on the medium by each of a plurality of character recognition processes that recognize characters included in the image and calculate the accuracy of the recognized characters; a determination unit that determines one of a plurality of candidate characters obtained as a recognition result of the one character by each of the plurality of character recognition processes as a confirmed character corresponding to the recognition result of the one character; Equipped with A priority of a character recognition process to be used for character recognition is determined for each of the above items, The determination unit is determining one of the plurality of candidate characters as the determined character based on the certainty of each of the plurality of candidate characters and the priority of each of the plurality of character recognition processes; 23. An information processing apparatus comprising:

2. a weight value based on the priority level is determined for each of the plurality of character recognition processes; The determination unit is calculating an evaluation value for each of the plurality of candidate characters by weighting the likelihood of each of the plurality of candidate characters with the weight value for each of the plurality of character recognition processes; determining, from among the plurality of candidate characters, the candidate character having the highest evaluation value as the confirmed character; The information processing device according to claim 1 .

3. The determination unit is determining, among the plurality of candidate characters, the first candidate character as the confirmed character when the accuracy of a first candidate character obtained as a recognition result of the one character by the character recognition process having the highest priority among the plurality of character recognition processes is equal to or greater than a predetermined value; The information processing device according to claim 1 .

4. A display control unit that displays the finalized character determined by the determination unit on a display unit, The display control unit is a display mode of the confirmed character when the confirmed character is not recognized in the character recognition process having the highest priority among the plurality of character recognition processes is made different from a display mode of the confirmed character when the confirmed character is recognized in the character recognition process having the highest priority among the plurality of character recognition processes; The information processing device according to claim 1 .

5. a recognition unit that recognizes a character included in a character image by a first character recognition process that recognizes a character included in an image and calculates a probability of the recognized character, and a second character recognition process that recognizes a character included in an image and calculates a probability of the recognized character; a determination unit that, when a common character exists that is included in both one or more first candidate characters obtained as a recognition result of the one character by the first character recognition process and one or more second candidate characters obtained as a recognition result of the one character by the second character recognition process, determines whether or not the common character is to be a confirmed character corresponding to the recognition result of the one character; Equipped with The determination unit is determining the common character as a confirmed candidate character that is a candidate for the confirmed character when the common character satisfies both a first condition that is a condition on a higher probability out of a first probability that is a probability of the common character included in the one or more first candidate characters and a second probability that is a probability of the common character included in the one or more second candidate characters, and a second condition that is a condition on a degree of difference between the first probability and the second probability; 23. An information processing apparatus comprising:

6. The first condition is: The condition is that the higher probability is equal to or greater than a predetermined probability, The second condition is: A condition that a difference between the first accuracy and the second accuracy is equal to or smaller than a predetermined value. The information processing device according to claim 5 .

7. The determination unit is When there are a plurality of confirmed candidate characters, determining a confirmed candidate character having the highest degree of certainty among the plurality of confirmed candidate characters as the confirmed character; The information processing device according to claim 6.

8. The determination unit is When there are a plurality of confirmed candidate characters, determining, as the confirmed character, a confirmed candidate character having the highest average value of the first probability and the second probability among the plurality of confirmed candidate characters; The information processing device according to claim 6.

9. a recognition unit that recognizes a character included in a character image by a first character recognition process that recognizes a character included in an image and calculates a probability of the recognized character, and a second character recognition process that recognizes a character included in an image and calculates a probability of the recognized character; a determination unit that, when a common character exists that is included in both one or more first candidate characters obtained as a recognition result of the one character by the first character recognition process and one or more second candidate characters obtained as a recognition result of the one character by the second character recognition process, determines whether or not the common character is to be a confirmed character corresponding to the recognition result of the one character; Equipped with The determination unit is If there are multiple common characters, determining, among the plurality of common characters, a common character having the highest average value of a first probability which is a probability of the common character included in the one or more first candidate characters and a second probability which is a probability of the common character included in the one or more second candidate characters, as a confirmed candidate character which is a candidate for the confirmed character; 23. An information processing apparatus comprising:

10. The processor, a recognition unit that recognizes, in a character image that is an image of a medium, one character of a character string represented for each item of contents to be written on the medium by each of a plurality of character recognition processes that recognize characters included in the image and calculate the accuracy of the recognized characters; a determination unit that determines one of a plurality of candidate characters obtained as a recognition result of the one character by each of the plurality of character recognition processes as a confirmed character corresponding to the recognition result of the one character; Function as a A priority of a character recognition process to be used for character recognition is determined for each of the above items, The determination unit is determining one of the plurality of candidate characters as the determined character based on the certainty of each of the plurality of candidate characters and the priority of each of the plurality of character recognition processes; program.

11. An information processing method for recognizing one character of a character string represented for each item of contents to be written on a medium in a character image that is an image of a medium by performing a plurality of character recognition processes that recognize characters included in an image and calculate the accuracy of the recognized character, and determining one of a plurality of candidate characters obtained as a recognition result of the one character as a confirmed character corresponding to the recognition result of the one character, comprising: A priority is assigned to each of the plurality of character recognition processes, determining one of the plurality of candidate characters as the determined character based on the certainty of each of the plurality of candidate characters and the priority of each of the plurality of character recognition processes; 23. An information processing method comprising:

Images