Image processing system, image processing method, and image processing program

Abstract

This invention provides an image processing system, an image processing method, and an image processing program capable of improving the accuracy of character recognition in input images, including handwritten characters and printed characters. [Solution] The recognition processing unit 113 performs a first character recognition process corresponding to the first type and a second character recognition process corresponding to the second type on the string of characters when the string of characters contained in the image of the document contains both first type and second type characters. The output processing unit 114 outputs the character recognition result of the string based on the result of the first character recognition process and the result of the second character recognition process.

Description

Technical Field

[0001] The present disclosure relates to a technique for performing processing such as character recognition on an image such as a document.

Background Art

[0002] Conventionally, techniques for recognizing characters (OCR processing) in documents, forms, etc. are known. For example, a technique for performing different OCR processes according to the character type when a document contains handwritten characters and printed characters is known (see, for example, Patent Document 1).

Prior Art Documents

Patent Documents

[0003] [[ID= others]] [[ID= others]]

Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0004] However, in the conventional technology, since the character type is determined based on the character size and the OCR process to be executed is selected, when the character type cannot be appropriately determined by the character size, an appropriate OCR process cannot be executed and the character recognition accuracy decreases.

[0005] An object of the present disclosure is to provide an image processing system, an image processing method, and an image processing program capable of improving the character recognition accuracy of an input image including handwritten characters and printed characters.

Means for Solving the Problems

[0006] An image processing system according to one aspect of the present disclosure comprises a recognition processing unit and an output processing unit. The recognition processing unit performs a first character recognition process corresponding to the first type and a second character recognition process corresponding to the second type on the string of characters contained in an image of a document, when the string of characters contained in the string of characters contains a first type of character and a second type of character. The output processing unit outputs the character recognition result of the string of characters based on the result of the first character recognition process and the result of the second character recognition process.

[0007] An image processing method according to another aspect of the present disclosure involves one or more processors performing, when a string of characters contained in an image of a document contains characters of a first type and characters of a second type, a first character recognition process corresponding to the first type and a second character recognition process corresponding to the second type on the string of characters, and outputting a character recognition result of the string of characters based on the result of the first character recognition process and the result of the second character recognition process.

[0008] An image processing program according to another aspect of this disclosure is a program that causes one or more processors to perform the following actions when a string of characters contained in an image of a document contains characters of a first type and characters of a second type: a first character recognition process corresponding to the first type and a second character recognition process corresponding to the second type on the string of characters; and outputting a character recognition result of the string of characters based on the result of the first character recognition process and the result of the second character recognition process. [Effects of the Invention]

[0009] According to this disclosure, it is possible to provide an image processing system, an image processing method, and an image processing program that can improve the accuracy of character recognition in input images, including handwritten characters and printed characters. [Brief explanation of the drawing]

[0010] [Figure 1] Figure 1 is a functional block diagram showing the configuration of an image processing apparatus according to the present disclosure. [Figure 2A]Figure 2A shows an example of a document (receipt) according to an embodiment of this disclosure. [Figure 2B] Figure 2B shows an example of a document (receipt) according to an embodiment of this disclosure. [Figure 2C] Figure 2C shows an example of a document (receipt) according to an embodiment of this disclosure. [Figure 3A] Figure 3A shows an example of an issue date included in a document according to an embodiment of this disclosure. [Figure 3B] Figure 3B shows an example of an issue date included in a document according to an embodiment of this disclosure. [Figure 3C] Figure 3C shows an example of an issue date included in a document according to an embodiment of this disclosure. [Figure 4] Figure 4 is a flowchart showing an example of the procedure for character recognition processing performed in the image processing apparatus according to the embodiment of this disclosure. [Figure 5] Figure 5 shows the results of OCR processing for typefaces on the issue date included in a document according to the embodiment of this disclosure. [Figure 6] Figure 6 shows the results of OCR processing for handwritten characters on the issue date included in a document according to the embodiment of this disclosure. [Figure 7] Figure 7 shows an example of a method for selecting the results of OCR processing for the issue date included in a document according to the embodiment of this disclosure. [Figure 8] Figure 8 shows the results of OCR processing for typefaces on the issue date included in a document according to the embodiment of this disclosure. [Figure 9] Figure 9 shows the results of OCR processing for handwritten characters on the issue date included in a document according to the embodiment of this disclosure. [Figure 10] Figure 10 shows an example of a method for calculating the similarity score in the results of OCR processing for printed characters according to the embodiment of this disclosure. [Figure 11] Figure 11 shows an example of a method for calculating the similarity score in the results of OCR processing for handwritten characters according to the embodiment of this disclosure. [Figure 12]FIG. 12 is a diagram showing an example of a method for calculating a character count score in the result of OCR processing for printed characters according to an embodiment of the present disclosure. [Figure 13] FIG. 13 is a diagram showing an example of a method for calculating a character count score in the result of OCR processing for handwritten characters according to an embodiment of the present disclosure. [Figure 14] FIG. 14 is a diagram showing an example of a method for calculating a specific character score in the result of OCR processing for printed characters according to an embodiment of the present disclosure. [Figure 15] FIG. 15 is a diagram showing an example of a method for calculating a specific character score in the result of OCR processing for handwritten characters according to an embodiment of the present disclosure. [Figure 16] FIG. 16 is a diagram showing an example of a method for calculating a total score of OCR processing for printed characters and OCR processing for handwritten characters according to an embodiment of the present disclosure.

Mode for Carrying Out the Invention

[0011] Hereinafter, embodiments of the present disclosure will be described with reference to the accompanying drawings. The following embodiments are an example of embodying the present disclosure and do not have the character of limiting the technical scope of the present disclosure.

[0012] [Image Processing Apparatus 1] FIG. 1 is a block diagram showing the configuration of an image processing apparatus 1 according to an embodiment of the present disclosure. The image processing apparatus 1 recognizes a character string included in an input image (image data) and executes a well-known character recognition process (OCR process) on the recognized character string to output a character recognition result.

[0013] As shown in FIG. 1, the image processing apparatus 1 includes a control unit 11, a storage unit 12, an operation display unit 13, a communication unit 14, and the like. The image processing apparatus 1 is an information processing apparatus such as a personal computer or a cloud server. The image processing apparatus 1 may be one or more cloud servers or one or more physical servers.

[0014] The communication unit 14 is a communication interface for connecting the image processing device 1 to a network by wire or wireless connection and for performing data communication with external devices via the network in accordance with a predetermined communication protocol. The network consists of, for example, the internet, a LAN, etc.

[0015] The operation display unit 13 is a user interface comprising a display unit such as a liquid crystal display or an organic EL display that displays various types of information, and an operation unit such as a mouse, keyboard, or touch panel that accepts input.

[0016] The storage unit 12 is a non-volatile storage unit such as an HDD (Hard Disk Drive), SSD (Solid State Drive), or flash memory that stores various types of information. The storage unit 12 stores a control program, such as a character recognition program (an example of an image processing program in this disclosure), which causes the control unit 11 to execute the character recognition processing described later (see Figure 4). For example, the character recognition program is non-temporarily recorded on a computer-readable recording medium such as a CD or DVD, read by a reader device (not shown) such as a CD drive or DVD drive provided by the image processing device 1, and stored in the storage unit 12. The character recognition program may also be distributed from a cloud server and stored in the storage unit 12.

[0017] Furthermore, the storage unit 12 stores image data (such as scanned data) of documents and other information acquired from external devices.

[0018] Figure 2 shows a receipt as an example of a document (form). As shown in Figure 2, a receipt includes multiple items such as the date of issue, recipient, issuer's contact information, and amount. The receipt in Figure 2 also includes both printed and handwritten text. For example, the date of issue "October 21, 2023" (see Figure 3A) on the receipt in Figure 2A is entirely handwritten. The date of issue "October 21, 2023" (see Figure 3B) on the receipt in Figure 2B is composed of handwritten "2023," "10," and "21," and printed "year," "month," and "day." The date of issue "October 21, 2023" (see Figure 3C) on the receipt in Figure 2C is entirely printed.

[0019] For example, the user scans the receipt using a scanner, multifunction printer, etc., and uploads the image data (input image) to the image processing device 1. Alternatively, the user takes a picture of the receipt with a camera on an operating terminal (e.g., a smartphone) and uploads the image data to the image processing device 1. When the control unit 11 acquires the image data of the receipt, it stores it in the storage unit 12. In another embodiment, the control unit 11 may acquire a document file of the receipt created on an external device and store the document file in the storage unit 12.

[0020] The control unit 11 includes control devices such as a CPU, ROM, and RAM. The CPU is a processor that performs various arithmetic operations. The ROM stores control programs such as a BIOS and OS in advance to allow the CPU to perform various operations. The RAM stores various information and is used as a temporary storage memory (work area) for the various operations performed by the CPU. The control unit 11 controls the image processing device 1 by executing various control programs stored in advance in the ROM or storage unit 12 using the CPU.

[0021] When the control unit 11 acquires image data of a document, it performs preprocessing on the image data, such as resolution conversion, skew correction, and background removal, and then performs recognition processing (extraction processing) of text characters (document parts). For example, in the image data after the preprocessing has been performed, the control unit 11 recognizes each part, such as text characters, tables, illustrations, and stamps, as rectangular areas of objects. Furthermore, after performing the document part recognition processing, the control unit 11 performs character recognition processing (OCR processing) on ​​the entire image data (overall image).

[0022] In this embodiment, the control unit 11 performs OCR processing on strings containing numbers such as issue date, expiration date, and amount included in the input image.

[0023] Specifically, as shown in Figure 1, the control unit 11 includes various processing units such as an acquisition processing unit 111, a detection processing unit 112, a recognition processing unit 113, and an output processing unit 114. The control unit 11 functions as these various processing units by executing various processes according to the character recognition program. Some or all of the processing units included in the control unit 11 may be composed of electronic circuits. The character recognition program may be a program that causes multiple processors to function as these various processing units.

[0024] The acquisition processing unit 111 acquires an input image containing the characters to be detected. Specifically, the acquisition processing unit 111 acquires an image (character image data) that is subject to character recognition. For example, the acquisition processing unit 111 acquires character image data from a document image, such as a receipt shown in Figures 2A to 2C, which contains handwritten characters, printed characters, etc.

[0025] The detection processing unit 112 detects character rectangles, which are the character regions of each character in a string, from the input image. For example, as shown in Figures 3A to 3C, it detects character rectangles (single character rectangles) for each character in the issue date.

[0026] The recognition processing unit 113 performs character recognition processing (OCR processing). Specifically, the recognition processing unit 113 performs OCR processing on the characters of the character rectangle detected by the detection processing unit 112. The recognition processing unit 113 extracts characters one by one through OCR processing. The recognition processing unit 113 also performs word formation processing to group the individual characters into word units (strings). Once the recognition processing unit 113 has extracted each string, it registers it in a string list (not shown). In the string list, information such as a number (string number), a management ID for each character, the string, and position coordinates (character rectangle coordinates) is associated with each extracted string and registered. The position coordinates are, for example, represented by the starting point coordinates (top left coordinates) and ending point coordinates (bottom right coordinates) of the string rectangle.

[0027] Furthermore, the recognition processing unit 113 performs OCR processing according to the type of extracted characters (printed characters, handwritten characters). Specifically, the recognition processing unit 113 performs OCR processing for printed characters and OCR processing for handwritten characters. For example, if the string of characters contained in the image of a document contains both printed and handwritten characters, the recognition processing unit 113 performs OCR processing for printed characters and OCR processing for handwritten characters on the string of characters. The OCR processing for handwritten characters uses an OCR engine capable of recognizing handwritten characters, and the OCR processing for printed characters uses an OCR engine capable of recognizing printed characters.

[0028] The output processing unit 114 outputs the results of the OCR processing performed by the recognition processing unit 113. Specifically, the output processing unit 114 outputs the character recognition results of the string based on the results of the OCR processing for printed characters and the OCR processing for handwritten characters. In other words, the recognition processing unit 113 executes multiple OCR processes (OCR processing for handwritten characters and OCR processing for printed characters) in parallel, selects the OCR result suitable for the item to be extracted from the multiple OCR results obtained, and the output processing unit 114 outputs the selected OCR result. The following describes specific examples of OCR processing and the method of selecting OCR results.

[0029] [Character recognition processing] Figure 4 is a flowchart showing an example of the character recognition process performed in the image processing apparatus 1 according to this embodiment.

[0030] This disclosure can be understood as a character recognition processing method (image processing method of this disclosure) that performs one or more steps included in the character recognition processing. Furthermore, one or more steps included in the character recognition processing described herein may be omitted as appropriate. In addition, the execution order of each step in the character recognition processing may differ to the extent that similar effects are produced. Furthermore, although this description uses the case in which the control unit 11 of the image processing device 1 performs each step in the character recognition processing as an example, in other embodiments, one or more processors may distribute and execute each step in the character recognition processing.

[0031] <Step S1> In step S1, the control unit 11 acquires an input image and extracts character rectangles, which are the character regions of each character in the string, from the acquired input image. For example, when the control unit 11 acquires image data of a document image (receipt) shown in Figure 2, it extracts character rectangles for each character from the receipt image. In the following, it will be assumed that the control unit 11 has extracted character rectangles corresponding to each character of the issue date "October 21, 2023".

[0032] <Step S2> In step S2, the control unit 11 performs OCR processing for printed characters on each extracted character. The control unit 11 performs OCR processing for printed characters on each character regardless of the character type.

[0033] <Step S3> In step S3, the control unit 11 determines whether or not there is a handwritten character area. Specifically, the control unit 11 determines whether or not the character area extracted in step S1 includes a handwritten character area. If there is a handwritten character area (S3: Yes), the control unit 11 proceeds to step S4. On the other hand, if there is no handwritten character area (S3: No), the control unit 11 proceeds to step S5. For example, the character areas shown in Figures 3A and 3B include a handwritten character area, while the character area shown in Figure 3C does not.

[0034] <Step S4> In step S4, the control unit 11 performs OCR processing for handwritten characters on each extracted character. For example, the control unit 11 performs OCR processing for handwritten characters on each character in the character area shown in Figures 3A and 3B.

[0035] <Step S5> In step S5, the control unit 11 acquires strings. For example, for the character areas shown in Figures 3A and 3B, the control unit 11 acquires the strings of each character recognized by the OCR processing for printed characters (step S2) and the strings of each character recognized by the OCR processing for handwritten characters (step S4). Also, for example, for the character area shown in Figure 3C, the control unit 11 acquires the strings of each character recognized by the OCR processing for printed characters (step S2).

[0036] <Step S6> In step S6, the control unit 11 determines whether the acquired string contains the results of OCR processing for handwritten characters. In the character areas shown in Figures 3A and 3B, the results of OCR processing for handwritten characters are included, while in the character area shown in Figure 3C, the results of OCR processing for handwritten characters are not included. If the acquired string contains the results of OCR processing for handwritten characters (S6: Yes), the control unit 11 proceeds to step S7. On the other hand, if the acquired string does not contain the results of OCR processing for handwritten characters (S6: No), the control unit 11 proceeds to step S74.

[0037] <Step S7> In step S7, the control unit 11 determines whether the number of characters in the string recognized by the OCR process for printed characters is the same as the number of characters in the string recognized by the OCR process for handwritten characters. When the number of characters in each is the same (S7: Yes), the control unit 11 transfers the process to step S8. When the number of characters in each is different (S7: No), the control unit 11 transfers the process to step S71.

[0038] <Step S8> In step S8, the control unit 11 compares the characters recognized by the OCR process for printed characters with the characters recognized by the OCR process for handwritten characters and selects the optimal characters. Hereinafter, a specific example of the character selection method will be described.

[0039] FIG. 5 shows the result of the OCR process for printed characters with the issue date "October 21, 2023". The control unit 11 extracts candidate characters for each of the 11 character rectangles and calculates the likelihood. Further, the control unit 11 selects the character of the first candidate with the highest likelihood among the plurality of candidate characters. Then, the control unit 11 obtains a character string ("ユ0ユ年|0月ユ|日") composed of the characters selected at each character position for each character rectangle of the printed characters.

[0040] FIG. 6 shows the result of the OCR process for handwritten characters with the issue date "October 21, 2023". The control unit 11 extracts candidate characters for each of the 11 character rectangles and calculates the likelihood. Further, the control unit 11 selects the character of the first candidate with the highest likelihood among the plurality of candidate characters. Then, the control unit 11 obtains a character string ("2023年10円218") composed of the characters selected at each character position for each character rectangle of the handwritten characters.

[0041] FIG. 7 shows the result of comparing the candidate characters of the first candidate of the acquired characters with the candidate characters of the first candidate of the handwritten characters. Here, the control unit 11 sets "〇" for characters corresponding to the numbers 0 to 9 in the character string, and sets "×" for characters other than the numbers. Also, the control unit 11 sets "〇" for characters corresponding to delimiter characters ("year", "month", "day", " / ", "-", ".", etc.) in the character string, and sets "×" for characters other than the delimiter characters.

[0042] Based on the determination results of the numbers 0 to 9 and the determination results of the delimiter characters at the same character positions, the control unit 11 selects the optimal character.

[0043] Specifically, the control unit 11 selects the character "2" of the result of the OCR process for handwritten characters for character ID "0", selects the character "0" of the result of the OCR process for handwritten characters for character ID "1", selects the character "2" of the result of the OCR process for handwritten characters for character ID "2", selects the character "3" of the result of the OCR process for handwritten characters for character ID "3", selects the character "year" of the result of the OCR process for handwritten characters for character ID "4", selects the character "1" of the result of the OCR process for handwritten characters for character ID "5", selects the character "0" of the result of the OCR process for handwritten characters for character ID "6", selects the character "month" of the result of the OCR process for typeset characters for character ID "7", selects the character "2" of the result of the OCR process for handwritten characters for character ID "8", selects the character "1" of the result of the OCR process for handwritten characters for character ID "9", and selects the character "day" of the result of the OCR process for typeset characters for character ID "10".

[0044] In this way, for each character included in the character string, the control unit 11 compares the candidate characters extracted by the OCR process for typeset characters with the candidate characters extracted by the OCR process for handwritten characters, and selects the character according to the type. Here, the control unit 11 selects the characters "2", "0", "2", "3", "year", "1", "0", "month", "2", "1", "day" based on the result of the OCR process for handwritten characters and the result of the OCR process for typeset characters.

[0045] <Step S9> In step S9, the control unit 11 integrates each selected character. In the above example, the control unit 11 integrates the characters '2', '0', '2', '3', 'year', '1', '0','month', '2', '1', 'day' to obtain the character string 'October 21, 2023'. After step S9, the control unit 11 transfers the process to step S10.

[0046] <Step S71> In step S71, the control unit 11 calculates a score for a predetermined evaluation item. Specifically, the control unit 11 calculates an evaluation value (evaluation value based on similarity; referred to as "similarity score A") based on the result (similarity) of the OCR process, an evaluation value (evaluation value based on the number of characters; referred to as "character count score B") based on the number of specific characters in the character string, and an evaluation value (evaluation value based on specific characters; referred to as "specific character score C") based on the presence or absence of specific characters or symbols. Hereinafter, specific examples of the calculation methods for each evaluation value will be described.

[0047] The control unit 11 calculates the similarity score A by multiplying a value obtained by calculating a statistical quantity (sum, average value, variance, etc.) of the evaluation values in the character string unit by a coefficient for the evaluation value (similarity 0 to 1) when each character is specified in the OCR process. [[ID=))

[0048] [[ID=))

[0049] Figure 8 shows the result of the OCR process for typeset characters with 13 characters in the character string, and Figure 9 shows the result of the OCR for handwritten characters with 11 characters in the character string. First, the control unit 11 selects the first candidate character for each character rectangle of the typeset characters (see Figure 8) based on the result of the OCR process for typeset characters to obtain a character string ('Z. 'Z3 year 1. month 2] day). Similarly, the control unit 11 selects the first candidate character for each character rectangle of the handwritten characters based on the result of the OCR for handwritten characters (see Figure 9) to obtain a character string (October 21, 2023).Next, the control unit 11 calculates a similarity evaluation value (similarity score A) for the acquired string. For example, as shown in Figure 10, the control unit 11 calculates the sum of the certainties, mean, and variance of each character in the acquired string for printed characters. Then, the control unit 11 calculates the difference between the mean and the variance as the similarity score A ("83.14"). Similarly, as shown in Figure 11, the control unit 11 calculates the sum of the certainties, mean, and variance of each character in the acquired string for handwritten characters. Then, the control unit 11 calculates the difference between the mean and the variance as the similarity score A ("99.59").

[0050] Next, the control unit 11 calculates an evaluation value (character count score B) based on the number of characters in the acquired string. For example, as shown in Figure 12, the control unit 11 calculates the total number of characters in the acquired string and the number of characters corresponding to the digits 0 through 9 for printed characters. Then, the control unit 11 calculates the ratio of the number of digits to the total number of characters as the character count score B ("23.08%"). Similarly, as shown in Figure 13, the control unit 11 calculates the total number of characters in the acquired string and the number of characters corresponding to the digits 0 through 9 for handwritten characters. Then, the control unit 11 calculates the ratio of the number of digits to the total number of characters as the character count score B ("72.73%").

[0051] Next, the control unit 11 calculates an evaluation value (specific character score C) for specific characters in the acquired string. The specific characters are delimiters such as "year", "month", "day", " / ", "-", and ".". For example, as shown in Figure 14, the control unit 11 calculates the total number of characters in the acquired string and the number of characters corresponding to the specific characters for printed characters. Then, the control unit 11 multiplies the number of characters corresponding to the specific characters by 10 to calculate the specific character score C ("30"). Similarly, as shown in Figure 15, the control unit 11 calculates the total number of characters in the acquired string and the number of characters corresponding to the specific characters for handwritten characters. Then, the control unit 11 multiplies the number of characters corresponding to the specific characters by 10 to calculate the specific character score C ("30").

[0052] Furthermore, if the item to be recognized by character recognition is an amount, the specified characters will include characters such as "total," "money," "¥," and "-." If the item to be recognized by character recognition is a recipient, the specified characters will include characters such as "Mr. / Ms.," and "To whom it may concern." If the item to be recognized by character recognition is a company name, the specified characters will include characters such as "company" and "(Co., Ltd.)." If the item to be recognized by character recognition is a telephone number, the specified characters will include characters such as "Tel" and "Fax."

[0053] <Step S72> In step S72, the control unit 11 determines whether the OCR processing result for handwritten characters is good or not based on the calculated score. Specifically, the control unit 11 calculates a total score (see Figure 16) of similarity score A, character count score B, and specific character score C based on the OCR processing result for handwritten characters, and calculates a total score of similarity score A, character count score B, and specific character score C based on the OCR processing result for printed characters, and compares each total score to determine which OCR processing result is better. If the control unit 11 determines that the OCR processing result for handwritten characters is good (S72: Yes), it moves the process to step S73, and if it determines that the OCR processing result for printed characters is good (S72: No), it moves the process to step S74. In the above example, the control unit 11 determines that the OCR processing result for handwritten characters, which has a larger total score, is good.

[0054] <Step S73> In step S73, the control unit 11 adopts the result of the OCR processing for handwritten characters. After step S73, the control unit 11 moves the processing to step S10.

[0055] <Step S74> In step S74, the control unit 11 adopts the result of the OCR processing for printed characters. In step S6, even if the acquired string does not include the result of the OCR processing for handwritten characters (S6: No), the control unit 11 adopts the result of the OCR processing for printed characters. After step S74, the control unit 11 moves the process to step S10. In this way, the control unit 11 selects the result of the OCR processing for printed characters and the OCR processing for handwritten characters that has a higher total score (see Figure 16) of similarity score A, character count score B, and specific character score C.

[0056] <Step S10> In step S10, the control unit 11 outputs the result of the OCR processing. Specifically, the control unit 11 outputs a string that integrates the results of the OCR processing for handwritten characters and the OCR processing for printed characters (step S9), a string that is the result of the OCR processing for handwritten characters (step S73), or a string that is the result of the OCR processing for printed characters (step S74). For example, if the total number of characters recognized by the OCR processing for printed characters is different from the total number of characters recognized by the OCR processing for handwritten characters (S7: No), the control unit 11 uses multiple evaluation indicators to select either the result of the OCR processing for printed characters or the result of the OCR processing for handwritten characters. Also, if the total number of characters recognized by the OCR processing for printed characters is the same from the total number of characters recognized by the OCR processing for handwritten characters (S7: Yes), the control unit 11 integrates the results of the OCR processing for printed characters and the OCR processing for handwritten characters and outputs the integrated result as the character recognition result for the string.

[0057] When the control unit 11 performs the character recognition process described above, it outputs "October 21, 2023" as the character recognition result for the issue date, which is the target of extraction.

[0058] As described above, when the image processing device 1 contains a string of characters of a first type (e.g., printed characters) and a second type of characters (handwritten characters) in the image of a document, it performs a first character recognition process (OCR processing for printed characters) corresponding to the first type and a second character recognition process (OCR processing for handwritten characters) corresponding to the second type on the string of characters, and outputs a character recognition result for the string of characters based on the result of the first character recognition process and the result of the second character recognition process.

[0059] Specifically, the image processing device 1 calculates evaluation values ​​for multiple evaluation indicators for both the result of the first character recognition process and the result of the second character recognition process, selects either the result of the first character recognition process or the result of the second character recognition process based on the calculated evaluation values, and outputs the selected result as the character recognition result of the string. For example, the image processing device 1 selects either the result of the first character recognition process or the result of the second character recognition process based on a first evaluation value (similarity score A) regarding the similarity of each character in the string, a second evaluation value (character count score B) regarding the number of specific characters in the string, and a third evaluation value (specific character score C) regarding the presence or absence of specific characters in the string.

[0060] According to the above configuration, for example, for a string containing both handwritten and printed characters, OCR processing for handwritten characters and OCR processing for printed characters can be performed in parallel, and the results appropriate to the items to be extracted can be selected from the obtained results and output. Therefore, it is possible to improve the accuracy of character recognition in input images containing both handwritten and printed characters.

[0061] The image processing device 1 may also use one or two of the following indicators to select the result of the first character recognition process or the result of the second character recognition process: similarity score A, character count score B, and specific character score C.

[0062] The image processing device 1 relating to this disclosure is an example of an image processing system relating to this disclosure. That is, the image processing system relating to this disclosure may consist of the image processing device 1 alone. Alternatively, the image processing system relating to this disclosure may consist of the image processing device 1 and a character recognition device that performs OCR processing. Alternatively, the image processing system relating to this disclosure may consist of the image processing device 1, the character recognition device, and an output device that outputs (displays) the character recognition results. For example, the output device may be a user terminal (personal computer, smartphone, etc.), and the user terminal may run a character recognition application (web application) to request character recognition processing of a document image from the image processing device 1, and obtain and display the character recognition results from the image processing device 1.

[0063] The control unit 11 of the image processing device 1 controls the entire image processing device 1. The control unit 11 realizes various functions by reading and executing various programs stored in the storage unit 12 (for example, storage or ROM). The control unit 11 may be realized by one or more control devices / arithmetic units (CPU (Central Processing Unit), SoC (System on a Chip)). The control unit 11 may also be composed of one or more control circuits (electronic circuits).

[0064] [Disclosure Note] The following is an overview of the disclosures extracted from the above-described embodiments. Note that each configuration and processing function described in the following notes can be selected and combined as desired.

[0065] <Note 1> A recognition processing unit that, when a string of characters contained in an image of a document contains both a first type of character and a second type of character, performs a first character recognition process corresponding to the first type and a second character recognition process corresponding to the second type on the string of characters, An output processing unit that outputs the character recognition result of the string based on the result of the first character recognition process and the result of the second character recognition process, An image processing system equipped with the following features.

[0066] <Note 2> The recognition processing unit calculates evaluation values ​​for each of the multiple evaluation indicators for the result of the first character recognition process and the result of the second character recognition process, and selects the result of the first character recognition process or the result of the second character recognition process based on the calculated evaluation values. The output processing unit outputs the selected result as the character recognition result of the string. The image processing system described in Appendix 1.

[0067] <Note 3> The recognition processing unit selects the result of the first character recognition process or the result of the second character recognition process based on a first evaluation value relating to the similarity of each character in the string, a second evaluation value relating to the number of specific characters in the string, and a third evaluation value relating to the presence or absence of specific characters in the string. The image processing system described in Appendix 2.

[0068] <Note 4> The recognition processing unit selects the result from the result of the first character recognition process and the result of the second character recognition process that has a higher sum of the first evaluation value, the second evaluation value, and the third evaluation value. The image processing system described in Appendix 3.

[0069] <Note 5> The recognition processing unit selects the result of the first character recognition process or the result of the second character recognition process using the multiple evaluation indicators if the total number of characters recognized in the first character recognition process is different from the total number of characters recognized in the second character recognition process. An image processing system as described in any of the appendices 2 to 4.

[0070] <Note 6> The recognition processing unit, when the total number of characters recognized in the first character recognition process is the same as the total number of characters recognized in the second character recognition process, integrates the results of the first character recognition process and the results of the second character recognition process. The output processing unit outputs the integrated result as the character recognition result of the string. An image processing system as described in any of the appendices 1 to 5.

[0071] <Note 7> The recognition processing unit compares the candidate character extracted in the first character recognition process with the candidate character extracted in the second character recognition process for each character included in the string, and selects a character according to the type. The image processing system described in Appendix 6.

[0072] <Note 8> One of the first character recognition process and the second character recognition process is an OCR process capable of recognizing handwritten characters, and the other is an OCR process capable of recognizing printed characters. An image processing system as described in any of the appendices 1 to 7.

[0073] <Note 9> The recognition processing unit performs character recognition processing on the string of a date or amount. An image processing system as described in any of the appendices 1 to 8.

[0074] <Note 10> When a string of characters contained in an image of a document contains both a first type of character and a second type of character, a first character recognition process corresponding to the first type and a second character recognition process corresponding to the second type are performed on the string of characters. Based on the results of the first character recognition process and the results of the second character recognition process, the character recognition result of the string is output. An image processing method performed by one or more processors.

[0075] <Note 11> When a string of characters contained in an image of a document contains both a first type of character and a second type of character, a first character recognition process corresponding to the first type and a second character recognition process corresponding to the second type are performed on the string of characters. Based on the results of the first character recognition process and the results of the second character recognition process, the character recognition result of the string is output. An image processing program for causing one or more processors to execute, or a non-temporary computer-readable recording medium on which the image processing program is recorded. [Explanation of Symbols]

[0076] 1: Image processing device 11: Control Unit 12: Storage section 13: Operation display section 14: Communications Department 111: Acquisition Processing Unit 112: Detection Processing Unit 113: Recognition Processing Unit 114: Output Processing Unit

Claims

1. A recognition processing unit that, when a string of characters contained in an image of a document contains both a first type of character and a second type of character, performs a first character recognition process corresponding to the first type and a second character recognition process corresponding to the second type on the string of characters, An output processing unit that outputs the character recognition result of the string based on the result of the first character recognition process and the result of the second character recognition process, An image processing system equipped with the following features.

2. The recognition processing unit calculates evaluation values ​​for each of a plurality of evaluation indicators for the result of the first character recognition process and the result of the second character recognition process, and selects the result of the first character recognition process or the result of the second character recognition process based on the calculated evaluation values. The output processing unit outputs the selected result as the character recognition result of the string. The image processing system according to claim 1.

3. The recognition processing unit selects the result of the first character recognition process or the result of the second character recognition process based on a first evaluation value relating to the similarity of each character in the string, a second evaluation value relating to the number of specific characters included in the string, and a third evaluation value relating to the presence or absence of specific characters in the string. The image processing system according to claim 2.

4. The recognition processing unit selects the result from the result of the first character recognition process and the result of the second character recognition process that has a higher sum of the first evaluation value, the second evaluation value, and the third evaluation value. The image processing system according to claim 3.

5. The recognition processing unit selects the result of the first character recognition process or the result of the second character recognition process using the plurality of evaluation indicators if the total number of characters recognized in the first character recognition process is different from the total number of characters recognized in the second character recognition process. The image processing system according to claim 2.

6. The recognition processing unit, when the total number of characters recognized in the first character recognition process is the same as the total number of characters recognized in the second character recognition process, integrates the results of the first character recognition process and the results of the second character recognition process. The output processing unit outputs the integrated result as the character recognition result of the string. The image processing system according to claim 1.

7. The recognition processing unit compares the candidate character extracted in the first character recognition process with the candidate character extracted in the second character recognition process for each character included in the string, and selects a character according to the type. The image processing system according to claim 6.

8. One of the first character recognition process and the second character recognition process is an OCR process capable of recognizing handwritten characters, and the other is an OCR process capable of recognizing printed characters. The image processing system according to any one of claims 1 to 7.

9. The recognition processing unit performs character recognition processing on the string of a date or amount. The image processing system according to any one of claims 1 to 7.

10. When a string of characters contained in an image of a document contains both a first type of character and a second type of character, a first character recognition process corresponding to the first type and a second character recognition process corresponding to the second type are performed on the string of characters. Based on the results of the first character recognition process and the results of the second character recognition process, the character recognition result of the string is output. An image processing method performed by one or more processors.

11. When a string of characters contained in an image of a document contains both a first type of character and a second type of character, a first character recognition process corresponding to the first type and a second character recognition process corresponding to the second type are performed on the string of characters. Based on the results of the first character recognition process and the results of the second character recognition process, the character recognition result of the string is output. An image processing program that is executed by one or more processors.

Images