Image processing apparatus, image processing method, and storage medium

Abstract

The present application relates to an image processing apparatus, an image processing method, and a storage medium. In a character region in a scanned image of a document, character recognition processing suitable for a handwritten character region and a printed character region is performed. Next, a character recognition result for the handwritten character region and a character recognition result for the printed character region are integrated, a likelihood indicating a probability of becoming an extraction target is calculated for a candidate character string that is a candidate for extraction among the integrated character recognition results, and a character string that is a project value is determined. Then, at the time of determination, in the case where a character originating from the handwritten character region is included in the characters constituting the candidate character string and in the case where the character originating from the handwritten character region is not included, different evaluation indexes are used.

Description

TECHNICAL FIELD

[0001] The technology of the present disclosure relates to a technology of extracting character information from a document image. BACKGROUND

[0002] In recent years, reading characters by performing a character recognition process on an image obtained by scanning a document generally called a "business form" such as a voucher and a receipt, and extracting an item value (for example, "$1000") corresponding to a specific item (for example, "amount") has been performed. This technology is generally called a key value extraction process. Japanese Patent Application Publication No. 2015-118488 has disclosed a technology of specifying a necessary item value by using information related to the size and thickness of a recognized character string even in a case where a specific item cannot be recognized by a character recognition process.

[0003] For example, in the case of a receipt, it is not uncommon that a company name or an amount section is filled in with a handwritten character. In a case of extracting an item value corresponding to a key item "amount" from a scanned image of a receipt including a handwritten character, OCR (Optical Character Recognition) for a handwritten character is generally used. The OCR for a handwritten character is an OCR engine specialized for extracting a handwritten character such as a number and a currency symbol (hereinafter, described as "numbers and the like"). Here, due to the characteristics of the OCR for a handwritten character, there is a case where a handwritten character section other than a number and the like of a handwritten character within a scanned image is erroneously recognized as a character string of a number and the like. For example, there is a case where a letter character written as a company name is erroneously recognized as a number and the like. Applying the method of the above-described Japanese Patent Application Publication No. 2015-118488 to such a case will cause a character string of a handwritten character that is not a number and the like of a character string erroneously recognized as a number and the like to be extracted as an item value corresponding to a specific item "amount". SUMMARY

[0004] An image processing apparatus according to the present disclosure is an image processing apparatus for extracting a string to be a project value corresponding to a specific project from among strings described in a document, the image processing apparatus including: an obtaining unit configured to obtain a handwritten character region representing a handwritten character and a printed character region representing a printed character among character regions in a scanned image of the document; a first OCR unit configured to perform a character recognition process for the handwritten character region; a second OCR unit configured to perform a character recognition process for the printed character region; a consolidation unit configured to consolidate a character recognition result of the handwritten character region and a character recognition result of the printed character region; and a determination unit configured to determine a string as a project value based on a result of calculating a likelihood indicating a probability of becoming an extraction target for a candidate string as an extraction candidate among the consolidated character recognition results, wherein in a case where a character originating from the handwritten character region is included in a character constituting the candidate string and in a case where the character originating from the handwritten character region is not included, the determination unit calculates the likelihood by using different evaluation indexes.

[0005] Further features of the present disclosure will become apparent from the following description of example embodiments with reference to the drawings. BRIEF DESCRIPTION OF DRAWINGS

[0006] Figure 1 FIG. 1A is a diagram showing an example of the configuration of an image processing system, and Figure 1 FIG. 1B is a functional block diagram showing a software configuration of a PC;

[0007] Figure 2 FIG. 2 is a flowchart showing a series of processes of extracting a project value corresponding to a key project from a document image;

[0008] Figure 3A FIG. 3 is a diagram showing an example of a scanned image of a receipt, and Figure 3B and Figure 3C is a diagram showing a result of a region separation process;

[0009] Figure 4 is a diagram showing a consolidated character recognition result;

[0010] Figure 5 is a flowchart showing details of a key value extraction process;

[0011] Figure 6 is a diagram showing value candidates detected from a consolidated character recognition result;

[0012] Figures 7A to 7C is a diagram each showing an example of a penalty;

[0013] Figure 8A is a graph showing a result of likelihood calculation obtained by the method of the present embodiment, and Figure 8B is a graph showing a result of likelihood calculation obtained by a conventional method; and

[0014] Figure 9 is a graph showing an example of a scanned image of a receipt. DETAILED DESCRIPTION

[0015] Hereinafter, the present disclosure will be explained in detail according to a preferred embodiment with reference to the accompanying drawings. The configuration shown in the following embodiment is merely exemplary, and the present disclosure is not limited to the configuration shown in the schematic view.

[0016] [First Embodiment]

[0017] <Configuration of System>

[0018] Figure 1 A of FIG. 1 is a graph showing an example of a configuration of an image processing system according to the present embodiment. The image processing system includes a copier 100 having a function of scanning a document and a PC 110 as an image processing apparatus having a function of performing key value extraction processing on an image (hereinafter, described as "document image") obtained by scanning the document.

[0019] The copier 100 has a scanner 101, a printer 102, and a communication unit 103. The scanner 101 generates a document image by scanning a document such as a business form, which is placed on a document table not shown schematically. The printer 102 forms an image on a print medium such as paper based on the document image. The communication unit 103 communicates with external devices including the PC 110 via a network. The document image generated by the scanner 101 is transmitted to the PC 110 via the communication unit 103.

[0020] The PC 110 has a CPU 111, a ROM 112, a RAM 113, an HDD 114, a display unit 115, an input unit 116, and a communication unit 117. The CPU 111 performs various processing by reading a control program stored in the ROM 112. The RAM 113 is used as a temporary storage area of the CPU 111, such as a main memory and a work area. The HDD 114 stores various data, various programs, and the like. A program to be described later Figure 1The respective functional units of the CPU 110 shown in A are realized by the CPU 111 reading a predetermined program stored in the ROM 112 or the HDD 114, loading the program to the RAM 113, and executing the program. The communication unit 117 receives the document image from the copier 100 by communicating with external devices including the copier 110 via a network. The display unit 115 is, for example, a liquid crystal display or the like, and displays various information to the user. The input unit 116 is a keyboard, a mouse, or the like, and receives various operations of the user. The display unit 115 and the input unit 116 can be provided integrally, such as a touch panel. Further, the display unit 115 can be a projector, in which case the input unit 116 can have a configuration in which the position of a fingertip to a projected image is recognized by a camera.

[0021] <Outline of key value extraction processing>

[0022] Figure 2 is a flowchart showing a series of processes for extracting, in the present embodiment, an item value corresponding to a predetermined specific item (hereinafter referred to as "key item") from a document image. In the present embodiment, with reference to the software functional block diagram of the PC 110 shown in B, an example is explained in which, from a document image of a receipt on which a company name and an amount are handwritten, a string indicating a description value corresponding to the key item "amount" is extracted as its item value. In the following explanation, the symbol "S" indicates a step. Further, the "string" which is an object to be extracted from the document image includes numbers and symbols. Figure 1

[0023] In S201, the region separation unit 121 performs processing (region separation processing) to separate a handwritten character region and a printed character region, and obtains two regions of the document image of the receipt input from the copier 100. For the recognition between the handwritten character region and the printed character region, a publicly known method is sufficient. For example, there is a method of recognizing the two regions by finding a separation coefficient of a feature amount of each character, or a method of recognizing the two regions by estimation based on a result of machine learning (trained model) on handwritten characters and printed characters. Figure 3A is an example of a document image of a receipt in which respective string portions indicating a company name, an issuance date, and an amount received are handwritten, and other portions are printed by printed characters. Then, Figure 3B and Figure 3C are results of the region separation processing for the document image in Figure 3A Figure 3B shows the printed character region, and Figure 3C shows the handwritten character region. Further processing can be performed to recognize and separate other regions, such as character information, an illustration, and a background other than a company logo.​​

[0024] In S202, the printed character OCR unit 122 performs a character recognition process for printed characters on the printed character regions obtained by the region separation process in S201. In Figure 3B In the case of the printed regions 301 to 307 in Table 1, the character recognition results shown in the following Table 1 are obtained by the character recognition process for printed characters in this step.

[0025] [Table 1]

[0026]

[0027]

[0028] The coordinate information and the character codes of the respective recognized characters are added to the character recognition results shown in Table 1, and using the coordinate information, the size and position of a rectangle indicating the recognized printed character region can be specified. Here, for the sake of explanation, it is assumed that all correct character recognition results have been obtained in the character recognition process for printed characters.

[0029] In S203, the handwritten character OCR unit 123 performs a character recognition process for handwritten characters on the handwritten character regions obtained by the region separation process in S201. Generally, handwritten characters differ in character and size in many cases, and differ greatly from person to person due to the individual habits of the writers. For this reason, in the character recognition process for handwritten characters, it is difficult to recognize characters compared to the character recognition process for printed characters, and the recognition accuracy of the character recognition process for handwritten characters is often low. Therefore, in order to improve the recognition accuracy, the handwritten character OCR unit 123 restricts the character types output as the character recognition results according to the purpose and use. Specifically, in the case where characters related to an amount of money are recognized, the character types are restricted to Arabic numerals and specific letters and symbols. In this case, the specific letters and symbols include, for example, "month names (e.g., January, December, etc.), "$ (currency symbol), ", (comma), ". (decimal point, dot), -(hyphen), etc. Thus, by performing the character recognition process on the handwritten characters in the handwritten character regions 311 to 315 in Table 2, the character recognition results shown in the following Table 2 are obtained. Figure 3C

[0030] [Table 2]

[0031] Handwritten character region Character recognition result 311 December 312 5,. 313 2020 314 11,286 315 $8？？

[0032] ​As can be seen from Table 2 above, correct character recognition results were obtained for handwritten character areas 311 to 314. However, for handwritten character area 315, which contains the company name, the characters within this area are not the target characters for recognition. Therefore, instead of the correct character string "ABC Company," a character recognition result such as "$8??" was obtained. In this case, the "?" indicates a character recognition failure. Originally, it is expected that all characters that are not the target characters for recognition would be "?", but in the example in Table 2, the letter "A" was incorrectly recognized as the dollar sign "$" and the letter "B" was incorrectly recognized as the number "8."

[0033] In step S204, the OCR result integration unit 124 performs processing to integrate the character recognition results of the printed character area obtained in step S202 and the character recognition results of the handwritten character area obtained in step S203. If the document related to the processing target document image is horizontally written, processing is performed to connect one or more character areas adjacent in the horizontal direction. Figure 4 The character recognition results of each character area are shown. These character recognition results are obtained by integrating Figure 3B The character recognition results of the printed character areas 301 to 307 shown in Table 1 and Figure 3C 2020) 。 The character recognition results of the handwritten character areas 311 to 315 shown in FIG4 are obtained (see Table 2). Here, the integrated character area 401 is obtained by connecting the printed character area 301, the handwritten character area 315 on its right side, and the printed character area 302 on its further right side, and the integrated character recognition result is "To $8?? Receipt". Similarly, the integrated character area 402 is obtained by connecting one printed character area 303 and three handwritten character areas 311, 312 and 313, and the integrated character recognition result is "Issue Date: December 5, 2020 (December 5, 2020)". Further, the integrated character area 404 is obtained by connecting two printed character areas 305 and 306 and the handwritten character area 314, and the integrated character recognition result is "Amount 11,286 US dollars". The integrated character area 403 is the printed character area 304 itself, and the integrated character area 405 is also the printed character area 307 itself. Then, to each integrated character recognition result, attribute information is added, the attribute information indicating whether the characters constituting each integrated character recognition result are character recognition results originating from the handwritten character area or character recognition results originating from the printed character area.

[0034] In S205, the key value extraction unit 125 performs processing to extract a string as an item value corresponding to a key item from the integrated character recognition result obtained in S204. In the present embodiment, the string "Amount" in the printed character region 305 is taken as a key item, and a string of the description value corresponding thereto is extracted as an item value thereof. Details of the key value extraction processing will be described later.

[0035] The above is a general flow of the processing of extracting an item value corresponding to a key item from a document image. In the flowchart described above Figure 2 , the character recognition processing for the printed character region (S202) and the character recognition processing for the handwritten character region (S203) are explained as serial processing, but the processing order is not limited, and the processing can be performed in the reverse order, or can be performed in parallel.

[0036] <Detail of Key Value Extraction Unit>

[0037] Next, with reference to the flowchart of Figure 5 , the processing in the key value extraction unit 125 will be described in detail, taking the case of the document image of the receipt shown in Figure 3A .

[0038] First, in S501, a string of a pre-designated key item is detected from the integrated character recognition result. Here, the string of the key item can be detected by, for example, fuzzy search using dictionary data. For example, in the case where the string "Amount" within the receipt is taken as a key item, the detection range can also be left with a margin so that the detection range includes "Amount Received", "Amount", and the like, and these strings are searched from the integrated character recognition result. Here, as the character recognition result of the character region 404 in Figure 4 , the part of "Amount" of "Amount 11,286 USD" is detected as a string of a key item. In the case of a receipt including only printed characters, details are often described in detail and numbers and the like indicating various amounts are printed. In this case, the case can occur where even if the string "Amount" itself is used as a key item, it is not possible to specify a string of an amount received desired to be extracted as an item value. Therefore, in the case of a receipt including only printed characters, it is desired that the above fuzzy search is not performed, and a specific string such as "Amount Received" and "Total Amount" is detected as a string of a key item. Alternatively, in the case where a plurality of strings of a key item are detected, these strings can also be weighted, and at the stage of determining an item value, a string with a large weight is given priority.

[0039] Next, at S502, a string (candidate string) that is a candidate for the item value is detected from the integrated character recognition result. In the following explanation, the candidate string is referred to as a "value candidate". The value candidate can be detected by pattern matching such as a regular expression. At this time, considering the possibility of erroneous recognition, in particular, in the character recognition processing (S203) for handwritten characters, it is desirable to set a fairly loose pattern restriction, and to detect as many value candidates as possible. In the case where the string "amount" is set as the key item, it is sufficient to set the pattern restriction when detecting the value candidate, for example, to "$? [0-9.,] {1, 10} dollars?" (for explanation, escape characters are omitted). In the case of this pattern restriction, the string is the search object, which starts with no character or one character "$ (dollar sign)", followed by zero to ten Arabic numerals, "." (period), and "," (comma), and ends with no character or one word "dollars". Thus, a string including numerals is detected as a value candidate, which is likely to indicate an amount, such as "$1,000", "50 dollars", and "10.0". Figure 6 The value candidates detected from the integrated character recognition result in Figure 4 are shown. In Figure 6 , the value candidate 601 is a string obtained by extracting the part of "$8" from the character recognition result "for $8?? receipt" of the integrated character region 401. Further, the value candidates 602, 603, and 604 are strings obtained by extracting the parts of "December", "5,", and "2020" from the character recognition result "issued date: December 5, 2020" of the integrated character region 402. Similarly, the value candidate 605 is a string obtained by extracting the part of "11,286 dollars" from the character recognition result "amount 11,286 dollars" of the integrated character region 404. Then, the value candidate 606 is a string obtained by extracting the part of "1001" from the character recognition result "No. 1001" of the integrated character region 405. For each value candidate detected from the integrated character recognition result in this way, in the subsequent step, the probability that each value candidate is the item value of the "amount" as the key item is evaluated, and the value candidate whose probability is evaluated to be the highest is finally determined as the item value.

[0040] At S503, based on the attribute information described above for the value candidate of interest among all the value candidates detected at S502, it is determined whether or not the handwritten character is included, and the processing branches according to the determination result. In the case where the handwritten character is included in the value candidate of interest, the processing proceeds to S505, and in the case where the handwritten character is not included (i.e., the value candidate of interest is composed only of printed characters), the processing proceeds to S504.

[0041] At S504, for the value candidate of interest consisting only of printed characters, a likelihood degree is calculated by using the evaluation index of the printed characters, which indicates the degree of probability of becoming the item value. In the present embodiment, by taking 100 points as the full score, the likelihood degree is found by a scoring system that penalizes the evaluation index, and the value candidate whose likelihood degree is the greatest is determined as the item value. Here, the evaluation index and the penalty depend on the type of the document to be processed and the characteristics of the item value expected to be extracted. Here, the processing object is a receipt, and the key item is "amount", so the more appropriate the string as the representation format of the amount, the lower the penalty given. In the case of printed characters, the character size in each character region is constant, but among the printed characters printed in advance within the receipt, the printed character portion that represents the received amount has a high likelihood of having a large character size. Therefore, the height of the character region (the height of the rectangle) of the value candidate is taken as the evaluation index, the larger the character size, the lower the penalty given, and the smaller the height of the character region (the height of the rectangle), the higher the penalty given. Figure 7A is a table that puts together the penalties according to the height of the rectangle, and gives different penalties according to predetermined height-of-rectangle segments. Here, the height of the rectangle is a relative value with respect to the largest height of the rectangle within the document image (corresponding to the height of a line due to horizontal writing), and the height of the rectangle is expressed by an integer, with the largest height of the rectangle taking "100". Figure 7B is a table indicating the penalty in the case where the presence or absence of an amount symbol as an evaluation index in the value candidate. In the case where an amount symbol such as "$" and "dollar" is present, the penalty is "0 (none)", and in the case where the amount symbol is not present, the penalty is "20". Figure 7A and Figure 7B The evaluation indexes and the penalties shown in and are merely examples, and they are not limited to these.

[0042] At S505, for the value candidate of interest including handwritten characters, the above-described likelihood degree is calculated by using the evaluation index of the handwritten characters. Here, the character size as the evaluation index of the printed characters is not used as the evaluation index of the handwritten characters. Instead, the number of digits of the number constituting the string of the value candidate is used as the evaluation index. In the case of handwritten characters, the character size is not constant in many cases. Furthermore, as described above, in the character recognition processing by the handwritten character OCR unit 123, the object character type is limited, and to some extent, there is a possibility that a character that is not originally a number is erroneously recognized. For this reason, the number of digits is used as the evaluation index instead of the character size. Then, the greater the number of digits, the lower the penalty given, and the reason is that it is considered that the probability of continuously erroneously recognizing a character that is not a number as a number is low, and the smaller the number of digits, the higher the probability of erroneous recognition.

[0043] At S506, it is determined whether the above likelihood calculation process is completed for all value candidates detected at S502, and the process is branched according to the determination result. In the case where the calculation of the likelihood of all value candidates is completed, the process proceeds to S507, whereas in the case where there is an unprocessed value candidate, the process returns to S503, and the next value candidate of interest is determined and the process is continued. Figure 8A is a table in which the results of the likelihood calculation of the six value candidates shown in Figure 6 are put together. Figure 8A is a table showing the results of the likelihood calculation obtained by the method of the embodiment of the present application. In the table of Figure 8A , in the "Candidate ID" column, the ID identifying the value candidate is input. In the "Identified String" column, information about the string relating to the value candidate is input. Then, in the "Attribute" column, information indicating whether the string of the value candidate includes handwritten characters or all characters are printed characters is input. In the "Number of Digits" column, the value obtained by counting the number of characters in the case where there are digits within the string of the value candidate is input, and the penalty according to this count value is indicated in parentheses. In the "Rectangular Height" column, the height value of the character region in the case where the string of the value candidate has the attribute of printed characters is input, and the penalty resulting therefrom is indicated in parentheses. In the "Amount Symbol" column, information indicating whether there is an amount symbol in the string of the value candidate is input, and the penalty resulting therefrom is indicated in parentheses. Then, in the "Likelihood" column on the leftmost side, the value obtained by subtracting the total penalty from 100 is input. The calculation process of the likelihood of the value candidates 601 to 606 by the method of the present embodiment is explained in detail as follows.

[0044] • Value candidate 601: The string includes handwritten characters and the number of digits is only "8", and the number of digits is "1", so the penalty resulting from the number of digits is "50", and "$" is included, so the penalty resulting from the amount symbol is "0". Therefore, the likelihood is "50".

[0045] • Value candidate 602: The string includes handwritten characters and the number of digits is "0", so the penalty resulting from the number of digits is "80" and "$" and the like are not included, so the penalty resulting from the amount symbol is "20". Therefore, the likelihood is "0".

[0046] • Value candidate 603: The string includes handwritten characters and the number of digits is "1", so the penalty resulting from the number of digits is "50" and "$" and the like are not included, so the penalty resulting from the amount symbol is "20". Therefore, the likelihood is "30".

[0047] • Value candidate 604: The string includes a handwritten character and the number of digits is "4", so the penalty due to the number of digits is "10" and does not include "$" or the like, so the penalty due to the amount symbol is "20". Therefore, the likelihood is "70".

[0048] • Value candidate 605: The string includes a handwritten character and the number of digits is "5", so the penalty due to the number of digits is "0" and includes "Dollars", so the penalty due to the amount symbol is "0". Therefore, the likelihood is "100".

[0049] • Value candidate 606: The string does not include a handwritten character (composed only of printed characters) and the rectangular height is "50", so the penalty due to the rectangular height is "20" and does not include "$" or the like, so the penalty due to the amount symbol is "20". Therefore, the likelihood is "60".

[0050] From Figure 8A the "Likelihood" column in the table in Figure 8B , it can be seen that the likelihood of "11,286 Dollars" of value candidate 604 is the largest. In , for comparison, a table in which the likelihood calculation results obtained by the conventional method are put together is shown. In the case of the conventional method, as a common evaluation index of handwritten characters and printed characters, the rectangular height and the presence / absence of the amount symbol are applied, and it can be seen that the likelihood of value candidate "$8" is the largest. The calculation processing of the likelihood of each value candidate by the conventional method is described in detail below.

[0051] • Candidate ID 1: The rectangular height is "100", so the penalty due to the rectangular height is "0" and includes "$", so the penalty due to the amount symbol is "0". Therefore, the likelihood is "100".

[0052] • Candidate ID 2: The rectangular height is "70", so the penalty due to the rectangular height is "10" and does not include "$" or the like, so the penalty due to the amount symbol is "20". Therefore, the likelihood is "70".

[0053] • Candidate ID 3: The rectangular height is "70", so the penalty due to the rectangular height is "10" and does not include "$" or the like, so the penalty due to the amount symbol is "20". Therefore, the likelihood is "70".

[0054] • Candidate ID 4: The rectangular height is "70", so the penalty due to the rectangular height is "10" and does not include "$" or the like, so the penalty due to the amount symbol is "20". Therefore, the likelihood is "70".

[0055] • Candidate ID_5: The rectangle height is "80", so the penalty caused by the rectangle height is "10" and includes "$", so the penalty caused by the amount symbol is "0". Therefore, the likelihood is "90".

[0056] • Candidate ID_6: The rectangle height is "50", so the penalty caused by the rectangle height is "20" and does not include "$" or the like, so the penalty caused by the amount symbol is "20". Therefore, the likelihood is "60".

[0057] The explanation returns to the flow in Figure 5 .

[0058] In S507, among all the value candidates detected in S502, the value candidate whose likelihood calculated in S504 or S505 is the greatest is determined as the string of the item value corresponding to the predetermined key item. In the case of the example of Figure 6 , as described above, the value candidate "11,286 dollars" whose likelihood is the greatest is determined as the item value. Then, the value candidate "11,286 dollars" is a string indicating the received amount described in the amount field of the receipt shown in Figure 3A , which means that the correct item value has been extracted here.

[0059] The above is the content of the key value extraction processing according to the present embodiment. The likelihood calculation method explained in S504 and S505 is an example, and the likelihood can be calculated by a dot product system, for example. Further, in the present embodiment, one value candidate whose likelihood is the greatest is determined as the item value, but all the value candidates whose likelihood is greater than or equal to a predetermined value can be determined as the item value. Further, the kinds of evaluation indexes can be increased. For example, in the case where the received amount in the receipt is taken as the key item, it can be said that the attribute of handwriting itself indicates a high probability as the item value. Therefore, it is also possible to give a penalty uniformly to the detected value candidates having the attribute of printed characters. Therefore, in the case where both the handwritten amount and the printed character amount are obtained as the value candidates, the handwritten amount can be determined as the item value preferentially.

[0060] [Modified Example]

[0061] In the above embodiment, the item value is determined based on only the likelihood calculated for the value candidates, but the item value can be determined by further considering information related to the key item. Hereinafter, an aspect as a modified example is explained, in which the item value is determined by considering the positional relationship between each value candidate and the character region of the string of the key item in addition to the likelihood of the value candidates.

[0062] Figure 9 is an example of a document image of a receipt applicable to the present modified example. First, by performing the processing of S101 to S103 on Figure 9The document image shown is subjected to the processing at S501 described earlier, and "Amount" in the character region 901 is detected as a string of the key item. Then, in the loop processing of S502 to S506 described earlier, "110,000 dollars" in the character region 902 and "10,000 dollars" in the character region 903 are detected as value candidates, and the number of digits of both is greater than or equal to 5, and includes the amount symbol, so the same likelihood is obtained for both. In the case where there is no difference in likelihood among the plurality of value candidates as described above, the item value is determined by referring to the positional relationship between each value candidate and the character region of the string of the key item. Specifically, in the processing of S507, the coordinate information relating to the character region 901 of "Amount" that is the string of the key item detected at S501 is obtained, and "110,000 dollars" in the character region 902 that is the same in the position coordinate in the y direction is determined as the item value. In this case, the position coordinate in the y direction need not be exactly matched, but only needs to be approximately matched. According to this modification example, even in the case where it is difficult to narrow down the value candidates by the likelihood alone, the string of the item value can be appropriately determined from among the plurality of value candidates based on the positional relationship between each value candidate and the string of the key item.

[0063] As described above, according to the present embodiment, even from a document image of a receipt such as a case where there is handwritten numerals and the like, a string such as a handwritten numeral and the like can be extracted with high precision as an item value corresponding to a particular key item.

[0064] (Other Embodiments)

[0065] Embodiments of the present disclosure can also be realized by a computer of a system or apparatus that reads out and executes computer executable instructions (e.g., one or more programs) recorded on storage media (which can also be referred to more fully as a 'non-transitory computer-readable storage medium') to perform the functions of one or more of the above-described embodiments and / or that includes one or more circuits (e.g., application specific integrated circuits (ASICs)) to perform the functions of one or more of the above-described embodiments and by a method realized by such a system or apparatus, the procedure of the method being performed by, for example, the computer of the system or apparatus reading out and executing the computer executable instructions to perform the functions of one or more of the above-described embodiments and / or controlling the one or more circuits to perform the functions of one or more of the above-described embodiments. The computer can comprise one or more processors (e.g., central processing units (CPUs), micro processing units (MPUs)) and can include a computer network or a combination of a plurality of processing circuits. The computer executable instructions can be provided to the computer, for example, from a network or the storage media. The storage media can include, for example, one or more of a hard disk, a random access memory (RAM), a read only memory (ROM), a storage of distributed computing systems, an optical disk (such as a compact disc (CD), digital versatile disc (DVD), or Blu-ray Disc (BD)®), a flash memory device, a memory card, and the like. TM

[0066] Other embodiments

[0067] Embodiments of the present application can also be realized by a method that provides software (a program) to a system or apparatus that performs the functions of the above-described embodiments by a network or various storage media, and a computer of the system or apparatus or a central processing unit (CPU) or a micro processing unit (MPU) that reads out and executes the program.

[0068] According to the technology of the present disclosure, even from a document image such as a receipt in which there is a case where part of the characters are handwritten, a desired handwritten character can be appropriately extracted as a project value corresponding to a specific project.

[0069] While the present disclosure has been described with reference to exemplary embodiments, it is to be understood that the present disclosure is not limited to the disclosed exemplary embodiments. The scope of the following claims is to be accorded the broadest interpretation so as to encompass all the modifications and equivalent structures and functions.​

Claims

1. An image processing apparatus for extracting a string among strings described in a document to be a project value corresponding to a specific project, the image processing apparatus comprising: an obtaining unit configured to obtain a handwritten character region representing a handwritten character and a printed character region representing a printed character among character regions in a scanned image of the document; a first OCR unit configured to perform a character recognition process for the handwritten character on the handwritten character region; a second OCR unit configured to perform a character recognition process for the printed character on the printed character region; an integrating unit configured to integrate a character recognition result for the handwritten character region and a character recognition result for the printed character region; and a determination unit configured to determine a string as the project value based on a result of calculating a likelihood indicating a probability of becoming an extraction target by calculating the likelihood for a candidate string as an extraction candidate among the integrated character recognition results.

2. The image processing apparatus according to claim 1, wherein in a case where a character included in the candidate string is a number, the determination unit uses: a number of digits of the number as an evaluation index under a condition where the character originating from the handwritten character region is included; and a character size of the number as an evaluation index under a condition where the character originating from the handwritten character region is not included.

3. The image processing apparatus according to claim 2, wherein in a case where the specific project is a project related to an amount of money, the determination unit performs the calculation of the likelihood such that the larger the number of digits of the number or the larger the character size, the larger the obtained likelihood.

4. The image processing apparatus according to claim 1, wherein the specific project is a project related to an amount of money, the character recognition process for the handwritten character is a character recognition process that limits a recognition target character type to at least a predetermined symbol including a number and a currency symbol, and the determination unit further uses presence / absence of a character indicating the predetermined symbol as an evaluation index.

5. The image processing apparatus according to claim 4, wherein the determination unit performs the calculation of the likelihood such that a larger likelihood is obtained in a case where the character indicating the predetermined symbol is present than in a case where the character indicating the predetermined symbol is not present.

6. The image processing apparatus according to claim 1, wherein the determination unit further uses whether the character constituting the candidate string originates from the handwritten character region as an evaluation index.

7. The image processing apparatus according to claim 6, wherein the determination unit performs the calculation of the likelihood such that a larger likelihood is obtained in a case where the character constituting the candidate string originates from the handwritten character region than in a case where the character does not originate from the handwritten character region.

8. The image processing apparatus according to claim 1, wherein ​ ​ In a case where the likelihoods calculated with respect to the plurality of candidate strings do not differ, the determination unit determines the string to be the item value based on a positional relationship between each candidate string and the string of the specific item.

9. The image processing apparatus according to claim 1, wherein The document is a receipt.

10. An image processing method for extracting, among strings described in a document, a string to be an item value corresponding to a specific item, the image processing method comprising the steps of: obtaining a handwritten character region representing a handwritten character and a printed character region representing a printed character among character regions in a scanned image of the document; performing a character recognition process for a handwritten character on the handwritten character region; performing a character recognition process for a printed character on the printed character region; integrating a character recognition result for the handwritten character region and a character recognition result for the printed character region; and determining the string to be the item value based on a result of calculating a likelihood indicating a probability of becoming an extraction target with respect to a candidate string that is a candidate for extraction among the integrated character recognition results, In the determination step, the likelihood is calculated by using different evaluation indexes in a case where a character originating from the handwritten character region is included in the characters constituting the candidate string and in a case where the character originating from the handwritten character region is not included.

11. A computer-readable storage medium storing a program for causing a computer to execute an image processing method for extracting, among strings described in a document, a string to be an item value corresponding to a specific item, the image processing method comprising the steps of: obtaining a handwritten character region representing a handwritten character and a printed character region representing a printed character among character regions in a scanned image of the document; performing a character recognition process for a handwritten character on the handwritten character region; performing a character recognition process for a printed character on the printed character region; integrating a character recognition result for the handwritten character region and a character recognition result for the printed character region; and determining the string to be the item value based on a result of calculating a likelihood indicating a probability of becoming an extraction target with respect to a candidate string that is a candidate for extraction among the integrated character recognition results, In the determination step, the likelihood is calculated by using different evaluation indexes in a case where a character originating from the handwritten character region is included in the characters constituting the candidate string and in a case where the character originating from the handwritten character region is not included.

Images