Information processing device, information processing method, and program

Abstract

To improve character recognition accuracy for a position tend to cause error in acquisition of a character string region in a document image.SOLUTION: An information processing device performs a character recognition process to a character string region acquired from a document image obtained by reading a document, specifies name characters constituting the name of a person in the document on the basis of a result of the character recognition process, re-acquires the character string region corresponding to the character string indicating the name, and re-performs the character recognition process. Then, the information processing device determines the recognition result of the part corresponding to the name of the person on the basis of the result of the initial character recognition process and the result of the re-processing of character recognition.SELECTED DRAWING: Figure 3

Description

[Technical field]

[0001] The present invention relates to a technique for recognizing characters contained in a document image. [Background technology]

[0002] Conventionally, there is a technology for reading and recognizing characters in a document image obtained by scanning a document. This technology is generally called OCR (Optical Character Recognition). OCR processing is usually composed of image preprocessing for modifying a document image into an image suitable for OCR, character string detection processing for identifying a region in which characters are written (character string region) from the document image, and character recognition processing for identifying each character included in the detected character string region. Regarding such OCR processing, a technology is known in which rules and classification items for classifying acquired character string regions are determined in advance, and optimal character recognition is performed for each character string region. For example, Patent Document 1 discloses a technology for classifying character string regions obtained from a document image based on the positional relationship between predetermined character string regions (layout information) and characteristics of classification items, and for improving recognition accuracy by matching the character string regions with a word dictionary corresponding to each classification item. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Japanese Patent Application Publication No. 5-20505 Summary of the Invention [Problem to be solved by the invention]

[0004] The technology of the above-mentioned Patent Document 1 requires that rules for classification, such as layout information of character string regions and characteristics of classification items, be determined in advance, but for documents that do not fit the rules, character string regions cannot be properly classified, and as a result, characters cannot be recognized correctly. For example, in the name portion of a business card, which may be vertically long or horizontally long or may be written vertically or horizontally, the number of characters is small and the spacing between characters is wide, making it difficult to obtain an appropriate character string region and perform character recognition with high accuracy.

[0005] The technology disclosed herein has been made in consideration of the above-mentioned problems, and aims to improve the character recognition accuracy for parts of a document image where acquisition of a character string area is likely to fail.

[0006] The information processing device according to the present disclosure is characterized by comprising: a first processing means for performing a first character recognition process on a character string area obtained by performing a character string detection process on a document image obtained by reading a document; an identification means for identifying name characters constituting a person's name in the document based on a result of the first character recognition process; an acquisition means for integrating an area corresponding to the name characters identified by the identification means and an area corresponding to one or more characters belonging to the same line as the name characters to acquire a character string area corresponding to a character line representing the name; a second processing means for performing a second character recognition process on the character string area corresponding to the character line representing the name; and a determination means for determining character codes for characters constituting the character line representing the name based on a result of the first character recognition process and a result of the second character recognition process. Effect of the Invention

[0007] According to the technology disclosed herein, it is possible to improve character recognition accuracy even in cases where it is difficult to properly acquire a character string area, such as the characters of a name on a business card. [Brief description of the drawings]

[0008] [Figure 1] FIG. 1 is a diagram showing a hardware configuration of an information processing system. [Diagram 2] 4 is a flowchart showing the flow of processing in the entire system. [Diagram 3] 11 is a flowchart showing details of a recognition result improvement process. [Figure 4] 11 is a flowchart showing details of a name line acquisition process. [Diagram 5] 11 is a flowchart showing details of a name character identification process. [Figure 6] 11 is a flowchart showing details of a character code replacement process. [Figure 7] 11 is a flowchart showing details of a name row reevaluation process. [Figure 8] 11 is a flowchart showing details of a blob integration process. [Figure 9] 11 is a flowchart showing details of a blob attribute determination process. [Figure 10] 10 is a flowchart showing details of a recognition result comparison process. [Figure 11] 1A is a diagram showing an example of a business card image, and FIG. 1B is a diagram showing an example of a character string region before a recognition result improvement process is performed. [Figure 12] FIG. 11 is a diagram showing an example of a recognition result before a recognition result improvement process is performed. [Figure 13] 11 is a diagram showing an example of a character string area corresponding to a name line acquired from a business card image. [Figure 14] 14 is a diagram showing an example of a recognition result for the character string region in FIG. 13. [Figure 15] FIG. 11 is a diagram showing an example of a result of a name character identification process. [Figure 16] 1A is a diagram showing a state in which the inter-character distance is positive when written horizontally, and FIG. 1B is a diagram showing a state in which the inter-character distance is positive when written vertically. [Figure 17] 11A and 11B are diagrams for explaining overlap determination of rectangular areas. [Figure 18] 11A and 11B are diagrams for explaining overlap determination between original blobs. [Figure 19] FIG. 11 is a diagram showing a character string region after a recognition result improvement process is performed. [Figure 20] FIG. 11 is a diagram showing an example of a recognition result after a recognition result improvement process is performed. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0009] Hereinafter, an embodiment of the present invention will be described with reference to the drawings. Note that the embodiment does not limit the present invention, and all of the configurations described in the embodiment are not necessarily essential means for solving the problems of the present invention.

[0010] [Embodiment 1] [Hardware configuration] 1 is a diagram showing the hardware configuration of an information processing system according to this embodiment. The information processing system includes a copying machine 100 and an information processing device 110.

[0011] The copier 100 includes a scanner 101 and a copier communication unit 102. The scanner 101 scans a document and generates a document image. The copier communication unit 102 communicates with external devices including an information processing device 110 via a network.

[0012] The information processing device 110 includes a system control unit 111, a ROM 112, a RAM 113, a HDD 114, a display unit 115, an input unit 116, and an information processing device side communication unit 117. The system control unit 111 is composed of a calculation device such as a CPU, and reads out a control program stored in the ROM 112 to execute various processes. The RAM 113 is used as a temporary storage area such as a main memory and a work area of ​​the system control unit 111. The HDD 114 stores various data and various programs. The functions and processes of the information processing device 110 described later are realized by the system control unit 111 reading out a program stored in the ROM 112 or the HDD 114 and executing the program. The information processing device side communication unit 117 performs communication processing with external devices including the copying machine 100 via a network. The display unit 115 displays various information. The input unit 116 includes a keyboard and a mouse, and accepts various operations by a user. The display unit 115 and the input unit 116 may be provided integrally, such as a touch panel. The display unit 115 may project an image using a projector, and the input unit 116 may recognize the position of a fingertip relative to the projected image using a camera.

[0013] In this embodiment, a scanner 101 of a copying machine 100 scans a paper document such as a business card to generate a document image. The document image is transmitted to an information processing device 110 by a copying machine side communication unit 102. In the information processing device 110, an information processing device side communication unit 117 receives the document image and stores it in a storage device such as an HDD 114.

[0014] 1 is an example of a configuration for implementing this embodiment, and for example, some functions of the display unit 115 and the input unit 116 may be included in the copier 100. Also, the copier 100 and the information processing device 110 may be integrated into one configuration.

[0015] [Overall processing flow] Next, the operation flow of software for implementing OCR processing on document images in the information processing system according to this embodiment will be described with reference to Fig. 2. A series of processes shown in the flowchart in Fig. 2 is implemented by the system control unit 111 reading a predetermined program from the ROM 112 or the like and executing it. In the following description, the symbol "S" means a step.

[0016] First, in S201, data of a document image is acquired from the HDD 114. Next, in S202, a process of determining the type of the document of the acquired document image is executed. In this embodiment, it is determined whether the document type is a business card or a non-business card. This determination is made based on the scan resolution and the document image size. For example, when the scan resolution is 300 DPI, if the long side of the document image obtained by scanning is within the range of 1040 pixels ± 5% and the short side is within the range of 615 pixels ± 5%, it is determined to be a business card, and if it is outside the range, it is determined to be a non-business card. Note that the method of determining the document type is not limited to the method based on the image size. For example, information specifying the document type may be received from the input unit 116, or the feature amount may be calculated from the document image and the determination may be made using a discrimination model that has been learned in advance.

[0017] Based on the result of the determination in S202, the process flow branches in S203. If it is determined that the object is not a business card, the process proceeds to S204, and if it is determined that the object is a business card, the process proceeds to S205.

[0018] In S204, a general-purpose character recognition process that is independent of the document type is executed. When the character recognition process is completed, this process ends.

[0019] In S205, a process for correcting the skew of the document image is executed. The document image generated by the scanner 101 is skewed depending on how the document is placed on the platen. Therefore, a skew correction process is performed to obtain a document image without skew. The skew correction process may be performed by applying a known method disclosed in Japanese Patent No. 4114959 or the like.

[0020] Next, in S206, a process is executed to determine the character direction in the document image. Here, the character direction is defined as the direction of the characters in the document image when the direction in which the characters are upright is set to 0 degrees. In the document image that has been subjected to the tilt correction process in S205, the original may be rotated in 90 degree increments. Therefore, the known method disclosed in Japanese Patent No. 3727971 and the like is applied to obtain the character direction of the document image.

[0021] Next, in S207, a process of rotating the document image is executed based on the character direction determined in S206, thereby obtaining a document image in which the characters are upright.

[0022] Next, in S208, a process of extracting character pixels from the document image is executed. In this embodiment, character pixels are extracted by binarization, which involves performing threshold processing on the luminance value of each pixel. For binarization, a known method such as Otsu's binarization may be applied. Since characters in a document are usually printed in a darker color than the background, pixels with a luminance value smaller than the threshold value are regarded as character pixels. Note that the method of extracting character pixels is not limited to the method using threshold processing. For example, it may be realized by a method of extracting image features from a neighboring area centered on an arbitrary pixel of interest, and inferring whether the pixel of interest is a character pixel or not based on a discrimination model previously learned based on the features.

[0023] Next, in S209, a process of determining the line direction is executed based on the character pixels extracted in S208. The line direction here refers to the direction in which characters are arranged in the document image, and in this embodiment, it is either the horizontal direction or the vertical direction. As a determination method, for example, a method of generating a projection histogram for each of the vertical and horizontal directions for the entire document image and determining the direction with the smallest variance as the line direction is used. By this line direction determination process, for example, the line direction of the business card image shown in FIG. 11(a) is determined to be vertical (vertical).

[0024] In the next step S210, the process flow branches based on the result of the determination in step S209. If the result of the determination is that the row direction is horizontal, the process proceeds to step S211, and if the row direction is vertical, the process proceeds to step S212.

[0025] In S211, when the line direction is horizontal, a character string detection process suitable for identifying a horizontal character line is executed. When the characters in a document image are arranged horizontally, the gap between characters (the gap between black pixel clusters = the number of white pixels) is usually narrower in the horizontal direction (left-right direction) than in the vertical direction (up-down direction). Therefore, a character string region is detected from the document image by performing region analysis with processing parameters adjusted to be suitable for identifying a group of characters written horizontally. As a specific method for detecting a character string, a publicly known method disclosed in JP-A-7-200733 and the like may be applied.

[0026] In S212, when the line direction is vertical, a rotation process for rotating the document image by 90 degrees is executed prior to the character string detection process. For example, the business card image in Fig. 11(a), which has a vertical line direction, is rotated by 90 degrees to generate a business card image in which the top and bottom of the business card are aligned left and right. As a result, the characters arranged vertically on the original (business card) are arranged horizontally in the rotated image.

[0027] In the next S213, similar to S211 described above, a character string detection process suitable for identifying horizontally written lines is executed. When the processing target is a vertically long business card with vertical writing, the character string detection process is executed after rotating the image by 90 degrees as in this embodiment, so that the character string area of ​​the character portion where the characters are widely spaced and discretely arranged, such as a name, can be more appropriately detected.

[0028] Next, in S214, the character string area detected in S213 is rotated by -90 degrees, that is, the character string area is rotated by the same angle in the opposite direction to the direction of rotation in S212. This causes the characters in the character string area to return to an upright state. At this time, a partial image corresponding to the detected character string area may be rotated by -90 degrees, or the coordinate information of the detected character string area may be rotated by -90 degrees and the coordinate information after the rotation may be applied to the document image obtained in S207 in the direction in which the characters are formed.

[0029] In S215, character recognition processing is performed on each string region detected in S211 or acquired in S214. As the character recognition processing, for example, there is a method of inferring the character code of the characters included in the partial image corresponding to the string region using a learned model. The learned model is an identification model that has been learned to output the character code with a character image as the input. Further, in the character recognition processing of the present embodiment, a process of obtaining the reliability with respect to the characters as the recognition result (hereinafter referred to as "recognized characters") is also performed together. The reliability is a numerical value representing the certainty of the recognized characters. The higher the numerical value, the higher the accuracy of the recognized characters. A high numerical value of reliability tends to be obtained when the state of the image portion corresponding to the recognized characters is good (such as less noise) and there is no overlap with other characters. Conversely, when the state of the image portion corresponding to the recognized characters is bad (such as more noise) and there is an overlap with other characters, the numerical value of the reliability tends to be low. Also, the numerical value is affected by the font size. Thus, in the character recognition processing of the present embodiment, the character code of the recognized characters, the region information (size information) indicating the circumscribed rectangle in units of the recognized characters, and the reliability with respect to the recognized characters are obtained.

[0030] Next, in S216, a process of improving the character recognition result obtained in S215 (recognition result improvement process) is executed. As described above, the accuracy of character recognition decreases if the results of the string detection processes in S211 and S213 are inappropriate. FIG. 11(b) is a diagram showing an example in which the string region of the name part could not be appropriately extracted from the business card image of FIG. 11(a) described above, and FIG. 12 shows the result of the character recognition process for the partial image (string rectangle image) corresponding to the string region. As a result of the detected string region being inappropriate, it can be seen that the Chinese character "Shin", which is part of the surname, has not been correctly recognized. Therefore, in the present embodiment, for the part where there is a high possibility that an appropriate string region has not been detected in the string detection process, the string region is re-acquired, and the character recognition process is performed again to improve the recognition result. The details of the recognition result improvement process will be described later. When the recognition result improvement process is completed, this process ends.

[0031] The above is the operational flow of the software that realizes OCR processing on document images.

[0032] <Outline of recognition result improvement process> Next, the recognition result improvement process in S216 will be outlined with reference to the flowchart in Fig. 3. This process is executed for each character string region extracted from the document image.

[0033] In S301, it is determined whether the character recognition process executed in S216 was character recognition process in Japanese. The target language of the character recognition process is set in advance, and the determination is made by referring to the setting information. If it was character recognition process in Japanese, the process proceeds to S302, and if it was character recognition process in another language, the process proceeds to S303.

[0034] In S302, a process is executed to obtain a character string area corresponding to a specific character line to be subjected to character recognition processing again from the document image based on the recognition result. Here, the specific character line is a character line constituting the name of a person in a business card (hereinafter, referred to as a "name line"). The name line obtaining process will be described in detail later.

[0035] Next, in S303, the process flow branches based on the result of the name line acquisition process. If the character string area corresponding to the name line can be acquired, the process proceeds to S304, and if not, the process ends. In S304, the same character recognition process as that performed in S215 is executed again on the character string area of ​​the name line acquired in S302. However, it is not necessarily necessary to perform the same character recognition process. For example, a different character recognition process may be performed using a trained model that limits the character types to be recognized to those that can be used in personal names, such as kanji, katakana, and hiragana.

[0036] Next, in S305, a process (character code replacement process) is executed to replace the character code of the recognized character obtained in the second character recognition process in S304 with the character code obtained in the first character recognition process under certain conditions. The details of the character code replacement process will be described later. When the character code replacement process is completed, this process ends.

[0037] The above is an overview of the recognition result improvement process.

[0038] <Name line acquisition process> Next, the name line acquisition process of S302 will be described with reference to the flowchart of FIG.

[0039] In S401, a process (name character identification process) is executed to identify characters that seem to be a person's name (hereinafter referred to as "name characters") from the recognition result obtained in S215. Each recognized character identified as a name character by this name character identification process is hereinafter treated as a unit called a "blob." This "blob" includes the coordinate information (rectangle information) of a circumscribing rectangle that surrounds the corresponding character or character string area and its character code. The name character identification process will be described in detail later. Table 1 below summarizes various types of blobs that will appear in the following explanation.

[0040] [Table 1]

[0041] In S402, candidates for name lines (character strings consisting of multiple name characters) are generated by combining name characters that are aligned in lines obtained in the name character identification process. Whether or not line alignment is possible is determined by the blob of each name character, and when they are combined, a blob corresponding to the name line candidate is generated.

[0042] In S403, characters existing in the range before and after the name line candidate obtained in S402 are combined as part of the name line candidate and incorporated into the name line candidate.

[0043] In S404, pixel blocks existing in the range before and after the name line candidate obtained in S403 are extracted from the document image, and are further added to the name line candidate.

[0044] In S405, a process is performed to delete unnecessary name line candidates from all the name line candidates obtained up to this point, and the remaining name line candidates are determined to be name lines. Whether or not a name line candidate is unnecessary is determined using rectangular information within the blob and information on the reliability of the recognition result. If a judgment condition that combines these pieces of information is not met, the name line candidate is determined to be unnecessary. Examples of the conditions include when there are two or more blobs of name characters, or when there are no name character blobs but the name line consists of multiple lines, etc.

[0045] The above is the content of the name line acquisition process. Fig. 13 is a diagram showing an example of a character string area corresponding to the name line acquired from the business card image of Fig. 11(a) by this process. The area including the title and name in the business card image of Fig. 11(a) (solid line frame in the figure) is acquired as the character string area corresponding to the name line. Fig. 14 shows the recognition result when character recognition process is performed on a partial image (character string rectangular image) corresponding to the character string area shown in Fig. 13. In this case, it can be seen that the name part (four kanji characters "Arata", "No", "Yasu", and "Suke") is correctly recognized, but the title part spanning two lines is not correctly recognized.

[0046] <Name character specific processing> Next, the name character identification process of S401 will be described with reference to the flowchart of Fig. 5. The series of processes shown below are executed for each recognized character obtained in the character recognition process of S215. The results of the character recognition process referred to in this flow include information on the number of first candidate characters for each recognized character, in addition to information on the character code, size, and reliability of the recognized characters.

[0047] In S501, it is determined whether the size of the recognition character of interest in the line height direction is less than a predetermined threshold. In this case, the threshold may be determined by a method such as aggregating the appearance frequencies of characters of each size that can be used as valid recognition results from the results of character recognition for the entire document image, and setting the size with the highest frequency as the threshold. In this case, the conditions for obtaining a valid recognition result are as follows. · The character width and aspect ratio are equal to or greater than a predetermined value. (Characters of a size like a barcode are not regarded as characters) · The confidence level for the recognition result is equal to or greater than a predetermined value. · The character size in the line height direction (width for vertical writing, height for horizontal writing) is equal to or greater than a predetermined value.

[0048] Note that the predetermined values in the above conditions are values determined based on empirical rules. As a result of the determination, if the size of the recognition character of interest in the line height direction is less than the threshold, the process proceeds to S508; if it is equal to or greater than the threshold, the process proceeds to S502.

[0049] In S502, it is determined whether the number of characters obtained as the first candidate of the recognition result for the recognition character of interest is plural (i.e., other than 1). Here, "the first candidate" means the one with the highest accuracy among the multiple recognition candidates for that character. And "the number of characters obtained is plural" means the case where two or more recognition candidates are output for one character. For example, a case where the Chinese character "誠" is recognized separately as "言" and "成" is applicable. If the number of characters obtained as the first candidate of the recognition result is other than 1, the process proceeds to S508; if the number of characters is 1, the process proceeds to S503.

[0050] In S503, it is determined whether the character code of the first candidate for the recognition character of interest is a character code other than a Chinese character or kana (specifically, a character code such as alphanumeric characters or symbols). If the character code of the first candidate is a character code other than a Chinese character or kana, the process proceeds to S508; if it is a character code of a Chinese character or kana, the process proceeds to S504.

[0051] In S504, it is determined whether the character code of the recognition character of interest is the character code to be excluded. The character codes to be excluded are, for example, the character codes of lowercase characters such as "ぁ" and "ァ", which are specified in advance. If it is the character code to be excluded, the process proceeds to S508; if it is a character code other than the character code to be excluded, the process proceeds to S505.

[0052] In S505, it is determined whether the confidence level for the recognition character of interest is lower than the threshold value. The threshold value here can be set based on an empirical rule as a value that can be considered reliable if it is a certain value or more. If the confidence level is lower than the threshold value, the process proceeds to S508; if it is equal to or higher than the threshold value, the process proceeds to S506.

[0053] In S506, it is determined whether the aspect ratio of the circumscribed rectangle of the recognition character of interest is approximately 1. Since the recognition result may include errors, it can be determined as "approximately 1" if the aspect ratio is within 1 ± the threshold value (for example, about 0 to 0.6). If the aspect ratio is other than approximately 1, the process proceeds to S508; if it is approximately 1, the process proceeds to S507.

[0054] In S507 when none of the above conditions of S501 to S506 are met, it is determined that the recognition character of interest is a name character. Also, in S508 when any of the conditions of S501 to S506 are met, it is determined that the recognition character of interest is not a name character. When the determination is completed, this process ends. FIG. 15 is a diagram showing an example of the result of performing this process on the character recognition result for the business card image in FIG. 11(a), indicating that each of the characters "株", "式", "会", "社", "野", "康", "介" has been determined as a name character.

[0055] <Character code replacement process> Subsequently, the character code replacement process of S305 will be described with reference to the flowchart of FIG. 6.

[0056] In S601, it is determined whether or not there is a valid recognized character in the recognition result of the second character recognition process (S304). Specifically, if the recognized character acquired by the second character recognition process is a character other than a space, the recognized character is determined to be a valid recognized character. If the recognized character is a character other than a space, it may be a character that can be used to replace the recognition result of the first character recognition process. Therefore, if it is determined to be a valid recognized character, the blob is collected as a valid recognized blob (see Table 1 above), and the process proceeds to S602. If there is no valid recognized character, the process is terminated. Note that even if the character is not a space, if its character height is less than a threshold and its character length is less than a threshold, it is collected as a valid recognized blob and also counted as a noise character. In this case, the threshold value is appropriately about one-fourth of the line height (width if written vertically, height if written horizontally).

[0057] In S602, a process of re-evaluating the name line (name line re-evaluation process) is executed based on the result of the second character recognition process (S304) and the result of the initial character recognition process (S215). The name line re-evaluation process will be described in detail later.

[0058] In S603, the process flow branches based on the result of the name line reevaluation process. If the result of the reevaluation shows that a name line containing name characters for which replacement is valid exists, the process proceeds to S604, and if not, the process ends.

[0059] In S604, the character codes of the name characters in the name line for which replacement is valid are replaced with the character codes obtained by the second character recognition process. When the character code replacement is completed, this process ends.

[0060] The above is the content of the character code replacement process.

[0061] <Name line reevaluation process> Next, the name line reevaluation process of S602 will be described with reference to the flowchart of FIG.

[0062] In S701, a process (blob integration process) is executed for integrating blobs that overlap in the row direction for the valid recognition blobs collected in S601. At this time, an attribute flag is set for the valid recognition blobs after integration. The blob integration process will be described in detail later.

[0063] In S702, the attributes of the valid recognized blobs after the blob integration process are determined, and a process of setting an attribute flag (blob attribute determination process) is executed. The blob attribute determination process will be described in detail later.

[0064] In S703, a process is executed to determine whether or not a recognized character corresponding to a valid recognition blob for which no attribute flag was set in S701 or S702 is a name character. In this case, there are four types of attribute flags: a noise flag, an integrated flag, a highly reliable original multi-line flag, and a highly reliable original single-line flag. For a recognized character of a valid recognition blob for which any of these four types of flags is set, a determination is not made as to whether or not the recognized character is a name character. Note that this determination can be made by carrying out the same process as the name character identification process described above.

[0065] In S704, the recognized characters determined to be name characters in S703 are integrated to generate a name row. This means that the rectangular information of both recognized characters determined to be name characters is combined based on the blobs of the recognized characters and treated as one character string area.

[0066] In S705, among the recognized characters located before and after the recognized characters determined to be name characters in S703, the characters that satisfy a predetermined condition are added to the name line generated in S704. This means that the rectangular information of the blob of the recognized characters determined to be name characters and the rectangular information of the blog of the characters before and after it are integrated into one character string area. For example, in the case of a character sequence "ai", if the recognized character "ai" is determined to be a name character, if the recognized character "a" located before "ai" satisfies a predetermined condition, it is taken in as a name line and a character string area including the two characters "ai" is obtained. Here, the predetermined condition is a combination of information such as the character code obtained by the character recognition process, the reliability of the recognition result, and the distance between the character determined to be a name character. For example, conditions such as the size of the character being equal to or larger than a threshold value and the reliability of the recognition result being equal to or larger than a threshold value are set, and characters that satisfy all the conditions are taken in to the name line. As a result, a character string area of ​​a character line consisting of recognized characters that are approximately like a person's name is obtained.

[0067] The above is the content of the name line reevaluation process.

[0068] <Blob integration processing> Next, the blob integration process of S701 will be described with reference to the flowchart of Fig. 8. Note that the series of processes shown below are executed in order for all valid recognition blobs.

[0069] In S801, it is determined whether a value indicating the distance between the end of a recognized character corresponding to a valid recognition blob of interest (hereinafter referred to as a "blob of interest") and the beginning of a recognized character corresponding to a next valid recognition blob (hereinafter referred to as a "next blob") is positive or negative. Here, at the start of processing, each valid recognition blob is sorted by distance from the beginning of the line. Therefore, the "next blob" means the valid recognition blob that is closest among the unprocessed valid recognition blobs in the same line. FIG. 16(a) is a diagram showing a state in which the distance (inter-character distance) between the end of a character of the blob of interest and the beginning of a character of the next blob is positive in the case of horizontal writing, and FIG. 16(b) is a diagram showing a state in which the inter-character distance is positive in the case of vertical writing. In other words, when the inter-character distance is negative, it means that the recognized character of the blob of interest and the recognized character of the next blob overlap. Hereinafter, the absolute value of the inter-character distance when it is negative will be called the "inter-character overlap size." If the inter-character distance value is positive, the process transitions to S805, and if it is negative, the process transitions to S802.

[0070] In S802, it is determined whether the overlap size between characters is equal to or smaller than a threshold. The threshold is determined based on the line height (character width for vertical writing, character height for horizontal writing). For example, a value less than one time the line height is set as the threshold. If it is equal to or smaller than the threshold, the process proceeds to S804, and if it is larger than the threshold, the process proceeds to S803.

[0071] In S803, the overlapping character integration flag is set for the blob of interest. After the flag is set, the process proceeds to S809.

[0072] In S804, it is determined whether the size of the recognized characters corresponding to the blob of interest and the next blob is larger than a threshold. The threshold is determined based on the line height (character width for vertical writing, character height for horizontal writing) like the threshold in S802. For example, a value less than one time the line height is set as the threshold. If the size of both recognized characters is larger than the threshold, the recognized characters of the blob of interest and the next blob are both determined to be the size of name characters, and processing for the blob of interest is completed. On the other hand, if the size of both recognized characters is equal to or smaller than the threshold, the process proceeds to S803 above, and an overlapping character integration flag is set for the blob of interest.

[0073] In S805, if the inter-character distance is positive, it is determined whether the inter-character distance is greater than a threshold. The threshold in this case is also determined based on the line height (character width for vertical writing, character height for horizontal writing) as with the threshold in S802 and S804. For example, a value less than one time the line height is set as the threshold. If the inter-character distance is greater than the threshold, it is determined that the name characters are separated, and processing of the target blob is completed without merging with the recognized characters of the next blob (S809). On the other hand, if the inter-character distance is equal to or less than the threshold, the process proceeds to S806.

[0074] In S806, it is determined whether the recognized characters corresponding to the valid recognized blob immediately preceding the target blob (hereinafter referred to as the "previous blob") are closer than twice the distance between the recognized characters of the target blob and the previous blob. If they are closer, the processing for the target blob ends without merging with the recognized characters of the previous blob (S809). On the other hand, if they are not closer, the processing proceeds to S807. Note that if there is no previous blob for the target blob, the line height of the target blob (character width in the case of vertical writing, character height in the case of horizontal writing) can be treated as the distance between the two characters and the determination can be made.

[0075] In S807, it is determined whether the length of the characters when the recognized characters of the target blob and the recognized characters of the next blob are combined is greater than a threshold value. Here, the "length of the characters when combined" is the character height in the case of vertical writing, and the character width in the case of horizontal writing. The threshold value is determined based on the line height (character width in the case of vertical writing, character height in the case of horizontal writing). For example, a value that is greater than one time the line height is set as the threshold. If the length of the characters when combined is greater than the threshold value, the processing for the target blob is completed without combining with the recognized characters of the next blob. On the other hand, if the length of the characters when combined is equal to or less than the threshold value, the processing proceeds to S808.

[0076] In S808, a separated character integration flag is set for the blob of interest. After the flag is set, the process proceeds to S809.

[0077] In S809, the blob of interest and the next blob are integrated to generate a new integrated valid recognition blob. After the integration, the process for the blob of interest is completed. Then, when the above process is completed for all valid recognition blobs, the process proceeds to S810.

[0078] In S810, an attribute flag is set for the valid recognition blob after integration generated in S809. Specifically, either a noise flag or an integration flag is set as the attribute flag according to the following determination conditions.

[0079] <Judgment conditions> If the merged blob contains a certain amount of noise characters, the noise flag is set. -If the merged blob has a duplicate character merge flag or a separated character merge flag, set the merge flag.

[0080] When the attribute flags are set for all the integrated valid recognition blobs according to the above-mentioned judgment conditions, this process ends.

[0081] The above is the content of the blob integration process.

[0082] <Blob attribute determination process> Next, the blob attribute determination process of S702 will be described with reference to the flowchart of Fig. 9. The following series of processes are executed in order for all valid recognition blobs. The valid recognition blobs that are the subject of this process include the valid recognition blobs that have been integrated in the above-mentioned blob integration process and the valid recognition blobs that remain unintegrated.

[0083] In S901, a process (recognition result comparison process) is executed to compare the recognition result of the first character recognition process (S215) with the recognition result of the second character recognition process (S304) for a valid recognition blob of interest (hereinafter, referred to as a "blob of interest"). The details of the recognition result comparison process will be described later.

[0084] In S902, it is determined whether the blob of interest is a blob that constitutes a multiline, based on the comparison result in S901. Here, a multiline refers to a character string consisting of multiple character lines, such as the character string enclosed in the dashed frame 1301 in Fig. 13. If it is determined that the blob of interest is a blob that constitutes a multiline, the process proceeds to S903, and if it is determined that the blob of interest is a blob that does not constitute a multiline, the process proceeds to S904.

[0085] In S903, a high-reliability original multiline flag is set for the blob of interest. After the flag is set, the process for the blob of interest ends.

[0086] In S904, based on the comparison result in S901, it is determined whether the number of highly reliable original characters is equal to or greater than a threshold value (whether the character string region is strip-shaped). Here, the "number of highly reliable original characters" represents the number of characters determined to have a higher reliability for the recognized characters in the first character recognition process than in the second character recognition process in the recognition result comparison process described next. If the number of highly reliable original characters is equal to or greater than the threshold value, the process proceeds to S905, and if it is less than the threshold, the process for the blob of interest is terminated.

[0087] In S905, a high-reliability original single-line flag is set for the blob of interest. After the flag is set, the process for the blob of interest ends.

[0088] When the above process has been performed on all valid recognition blobs, this process ends.

[0089] The above is the content of the blob attribute determination process.

[0090] <Recognition result comparison process> Next, the recognition result comparison process of S703 will be described with reference to the flowchart of Fig. 10. Note that the series of processes shown below are executed in order for all original blobs.

[0091] In S1001, it is determined whether or not the area indicated by the rectangular information of the valid recognition blob of interest overlaps with the area indicated by the rectangular information of the original blob to be processed (see Table 1 above). FIG. 17 is a diagram for explaining the overlap determination of rectangular areas. In FIG. 17, two vertically long frames indicate rectangular areas of the original blob to be processed, and a substantially square frame indicates a rectangular area of ​​the valid recognition blob of interest. In the example shown in FIG. 17, both of the vertically long rectangular areas are determined to overlap with the substantially square rectangular area. In this way, if both rectangular areas overlap, the process proceeds to S1002. On the other hand, if both rectangular areas do not overlap, the process for the original blob is terminated.

[0092] In S1002, it is determined whether there are recognized characters corresponding to the original blob to be processed. In the case where no character recognition results are returned for a character string region, information indicating this is stored in the blob, so this process uses that information to determine whether any recognized characters are associated. If there are recognized characters, the process proceeds to S1003. If there are no recognized characters, the process for the original blob ends.

[0093] In S1003, the reliability of the valid recognition blob of interest is compared with the reliability of the original blob to be processed. If the reliability of the original blob is higher than that of the valid recognition blob, that is, if the original blob to be processed is a "high reliability original blob (see Table 1 above)", the process proceeds to S1004. On the other hand, if the reliability of the original blob to be processed is lower than that of the valid recognition blob, the process for the original blob is terminated.

[0094] In S1004, it is determined whether the original blob to be processed and another high-reliability original blob are separated in the height direction and overlap in the row direction. If they are separated in the height direction and overlap in the row direction, the process proceeds to S905. FIG. 18 is a diagram for explaining this overlap determination, and in the example shown in the diagram, they are separated in the height direction and overlap in the row direction, so the process proceeds to S1005. On the other hand, if they are not separated in the height direction or do not overlap in the row direction, the process for the original blob is terminated.

[0095] In S1005, it is determined that the recognized character corresponding to the original blob to be processed is a character that constitutes a multiline consisting of multiple lines. In the following S1006, the blob to be processed is counted as a high-confidence original blob, and the process for the original blob is terminated.

[0096] When the above process has been performed for all the original blobs, this process ends.

[0097] The above is the content of the recognition result comparison process.

[0098] <Effects of the recognition result improvement process> Finally, the effect of the recognition result improvement process explained using the flowcharts of Figs. 3 to 10 will be explained. Fig. 19 is a diagram showing character string areas when the recognition result improvement process is performed on the business card image shown in Fig. 11(a) described above. As is clear from a comparison with Fig. 11(b), it can be seen that the character string areas of the name portion have been properly extracted. Fig. 20 shows the results of character recognition processing for each character string area shown in Fig. 19. As is clear from a comparison with Fig. 12, it can be seen that the character string areas have been properly extracted, resulting in correct character recognition.

[0099] As described above, according to this embodiment, the character lines (name lines) constituting the name of a person on a business card are identified from the business card image, and character recognition processing is performed again on the character string area. The result of the first character recognition processing is then compared with the result of the second character recognition processing, and the more likely result is determined as the final recognition result. This makes it possible to make the recognition result of the name area on the business card more likely, and improves the accuracy of the character recognition processing.

[0100] <Other embodiments> The present invention can also be realized by a process in which a program for implementing one or more of the functions of the above-described embodiments is supplied to a system or device via a network or a storage medium, and one or more processors in a computer of the system or device read and execute the program. The present invention can also be realized by a circuit (e.g., ASIC) that implements one or more of the functions.

Claims

1. a first processing means for performing a first character recognition process on a character string area obtained by performing a character string detection process on a document image obtained by reading a document; an identifying means for identifying name characters constituting a person's name in the document based on a result of the first character recognition process; an acquiring means for acquiring a character string area corresponding to a character line representing the name by integrating an area corresponding to the name characters identified by the identifying means and an area corresponding to one or more characters belonging to the same line as the name characters; a second processing means for performing a second character recognition process on a character string area corresponding to the character line representing the name; a determination means for determining character codes for characters constituting the character line representing the name based on a result of the first character recognition process and a result of the second character recognition process; An information processing device comprising:

2. 2 . The information processing apparatus according to claim 1 , wherein the second character recognition process is a character recognition process in which a character type to be recognized is limited to a character type that can be used in a person's name.

3. 3. The information processing apparatus according to claim 2, wherein the types of characters that can be used in the person's name include at least kanji and kana.

4. The determining means is generating a line of characters representing the name using recognized characters other than spaces from the result of the second character recognition process; if a recognized character included in the generated character line representing the name is a valid name character, the character code obtained by the first character recognition process is replaced with the character code of the recognized character that is the valid name character; 4. The information processing apparatus according to claim 1, wherein the information processing apparatus is a computer.

5. The information processing apparatus according to claim 4, characterized in that the determining means generates the character line representing the name by integrating recognized characters other than the space, the recognized characters having a negative inter-character distance, and treating them as a single recognized character.

6. The information processing device according to any one of claims 1 to 5, characterized in that the identification means identifies a recognized character as the name character when a size in the line height direction of the recognized character included in the result of the first character recognition process is equal to or larger than a threshold value.

7. The information processing device according to claim 1 , wherein the identifying means identifies a recognized character as the name character when the number of first candidate characters for the recognized character included in the result of the first character recognition process is one.

8. The information processing device according to any one of claims 1 to 5, characterized in that when a character code of a first candidate for a recognized character included in the result of the first character recognition process is kanji or kana, the identification means identifies the recognized character as the name character.

9. The information processing device according to any one of claims 1 to 5, characterized in that the identification means identifies the recognized character as the name character when the character code of the recognized character included in the result of the first character recognition process is a character code other than a character code specified as an exclusion target.

10. 6. The information processing apparatus according to claim 1, wherein the identifying means identifies a recognized character included in the result of the first character recognition process as the name character when the reliability of the recognized character is equal to or higher than a threshold value.

11. The information processing device according to claim 1, characterized in that the acquiring means acquires a character string area corresponding to the character line representing the name by combining name characters that are aligned with each other for the plurality of name characters identified by the identifying means.

12. 12. The information processing apparatus according to claim 11, wherein the acquiring means acquires a character string area corresponding to the character line representing the name by further combining characters before and after the character string area obtained by the combining.

13. The information processing device according to claim 12, characterized in that the acquiring means acquires a character string area corresponding to the line of characters representing the name by adding pixel chunks before and after the character string area obtained by combining the characters before and after the character string area.

14. 14. The information processing apparatus according to claim 11, wherein the acquiring means comprises a means for deleting a character string area that satisfies a certain condition from among the acquired character string areas corresponding to the character lines representing the name.

15. a processing step of performing a first character recognition process on a character string area obtained by performing a character string detection process on a document image obtained by reading the document; identifying name characters constituting a person's name in the document based on a result of the first character recognition process; an acquiring step of acquiring a character string area corresponding to a character line representing the name by integrating an area corresponding to the name characters identified in the identifying step and an area corresponding to one or more characters belonging to the same line as the name characters; a processing step of performing a second character recognition process on a character string area corresponding to the character line representing the name; a determining step of determining character codes for characters constituting the character line representing the name based on a result of the first character recognition process and a result of the second character recognition process; 13. An information processing method comprising:

16. A program for causing a computer to function as the information processing device according to any one of claims 1 to 14.

Images