Character recognition device, character recognition method, and program

Abstract

To provide a character recognition device, a character recognition method, and a program capable of correctly recognizing characters included in an input image.SOLUTION: A character recognition device comprises a first score calculation unit, a character region estimation unit, a second score calculation unit, and a selection unit. The first score calculation unit calculates a first score showing likelihood of a character string, the first score being for each of a plurality of candidate character strings which are candidates of the character string included in an input image. The character region estimation unit estimates a region corresponding to each character included in the candidate character string, among the regions of the input image. The second score calculation unit calculates a second score indicating the integrity of a character included in the candidate character string, on the basis of the estimated region. The selection unit selects one or more character strings among the plurality of candidate character strings, on the basis of the calculated first score and second score.SELECTED DRAWING: Figure 1

Description

[Technical field]

[0001] An embodiment of the present invention relates to a character recognition device, a character recognition method, and a program. [Background technology]

[0002] Conventionally, character recognition technology that recognizes characters included in an input image has been known. It is known that such conventional technology can improve recognition accuracy by recognizing characters without explicitly separating their boundaries. However, when recognizing characters without separating their boundaries, there are cases where one character is recognized as being duplicated. Also, depending on the input image, there are cases where a character is skipped during recognition. In other words, there is a problem with conventional technology in that characters included in an input image cannot be correctly recognized. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] JP 2013-097590 A Summary of the Invention [Problem to be solved by the invention]

[0004] An object of the present invention is to provide a character recognition device, a character recognition method, and a program that are capable of correctly recognizing characters included in an input image. [Means for solving the problem]

[0005] A character recognition device according to an embodiment includes a first score calculation unit, a character region estimation unit, a second score calculation unit, and a selection unit. The first score calculation unit is a first score indicating the likelihood of a character string, and calculates the first score for each of a plurality of candidate character strings that are candidates for character strings included in an input image. The character region estimation unit estimates areas of the input image that correspond to each of the characters included in the candidate character strings. The second score calculation unit calculates a second score indicating consistency of characters included in the candidate character strings based on the estimated areas. The selection unit selects one or more character strings from the plurality of candidate character strings based on the calculated first score and second score. The input image includes a plurality of character input areas, and the second score calculation unit calculates the second score based on areas corresponding to each of the characters included in the candidate character string and the character input areas. [Brief description of the drawings]

[0006] [Figure 1] FIG. 1 is a schematic diagram showing an example of the functional configuration of a character recognition system according to a first embodiment. [Diagram 2] FIG. 2 is a view showing an example of an input image according to the first embodiment. [Diagram 3] FIG. 4 is a diagram for explaining a first score according to the first embodiment. [Figure 4] FIG. 4 is a diagram for explaining a second score according to the first embodiment. [Diagram 5] 4 is a flowchart showing a series of operations of the character recognition system according to the first embodiment. [Figure 6] FIG. 11 is a diagram showing a modified example of the functional configuration of a second score calculation unit according to the first embodiment. [Figure 7] FIG. 2 is a diagram for explaining overlapping reading according to the first embodiment. [Figure 8] FIG. 4 is a diagram showing an example of the functional configuration of a skip score calculation unit according to the first embodiment. [Figure 9] FIG. 4 is a diagram for explaining a character-likeness map according to the first embodiment; [Figure 10] FIG. 11 is a diagram showing a modified example of the functional configuration of a first score calculation unit according to the first embodiment. [Figure 11] FIG. 11 is a diagram for explaining a beam search according to the second embodiment. [Figure 12] 13 is a diagram for explaining a match score for each partial input image in the beam search according to the second embodiment. FIG. [Figure 13] 10 is a flowchart showing a series of operations of a character recognition system according to a second embodiment. [Figure 14] FIG. 13 is a view showing an example of an input image according to the third embodiment. [Figure 15] FIG. 13 is a diagram showing an example of a second score according to the third embodiment. [Figure 16] 13A to 13C are diagrams for explaining an example of the operation of a character region estimation unit according to the fourth embodiment. [Figure 17] 13A to 13C are diagrams for explaining a modified example of the operation of the character region estimation unit according to the fourth embodiment. [Figure 18] FIG. 13 is a diagram for explaining a modified example of the character-likeness map according to the fifth embodiment. [Figure 19] FIG. 13 is a diagram showing an example of the functional configuration of a modified example of a character-likeliness map generation unit according to the fifth embodiment. [Figure 20] 13A and 13B are diagrams showing an example of input data and teacher data according to the fifth embodiment. [Figure 21] FIG. 1 is a diagram for explaining overlapping reading and skipping according to the conventional technology. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0007] Hereinafter, a character recognition device, a character recognition method, and a program according to an embodiment will be described with reference to the drawings.

[0008] [Prior Art] Problems that arise when using a character recognition method according to the prior art will be described with reference to Fig. 21. Fig. 11 is a diagram for explaining overlapping reading and skipping reading according to the prior art. Column 90 shows an example of overlapping reading, and column 95 shows an example of skipping.

[0009] In an example of duplicate reading, the case where the input image 91 is input will be described. When correctly recognizing the characters described in the input image 91, the regions 92 and 93 are estimated as the positions where the characters are described. Since "川 (river)" is described in region 92 and "崎 (cape)" is described in region 93, according to the character recognition method of the prior art, it can be recognized that "川崎 (Kawasaki)" is described in the input image 91. On the other hand, when the characters described in the input image 91 are misrecognized, the regions 92, 93, and 94 are estimated as the positions where the characters are described. Since "川 (river)" is described in region 92, "崎 (cape)" is described in region 93, and "山 (mountain)" is described in region 94, according to the character recognition method of the prior art, it is misrecognized that "川山崎 (Kawayama崎)" is described in the input image 91. Thus, in an example of duplicate reading, after recognizing the deviation of "崎 (cape)" as "山 (mountain)", "崎 (cape)" is recognized again, so there is a problem of recognizing the character "山 (mountain)" repeatedly.

[0010] Next, in an example of skipping, the case where the input image 96 is input will be described. When correctly recognizing the characters described in the input image 96, the regions 97, 98, and 99 are estimated as the positions where the characters are described. Since "長 (long)" is described in region 97, "谷 (valley)" is described in region 98, and "川 (river)" is described in region 99, according to the character recognition method of the prior art, it can be recognized that "長谷川 (Hasegawa)" is described in the input image 96. On the other hand, when the characters described in the input image 96 are misrecognized, the regions 97 and 98 are estimated as the positions where the characters are described. Since "長 (long)" is described in region 97 and "川 (river)" is described in region 99, according to the character recognition method of the prior art, it is misrecognized that "長川 (Nagakawa)" is described in the input image 96. Thus, in an example of skipping, there is a problem of skipping and recognizing the character "谷 (valley)" described in region 98.

[0011] [First Embodiment] The character recognition device according to this embodiment suppresses the problems of the conventional technology as described above. The character recognition device according to this embodiment performs character string recognition on a character string written in an input image. Character string recognition is a task of recognizing a character string in an image by inputting an image including the character string. In this embodiment, a horizontally written character string read from left to right will be described. Note that this embodiment is not limited to a horizontally written character string read from left to right, and the same applies to a vertically written character string. Note that an image including a character string broadly includes an image of handwritten characters, a photographed signboard, a road sign, and the like. Note that in this embodiment, a case will be described in which the number of characters included in the character string is 0 or more.

[0012] 1 is a schematic diagram showing an example of the functional configuration of a character recognition system according to a first embodiment. The functional configuration of the character recognition system 1 will be described with reference to the diagram. The character recognition system 1 includes an input image acquisition unit 21, a candidate character string calculation unit 22, a character recognition device 10, and an output unit 23.

[0013] The input image acquisition unit 21 acquires an input image IM. Fig. 2 is a diagram showing an example of the input image IM according to the first embodiment. A character string S is written in the input image IM. Specifically, the input image IM includes the character string S, which is handwritten characters of "Kawasaki." In this embodiment, an example will be described in which the input image IM includes the handwritten character string S.

[0014] Returning to Fig. 1, the candidate character string calculation unit 22 calculates candidate character strings CS that are candidates for the character string S written in the input image IM acquired by the input image acquisition unit 21. The candidate character string calculation unit 22 calculates the candidate character strings CS using a known character recognition technique (e.g., pattern matching, feature detection, etc.). The candidate character string calculation unit 22 outputs a plurality of character strings S as candidate character strings CS.

[0015] Character recognition device 10 selects, based on input image IM acquired by input image acquisition unit 21 and multiple candidate character strings CS calculated by candidate character string calculation unit 22, a likely character string as a selected character string SS from the multiple candidate character strings CS calculated by candidate character string calculation unit 22. Character recognition device 10 includes a first score calculation unit (character recognition unit) 110, a character region estimation unit 120, a second score calculation unit (region consistency score calculation unit) 130, and a selection unit 140.

[0016] The output unit 23 outputs the selected character string SS selected by the character recognition device 10. The output unit 23 outputs the selected character string SS, for example, by outputting information for displaying the selected character string SS on a display unit (not shown), by outputting information for outputting the selected character string SS as voice from a voice output unit (not shown), or by wirelessly outputting the selected character string SS to an information processing device (not shown).

[0017] The first score calculation unit 110 calculates a first score S1 for each of the multiple candidate strings CS calculated by the candidate string calculation unit 22. A candidate string CS is a candidate for a string S included in the input image IM. The first score S1 indicates the likelihood of a string. That is, the first score calculation unit 110 calculates the first score S1 indicating the likelihood of a string, for each of the multiple candidate strings CS that are candidates for the string S included in the input image IM.

[0018] FIG. 3 is a diagram for explaining the first score according to the first embodiment. A specific example of the first score S1 calculated by the first score calculation unit 110 will be described with reference to the figure. The figure shows an example in which the input image IM includes the character string "Kawasaki". In this example, the candidate character string calculation unit 22 calculates "Kawayamazaki" as the candidate character string CS-1, "Kawasaki" as the candidate character string CS-2, and "Kawayamaki" as the candidate character string CS-3. The first score calculation unit 110 calculates the first score S1 for each of the candidate character strings CS-1, CS-2, and CS-3. In this example, the first score S1-1 of the candidate character string CS-1 is "0.5", the first score S1-2 of the candidate character string CS-2 is "0.5", and the first score S1-3 of the candidate character string CS-3 is "0.1".

[0019] Returning to Fig. 1, character area estimation unit 120 estimates character area CA based on candidate character string CS and input image IM. Character area CA is an area corresponding to each character C included in candidate character string CS. In other words, character area estimation unit 120 estimates character area CA, which is an area of ​​input image IM corresponding to each character C included in candidate character string CS.

[0020] The second score calculation unit 130 calculates a second score S2 indicating the consistency of characters included in the candidate character string CS based on the character area CA, which is the area estimated by the character area estimation unit 120. Here, the consistency of characters included in the candidate character string CS refers to spatial consistency. If there is no spatial consistency, characters may overlap or characters may be skipped.

[0021] FIG. 4 is a diagram for explaining the second score according to the first embodiment. With reference to the figure, the estimation of the character area CA performed by the character area estimation unit 120 and a specific example of the second score S2 calculated by the second score calculation unit 130 will be explained. The figure shows an example in which the input image IM includes a character string "Kawasaki". In this example, the character area estimation unit 120 estimates character areas CA, which are areas corresponding to the characters C included in the candidate character string CS. For example, the character area estimation unit 120 estimates the areas corresponding to the characters C included in the candidate character string CS-1, "Kawasaki", as the character areas CA1-1, CA2-1, and CA3-1. In addition, the character area estimation unit 120 estimates the areas corresponding to the characters C included in the candidate character string CS-2, "Kawasaki", as the character areas CA1-2 and CA2-2. Furthermore, character area estimation unit 120 estimates areas corresponding to the character C included in the candidate character string CS-3, "KAWAYAMAKI," as character areas CA1-3, CA2-3, and CA3-3.

[0022] The second score calculation unit 130 calculates a second score S2 for each of the multiple candidate character strings CS. In this example, the second score S2-1 of the candidate character string CS-1 is "0.1", the second score S2-2 of the candidate character string CS-2 is "1.0", and the second score S2-3 of the candidate character string CS-3 is "1.0".

[0023] 1, the selection unit 140 selects one or more character strings S from among the multiple candidate character strings CS as a selected character string SS based on the calculated first score S1 and second score S2. For example, the selection unit 140 selects the candidate character string CS that has the largest value as a result of multiplying the first score S1 and the second score S2 as the selected character string SS.

[0024] 5 is a flowchart showing a series of operations of the character recognition system according to the first embodiment. Below, a series of operations of the character recognition device 10 will be described with reference to the flowchart shown in the figure.

[0025] (Step S110) The input image acquisition unit 21 acquires an input image IM. The candidate character string calculation unit 22 calculates candidate character strings CS that are candidates for the character string S written in the input image IM. In this flowchart, a case will be described in which the candidate character string calculation unit 22 calculates n (n is an integer equal to or greater than 1) candidate character strings CS.

[0026] (Step S120) The first score calculation unit 110 calculates a first score S1 for each of the calculated candidate character strings CS. n , the first score S1 is α n In this case, the first score calculation unit 110 calculates (y 1, α1)…(y n, α n ) is calculated.

[0027] (Step S130) The second score calculation unit 130 sets 1 to a counter i.

[0028] (Step S140) Character region estimation unit 120 estimates regions in input image IM that correspond to each of a plurality of characters C included in candidate character string CS. In this flowchart, a case where candidate character string CS includes m characters (m is an integer equal to or greater than 1) is described. i y i,1 ,…,y i,m In this case, the character region estimation unit 120 estimates character regions CA, s1, ..., s2, which correspond to the respective characters C. m Estimate.

[0029] (Step S150) The second score calculation unit 130 calculates y i The second score S2 is calculated by dividing the second score S2 by β n The second score S2 is also written as β n is s1,…,s m It is calculated based on the following.

[0030] (Step S160) The selection unit 140 determines α which is the first score S1 i and β which is the second score S2 i to calculate γ i .

[0031] (Step S170) If i < n, the second score calculation unit 130 proceeds to step S190. That is, the second score calculation unit 130 repeats the steps from step S140 to step S160 until the counter i reaches n, which is the number of candidate strings CS calculated by the candidate string calculation unit 22. If i ≥ n, that is, if the counter i reaches n, which is the number of candidate strings CS calculated by the candidate string calculation unit 22, the process proceeds to step S180.

[0032] (Step S190) The second score calculation unit 130 increments the counter i and proceeds to step S140.

[0033] (Step S180) The selection unit 140 selects the candidate string CS for which γ n is the maximum as the selected string SS. In this flowchart, the selection unit 140 selects the selected string SS based on the maximum score set. Note that the selection unit 140 may select the selected string SS based on the minimum score set according to the calculation methods of α n and β n .

[0034] [Modification Example of the First Embodiment] FIG. 6 is a diagram showing a modification example of the functional configuration of the second score calculation unit according to the first embodiment. With reference to this figure, a second score calculation unit 130A, which is a modification example of the second score calculation unit 130, will be described. The second score calculation unit 130A includes a duplicate reading score calculation unit 131, a skipping score calculation unit 132, and a second score integration unit 133.

[0035] The overlapping reading score calculation unit 131 calculates an overlapping reading score S21 which is a score indicating the amount of overlapping of the candidate character string CS. The amount of overlapping of the candidate character string CS is, specifically, the amount of overlapping of the regions corresponding to the characters C included in the candidate character string CS. The second score calculation unit 130A calculates a second score S2 based on the calculated overlapping reading score S21. That is, in this embodiment, the second score calculation unit 130A calculates the second score S2 based on the amount of overlapping of the regions corresponding to the characters C included in the candidate character string CS.

[0036] FIG. 7 is a diagram for explaining overlapping readings according to the first embodiment. The amount of overlap calculated by overlapping reading score calculation unit 131 will be explained with reference to the same figure. In this example, the input image IM contains the characters "Kawasaki," and character area estimation unit 120 estimates character area CA1, character area CA2, and character area CA3 as character area CA. Here, overlapping area CA-DP, which is an area where character area CA2 and character area CA3 overlap each other, is the amount of overlap. Specifically, when the amount of overlap is m(y), overlapping reading score calculation unit 131 calculates the consistency score P of the overlap using the following formula (1): ovlp It is calculated as follows.

[0037]

number

[0038] Here, C OP is a constant between 0 and 1, and the smaller it is, the lower the consistency score P ovlp becomes smaller. C OP The value of may be determined experimentally.

[0039] 6, skipping score calculation unit 132 calculates a skipping score S22, which is a score indicating whether skipping has occurred, based on a character C included in candidate character string CS and the area estimated by character area estimation unit 120. Second score calculation unit 130 calculates a second score S2 based on the calculated skipping score S22. That is, second score calculation unit 130 calculates the second score S2 based on a character C included in candidate character string CS and the area estimated by character area estimation unit 120.

[0040] 8 is a diagram showing an example of the functional configuration of the skipping score calculation unit according to the first embodiment. An example of the functional configuration of the skipping score calculation unit 132 will be described with reference to the diagram. The skipping score calculation unit 132 includes a character likelihood map generation unit 1321 and a skipping score integration unit 1322. In this embodiment, the skipping score calculation unit 132 calculates a second score S2 based on the likelihood that some character C exists in an area of ​​the input image IM.

[0041] The character-likelihood map generating unit 1321 generates a character-likelihood map CLM. The character-likelihood map CLM indicates the likelihood that some character C exists in an image region of the input image IM.

[0042] The skipping score integrating unit 1322 calculates a skipping score S22 based on the character area CA estimated by the character area estimating unit 120 and the character-likeliness map CLM generated by the character-likeliness map generating unit 1321.

[0043] 9 is a diagram for explaining the character-likeliness map according to the first embodiment. With reference to the drawing, an overview of the character-likeliness map CLM and the process performed by the skippability score integrating unit 1322 will be described.

[0044] FIG. 9(A) is a diagram showing a character region CA when the character region estimation unit 120 estimates the character region CA for a candidate character string CS in which a skip misrecognition has occurred. In this figure, for the character "長" in the character string S "長谷川" included in the input image IM, it is estimated as the character region CA1, and for the character "川", it is estimated as the character region CA2. The character "谷" is not estimated as the character region CA. That is, a skip has occurred.

[0045] FIG. 9(B) is a diagram showing an example of a character-likeness map CLM. In the example shown in this figure, there is a high probability that characters exist in the regions AR1, AR2, and AR3. That is, the character-likeness map generation unit 1321 generates, as the character-likeness map CLM, the likelihood that some character C exists in the image region of the input image IM, and estimates the likelihood that the character C exists for the character string S including characters in which a skip has occurred.

[0046] FIG. 9(C) is a diagram showing an example of a mask MSK. The skip score integration unit 1322 generates the mask MSK based on the character region CA estimated by the character region estimation unit 120. The mask MSK indicates a region where the characters included in the candidate character string CS exist or do not exist. The skip score integration unit 1322 filters the character-likeness map CLM with the generated mask MSK. By filtering the character-likeness map CLM, the skip score integration unit 1322 estimates regions where there is a high probability that characters exist even though they are not included in the candidate character string CS.

[0047] FIG. 9(D) is a diagram showing the character-likeness map CLM after being filtered by the skip score integration unit 1322. The region AR2 is a region where there is a high probability that characters exist but is not included in the candidate character string CS. That is, it can be said that the larger the region AR2 is, the higher the possibility that a skip has occurred.

[0048] Here, the image region of the input image IM that has a high probability of containing some character C and is included in the character likelihood map CLM after filtering is called U j When the image area of ​​the input image IM is divided into a region with a width W and a height H, the skipping score integration unit 1322 calculates the skipping consistency score P SKIP The image area of ​​the input image IM may be divided into pixel units of the input image IM, or may be divided into units of a predetermined range made up of a plurality of pixels.

[0049]

number

[0050] Here, C SP is a constant greater than or equal to 0, and C SP The larger the skipping consistency score P SKIP becomes smaller. C SP The value of may be determined experimentally. If no skipping penalty is imposed, SP may be set to 0.

[0051] 6, the second score integrating unit 133 calculates the second score S2 based on the overlapping reading score S21 calculated by the overlapping reading score calculation unit 131 and the skipping score S22 calculated by the skipping score calculation unit 132. For example, the second score integrating unit 133 calculates a value obtained by multiplying the overlapping reading score S21 and the skipping score S22 as the second score S2.

[0052] 10 is a diagram showing a modified example of the functional configuration of the first score calculation unit according to the first embodiment. With reference to the figure, a first score calculation unit 110A which is a modified example of first score calculation unit 110 will be described. First score calculation unit 110A includes a character recognition score calculation unit 111, a knowledge processing score calculation unit 112, and a first score integration unit 113.

[0053] The character recognition score calculation unit 111 calculates a character recognition score S11 for each candidate character string CS. The character recognition score S11 indicates the likelihood of the character string.

[0054] The knowledge processing score calculation unit 112 calculates the knowledge processing score S12 for each candidate character string CS. The knowledge processing score calculation unit 112 is used when the candidate character strings CS to be written in the input image IM are limited. The case where the candidate character strings CS to be written in the input image IM are limited is, for example, when the input image IM has information such as a postal code, an address, a name, etc., obtained in advance. When it is known that the input image IM is a postal code, and when the candidate character string CS is not a number, the knowledge processing score S12 is calculated to be lower. Also, when it is known that the input image IM is an address, the knowledge processing score S12 is calculated to be lower when the input image is "Kawayamaki" than when the input image is "Kawasaki".

[0055] The first score integration unit 113 calculates a first score S1 based on the character recognition score S11 calculated by the character recognition score calculation unit 111 and the knowledge processing score S12 calculated by the knowledge processing score calculation unit 112. The selection unit 140 selects a selected character string SS based on the calculated first score S1 and second score S2.

[0056] Here, an example will be described in which the selection unit 140 selects the selected character string SS based on the character recognition score S11, the knowledge processing score S12, the overlapping reading score S21, and the skipping score S22. In this case, the selection unit 140 selects the selected character string SS based on the following formula (3).

[0057]

number

[0058] Specifically, the selection unit 140 selects P OCR and P, which is the knowledge processing score S12 LM and P, which is the overlapping reading score S21. ovlp and P, whose skipping score is S22.skip The candidate string CS for which the product of x and y is the largest is selected as the selected string SS.

[0059] [Summary of the first embodiment] According to the above-described embodiment, character recognition device 10 includes first score calculation unit 110 to calculate a first score S1 indicating the likelihood of character string S for each candidate character string CS, character region estimation unit 120 to estimate a region for each character C included in character string S, second score calculation unit 130 to calculate a second score S2 indicating the consistency of character C, and selection unit 140 to select a selected character string SS based on the first score S1 and the second score S2. That is, according to the above-described embodiment, the most likely character string is selected in consideration of the consistency of the region in which character C exists. Therefore, character recognition device 10 can correctly recognize character C included in input image IM.

[0060] According to the embodiment described above, the second score calculation unit 130 calculates the second score S2 based on the overlapping reading score S21. The overlapping reading score S21 is a score according to the amount of overlap between the regions corresponding to the characters C included in the complementary string CS. Therefore, according to the embodiment, the character recognition device 10 can suppress overlapping reading, and can therefore correctly recognize the character C included in the input image IM.

[0061] Furthermore, according to the above-described embodiment, the second score calculation unit 130 calculates the second score S2 based on the skipping score S22. The skipping score S22 is a score based on the character C included in the candidate character string CS and the character area CA estimated by the character area estimation unit 120, and if skipping occurs, a large penalty is given. Therefore, according to the present embodiment, the character recognition device 10 can prevent skipping, and can therefore correctly recognize the character C included in the input image IM.

[0062] According to the conventional technology, the improvement of duplicate reading and the improvement of skipping are in a trade-off relationship, and improving one of them makes the other problem more likely to occur. According to the above-mentioned embodiment, the duplicate reading score S21 and the skipping score S22 are calculated separately, and the selected character string SS is selected comprehensively, so that both the duplicate reading and the skipping problems can be improved.

[0063] Moreover, according to the above-described embodiment, the second score calculation unit 130 calculates the skipping score S22 by using the character-likelihood map CLM. The character-likelihood map CLM indicates the likelihood that some character C exists in the region of the input image IM. According to the present embodiment, skipping can be easily prevented.

[0064] [Second embodiment] An example of a character recognition device 10A according to the second embodiment will be described with reference to FIG. 11 to FIG. 13. The character recognition device 10A according to the second embodiment performs character recognition of a character string S included in an input image IM using a beam search algorithm. Here, when candidate characters C are calculated for each of a plurality of characters C included in the character string S and a combination of the candidates for each character C is set as a candidate character string CS, there is a problem that the more characters C included in the character string S, the more candidate character strings CS there are. When the number of candidate character strings CS increases, it takes time and resources to select a selection character string SS. Therefore, in this embodiment, an object is to perform character recognition with less time and resources by using a beam search algorithm.

[0065] FIG. 11 is a diagram for explaining beam search according to the second embodiment. With reference to the same figure, the beam search according to the second embodiment will be explained. In this embodiment, the character recognition device 10A divides the input image IM into a plurality of partial input images IMP and performs character recognition. In the example shown in the same figure, the input image IM is divided into a partial input image IMP-1, a partial input image IMP-2, and a partial input image IMP-3. The partial input image IMP may be divided according to, for example, a predetermined number of pixels. The predetermined number of pixels may be determined according to the width in which the character C is likely to be written.

[0066] Specifically, first, character recognition device 10A selects one or more selected character strings SS for partial input image IMP-1 of input image IM. Next, character recognition device 10A selects one or more selected character strings SS for partial input image IMP-1 and partial input image IMP-2. At this time, one or more selected character strings SS have already been selected for partial input image IMP-1, so the number of candidate character strings CS for partial input image IMP-1 and partial input image IMP-2 is reduced. Furthermore, character recognition device 10A selects final selected character strings SS for partial input image IMP-1, partial input image IMP-2, and partial input image IMP-3. At this time, one or more selected character strings SS have already been selected for partial input image IMP-1 and partial input image IMP-2, so the number of candidate character strings CS for partial input image IMP-1, partial input image IMP-2, and partial input image IMP-3 is reduced. In this manner, in this embodiment, the candidate character strings S are narrowed down for each partial input image IMP, thereby shortening the overall processing time.

[0067] FIG. 12 is a diagram for explaining the consistency score for each partial input image in the beam search according to the second embodiment. With reference to this figure, the consistency score for each partial input image IMP when the input image IM is described as "Kawasaki City" will be explained. FIG. 12(A) shows the correspondence between the candidate string CS and the consistency score when the character recognition device 10A performs character recognition on the partial input image IMP-1, and FIG. 12(B) shows the correspondence between the candidate string CS and the consistency score when the character recognition device 10A performs character recognition on the partial input image IMP-1 and the partial input image IMP-2. Here, the consistency score is a score used when the selection unit 140 selects the selected string SS, and is, for example, a score obtained by multiplying the first score S1 and the second score S2.

[0068] In FIG. 12(A), the character recognition device 10A calculates "Kawa" as the candidate string CS-11, "Kawa1" as the candidate string CS-12, and "Nori" as the candidate string CS-13. The consistency scores of the respective candidate strings CS are "1.0", "0.3", and "1.0", respectively. The character recognition device 10A selects the candidate string CS-11, which is a likely string, and the candidate string CS-13 as the selected string SS. In other words, the character recognition device 10A excludes the candidate string C-12 from the candidates.

[0069] In FIG. 12(B), the character recognition device 10A calculates "Kawayama崎" as the candidate character string CS-21, "Kawasaki" as the candidate character string CS-22, "Nori山崎" as the candidate character string CS-23, "Nori崎" as the candidate character string CS-24, "Kawayama奇" as the candidate character string CS-25, and "Nori山奇" as the candidate character string CS-26. The consistency scores of the respective candidate character strings CS are "0.1", "1.0", "0.1", "1.0", "1.0", and "1.0", respectively. The character recognition device 10A selects the candidate character string CS-22, which is the most likely character string, the candidate character string CS-24, the candidate character string CS-25, and the candidate character string CS-26 as the selected character string SS. In other words, the character recognition device 10A excludes the candidate character string C-21 and the candidate character string CS-23 from the candidates. Here, in the consideration of the partial input image IMP-1, since the candidate character string CS-12, which is "川1", has been excluded from the candidates, in the consideration of the partial input image IMP-1 and the partial input image IMP-2, the candidate character string S can be reduced.

[0070] FIG. 13 is a flowchart showing a series of operations of the character recognition system according to the second embodiment. While referring to the figure, a series of operations of the character recognition system 1A according to the second embodiment will be described. Step S100 is the same as the operation of the character recognition system according to the first embodiment described in FIG. 5, so the description will be omitted.

[0071] (Step S210) The character recognition device 10A sets x to δ. δ is a predetermined integer indicating the range of the partial input image IMP. x indicates the range within which the character recognition device 10A performs character recognition. In this flowchart, the character recognition device 10A first calculates the candidate character string CS for the range from 0 to x. Here, x, which is the range within which the character recognition device 10A performs character recognition, corresponds to the partial input image IMP in an example described with reference to FIG. 11.

[0072] (Step S220) The character recognition device 10A sets the empty set to the candidate set Φ.

[0073] (Step S230) The first score calculation unit 110 included in the character recognition device 10A calculates a first score S1 for each of the multiple candidate character strings CS included in the partial input image IMP. n , the first score S1 is α n In this case, the first score calculation unit 110 calculates (y 1, α1)…(y n, α n ) is calculated.

[0074] (Step S240) The character recognition device 10A selects a selection character string SS in the partial input image IMP. Specifically, the character recognition device 10A selects a selection character string SS in the partial input image IMP. i R large y's i and γ i The set of R is selected and set as the candidate set Φ. R is the number of character strings to be candidates when performing character recognition on the next partial input image IMP. By making R smaller, the processing time can be shortened, but if it is too small, the possibility of misrecognition may increase.

[0075] (Step S250) Character recognition device 10A determines whether character recognition has been performed on all of input image IM. Specifically, if x is smaller than W, character recognition device 10A proceeds to step S270. If x is not smaller than W, character recognition device 10A proceeds to step S260.

[0076] (Step S270) Character recognition device 10A expands the range in which character recognition is performed. Specifically, character recognition device 10A sets the value obtained by adding δ to x as x, and proceeds to step S230.

[0077] (Step S260) The character recognition device 10A k The string y for which k is output as the selected string SS.

[0078] [Summary of the second embodiment] According to the above-described embodiment, the first score calculation unit 110 included in the character recognition device 10A calculates a first score S1 for a partial input image IMP that is a part of the input image IM. In other words, the first score calculation unit 110 calculates a first score S1 for a candidate string CS that is a candidate for a string S that includes some characters among a plurality of characters C that constitute the string S included in the input image IM. In addition, the second score calculation unit 130 included in the character recognition device 10A calculates a second score S2 for a partial input image IMP that is a part of the input image IM. In other words, the second score calculation unit 130 calculates a second score S2 for a candidate string CS that is a candidate for a string S that includes some characters among a plurality of characters C that constitute the string S included in the input image IM. The character recognition device 10A calculates a candidate string CS for each part of the input image IM, so that the number of candidates for the entire string included in the input image IM can be reduced. Therefore, according to this embodiment, character recognition can be performed with less time and resources by using a beam search algorithm.

[0079] [Third embodiment] An example of a character recognition device 10B according to the third embodiment will be described with reference to Fig. 14 and Fig. 15. The third embodiment differs from the other embodiments in that a reference character spacing or an area of ​​a character to be written is defined in an input image IM. The present embodiment aims to perform character recognition based on the reference character spacing or the area of ​​a character to be written, thereby more accurately recognizing a character C included in an input image IM.

[0080] Fig. 14 is a diagram showing an example of an input image IM according to the third embodiment. With reference to the same figure, the reference character spacing or the area of ​​the character to be written, which is determined in the input image IM, will be described. Fig. 14(A) is an example of an input image IM in this embodiment. Fig. 14(B) is an example of the case where a character is written in the input image IM in this embodiment.

[0081] The input image IM shown in FIG. 14(A) includes a plurality of character input areas IAR. Specifically, the input image IM includes a character input area IAR1, a character input area IAR2, and a character input area IAR3. The character input area IAR is provided, for example, as a reference for a user who describes a character string S in the input image IM. That is, the character input area IAR determines the interval between reference characters or the area of the character to be described. In the following description, the character input area IAR may be described as a "frame".

[0082] In the input image IM shown in FIG. 14(B), a character C is described. Specifically, a character C-1 is described in the character input area IAR1, a character C-2 is described in the character input area IAR2, and a character C-3 is described in the character input area IAR3. The character C-1 is "川" (river), the character C-2 is "崎" (cape), and the character C-3 is "市" (city).

[0083] FIG. 15 is a diagram showing an example of the second score according to the third embodiment. With reference to this figure, the second score S2 calculated by the second score calculation unit 130 will be described. In an example shown in this figure, the candidate character string calculation unit 22 calculates candidate character strings CS of "川山奇市" (Kawayama Kishi) and "川崎市" (Kawasaki City).

[0084] When the candidate character string CS is "川山奇市" (Kawayama Kishi), the character input area IAR1 includes a character area CA1-1, the character input area IAR2 includes a character area CA2-1 and a character area CA3-1, and the character input area IAR3 includes a character area CA4-1. In this case, since the character input area IAR2 includes the character area CA2-1 and the character area CA3-1, there are two character areas CA in one frame (character input area IAR). In this case, when there are a plurality of character areas CA in one frame, the second score calculation unit 130 sets the overlapping amount between the smaller character area CA and the frame area as m(y), and based on the following formula (4), the consistency score P BOX is calculated.

[0085]

Equation

[0086] Here, C BP is a constant between 0 and 1, and the smaller it is, the lower the consistency score P BOX becomes smaller. C BP The value of may be determined experimentally.

[0087] Here, the selection unit 140 further selects the consistency score P BOX An example of a case where the selection unit 140 selects the selection character string SS based on the following formula (5) will be described.

[0088]

number

[0089] That is, in this embodiment, the second score calculation unit 130 calculates the second score S2 based on character areas CA, which are areas corresponding to the characters C included in the candidate string CS, and the character input area IAR.

[0090] [Summary of the third embodiment] According to the above-described embodiment, the character recognition device 10B includes the second score calculation unit 130, and thereby calculates the second score S2 based on the character area CA and the character input area IAR. For example, when a single box contains multiple characters, the second score calculation unit 130 calculates a low value for the second score S2. When a single box contains multiple characters, the second score calculation unit 130 calculates a low value for the second score S2, thereby preventing erroneous recognition such as character recognition by dividing a kanji character into its radicals, side characters, and the like. Therefore, according to this embodiment, the character C included in the input image IM can be more accurately recognized.

[0091] [Fourth embodiment] A fourth embodiment will be described with reference to Fig. 16 and Fig. 17. In the fourth embodiment, a specific example of character region estimation unit 120 will be described. An example of character region estimation unit 120 will be described with reference to Fig. 16, and a modified example of character region estimation unit 120 will be described with reference to Fig. 17.

[0092] 16 is a diagram for explaining an example of the operation of the character region estimation unit according to the fourth embodiment. An example of the character region estimation unit 120 will be described with reference to the same figure. First, input data DI is input to the neural network NN1. In this example, an example in which the input data DI is an image of a character string will be described. Specifically, an example in which the input data DI is an image of a character string "Kawasaki City" handwritten horizontally from left to right will be described.

[0093] The neural network NN1 calculates a series of feature values ​​F of an input character string. When the input data DI is a character string image handwritten horizontally from left to right, the neural network NN1 recognizes the series of feature values ​​F from left to right for the width of the judgment range. In this example, the neural network NN1 calculates feature values ​​F1 to F6. Here, the neural network NN1 calculates a number of feature values ​​F according to the length of a line of the input data DI.

[0094] The neural network NN2 calculates a probability distribution P for each feature F calculated by the neural network NN1. In this example, the neural network NN1 calculates features F1 to F6, and therefore the neural network NN2 calculates a probability distribution P1 corresponding to the feature F1 to a probability distribution P6 corresponding to the feature F6.

[0095] A CTC (Connectionist Temporal Classification) 80 integrates each of the calculated probability distributions, calculates a probability distribution P of a character string corresponding to the input data DI, and outputs a character string recognized from the calculated probability distribution P as output data DO.

[0096] The estimation unit 85 acquires the feature amount F calculated by the neural network NN1. The estimation unit 85 estimates, from the acquired feature amount F, a range in which an element to which a predetermined label should be assigned may exist, using the neural network NN3.

[0097] The estimation unit 85 associates each label of the output data DO recognized by the CTC 80 with each feature F. When one label in the label string of the output data DO is associated with multiple feature F, the estimation unit 85 integrates and outputs the range estimated from the multiple feature F associated with the one label. The output result output by the estimation unit 85 specifies the range of each label in the input data DI. In the example shown in the figure, range A1 specifies the range of "river", range A2 specifies the range of "cape", and range A3 specifies the range of "city".

[0098] 17 is a diagram for explaining a modified example of the operation of the character region estimation unit according to the fourth embodiment. With reference to the same figure, a modified example of the operation of character region estimation unit 120 will be explained. In the modified example of the operation of character region estimation unit 120, object detection is applied to estimate a character region.

[0099] First, input data DI is input to the neural network NN4. In this example, an example in which the input data DI is an image of a character string will be described. Specifically, an example in which the input data DI is an image of a character string "Kawasaki" handwritten horizontally from left to right will be described. The neural network NN4 is a detection DNN (Deep Neural Network). The neural network NN4 receives an image as input, and outputs a plurality of candidate rectangles R and the scores of characters corresponding to each of the candidate rectangles R.

[0100] Specifically, the neural network NN4 outputs a candidate rectangle R1, a candidate rectangle R2, a candidate rectangle R3, a candidate rectangle R4, a candidate rectangle R5, a candidate rectangle R6, and the score of the character corresponding to each candidate rectangle R. More specifically, it outputs the score "0.8" for the character "川" corresponding to the candidate rectangle R1 and the score "0.1" for the character "州", the score "0.5" for the character "り" corresponding to the candidate rectangle R2 and the score "0.2" for the character "い", the score "0.3" for the character "1" corresponding to the candidate rectangle R3 and the score "0.1" for the character "ノ", the score "0.8" for the character "崎" corresponding to the candidate rectangle R4 and the score "0.1" for the character "埼", the score "0.5" for the character "山" corresponding to the candidate rectangle R5 and the score "0.1" for the character "凸", and the score "0.7" for the character "奇" corresponding to the candidate rectangle R6 and the score "0.1" for the character "嵜".

[0101] [Summary of the Fourth Embodiment] According to the above-described embodiment, the character region estimation unit 120 includes the estimation unit 85, and based on the feature amount F obtained from the input data DI, estimates the region where the character C may exist, associates the character C with at least one of the plurality of feature amounts F, and integrates a plurality of ranges associated with one label to specify the region corresponding to each character C. By using this embodiment, efficient search by the beam search algorithm can be performed. Also, the estimation of the character region according to this embodiment can be easily implemented.

[0102] Also, according to the above-described embodiment, the character region estimation unit 120 takes an image as input and outputs a plurality of candidate rectangles R and the scores of the characters corresponding to each candidate rectangle R. By using this embodiment, the character region can be estimated with few resources.

[0103] [Fifth Embodiment] The fifth embodiment will be described with reference to Fig. 18 to Fig. 20. In the fifth embodiment, a modified example of the character-likeness map CLM generated by the character-likeness map generating unit 1321 will be described. Fig. 18 is a diagram for explaining a modified example of the character-likeness map according to the fifth embodiment. Fig. 18(A) is the character-likeness map CLM described in the first embodiment.

[0104] 18B shows a character-likeness map CLM1 which is a first modified example of the character-likeness map CLM according to the fifth embodiment. The character-likeness map CLM1 differs from the character-likeness map CLM in that the character-likeness is expressed in gradation for each region made up of a plurality of pixels. In this way, the character-likeness map CLM1 may be calculated for each predetermined range of the input image IM.

[0105] FIG. 18C shows a character-likeness map CLM2 which is a second modified example of the character-likeness map CLM according to the fifth embodiment. The character-likeness map CLM2 includes a correspondence relationship between the x-coordinate of the input image IM and the number of black pixels at each x-coordinate. That is, the character-likeness map CLM2 may be a luminance histogram. In this embodiment, since the input image IM is a character written horizontally, the X-coordinate which is the direction in which the character is written is used. If the input image IM is a character written vertically, the Y-coordinate may be used. Since the character-likeness map CLM2 uses information on the number of black pixels at each X-coordinate, the character-likeness map CLM2 can be easily created.

[0106] 18D shows a character-likeliness map CLM3 which is a third modified example of the character-likeliness map CLM according to the fifth embodiment. The character-likeliness map CLM3 is obtained by normalizing the character-likeliness map CLM2 so that the value ranges from 0 to 1.

[0107] Alternatively, the character-likeliness map CLM may be obtained by dividing the input image IM into small regions in a grid pattern, and may be based on the total number of black pixels in each small region.

[0108] 19 is a diagram showing an example of a functional configuration of a modified example of a character-likeliness map generating unit according to the fifth embodiment. With reference to the diagram, a character-likeliness map generating unit 1321A which is a modified example of the character-likeliness map generating unit 1321 will be described. The character-likeliness map generating unit 1321A differs from the character-likeliness map generating unit 1321 in that it includes a character-likeliness calculation neural network DNN.

[0109] The character likelihood calculation neural network DNN is a neural network that has been trained to predict character likelihood in advance. Fig. 20 is a diagram showing an example of input data DI and teacher data DT according to the fifth embodiment. An example of input data DI and an example of teacher data DT will be described with reference to the diagram.

[0110] In the example shown in Fig. 20, the input data DI1 corresponds to the teacher data DT1, and the input data DI2 corresponds to the teacher data DT2. The input data DI1 includes the character C-11 and the character C-12, and the teacher data DT1 includes the area AR11 corresponding to the character C-11 and the area AR12 corresponding to the character C-12. The input data DI2 includes the character C-21, the character C-22, and the character C-23, and the teacher data DT2 includes the area AR21 corresponding to the character C-21, the area AR22 corresponding to the character C-22, and the area AR23 corresponding to the character C-23.

[0111] [Summary of the fifth embodiment] According to the embodiment described above, the character-likeliness map generating unit 1321 can easily generate the character-likeliness map CLM by using the character-likeliness map CLM1, the character-likeliness map CLM2, or the character-likeliness map CLM3.

[0112] Moreover, according to the above-described embodiment, the character-likeliness map generating unit 1321 includes a character-likeliness calculation neural network DNN, and is therefore capable of generating a character-likeliness map CLM by machine learning. According to the above-described embodiment, the use of machine learning makes it possible to be resistant to noise and to prevent erroneous recognition. According to the above-described embodiment, the use of machine learning makes it possible to correctly recognize input images IM with different backgrounds.

[0113] As described above, in the embodiment, a number of modified examples have been described. Here, to the extent that combinations are possible, a number of embodiments and a number of modified examples may be combined and implemented.

[0114] The functions of the information processing device in the above-mentioned embodiment may be realized by a computer. In that case, a program for realizing the functions may be recorded in a computer-readable recording medium, and the program recorded in the recording medium may be read into a computer system and executed to realize the functions. The term "computer system" as used herein includes hardware such as an OS and peripheral devices. The term "computer-readable recording medium" refers to portable media such as flexible disks, optical magnetic disks, ROMs, CD-ROMs, DVD-ROMs, and USB memories, and storage devices such as hard disks built into computer systems. The term "computer-readable recording medium" may also include those that dynamically hold a program for a short period of time, such as a communication line when a program is transmitted via a network such as the Internet or a communication line such as a telephone line, and those that hold a program for a certain period of time, such as a volatile memory inside a computer system that is a server or client in such a case. The above-mentioned program may be for realizing a part of the above-mentioned functions, or may be capable of realizing the above-mentioned functions in combination with a program already recorded in the computer system.

[0115] According to at least one of the embodiments described above, by having a first score calculation unit, a character area estimation unit, a second score calculation unit, and a selection unit, it is possible to correctly recognize characters contained in an input image.

[0116] Although some embodiments of the present invention have been described, these embodiments are presented as examples and are not intended to limit the scope of the invention. These embodiments can be implemented in various other forms, and various omissions, substitutions, and modifications can be made without departing from the spirit of the invention. These embodiments and their modifications are included in the scope of the invention and its equivalents described in the claims, as well as in the scope and spirit of the invention. [Explanation of symbols]

[0117] 1...character recognition system, 10...character recognition device, 21...input image acquisition unit, 22...candidate character string calculation unit, 23...output unit, 24...input rule information storage unit, 110...first score calculation unit, 120...character area estimation unit, 130...second score calculation unit, 140...selection unit, 111...character recognition score calculation unit, 112...knowledge processing score calculation unit, 113...first score integration unit, 131...overlapping reading score calculation unit, 132...skipping score calculation unit, 133...second score integration unit , 1321...character-likeness map generation unit, 1322...skipping score integration unit, IM...input image, S...character string, C...character, CS...candidate character string, CA...character region, S1...first score, S2...second score, S11...character recognition score, S12...knowledge processing score, S21...overlapping reading score, S22...skipping score, SS...selected character string, IR...input rule, IAR...character input region, CLM...character-likeness map, MSK...mask, IMP...partial input image

Claims

1. a first score calculation unit that calculates a first score indicating a likelihood of a character string, the first score being a candidate for a character string included in an input image; a character region estimation unit that estimates regions of the input image corresponding to each of the characters included in the candidate character string; a second score calculation unit that calculates a second score indicating consistency of characters included in the candidate character string based on the region estimated by the character region estimation unit; a selection unit that selects one or more character strings from among the plurality of candidate character strings based on the calculated first score and the calculated second score, the input image includes a plurality of character input areas; The second score calculation unit calculates the second score based on an area corresponding to each character included in the candidate character string and the character input area. Character recognition device.

2. The first score calculation unit calculates the first score of the candidate character string included in a partial input image that is a part of the input image.

2. The character recognition device according to claim 1.

3. The second score calculation unit, The second score is calculated based on an amount of overlap between regions corresponding to the characters included in the candidate character string.

3. The character recognition device according to claim 1.

4. The second score calculation unit, The second score is calculated based on the characters included in the candidate character string and the area estimated by the character area estimation unit. The character recognition device according to any one of claims 1 to 3.

5. The second score calculation unit, Calculating the second score based on the likelihood that any character exists in the region of the input image.

5. The character recognition device according to claim 4.

6. A method for detecting a likelihood of a character string from an input image, comprising: a first score calculation step in which the computer calculates a first score indicating a likelihood of a character string for each of a plurality of candidate character strings that are candidates for a character string included in an input image; a character region estimation step of estimating, by a computer, regions of the input image corresponding to each of the characters included in the candidate character string; a second score calculation step of calculating a second score indicating consistency of characters included in the candidate character string based on the estimated region by the computer; a selection step of selecting one or more character strings from among the plurality of candidate character strings based on the calculated first score and the calculated second score, the input image includes a plurality of character input areas; The second score calculation step calculates the second score based on an area corresponding to each character included in the candidate character string and the character input area. Character recognition method.

7. On the computer, a first score calculation step of calculating a first score indicating a likelihood of a character string, the first score being a candidate for a character string included in the input image for each of a plurality of candidate character strings; a character region estimation step of estimating regions of the input image corresponding to each of the characters included in the candidate character string; a second score calculation step of calculating a second score indicating consistency of characters included in the candidate character string based on the estimated region; a selection step of selecting one or more character strings from among the plurality of candidate character strings based on the calculated first score and the calculated second score; the input image includes a plurality of character input areas; The second score calculation step calculates the second score based on an area corresponding to each character included in the candidate character string and the character input area. program.

Images