Character recognition device, character recognition method, and program

Abstract

A character recognition device according to an embodiment includes a first score calculation unit, a character region estimation unit, a second score calculation unit, and a selection unit. A first score calculation unit calculates a first score for each of a plurality of candidate character strings, which are candidates of character strings included in an input image, the first scores representing likelihood of the character strings. The character region estimation unit estimates a region corresponding to each of the characters included in the candidate character string in a region of the input image. A second score calculation unit calculates, on the basis of the estimated region, a second score indicating the matching of the characters included in the candidate character string. The selection unit selects one or more character strings from among the plurality of candidate character strings on the basis of the calculated first score and second score.

Description

Technical Field

[0001] Embodiments of the present invention relate to a character recognition device, a character recognition method, and a program. Background Art

[0002] In the past, there is a known character recognition technology for recognizing characters contained in an input image. According to such prior art, it is known that recognition accuracy can be improved by performing recognition without clearly dividing the boundaries of each character. However, when recognizing characters without dividing the boundaries of the characters, a character may be recognized repeatedly. In addition, depending on the input image, characters may be skipped and recognized. That is, according to the prior art, there is a problem that the characters contained in the input image cannot be accurately recognized.

[0003] Prior art literature

[0004] Patent Literature

[0005] Patent Document 1: Japanese Patent Application Publication No. 2013-097590 Summary of the invention

[0006] Technical problem to be solved by the invention

[0007] The technical problem to be solved by the present invention is to provide a character recognition device, a character recognition method and a program that can accurately recognize characters contained in an input image.

[0008] Means for solving technical problems

[0009] The character recognition device of the embodiment has a first score calculation unit, a character area estimation unit, a second score calculation unit and a selection unit. The first score calculation unit calculates a first score for each of the candidate character strings contained in the input image, i.e., a plurality of candidate character strings, wherein the first score indicates the likelihood of the character string. The character area estimation unit estimates the regions in the input image that respectively correspond to the characters contained in the candidate character strings. The second score calculation unit calculates a second score based on the estimated regions, wherein the second score indicates the matching of the characters contained in the candidate character strings. The selection unit selects one or more character strings from the plurality of candidate character strings based on the calculated first score and the second score. BRIEF DESCRIPTION OF THE DRAWINGS

[0010] Figure 1 This is a schematic diagram showing an example of the functional configuration of the character recognition system according to the first embodiment.

[0011] Figure 2 This is a diagram showing an example of an input image according to the first embodiment.

[0012] Figure 3This is a diagram for explaining the first score in the first embodiment.

[0013] Figure 4 This is a diagram for explaining the second score in the first embodiment.

[0014] Figure 5 This is a flowchart showing a series of operations of the character recognition system according to the first embodiment.

[0015] Figure 6 It is a diagram showing a modified example of the functional configuration of the second score calculation unit according to the first embodiment.

[0016] Figure 7 This is a diagram for explaining the repeated reading according to the first embodiment.

[0017] Figure 8 It is a diagram showing an example of the functional configuration of the skipping score calculation unit according to the first embodiment.

[0018] Fig. 9 This is a diagram for explaining the character similarity graph according to the first embodiment.

[0019] Fig.10 It is a diagram showing a modified example of the functional configuration of the first score calculation unit according to the first embodiment.

[0020] Fig.11 It is a diagram for explaining the beam search according to the second embodiment.

[0021] Fig.12 1 is a diagram for explaining the matching score of each partial input image in the beam search according to the second embodiment.

[0022] Fig.13 This is a flowchart showing a series of operations of the character recognition system according to the second embodiment.

[0023] Fig.14 is a diagram showing an example of an input image according to the third embodiment;

[0024] Fig.15 This is a diagram showing an example of the second score according to the third embodiment.

[0025] Fig.16 This is a diagram for explaining an example of the operation of the character region estimating unit according to the fourth embodiment.

[0026] Fig.17 It is a diagram for explaining a modified example of the operation of the character region estimating unit according to the fourth embodiment.

[0027] Fig.18 This is a diagram for explaining a modified example of the character similarity graph according to the fifth embodiment.

[0028] Fig.19 This is a diagram showing an example of the functional structure of a modified example of the character similarity map generation unit according to the fifth embodiment.

[0029] Fig. 20 This is a diagram showing an example of the input data and the supervision data according to the fifth embodiment.

[0030] Fig.21 This is a diagram for explaining repeated reading and skipping in the prior art. Detailed implementation manners

[0031] Hereinafter, a character recognition device, a character recognition method, and a program according to an embodiment will be described with reference to the accompanying drawings.

[0032] [Prior art]

[0033] Refer to Fig.21 to explain the problems that occur when using the prior art character recognition method. Fig.11 This is a diagram for explaining repeated reading and skipping in the prior art. Column 90 shows an example of repeated reading, and column 95 shows an example of skipping.

[0034] In an example of repeated reading, the case where the input image 91 is input is described. When the character string recorded in the input image 91 is accurately recognized, as the positions where characters are recorded, the estimation regions 92 and 93 are determined. Since "川" is recorded in the region 92 and "崎" is recorded in the region 93, according to the prior art character recognition method, it can be recognized that "川崎" is recorded in the input image 91. On the other hand, when the character string recorded in the input image 91 is incorrectly recognized, as the positions where characters are recorded, the estimation regions 92, 93, and 94 are determined. Since "川" is recorded in the region 92, "崎" is recorded in the region 93, and "山" is recorded in the region 94, according to the prior art character recognition method, it is misrecognized that "川山崎" is recorded in the input image 91. Thus, in an example of repeated reading, after recognizing the radical of "崎" as "山", "崎" is recognized again, so there is a problem of repeatedly recognizing the character "山".

[0035] Next, a case where the input image 96 is input in an example of skipping reading will be described. When the characters in the input image 96 are accurately recognized, as the positions where the characters are recorded, the estimation regions 97, 98, and 99 are obtained. "Naga" is recorded in region 97, "Tani" is recorded in region 98, and "Kawa" is recorded in region 99. Therefore, according to the character recognition method of the prior art, it is possible to recognize that "Nagatani Kawa" is recorded in the input image 96. On the other hand, when the characters in the input image 96 are erroneously recognized, as the positions where the characters are recorded, the estimation regions 97 and 98 are obtained. Since "Naga" is recorded in region 97 and "Kawa" is recorded in region 99, according to the character recognition method of the prior art, it is erroneously recognized that "Naga Kawa" is recorded in the input image 96. Thus, in an example of skipping reading, there is a problem that the character "Tani" recorded in region 98 is skipped and recognized.

[0036] [First Embodiment]

[0037] The character recognition device of the present embodiment suppresses the problems of the prior art as described above. The character recognition device of the present embodiment performs string recognition on the string recorded in the input image. String recognition refers to the task of recognizing the string in an image with an image containing the string as the input. In the present embodiment, a horizontally written string read from left to right will be described. In addition, in the present embodiment, it is not limited to a horizontally written string read from left to right, and the same applies to a vertically written string. Furthermore, an image containing a string widely includes an image of handwritten characters, a billboard photographed by a photo, a road sign, etc. In addition, in the present embodiment, a case where the number of characters included in the string is 0 or more will be described.

[0038] Figure 1 It is a schematic diagram showing an example of the functional structure of the character recognition system of the first embodiment. Referring to this figure, the functional structure of the character recognition system 1 will be described. The character recognition system 1 includes an input image acquisition unit 21, a candidate string calculation unit 22, a character recognition device 10, and an output unit 23.

[0039] The input image acquisition unit 21 acquires the input image IM. Figure 2 It is a diagram showing an example of the input image IM of the first embodiment. A string S is recorded in the input image IM. Specifically, the string S, which is the handwritten characters of "Kawasaki", is recorded in the input image IM. In the present embodiment, an example of the case where the handwritten string S is recorded in the input image IM will be described.

[0040] Return to Figure 1, the candidate string calculation unit 22 calculates a candidate string CS, which is a candidate for the string S described in the input image IM acquired by the input image acquisition unit 21. The candidate string calculation unit 22 calculates the candidate string CS using a known character recognition technique (e.g., pattern matching or feature detection, etc.). The candidate string calculation unit 22 outputs multiple strings S as the candidate string CS.

[0041] Based on the input image IM acquired by the input image acquisition unit 21 and the multiple candidate strings CS calculated by the candidate string calculation unit 22, the character recognition device 10 selects a similar string from the multiple candidate strings CS calculated by the candidate string calculation unit 22 as the selected string SS. The character recognition device 10 includes a first score calculation unit (character recognition unit) 110, a character region estimation unit 120, a second score calculation unit (region matching score calculation unit) 130, and a selection unit 140.

[0042] The output unit 23 outputs the selected string SS selected by the character recognition device 10. For example, the output unit 23 outputs information for causing a display unit (not shown) to display the selected string SS, outputs information for performing voice output from a voice output unit (not shown), or outputs the selected string SS to an information processing device (not shown) via wireless output.

[0043] The first score calculation unit 110 calculates a first score S1 for each of the multiple candidate strings CS calculated by the candidate string calculation unit 22. The candidate string CS refers to a candidate for the string S included in the input image IM. The first score S1 represents the likelihood of the string. That is, the first score calculation unit 110 calculates the first score S1 representing the likelihood of the string, that is, the first score S1 for each of the multiple candidate strings CS, which are candidates for the string S included in the input image IM.

[0044] Figure 3 This is a diagram for explaining the first score of the first embodiment. Referring to this diagram, a specific example of the first score S1 calculated by the first score calculation unit 110 will be described. In this diagram, an example of the case where the string "Kawasaki" is included in the input image IM is shown. In this example, the candidate string calculation unit 22 calculates "Kawayamaaki" as the candidate string CS-1, calculates "Kawasaki" as the candidate string CS-2, and calculates "Kawayamaqi" as the candidate string CS-3. The first score calculation unit 110 calculates the first score S1 for the candidate string CS-1, the candidate string CS-2, and the candidate string CS-3 respectively. In this example, the first score S1-1 of the candidate string CS-1 is "0.5", the first score S1-2 of the candidate string CS-2 is "0.5", and the first score S1-3 of the candidate string CS3 is "0.1".

[0045] Return to Figure 1 The character region estimation unit 120 estimates a character region CA based on the candidate string CS and the input image IM. The character region CA is a region corresponding to each character C included in the candidate string CS. That is, the character region estimation unit 120 estimates a region in the input image IM that corresponds to each character C included in the candidate string CS, namely, the character region CA.

[0046] The second score calculation unit 130 calculates a second score S2 indicating the matching degree of the characters included in the candidate string CS based on the region estimated by the character region estimation unit 120, that is, the character region CA. Here, the matching degree of the characters included in the candidate string CS refers to the spatial matching degree. In the case where there is no spatial matching degree, there are cases where each character is repeated or a character is skipped.

[0047] Figure 4 FIG. is a diagram for explaining the second score of the first embodiment. Referring to this figure, a specific example of the estimation of the character region CA by the character region estimation unit 120 and the second score S2 calculated by the second score calculation unit 130 will be described. In this figure, an example in which the string "Kawasaki" is included in the input image IM is shown. In this example, the character region estimation unit 120 estimates a region corresponding to each character C included in the candidate string CS, that is, the character region CA. For example, the character region estimation unit 120 estimates the regions corresponding to the characters C included in the candidate string CS-1, that is, "Chuan Shan Qi", as the character regions CA1-1, CA2-1, and CA3-1. In addition, the character region estimation unit 120 estimates the regions corresponding to the characters C included in the candidate string CS-2, that is, "Kawasaki", as the character regions CA1-2 and CA2-2. In addition, the character region estimation unit 120 estimates the regions corresponding to the characters C included in the candidate string CS-3, that is, "Chuan Shan Qi", as the character regions CA1-3, CA2-3, and CA3-3.

[0048] The second score calculation unit 130 calculates the second score S2 for each of the plurality of candidate strings CS. In this example, the second score S2-1 of the candidate string CS-1 is "0.1", the second score S2-2 of the candidate string CS-2 is "1.0", and the second score S2-3 of the candidate string CS3 is "1.0".

[0049] Return to Figure 1 The selection unit 140 selects one or more strings S from the plurality of candidate strings CS based on the calculated first score S1 and second score S2. For example, the selection unit 140 selects the candidate string CS whose result of multiplying the first score S1 by the second score S2 is the maximum value as the selected string SS.

[0050] Figure 5 1 is a flowchart showing a series of operations of the character recognition system according to the first embodiment. Hereinafter, a series of operations of the character recognition device 10 will be described according to the flowchart shown in the figure.

[0051] (Step S110) The input image acquisition unit 21 acquires the input image IM. The candidate character string calculation unit 22 calculates candidate character strings CS that are candidates for the character string S written in the input image IM. In this flowchart, the case where the candidate character string calculation unit 22 calculates n (n is an integer greater than or equal to 1) candidate character strings CS is described.

[0052] (Step S120) The first score calculation unit 110 calculates a first score S1 for each of the plurality of candidate character strings CS calculated. n , set the first score S1 to α n In the case of n , α n ).

[0053] (Step S130 ) The second score calculation unit 130 sets 1 to the counter i.

[0054] (Step S140) The character region estimation unit 120 estimates regions corresponding to the plurality of characters C included in the candidate character string CS in the input image IM. In this flowchart, the case where the candidate character string CS includes m characters (m is an integer greater than 1) is described. i Contains y i,1 , …, y i,m In this case, the character area estimation unit 120 estimates the character areas CA corresponding to the characters C, namely, s1, ..., s m .

[0055] (Step S150) The second score calculation unit 130 calculates the score of the candidate character string CS. i The second score S2 is also recorded as β n . As the β of the second score S2 n Based on s1,…,s m To figure out.

[0056] (Step S160) The selection unit 140 selects the first score S1 based on α i and β as the second score S2 i , calculate γ i .

[0057] (Step S170) When i<n, the second score calculation unit 130 proceeds to step S190. That is, the second score calculation unit 130 repeats the process from step S140 to step S160 until the counter i reaches n, which is the number of candidate strings CS calculated by the candidate string calculation unit 22. When i<n, that is, when the counter i reaches n, which is the number of candidate strings CS calculated by the candidate string calculation unit 22, the second score calculation unit 130 proceeds to step S180.

[0058] (Step S190 ) The second score calculation unit 130 increments the counter i and proceeds to step S140 .

[0059] (Step S180) The selection unit 140 selects γ n The largest candidate string CS is used as the selected string SS. In this flowchart, the selection unit 140 selects the selected string SS through the maximum point set. In addition, the selection unit 140 can also select the selected string SS based on α n and β n The calculation method is to select the string SS through the minimum point set selection.

[0060] [Modification of the first embodiment]

[0061] Figure 6 1 is a diagram showing a modified example of the functional configuration of the second score calculation unit of the first embodiment. Referring to this diagram, a second score calculation unit 130A as a modified example of the second score calculation unit 130 will be described. The second score calculation unit 130A includes a repeated reading score calculation unit 131, a skipped reading score calculation unit 132, and a second score merging unit 133.

[0062] The repeated reading score calculation unit 131 calculates a score indicating the amount of repetition of the candidate character string CS, namely, a repeated reading score S21. Specifically, the amount of repetition of the candidate character string CS refers to the amount of overlap between the regions corresponding to the characters C included in the candidate character string CS. The second score calculation unit 130A calculates the second score S2 based on the calculated repeated reading score S21. That is, in the present embodiment, the second score calculation unit 130A calculates the second score S2 based on the amount of overlap between the regions corresponding to the characters C included in the candidate character string CS.

[0063] Figure 7This is a diagram for explaining the repeated reading of the first embodiment. The amount of repetition calculated by the repeated reading score calculation unit 131 will be described with reference to this diagram. In this example, the character "Kawasaki" is recorded in the input image IM, and the character region estimation unit 120 estimates the character regions CA1, CA2, and CA3 as the character region CA. Here, the overlapping region of the character regions CA2 and CA3 with each other, that is, the repeated region CA-DP, is the amount of repetition. Specifically, when the amount of repetition is m(y), the repeated reading score calculation unit 131 calculates the following formula (1) as the overlapping matching score P ovlp .

[0064] [Equation 1]

[0065] P ovlp (y):=(C OP ) m(y) …(1)

[0066] Here, C OP is a constant from 0 to 1. The smaller it is, the smaller the repeated matching score P ovlp . The value of C OP can be obtained through experiments.

[0067] Return to Figure 6 , and the skip reading score calculation unit 132 calculates the skip reading score S22, which is a score indicating whether skip reading occurs, based on the characters C included in the candidate string CS and the regions estimated by the character region estimation unit 120. The second score calculation unit 130 calculates the second score S2 based on the calculated skip reading score S22. That is, the second score calculation unit 130 calculates the second score S2 based on the characters C included in the candidate string CS and the regions estimated by the character region estimation unit 120.

[0068] Figure 8 This is a diagram showing an example of the functional structure of the skip reading score calculation unit of the first embodiment. An example of the functional structure of the skip reading score calculation unit 132 will be described with reference to the accompanying drawings. The skip reading score calculation unit 132 includes a character similarity map generation unit 1321 and a skip reading score merging unit 1322. In this embodiment, the skip reading score calculation unit 132 calculates the second score S2 based on the likelihood of the presence of certain characters C in the region of the input image IM.

[0069] The character similarity map generation unit 1321 generates a character similarity map CLM. The character similarity map CLM represents the likelihood of the presence of certain characters C in the image region of the input image IM.

[0070] The skip reading score merging unit 1322 calculates the skip reading score S22 based on the character region CA estimated by the character region estimation unit 120 and the character similarity map CLM generated by the character similarity map generation unit 1321.

[0071] Fig. 9 This is a diagram for explaining the character similarity map of the first embodiment. With reference to this diagram, an outline of the processing performed by the character similarity map CLM and the skip reading score merging unit 1322 will be described.

[0072] Fig. 9 The (A) of is a diagram showing the character region CA in the case where the character region CA is estimated by the character region estimation unit 120 for the candidate string CS for which skip reading is performed. In this diagram, for the character "長" in the string S, i.e., "長谷川", included in the input image IM, the character region CA1 is estimated, and for the character "川", the character region CA2 is estimated. For the character "谷", the character region CA is not estimated. That is, skip reading occurs.

[0073] Fig. 9 The (B) of is a diagram showing an example of the character similarity map CLM. In the example shown in this diagram, there is a high probability of characters existing in the regions AR1, AR2, and AR3. That is, the character similarity map generation unit 1321 generates the likelihood of certain characters C existing in the image region of the input image IM as the character similarity map CLM. Therefore, for the string S that also includes the characters where skip reading occurs, the likelihood of the existence of the characters C is estimated.

[0074] Fig. 9 The (C) of is a diagram showing an example of the mask MSK. The skip reading score merging unit 1322 generates the mask MSK based on the character region CA estimated by the character region estimation unit 120. The mask MSK represents the region where the characters included in the candidate string CS exist or do not exist. The skip reading score merging unit 1322 filters the character similarity map CLM with the generated mask MSK. The skip reading score merging unit 1322 estimates the region where there is a high probability of characters existing even though they are not included in the candidate string CS by filtering the character similarity map CLM.

[0075] Fig. 9 The (D) of is a diagram showing the character similarity map CLM filtered by the skip reading score merging unit 1322. The region AR2 is a region where there is a high probability of characters existing but is not included in the candidate string CS. That is, it can be said that the larger the region AR2, the higher the possibility of skip reading.

[0076] Here, when there is a high probability of certain characters C existing in the image region of the input image IM, the region included in the filtered character similarity map CLM is taken as U j (y). When the image region of the input image IM is divided into a width W and a height H, the skip reading score merging unit 1322 calculates the following formula (2) as the skip reading matching score P SKIP(y). In addition, the image area of the input image IM can be divided in units of pixels of the input image IM, or can be divided in units of a predetermined range composed of a plurality of pixels.

[0077] [Formula 2]

[0078]

[0079] Here, C SP is a constant of 0 or more. The larger C SP is, the smaller the skip matching score P SKIP . The value of C SP can be obtained through experiments. In addition, when no skip penalty is imposed, C SP can also be set to 0.

[0080] Return to Figure 6 . The second score combining unit 133 calculates the second score S2 based on the repeated reading score S21 calculated by the repeated reading score calculation unit 131 and the skip reading score S22 calculated by the skip reading score calculation unit 132. For example, the second score combining unit 133 calculates the value obtained by multiplying the repeated reading score S21 by the skip reading score S22 as the second score S2.

[0081] Fig.10 is a diagram showing a modified example of the functional structure of the first score calculation unit of the first embodiment. Referring to this diagram, the first score calculation unit 110A, which is a modified example of the first score calculation unit 110, will be described. The first score calculation unit 110A includes a character recognition score calculation unit 111, a knowledge processing score calculation unit 112, and a first score combining unit 113.

[0082] The character recognition score calculation unit 111 calculates the character recognition score S11 for each candidate string CS. The character recognition score S11 represents the likelihood of the string.

[0083] The knowledge processing score calculation unit 112 calculates the knowledge processing score S12 for each candidate string CS. The knowledge processing score calculation unit 112 is used when the candidate string CS to be recorded in the input image IM is limited. The case where the candidate string CS to be recorded in the input image IM is limited means, for example, the case where information such as the input image IM being a postal code, an address, a name, etc. is obtained in advance. When it is known that the input image IM is a postal code, if the candidate string CS is not a number, the knowledge processing score S12 is calculated to be low. In addition, when it is known that the input image IM is an address, if it is "Kawasaki" compared to "Kawayamaqi", the knowledge processing score S12 is calculated to be low.

[0084] The first score merging unit 113 calculates a first score S1 based on the character recognition score S11 calculated by the character recognition score calculating unit 111 and the knowledge processing score S12 calculated by the knowledge processing score calculating unit 112. The selecting unit 140 selects a selection character string SS based on the calculated first score S1 and second score S2.

[0085] Here, an example is described in which the selection unit 140 selects the selected character string SS based on the character recognition score S11, the knowledge processing score S12, the repeated reading score S21, and the skipped reading score S22. In this case, the selection unit 140 selects the selected character string SS based on the following formula (3).

[0086] [Formula 3]

[0087]

[0088] Specifically, the selection unit 140 selects the character recognition score S11, that is, P OCR , knowledge processing score S12, namely P LM , repeated reading score S21 is P ovlp , Skip reading score S22 is P SKIP The candidate character string CS having the largest value obtained by the multiplication is set as the selected character string SS.

[0089] [Summary of the First Embodiment]

[0090] According to the above-mentioned embodiment, the character recognition device 10 is provided with a first score calculation unit 110, which calculates a first score S1 indicating the likelihood of the character string S for each candidate character string CS, is provided with a character region estimation unit 120, which estimates the region of each character C included in the character string S, is provided with a second score calculation unit 130, which calculates a second score S2 indicating the matching of the character C, and is provided with a selection unit 140, which selects the character string SS based on the first score S1 and the second score S2. That is, according to the above-mentioned embodiment, the maximum likelihood character string is selected in consideration of the matching of the region where the character C exists. Therefore, the character recognition device 10 can accurately perform character recognition on the character C included in the input image IM.

[0091] In addition, according to the above-mentioned embodiment, the second score calculation unit 130 calculates the second score S2 based on the repeated reading score S21. The repeated reading score S21 refers to the score corresponding to the amount of overlap between the regions corresponding to the characters C included in the auxiliary character string CS. Therefore, according to this embodiment, the character recognition device 10 can suppress repeated reading, and thus can accurately recognize the characters C included in the input image IM.

[0092] In addition, according to the above-mentioned embodiment, the second score calculation unit 130 calculates the second score S2 based on the skip score S22. The skip score S22 refers to a score based on the character C included in the candidate character string CS and the character area CA estimated by the character area estimation unit 120. When skip occurs, the penalty given becomes larger. Therefore, according to this embodiment, the character recognition device 10 can suppress skip, and thus can accurately recognize the character C included in the input image IM.

[0093] Here, according to the prior art, there is an antimony relationship between the improvement of repeated reading and the improvement of skip reading. If one is improved, the problem of the other is likely to occur. According to the above-mentioned embodiment, the repeated reading score S21 and the skip reading score S22 are calculated separately, and the selection character string SS is comprehensively selected, so that both the problem of repeated reading and the problem of skip reading can be improved.

[0094] According to the above-described embodiment, the second score calculation unit 130 calculates the skip score S22 by using the character similarity map CLM. The character similarity map CLM indicates the likelihood that a certain character C exists in a region of the input image IM. According to this embodiment, skip can be easily suppressed.

[0095] [Second Embodiment]

[0096] Reference Figures 11 to 13 , an example of a character recognition device 10A of the second embodiment is described. The character recognition device 10A of the second embodiment uses a beam search algorithm to perform character recognition of a string S contained in an input image IM. Here, for each of the multiple characters C contained in the string S, a candidate character C is calculated, and when the combination of candidates for each character C is set as a candidate string CS, there is a problem that the more characters C contained in the string S, the more candidate strings CS there are. If the number of candidate strings CS increases, it will take time and resources to select the string SS. Therefore, in the present embodiment, the purpose is to perform character recognition with less time and resources by using a beam search algorithm.

[0097] Fig.11 This is a diagram for explaining the beam search of the second embodiment. The beam search of the second embodiment is explained with reference to this diagram. In this embodiment, the character recognition device 10A divides the input image IM into a plurality of partial input images IMP to perform character recognition. In the example shown in this figure, the input image IM is divided into partial input image IMP-1, partial input image IMP-2, and partial input image IMP-3. The partial input image IMP may also be divided, for example, according to a specified number of pixels. The specified number of pixels may also be determined based on the width of the recorded character C.

[0098] Specifically, first, the character recognition device 10A selects one or more selection strings SS for a partial input image IMP-1 in the input image IM. Next, the character recognition device 10A selects one or more selection strings SS for the partial input image IMP-1 and the partial input image IMP-2. At this time, regarding the partial input image IMP-1, one or more selection strings SS have already been selected, so the candidate strings CS for the partial input image IMP-1 and the partial input image IMP-2 become fewer. In addition, the character recognition device 10A selects a final selection string SS for the partial input image IMP-1, the partial input image IMP-2, and the partial input image IMP-3. At this time, regarding the partial input image IMP-1 and the partial input image IMP-2, one or more selection strings SS have already been selected, so the candidate strings CS for the partial input image IMP-1 and the partial input image IMP-2, and the partial input image IMP-3 become fewer. Thus, in the present embodiment, by reducing the strings S that become candidates for each partial input image IMP, the overall processing time is shortened.

[0099] Fig.12 It is a diagram for explaining the matching score of each partial input image in the beam search of the second embodiment. Referring to this diagram, the matching score of each partial input image IMP in the case where "Kawasaki City" is described in the input image IM will be described. Fig.12 (A) thereof shows the correspondence between the candidate strings CS and the matching scores when the character recognition device 10A performs character recognition on the partial input image IMP-1, Fig.12 and (B) thereof shows the correspondence between the candidate strings CS and the matching scores when the character recognition device 10A performs character recognition on the partial input image IMP-1 and the partial input image IMP-2. Here, the matching score is a score used when the selection unit 140 selects the selection string SS, and is, for example, a score obtained by multiplying the first score S1 by the second score S2.

[0100] In Fig.12 In (A) thereof, the character recognition device 10A calculates "Kawa" as the candidate string CS-11, "Kawa1" as the candidate string CS-12, and "Nori" as the candidate string CS-13. The matching scores of the respective candidate strings CS are "1.0", "0.3", and "1.0". The character recognition device 10A selects the candidate strings CS-11 and CS-13, which are similar strings, as the selection string SS. In other words, the character recognition device 10A excludes the candidate string C12 from the candidates.

[0101] In Fig.12In (B) of this, the character recognition device 10A calculates "Chuan Shanqi" as the candidate string CS-21, calculates "Kawasaki" as the candidate string CS-22, calculates "Nori Shanqi" as the candidate string CS-23, calculates "Noriqi" as the candidate string CS-24, calculates "Chuan Shanqi" as the candidate string CS-25, and calculates "Nori Shanqi" as the candidate string CS-26. The matching scores of the respective candidate strings CS are "0.1", "1.0", "0.1", "1.0", "1.0", and "1.0". The character recognition device 10A selects the candidate strings CS-22, candidate string CS-24, candidate string CS-25, and candidate string CS-26 as the selected strings SS as the similar strings. In other words, the character recognition device 10A excludes the candidate string C-21 and the candidate string CS-23 from the candidates. Here, in the study of the partial input image IMP-1, since "Chuan 1" as the candidate string CS-12 is excluded from the candidates, in the study of the partial input image IMP-1 and the partial input image IMP-2, the strings S that become candidates can be reduced.

[0102] Fig.13 is a flowchart showing a series of operations of the character recognition system according to the second embodiment. Referring to this figure, a series of operations of the character recognition system 1A according to the second embodiment will be described. Step S100 is the same as the operation of the character recognition system according to the first embodiment described in Figure 5 and thus the description thereof is omitted.

[0103] (Step S210) The character recognition device 10A sets x to δ. δ is a prescribed integer representing the range of the partial input image IMP. x represents the range in which the character recognition device 10A performs character recognition. In this flowchart, the character recognition device 10A first calculates the candidate string CS for the range from 0 to x. Here, the range in which the character recognition device 10A performs character recognition, that is, x, corresponds to the partial input image IMP in an example described with reference to Fig.11 the description.

[0104] (Step S220) The character recognition device 10A sets the candidate set Φ to an empty set (null set).

[0105] (Step S230) The first score calculation unit 110 included in the character recognition device 10A calculates the first score S1 for each of the plurality of candidate strings CS included in the partial input image IMP. That is, when the candidate string CS is set to y n and the first score S1 is set to α n the first score calculation unit 110 calculates (y1, α1)…(y n , α n).

[0106] (Step S240) The character recognition device 10A selects the selected character string SS in the partial input image IMP. Specifically, the character recognition device 10A selects γ i Big R y i With γ i The set of is taken as the candidate set Φ. R is the number of character strings that are candidates when character recognition is performed on the next partial input image IMP. If R is reduced, the processing time can be shortened, but if it is too small, the possibility of misrecognition increases.

[0107] (Step S250) The character recognition device 10A determines whether character recognition has been performed on the entire input image IM. Specifically, when x is less than W, the character recognition device 10A proceeds to step S270. When x is not less than W, the character recognition device 10A proceeds to step S260.

[0108] (Step S270 ) The character recognition device 10A expands the range in which character recognition is performed. Specifically, the character recognition device 10A sets the value obtained by adding δ to x as x, and advances the process to step S230 .

[0109] (Step S260) The character recognition device 10A outputs γ k The largest string y k As the selection string SS.

[0110] [Summary of the Second Embodiment]

[0111] According to the above-mentioned embodiment, the first score calculation unit 110 of the character recognition device 10A calculates the first score S1 for the partial input image IMP as a part of the input image IM. In other words, the first score calculation unit 110 calculates the first score S1 of the candidate string CS, which is a candidate of the string S that includes a part of the multiple characters C constituting the string S included in the input image IM. In addition, the second score calculation unit 130 of the character recognition device 10A calculates the second score S2 for the partial input image IMP as a part of the input image IM. In other words, the second score calculation unit 130 calculates the second score S2 of the candidate string CS, which is a candidate of the string S that includes a part of the multiple characters C constituting the string S included in the input image IM. The character recognition device 10A calculates the candidate string CS for each part of the input image IM, thereby reducing the number of candidates for the entire string included in the input image IM. Therefore, according to the present embodiment, by using the beam search algorithm, character recognition can be performed with less time and resources.

[0112] [Third Embodiment]

[0113] Reference will be made to Fig.14 and Fig.15 to describe an example of the character recognition device 10B according to the third embodiment. In the third embodiment, it is different from other embodiments in that the interval of the character serving as a reference or the area of the character to be recorded is determined in the input image IM. In the present embodiment, the purpose is to perform character recognition based on the interval of the character serving as a reference or the area of the character to be recorded, thereby more accurately performing character recognition on the character C included in the input image IM.

[0114] Fig.14 FIG. is a diagram showing an example of the input image IM according to the third embodiment. Referring to this figure, the interval of the character serving as a reference or the area of the character to be recorded determined in the input image IM will be described. Fig.14 The (A) of is an example of the input image IM in the present embodiment. Fig.14 The (B) of is an example of the case where characters are recorded in the input image IM in the present embodiment.

[0115] Fig.14 The input image IM shown in the (A) of includes a plurality of character input areas IAR. Specifically, the input image IM includes a character input area IAR1, a character input area IAR2, and a character input area IAR3. The character input area IAR is provided to the user who records the character string S in the input image IM as a reference when recording characters. That is, the interval of the character serving as a reference or the area of the character to be recorded is determined by the character input area IAR. In the following description, the character input area IAR may sometimes be referred to as a "frame".

[0116] In Fig.14 The characters C are recorded in the input image IM shown in the (B) of. Specifically, the character C-1 is recorded in the character input area IAR1, the character C-2 is recorded in the character input area IAR2, and the character C-3 is recorded in the character input area IAR3. The character C-1 is "川", the character C-2 is "崎", and the character C-3 is "市".

[0117] Fig.15 FIG. is a diagram showing an example of the second score according to the third embodiment. Referring to this figure, the second score S2 calculated by the second score calculation unit 130 will be described. In the example shown in this figure, the candidate string calculation unit 22 calculates candidate strings CS of "川山奇市" and "川崎市".

[0118] When the candidate string CS is "Chuanshanqi City", the character input area IAR1 contains the character area CA1-1, the character input area IAR2 contains the character areas CA2-1 and CA3-1, and the character input area IAR3 contains the character area CA4-1. In this case, since the character input area IAR2 contains the character areas CA2-1 and CA3-1, there are two character areas CA in one box (character input area IAR). In this case, when there are multiple character areas CA in one box, the second score calculation unit 130 sets the duplication amount of the smaller character area CA and the box area as m(y), and calculates the matching score P based on the following formula (4). BOX 。

[0119] [Formula 4]

[0120] P BOX (y):=(C BP ) m(y) …(4)

[0121] Here, C BP is a constant from 0 to 1. The smaller it is, the smaller the matching score P BOX . The value of C BP can be obtained through experiments.

[0122] Here, an example of the case where the selection unit 140 further selects the selected string SS based on the matching score P BOX is described. In this case, the selection unit 140 selects the selected string SS based on the following formula (5).

[0123] [Formula 5]

[0124]

[0125] That is, in the present embodiment, the second score calculation unit 130 calculates the second score S2 based on the character area CA and the character input area IAR corresponding to the characters C included in the candidate string CS respectively.

[0126] [Summary of the Third Embodiment]

[0127] According to the above-mentioned embodiment, the character recognition device 10B calculates the second score S2 based on the character area CA and the character input area IAR by having a second score calculation unit 130. For example, when a plurality of characters are included in one frame, the second score calculation unit 130 calculates the value of the second score S2 to be low. The second score calculation unit 130 calculates the value of the second score S2 to be low when a plurality of characters are included in one frame, thereby suppressing erroneous recognition such as character recognition by dividing the character into the left half and the right half, etc. Therefore, according to this embodiment, the character C included in the input image IM can be recognized more accurately.

[0128] [Fourth Embodiment]

[0129] Reference Fig.16 and Fig.17 A fourth embodiment will be described. In the fourth embodiment, a specific example of the character region estimation unit 120 will be described. Fig.16 An example of the character region estimation unit 120 will be described below. Fig.17 A modification example of the character region estimating unit 120 will be described.

[0130] Fig.16 4 is a diagram for explaining an example of the operation of the character region estimation unit according to the fourth embodiment. Figure 1 Next, an example of the character region estimation unit 120 is described. First, input data DI is input to the neural network NN1. In this example, an example of a case where the input data DI is an image of a character string is described. Specifically, an example of a case where the input data DI is an image of a character string "Kawasaki City" and handwritten horizontally from left to right is described.

[0131] The neural network NN1 calculates a sequence of feature quantities F of the input character string. When the input data DI is a character string image handwritten horizontally from left to right, the neural network NN1 recognizes a sequence of feature quantities F from the left direction to the right direction by a width of a determination range. In this example, the neural network NN1 calculates feature quantities from feature quantity F1 to feature quantity F6. Here, the neural network NN1 calculates feature quantities F of a number corresponding to the length of a line of the input data DI.

[0132] Neural network NN2 calculates probability distribution P for each feature quantity F calculated by neural network NN1. In this example, neural network NN1 calculates feature quantities from feature quantity F1 to feature quantity F6, so neural network NN2 calculates probability distribution P6 corresponding to feature quantity F6 based on probability distribution P1 corresponding to feature quantity F1.

[0133] The CTC (Connectionist Temporal Classification) 80 merges the calculated probability distributions, calculates the probability distribution P of the string corresponding to the input data DI, and outputs the string recognized based on the calculated probability distribution P as the output data DO.

[0134] The estimation unit 85 obtains the feature quantity F calculated by the neural network NN1. The estimation unit 85 estimates the range where the element to which a prescribed label should be assigned may exist based on the obtained feature quantity F through the neural network NN3.

[0135] The estimation unit 85 associates the label of each output data DO recognized by the CTC 80 with the respective feature quantity F. When one label in the label column of the output data DO is associated with a plurality of feature quantities F, the estimation unit 85 merges and outputs the ranges estimated based on the plurality of feature quantities F associated with the one label. The output result output by the estimation unit 85 determines the range of each label in the input data DI. In an example shown in this figure, the range A1 determines the range of "川", the range A2 determines the range of "崎", and the range A3 determines the range of "市".

[0136] Fig.17 It is a diagram for explaining a modified example of the operation of the character region estimation unit according to the fourth embodiment. Referring to this diagram, a modified example of the operation of the character region estimation unit 120 will be described. In the modified example of the operation of the character region estimation unit 120, object detection is applied to estimate the character region.

[0137] First, the input data DI is input to the neural network NN4. In this example, an example of the case where the input data DI is an image of a string is described. Specifically, an example of the case where the input data DI is an image of a horizontally handwritten string "川崎" from left to right is described. The neural network NN4 is a detection DNN (Deep Neural Network). The neural network NN4 takes an image as input and outputs a plurality of candidate rectangles R and the scores of the characters corresponding to the respective candidate rectangles R.

[0138] Specifically, the neural network NN4 outputs candidate rectangles R1, R2, R3, R4, R5, R6, and the scores of the characters corresponding to each candidate rectangle R. More specifically, it outputs the score "0.8" that the character corresponding to candidate rectangle R1 is "川", the score "0.1" that it is the character "州", the score "0.5" that the character corresponding to candidate rectangle R2 is "り", the score "0.2" that it is the character "い", the score "0.3" that the character corresponding to candidate rectangle R3 is "1", the score "0.1" that it is the character "ノ", the score "0.8" that the character corresponding to candidate rectangle R4 is "崎", the score "0.1" that it is the character "埼", the score "0.5" that the character corresponding to candidate rectangle R5 is "山", the score "0.1" that it is the character "凸", the score "0.7" that the character corresponding to candidate rectangle R6 is "奇", and the score "0.1" that it is the character "嵜".

[0139] [Summary of the Fourth Embodiment]

[0140] According to the above-described embodiment, the character region estimation unit 120, by including the estimation unit 85, estimates the region where the character C may exist based on the feature amount F obtained from the input data DI, associates the character C with at least one of the plurality of feature amounts F, and merges the plurality of ranges corresponding to one label, thereby determining the region corresponding to each character C. By using this embodiment, an efficient search based on the beam search algorithm can be performed. In addition, the estimation of the character region of this embodiment can be easily installed.

[0141] In addition, according to the above-described embodiment, the character region estimation unit 120 takes an image as input and outputs a plurality of candidate rectangles R and the scores of the characters corresponding to each candidate rectangle R. By using this embodiment, the estimation of the character region can be performed with fewer resources.

[0142] [Fifth Embodiment]

[0143] Refer to Figures 18 to 20 , and the fifth embodiment will be described. In the fifth embodiment, a modified example of the character similarity map CLM generated by the character similarity map generation unit 1321 will be described. Fig.18 It is a diagram for explaining a modified example of the character similarity map of the fifth embodiment. Fig.18 (A) of is the character similarity map CLM described in the first embodiment.

[0144] Fig.18(B) is a character similarity map CLM1 as a first variant of the character similarity map CLM of the fifth embodiment. The character similarity map CLM1 is different from the character similarity map CLM in that the character similarity is expressed in grayscale for each region composed of a plurality of pixels. Thus, the character similarity map CLM1 can also calculate the character similarity for each specified range in the input image IM.

[0145] Fig.18 (C) is a character similarity map CLM2 which is a second variant of the character similarity map CLM of the fifth embodiment. The character similarity map CLM2 includes the correspondence between the x-coordinate of the input image IM and the number of black pixels in each x-coordinate. That is, the character similarity map CLM2 can be a brightness histogram. In the present embodiment, the input image IM is a horizontally written character, so the direction in which the character is recorded, that is, the X-coordinate is used. In the case where the input image IM is a vertically written character, the Y-coordinate can also be used. The character similarity map CLM2 uses the information of the number of black pixels in each X-coordinate, so the character similarity map CLM2 can be easily produced.

[0146] Fig.18 (D) is a character similarity map CLM3 as a third modification example of the character similarity map CLM of the fifth embodiment. The character similarity map CLM3 is normalized so that the character similarity map CLM2 has values ​​from 0 to 1.

[0147] Furthermore, the character similarity map CLM may be a map obtained by dividing the input image IM into small grid-like regions and based on the total number of black pixels in each small region.

[0148] Fig.19 1 is a diagram showing an example of the functional structure of a modified example of the character similarity graph generating unit of the fifth embodiment. With reference to this diagram, a character similarity graph generating unit 1321A as a modified example of the character similarity graph generating unit 1321 is described. The character similarity graph generating unit 1321A is different from the character similarity graph generating unit 1321 in that it includes a character similarity calculation neural network DNN.

[0149] The character similarity calculation neural network DNN is a neural network that has been pre-learned to be able to predict character similarity. Fig. 20 1 is a diagram showing an example of input data DI and supervisory data DT according to the fifth embodiment. An example of input data DI and an example of supervisory data DT will be described with reference to the drawings.

[0150] exist Fig. 20In the example shown, input data DI1 corresponds to supervisory data DT1, and input data DI2 corresponds to supervisory data DT2. Input data DI1 includes character C-11 and character C12, and training data DT1 includes area AR11 corresponding to character C-11 and area AR12 corresponding to character C12. Input data DI2 includes character C-21, character C-22, and character C-23, and supervisory data DT2 includes area AR21 corresponding to character C-21, area AR22 corresponding to character C-22, and area AR23 corresponding to character C-23.

[0151] [Summary of the Fifth Embodiment]

[0152] According to the above-described embodiment, the character similarity map CLM1 , the character similarity map CLM2 , or the character similarity map CLM3 is used, so that the character similarity map generation unit 1321 can easily generate the character similarity map CLM.

[0153] In addition, according to the above-mentioned embodiment, the character similarity graph generation unit 1321 is provided with a character similarity calculation neural network DNN, thereby being able to generate a character similarity graph CLM through machine learning. According to the above-mentioned embodiment, since machine learning is used, it is possible to resist noise and suppress misrecognition. In addition, according to the above-mentioned embodiment, since machine learning is used, input images IM with different backgrounds can also be accurately recognized.

[0154] As described above, a plurality of modified examples are described in the embodiment. Here, as long as they can be combined, a plurality of embodiments and a plurality of modified examples may be combined and implemented.

[0155] In addition, the function of the information processing device in the above-mentioned embodiment can also be realized by a computer. In this case, it can also be realized by recording the program for realizing the function in a computer-readable recording medium, so that the computer system reads and executes the program recorded in the recording medium. In addition, the "computer system" mentioned here includes hardware such as OS and peripheral devices. In addition, "computer-readable recording medium" refers to removable media such as floppy disks, optical magnetic disks, ROMs, CD-ROMs, DVD-ROMs, USB memories, and storage devices such as hard disks built into computer systems. In addition, "computer-readable recording medium" can also include a medium that dynamically maintains the program in a short time, such as a communication line in the case of sending a program via a network such as the Internet or a communication line such as a telephone line, and a medium that maintains the program for a certain period of time, such as a volatile memory inside the computer system of a server or client in this case. In addition, the above-mentioned program can also be used to realize part of the above-mentioned function, and the above-mentioned function can also be realized by a combination with a program already recorded in a computer system.

[0156] According to at least one of the above-described embodiments, by including the first score calculation unit, the character region estimation unit, the second score calculation unit, and the selection unit, it is possible to accurately perform character recognition on characters included in an input image.

[0157] Several embodiments of the present invention have been described, but these embodiments are presented as examples and are not intended to limit the scope of the invention. These embodiments can be implemented in various other ways, and various omissions, substitutions, and changes can be made without departing from the scope of the subject matter of the invention. These embodiments and their variations are included in the scope or subject matter of the invention, and are included in the invention described in the claims and the scope of their equivalents.

[0158] [Explanation of Reference Numerals]

[0159] 1…character recognition system, 10…character recognition device, 21…input image acquisition unit, 22…candidate character string calculation unit, 23…output unit, 24…input rule information storage unit, 110…first score calculation unit, 120…character area estimation unit, 130…second score calculation unit, 140…selection unit, 111…character recognition score calculation unit, 112…knowledge processing score calculation unit, 113…first score merging unit, 131…repeated reading score calculation unit, 132…skip reading score calculation unit, 133…second score merging unit merging part, 1321…character similarity graph generating part, 1322…skip score merging part, IM…input image, S…character string, C…character, CS…candidate character string, CA…character area, S1…first score, S2…second score, S11…character recognition score, S12…knowledge processing score, S21…repeat reading score, S22…skip score, SS…select character string, IR…input rule, IAR…character input area, CLM…character similarity graph, MSK…mask, IMP…partial input image

Claims

1. A character recognition device comprising: A first score calculation unit calculates a first score for each of a plurality of candidate character strings that are candidates for the character string included in the input image, wherein the first score indicates a likelihood of the character string; a character region estimating unit that estimates regions in the region of the input image that respectively correspond to characters included in the candidate character string; a second score calculation unit that calculates a second score indicating the matching of characters included in the candidate character string based on the region estimated by the character region estimation unit; as well as The selection unit selects one or more character strings from among the plurality of candidate character strings based on the calculated first score and the second score.

2. The character recognition device according to claim 1, wherein: The first score calculation unit calculates the first score of the candidate character string included in a partial input image which is a part of the input image.

3. The character recognition device according to claim 1 or 2, wherein: The second score calculation unit calculates the second score based on the amount of overlap between regions corresponding to characters included in the candidate character string.

4. The character recognition device according to any one of claims 1 to 3, wherein: The second score calculation unit calculates the second score based on characters included in the candidate character string and the region estimated by the character region estimation unit.

5. The character recognition device according to claim 4, wherein: The second score calculation section calculates the second score based on the likelihood that certain characters exist in the region of the input image.

6. The character recognition device according to any one of claims 1 to 5, wherein: The input image includes a plurality of character input areas. The second score calculation unit calculates the second score based on regions corresponding to characters included in the candidate character string and the character input region.

7. A character recognition method comprising the following steps: A first score calculation step of calculating a first score for each of a plurality of candidate character strings that are candidates for the character string included in the input image, wherein the first score indicates a likelihood of the character string; a character region estimating step of estimating regions in the region of the input image corresponding to the characters included in the candidate character string; a second score calculation step of calculating a second score based on the estimated area, the second score indicating the matching of characters included in the candidate character string; as well as A selecting step is to select one or more character strings from among the plurality of candidate character strings based on the calculated first score and the second score.

8. A program for causing a computer to perform the following steps: a first score calculation step of calculating a first score of each of the candidate strings contained in the input image, that is, a plurality of candidate strings, wherein the first score indicates the likelihood of the string; A character region estimation step of estimating regions in the input image that correspond to the characters included in the candidate character string; a second score calculation step of calculating a second score based on the estimated region, wherein the second score indicates the matching of characters included in the candidate character string; as well as A selecting step of selecting one or more character strings from among the plurality of candidate character strings based on the calculated first score and the second score.

Images