Determination device, determination method, and determination program

Abstract

To determine a category to which a character string included in an image belongs, with high accuracy.SOLUTION: A calculation unit 20B of a determination apparatus 10 calculates category score information that represents a likelihood score belonging to a category for each element region of an image. An extraction unit 20C extracts a character string region included in the image. A determination unit 20D determines a category to which a character string included in the character string region belongs based on the category score information. The calculation unit 20B calculates the category score information using an NNW in which the image and expectation category information are input and the category score information is output. The NNW outputs the category score information such that a score of an expectation category included in the expectation category information becomes higher.SELECTED DRAWING: Figure 1

Description

[Technical Field]

[0001] The present invention relates to a determination device, a determination method, and a determination program. [Background technology]

[0002] There are known techniques for determining a category to which a character string included in an image belongs. For example, a technique has been disclosed in which a character string included in an image is extracted and a character recognition result of the extracted character string is used to determine a category to which the character string belongs.

[0003] However, in the conventional technology, when a character string has multiple meanings, it may not be possible to narrow down the category to which the character string belongs to to just one category, which may result in a decrease in the accuracy of category determination. [Prior art documents] [Patent documents]

[0004] [Patent Document 1] Patent No. 4599896 [Patent Document 2] Patent No. 6795195 Summary of the Invention [Problem to be solved by the invention]

[0005] The present invention has been made in consideration of the above, and aims to provide a determination device, a determination method, and a determination program that can determine with high accuracy the category to which a character string included in an image belongs. [Means for solving the problem]

[0006] A determination device according to an embodiment includes an acquisition unit, a calculation unit, an extraction unit, and a determination unit. The acquisition unit acquires an image including a character string and expected category information including an expected category, which is a category expected to be output from among predetermined categories. The calculation unit calculates category score information representing a likelihood score of belonging to the category for each of one or more element regions of the image. The extraction unit extracts a character string region included in the image. The determination unit determines the category to which the character string included in the character string region belongs based on the category score information. The calculation unit calculates the category score information using a neural network that receives the image and the expected category information as input and outputs the category score information. The neural network outputs the category score information so as to increase the score of the expected category included in the expected category information. [Brief explanation of the drawings]

[0007] [Figure 1] FIG. 2 is a block diagram showing the configuration of a determination device. [Figure 2] FIG. 3 is a schematic diagram showing the overall flow of processing by a control unit. [Figure 3] FIG. 4 is an explanatory diagram of processing by an acquisition unit. [Figure 4] FIG. [Figure 5] Schematic diagram of training data. [Figure 6] An explanatory diagram of the internal processing of the NNW. [Figure 7] FIG. 10 is an explanatory diagram of processing by an extraction unit. [Figure 8] FIG. 4 is an explanatory diagram of a determination process performed by a determination unit. [Figure 9A] FIG. 4 is an explanatory diagram of processing by a determination unit. [Figure 9B] FIG. 4 is an explanatory diagram of processing by a determination unit. [Figure 10A] FIG. 4 is an explanatory diagram of processing by a determination unit. [Figure 10B] FIG. 4 is an explanatory diagram of processing by a determination unit. [Figure 10C] FIG. 4 is an explanatory diagram of processing by a determination unit. [Figure 11] 10 is a flowchart showing the flow of information processing. [Figure 12A] FIG. 10 is an explanatory diagram of the effect of the determination device. [Figure 12B] FIG. 10 is an explanatory diagram of the effect of the determination device. [Figure 13] FIG. 10 is an explanatory diagram of expected category information represented by a map. [Figure 14] FIG. [Figure 15] FIG. [Figure 16] Hardware configuration diagram. DETAILED DESCRIPTION OF THE INVENTION

[0008] The determination device, the determination method, and the determination program will be described in detail below with reference to the accompanying drawings.

[0009] FIG. 1 is a block diagram showing an example of the configuration of a determination device 10 according to this embodiment.

[0010] The determination device 10 is an information processing device for determining the category to which a character string included in an image belongs.

[0011] The determination device 10 includes a storage unit 12, a communication unit 14, a UI (user interface) unit 16, and a control unit 20. The storage unit 12, the communication unit 14, the UI unit 16, and the control unit 20 are communicatively connected via a bus 18 or the like.

[0012] The storage unit 12 stores various types of data. The storage unit 12 is, for example, a semiconductor memory element such as a RAM (Random Access Memory), a flash memory, a hard disk, an optical disk, etc. Note that the storage unit 12 may be a storage device provided outside the determination device 10.

[0013] The communication unit 14 is a communication interface that communicates with an external information processing device via a network or the like.

[0014] The UI unit 16 has a reception function for receiving operation inputs from a user and a display function for displaying various information. The reception function is realized by, for example, a pointing device such as a mouse, a keyboard, etc. The display function is realized by, for example, a display. The UI unit 16 may be a touch panel that integrates the reception function and the display function.

[0015] The control unit 20 executes various information processes in the determination device 10.

[0016] FIG. 2 is a schematic diagram showing an example of the overall flow of processing by the control unit 20 of this embodiment.

[0017] The control unit 20 of this embodiment executes a process of determining, for each character string region 54 of the character string 52 included in the image 50, a category to which the character string 52 included in the character string region 54 belongs.

[0018] The image 50 is an image to be judged by the judgment device 10. An image 50A is shown as an example of the image 50 in FIG.

[0019] Image 50A includes one or more character strings 52. A character string 52 is a group of one or more characters written in a writing direction. FIG. 2 shows, as examples, a character string 52A consisting of the characters "Kawasaki City, Saiwai Ward, Horikawa-cho 72-34" and a character string 52B consisting of the characters "1234-5678." Character strings 52A and 52B are examples of character strings 52.

[0020] The character string area 54 is an area of one line of character string 52 included in the image 50. Fig. 2 shows, as an example, a character string area 54A of a character string 52A and a character string area 54B of a character string 52B.

[0021] The control unit 20 of this embodiment determines, for each character string area 54, the category to which the character string 52 included in the character string area 54 belongs. Fig. 2 shows, as an example, a state in which the category of the character string 52A included in the character string area 54A is determined to be "address," and the category of the character string 52B included in the character string area 54B is determined to be "telephone number."

[0022] The control unit 20 of this embodiment will be described in detail below.

[0023] Returning to Figure 1, we continue the explanation.

[0024] The control unit 20 includes an acquisition unit 20A, a calculation unit 20B, an extraction unit 20C, a determination unit 20D, a character recognition unit 20E, and an output control unit 20F.

[0025] The acquisition unit 20A, calculation unit 20B, extraction unit 20C, determination unit 20D, character recognition unit 20E, and output control unit 20F are realized, for example, by one or more processors. For example, each of the above units may be realized by having a processor such as a CPU (Central Processing Unit) execute a program, i.e., by software. Each of the above units may be realized by a processor such as a dedicated IC, i.e., by hardware. Each of the above units may be realized by a combination of software and hardware. When multiple processors are used, each processor may realize one of the units, or two or more of the units.

[0026] The acquisition unit 20A acquires an image 50 including a character string 52 and expected category information.

[0027] FIG. 3 is an explanatory diagram of an example of processing by the acquisition unit 20A.

[0028] The expected category information 62 is information that includes at least an expected category 62A.

[0029] The expected category 62A is a category 60 that is expected as an output from one or more predetermined categories 60. In other words, the expected category 62A is a category 60 with a high expected value to which the character string 52 included in the image 50 belongs.

[0030] The category 60 is information that indicates the classification of the meaning indicated by the character string 52. In other words, the category 60 is information that becomes a label for each group when the character string 52 is classified into multiple groups according to predetermined classification conditions. Specifically, the category 60 is, for example, an address, a name, a telephone number, a postal code, an email address, a FAX (Facsimile) number, a company name, a store name, a URL (Uniform Resource Locator), a job title, a department name, etc., but is not limited to these. In this embodiment, an example will be described in which multiple categories 60 are predetermined. At least some of the multiple categories 60 are used as the expected category 62A.

[0031] The expected category information 62 may include an expected output level 62B in addition to the expected category 62A.

[0032] The output expectation 62B is information representing an expected value of the output for the corresponding expected category 62A. One output expectation 62B is associated with one expected category 62A. In other words, the output expectation 62B is information representing the degree of certainty or degree to which it is estimated that the character string 52 included in the image 50 belongs to the expected category 62A.

[0033] In this embodiment, a form will be described as an example in which the expected category information 62 is information including one or more expected categories 62A and an expected output degree 62B corresponding to each of the one or more expected categories 62A.

[0034] The data format of the expected category information 62 is not limited. For example, the expected category information 62 is represented by a vector representing an expected category 62A and an expected output degree 62B for the expected category 62A. For example, assume that the number of predetermined categories 60 is K (K is an integer equal to or greater than 1). Then, assume that the number of expected categories 62A selected from the plurality of categories 60 is M (M is an integer equal to or greater than 1, and K≧M). In this case, the expected category information 62 is represented by, for example, a K-dimensional vector, which is the number of categories 60.

[0035] Note that, among the K categories 60, the value of a dimension corresponding to a category 60 that is not used as the expected category 62A may be represented as zero (0), for example. In the example of FIG. 3, assume that the dimension corresponding to the address is the first dimension, the dimension corresponding to the name is the second dimension, the dimension corresponding to the telephone number is the third dimension, the dimension corresponding to the postal code is the fourth dimension, the dimension corresponding to the email address is the fifth dimension, the dimension corresponding to the fax number is the sixth dimension, the dimension corresponding to the company name is the seventh dimension, the dimension corresponding to the store name is the eighth dimension, the dimension corresponding to the URL is the ninth dimension, the dimension corresponding to the job title is the tenth dimension, and the dimension corresponding to the department name is the eleventh dimension. In this case, the K-dimensional vector is [0.9, 0.1, 0.7, 0.5, 0, 0, 0, 0, 0, 0]T, where T means transpose.

[0036] The acquisition unit 20A acquires, for example, expected category information 62 input by the user from the UI unit 16. For example, the control unit 20 displays a list of a plurality of predetermined categories 60 on the UI unit 16. The user operates the UI unit 16 while visually checking the list of categories 60 displayed, to select one or more expected categories 62A that the user expects to output. The user also operates the UI unit 16 to input an output expectation 62B corresponding to each of the selected one or more expected categories 62A. When the user inputs the output expectation 62B, the output expectation 62B is expressed by a numerical value representing, for example, "0" or "1." By accepting these operations, the acquisition unit 20A acquires the expected category information 62 input by the user.

[0037] The acquisition section 20A may also acquire the expected category information 62 using a model for deriving the expected category information 62 from the image 50.

[0038] In this case, for example, a deep learning model (DNN), a model using an estimation algorithm, or the like may be used as the model.

[0039] When using a DNN as a model, it is sufficient to use a DNN that takes an image 50 as input and outputs expected category information 62.

[0040] When an estimation algorithm is used as a model, the estimation algorithm may be, for example, an algorithm that statistically estimates the expected category 62A to which the character string 52 contained in the image 50 belongs from the character recognition results of the image 50 to be estimated and multiple other images similar to the image 50.

[0041] Specifically, for example, assume that image 50 to be estimated is a form. In this case, multiple images are prepared in which other character strings 52 belonging to the same category are written in the same area as image 50. Then, assume that the character recognition results of character strings 52 included in the same character string area 54 in image 50 and the multiple images include the most character recognition results belonging to the category "address." In this case, even if the character recognition results of character strings 52 included in the same character string area 54 in image 50 belong to both the categories "address" and "name," the expected category 62A "address" is statistically estimated by estimating it as the category "address."

[0042] The estimation algorithm may also be an algorithm that estimates expected category 62A to which character string 52 belongs from the character recognition results of characters placed outside the entry field of character string 52 included in image 50 to be estimated.

[0043] Specifically, for example, assume a situation in which the characters "Name" are located outside the entry field for character string 52 in image 50 to be estimated and in the vicinity of character string 52. In this case, an estimation algorithm for estimating expected category 62A of character string 52, "Name", may be used.

[0044] Furthermore, when a model for deriving expected category information 62 from image 50 is used, a continuous value within a specific range, such as 0 or more and less than 1, may be used for output expectation 62B.

[0045] Returning to Figure 1, we continue the explanation.

[0046] The calculation unit 20B calculates the category score information.

[0047] FIG. 4 is a diagram illustrating an example of processing by the calculation unit 20B.

[0048] The calculation unit 20B calculates category score information 74 using a neural network (NNW) 70.

[0049] The category score information 74 is information that represents the likelihood score of belonging to a category 60 for each of one or more element regions 72 of the image 50. In other words, the category score information 74 is information that represents the score for each of the multiple categories 60 for each of one or more element regions 72 of the image 50.

[0050] The element region 72 is a region having a size equal to the size of a processing unit by the NNW 70. In particular, the size of the element region 72 is determined by the structure of the NNW 70. Specifically, for example, the size of the element region 72 is determined by a kernel used during processing by the NNW 70. The kernel is sometimes referred to as a filter. For example, the element region 72 corresponds to a region of the image 50 having a size equal to one or more pixels.

[0051] The NNW 70 is an NNW that receives an image 50 and expected category information 62 as input and outputs category score information 74. The NNW 70 is a deep learning model (DNN) that is configured with a neural network such as a CNN (Convolutional Neural Network), a GCN (Graph Convolutional Network), or an RNN (Recurrent Neural Network).

[0052] The NNW 70 of this embodiment outputs the category score information 74 so as to increase the score of the expected category 62A included in the expected category information 62. In detail, the NNW 70 outputs the category score information 74 so that the score of the category 60 that matches the expected category 62A included in the expected category information 62 is higher than when the score of the category 60 is output without using the expected category information 62.

[0053] In this embodiment, the expected category information 62 includes an expected category 62A and an expected output degree 62B.

[0054] For this reason, in this embodiment, the NNW 70 outputs the category score information 74 so that the score of the expected category 62A included in the expected category information 62 becomes higher as the corresponding output expectation 62B becomes higher. In other words, the NNW 70 outputs category score information 74 such that the score of the category 60 that matches the expected category 62A included in the expected category information 62 is higher than when the score of that category 60 is output without using the expected category information 62, and the value of the height corresponds to the corresponding output expectation 62B.

[0055] The NNW 70 may be trained in advance to output category score information 74 that satisfies the above relationship.

[0056] 5 is a schematic diagram of an example of training data 80 used for training the NNW 70. For example, the NNW 70 is trained in advance using a plurality of pieces of training data 80 in which the image 50 and expected category information 62 are used as input data and category score information 74 is used as training data. The training data included in the training data 80 is category score information 74 that indicates a score that is higher for the expected category 62A included in the corresponding input data than if the expected category 62A is not used.

[0057] For example, a plurality of pieces of learning data 80 that satisfy the above conditions may be prepared in advance, and the NNW 70 may be learned in advance. The determination device 10 may learn the NNW 70 in advance using the plurality of pieces of learning data 80. Alternatively, an information processing device external to the determination device 10 may learn the NNW 70 in advance using the plurality of pieces of learning data 80.

[0058] Returning to FIG. 4, the explanation will be continued. The calculation unit 20B receives the image 50 and expected category information 62 acquired by the acquisition unit 20A from the acquisition unit 20A. As described above, for example, the expected category information 62 is represented by a K-dimensional vector. In addition, in this embodiment, an example will be described in which the image 50 is a black and white image, i.e., a grayscale image.

[0059] In this case, the calculation unit 20B replicates the expected category information 62, represented by a K-dimensional vector, W×H times, which is the number of pixels contained in the image 50, and bundles them into K images to generate a W×H×(K+1) tensor 64.

[0060] W is the number of pixels in a specific arrangement direction in a two-dimensional plane that is the arrangement plane of pixels of image 50 arranged in a multidimensional space used in processing by NNW 70. H is the number of pixels in the arrangement direction perpendicular to the direction along W in the two-dimensional plane. K is the same as above.

[0061] If the image 50 is a color image expressed by RGB, the calculation unit 20B may generate a tensor 64 of W×H×(K+3).

[0062] Then, the calculation unit 20B inputs the tensor 64 to the NNW 70. Then, the calculation unit 20B acquires the category score information 74 output from the NNW 70, and calculates the category score information 74.

[0063] FIG. 6 is an explanatory diagram of an example of internal processing of the NNW 70.

[0064] The NNW 70 calculates a score representing the likelihood that each element region 72 extracted by a sliding window of a fixed size and a fixed step width belongs to one of the K categories 60. As described above, the size and step width of the element region 72 are determined by the structure of the NNW 70. Fig. 6 shows an example in which the element region 72 is a region of a size corresponding to a plurality of pixels.

[0065] The NNW 70 may calculate a score for each element region 72 by filling in a predetermined value for an area outside the frame of the image 50 in the processing space of the NNW 70. The NNW 70 then outputs category score information 74 representing the score for each of the multiple categories 60 for each of the multiple element regions 72.

[0066] Continuing the explanation, returning to Fig. 4, the NNW 70 may be any NNW that receives the image 50 and expected category information 62 as input and outputs category score information 74, and the tensor 64 may also be generated within the NNW 70.

[0067] The calculation unit 20B acquires category score information 74 as an output from the NNW 70. That is, the calculation unit 20B acquires the category score information 74, which is information representing the score for each of the multiple categories 60, for each of the multiple element regions 72 of the image 50. As described above, the expected category information 62 is represented by, for example, a K-dimensional vector. Therefore, for example, the calculation unit 20B acquires the category score information 74 represented by a W′×H′×K tensor.

[0068] W' and H' are numbers determined by the structure of the NNW 70, and W' x H' is equal to the number of all element regions 72. In this embodiment, we consider the case where W' = W and H' = H. Note that W' and H' are not limited to W' = W and H' = H.

[0069] 4 shows an example where the element region 72 is an area of a size equivalent to one pixel. FIG. 4 shows a case where a certain element region 72 has a score of "10" for category 60 "furigana" and scores of "90" for other categories. FIG. 4 also shows a case where another element region 72 has a score of "90" for category 60 "address" and a score of "10" for category 60 "name."

[0070] In addition, when the NNW 70 is a convolutional neural network (CNN), the calculation unit 20B may calculate the category score information 74 using the NNW 70 to which at least one of downsampling by pooling and a stride of two or more of the convolution operation, which is the interval at which the convolution is applied, is applied.

[0071] By using a NNW 70 to which at least one of downsampling by pooling and a stride of 2 or more for the convolution operation is applied, it is possible to make W' = W / 2 and H' = H / 2. In this case, the number of element regions 72 can be thinned to 1 / 4 compared to when W' = W and H' = H, thereby improving calculation efficiency.

[0072] Continuing the explanation, returning to Fig. 1, the extraction unit 20C extracts a character string region 54 included in the image 50.

[0073] 7 is an explanatory diagram of an example of processing by extraction unit 20C. Extraction unit 20C extracts character string region 54 included in image 50. Extraction unit 20C may extract character string region 54 included in image 50 using a known method. For example, character string region 54 included in image 50 may be extracted using the methods described in the following publicly known documents A to C.

[0074] ·Public publication A: R. Joseph, et al., You Only Look Once: Unified, Real-Time Object Detection, 2015 ([1506.02640] You Only Look Once: Unified, Real-Time Object Detection (arxiv.org)) ·Public publication B: L. Minghui, et al., Real-time Scene Text Detection with Differentiable Binarization, 2019([1911.08947] Real-time Scene Text Detection with Differentiable Binarization (arxiv.org)) Publicly known document C: JP 2020-135272 A

[0075] FIG. 7 shows an example of a state in which character string regions 54A to 54C are extracted from image 50A.

[0076] Continuing the explanation, returning to Fig. 1, the determination unit 20D determines the category 60 to which the character string 52 included in the character string region 54 extracted by the extraction unit 20C belongs, based on the category score information 74 calculated by the calculation unit 20B.

[0077] 8 is a diagram illustrating an example of a determination process performed by the determination unit 20D. The determination unit 20D determines the category 60 to which each area E corresponding to the character string area 54 extracted by the extraction unit 20C belongs, in the category score information 74 calculated by the calculation unit 20B.

[0078] For example, for each character string region 54 included in the image 50, the judgment unit 20D judges the category 60 with the largest average score for each category 60 of the element regions 72 included in the character string region 54 as the category 60 to which the character string 52 included in the character string region 54 belongs.

[0079] In detail, the determination unit 20D identifies an area E corresponding to the character string area 54 in the category score information 74. FIG. 8 shows a state in which an area Ea corresponding to the character string area 54A has been identified. The area Ea is an example of the area E. Then, the determination unit 20D identifies the scores of each of the multiple categories 60 for each of the element areas 72 included in the identified area Ea, based on the category score information 74. Furthermore, the determination unit 20D calculates an average value for each of the same categories 60 for the scores identified for each of the element areas 72 included in the area Ea. The determination unit 20D determines the category 60 with the largest calculated average value as the category 60 to which the character string 52 included in the character string area 54A corresponding to the area Ea belongs.

[0080] Specifically, the determining unit 20D determines the category 60 using the following formulas (1) to (3).

[0081] For example, the i-th character string region 54 included in the image 50 is denoted by Si, and the score for each category 60 of the element region 72 is denoted by p, which is expressed by the following formula (1).

[0082]

number

[0083] In this case, the determining unit 20D determines the category 60 to which the character string 52 included in the i-th character string area 54 belongs, using the following formula (2).

[0084]

number

[0085] In formula (2), the left side represents the category 60 to which the character string 52 included in the i-th character string region 54 belongs. In formula (2), k is the number of the category 60, and takes the value k=1, 2, ..., K. x and y are the numbers of the element region 72 included in the i-th character string region 54.

[0086] In addition, the judgment unit 20D may identify, for each character string region 54 included in the image 50, the category 60 corresponding to the highest score for each element region 72 included in the character string region 54, and judge the category 60 identified most frequently within the character string region 54 to be the category 60 to which the character string 52 included in the character string region 54 belongs.

[0087] In detail, the determination unit 20D identifies an area E corresponding to the character string area 54 in the category score information 74. Fig. 8 shows a state in which an area Ea corresponding to the character string area 54A has been identified. Then, based on the category score information 74, the determination unit 20D identifies the category 60 with the highest score for each of the element areas 72 included in the identified area Ea. Then, the determination unit 20D determines that the category 60 most frequently identified in the area Ea is the category 60 to which the character string 52 included in the character string area 54A corresponding to the area Ea belongs.

[0088] Specifically, the determining unit 20D determines that the mode of the set expressed by the following formula (3) belongs to category 60.

[0089]

number

[0090] In formula (3), k, x, y, Si, and p have the same meanings as in formulas (1) and (2) above.

[0091] In addition, for each character string area 54, which is an area of a character string 52 corresponding to an expected category 62A included in the expected category information 62, the judgment unit 20D may determine the category 60 to which the character string 52 included in the character string area 54 belongs based on the score represented by the category score information 74.

[0092] 9A and 9B are explanatory diagrams of an example of processing by the determining unit 20D.

[0093] 9A, for example, assume a situation in which acquisition unit 20A acquires image 50C including character string 52 "Toshiba Taro" and expected category information 62 including expected category 62A "first name." In this case, determination unit 20D determines category 60 based on category score information 74 for character string region 54D in image 50C, which is the region (first name region) of character string 52 "Toshiba Taro" that corresponds to expected category 62A "first name."

[0094] 9B , for example, assume a situation in which the acquiring unit 20A acquires an image 50C including the character string 52 “Toshiba Taro” and expected category information 62 including the expected category 62A “Last name” and the expected category 62A “First name.” In this case, the determining unit 20D determines the category 60 for a character string region 54D1 in the image 50C that is the region (last name region) of the character string 52 “Toshiba” corresponding to the expected category 62A “Last name” based on the category score information 74. The determining unit 20D also determines the category 60 for a character string region 54D2 in the image 50C that is the region (first name region) of the character string 52 “Taro” corresponding to the expected category 62A “First name” based on the category score information 74.

[0095] In this way, the judgment unit 20D judges the category 60 for each character string region 54, which is a region of the character string 52 corresponding to the expected category 62A included in the expected category information 62, making it possible to adjust the granularity of the judgment depending on the expected category 62A.

[0096] Note that there may be cases where the scores of the categories 60 of the element regions 72 included in the character string region 54 vary by more than a threshold value within the character string region 54. The threshold value may be determined in advance. In this case, the determination unit 20D may determine the category 60 to which each divided character string region, obtained by dividing the character string region 54, belongs according to the threshold value.

[0097] 10A to 10C are explanatory diagrams of an example of processing by the determining unit 20D.

[0098] 10A, for example, assume a situation in which the determination unit 20D determines category 60 for an area Eb corresponding to the character string area 54B extracted by the extraction unit 20C in the category score information 74. The area Eb is an example of the area E.

[0099] Also, as shown in FIG. 10B, assume that the scores of each category 60 of the element regions 72 included in region Eb change by more than a threshold within region Eb. In FIG. 10B, higher scores are indicated by darker black. For example, assume that in region Eb, the score of the category "phone number" is low from the center toward one end in the direction along W (see the direction of arrow Wa) and high from the center toward the other end (see the direction of arrow Wb). Also, assume that in region Eb, the scores of the other categories are low from the center toward the other end in the direction along W (see the direction of arrow Wb) and high from the center toward one end (see the direction of arrow Wa).

[0100] 10C is an explanatory diagram of an example of character string 52. In the state shown in FIGS. 10A and 10B, determination unit 20D divides character string region 54 into multiple divided character string regions 53 according to a threshold. For example, consider a situation in which the position in character string region 54 where the score changes by more than the threshold is the center of region Eb corresponding to character string region 54 in the direction along W. In this case, determination unit 20D divides character string region 54B at the center position where the score changes by more than the threshold, thereby dividing character string region 54B into divided character region 53A and divided character region 53B.

[0101] Then, the determination unit 20D may determine the category 60 for each of the divided character regions 53A and 53B in the same manner as described above.

[0102] The process of dividing the character string region 54 into the divided character string regions 53 may be performed by the determination unit 20D or by the extraction unit 20C.

[0103] Returning to FIG. 1 , the description will continue. Character recognition unit 20E recognizes the characters included in character string region 54 according to the category determined by determination unit 20D. Character recognition unit 20E may perform character recognition of the characters included in character string region 54 using a known method. For example, for each character string region 54 included in image 50, character recognition unit 20E may perform character recognition of the characters included in that character string region 54 using a character recognition dictionary corresponding to category 60 determined by determination unit 20D. By performing character recognition according to the determined category 60, character recognition unit 20E can perform character recognition with high accuracy.

[0104] The output control unit 20F outputs recognition result information that associates characters recognized by the character recognition unit 20E with the category 60 determined for the character string area 54 including the characters. For example, the output control unit 20F outputs the recognition result information to the UI unit 16. By visually checking the recognition result information output to the UI unit 16, the user can confirm the character recognition result and the category 60 to which the character string 52 represented by the character recognition result belongs. Note that the recognition result information may further include at least one of expected category information 62 and the image 50 acquired by the acquisition unit 20A.

[0105] The output control unit 20F may also transmit the recognition result information to an external information processing device via the communication unit 14. The output control unit 20F may also store the recognition result information in the storage unit 12.

[0106] Next, an example of the flow of information processing executed by the control unit 20 of the determination device 10 of this embodiment will be described.

[0107] FIG. 11 is a flowchart showing an example of the flow of information processing executed by the control unit 20 of the determination device 10 of this embodiment.

[0108] The acquisition unit 20A acquires the image 50 and the expected category information 62 (step S100).

[0109] The calculation unit 20B calculates category score information 74 representing a score of likelihood that each of the element regions 72 of the image 50 acquired in step S100 belongs to each of the multiple categories 60 (step S102). That is, the calculation unit 20B inputs the image 50 and expected category information 62 acquired in step S100 to the NNW 70, and acquires the category score information 74 output from the NNW 70. The calculation unit 20B calculates the category score information 74 by acquiring the category score information 74.

[0110] Extraction unit 20C extracts character string region 54 included in image 50 acquired in step S100 (step S104). Determination unit 20D determines category 60 to which character string 52 included in character string region 54 extracted by extraction unit 20C belongs, based on category score information 74 calculated in step S102 (step S106).

[0111] The character recognition unit 20E recognizes the characters included in the character string region 54 according to the category 60 determined for each character string region 54 in step S106 (step S108). The output control unit 20F outputs recognition result information that associates the characters recognized in step S108 with the category 60 determined for the character string region 54 containing the characters (step S110). Then, this routine ends.

[0112] As described above, the determination device 10 of this embodiment includes an acquisition unit 20A, a calculation unit 20B, an extraction unit 20C, and a determination unit 20D. The acquisition unit 20A acquires an image 50 including a character string 52 and expected category information 62 including an expected category 62A, which is a category 60 expected to be output from among predetermined categories 60. The calculation unit 20B calculates category score information 74 representing a likelihood score of belonging to the category 60 for each element region 72 of the image 50. The extraction unit 20C extracts a character string region 54 included in the image 50. The determination unit 20D determines the category 60 to which the character string 52 included in the character string region 54 belongs based on the category score information 74. The calculation unit 20B calculates the category score information 74 using a network network 70 that receives the image 50 and the expected category information 62 as input and outputs the category score information 74. The NNW 70 outputs the category score information 74 so that the score of the expected category 62A included in the expected category information 62 becomes high.

[0113] Here, as a conventional technique, a technique is disclosed in which a character string 52 contained in an image 50 is extracted, and the extracted character string 52 is subjected to character recognition, and then the category 60 to which the character string 52 belongs is determined using the character recognition results.

[0114] For example, a conventional technique has been disclosed in which a plurality of categories 60 are identified based on the character recognition results of a character string 52, and the category 60 with the highest priority among the identified plurality of categories 60 is determined as the category 60 to which the image 50 belongs. However, with this conventional technique, depending on the meaning indicated by the character string 52 or the content indicated by the character recognition results of the character string 52, it may not be possible to narrow down the categories 60 to which the image 50 belongs to one category 60. As a result, with the conventional technique, the accuracy of determining the category 60 may be reduced.

[0115] Furthermore, in the conventional technology, the category 60 to which the character string 52 belongs is determined using the character recognition result of the character string 52, so it is unclear which character recognition dictionary corresponding to the category 60 is to be learned, making it difficult to improve the accuracy of character recognition. As a result, in the conventional technology, there are cases where the accuracy of determining the category 60 is reduced.

[0116] On the other hand, the determination device 10 of this embodiment acquires the image 50 to be determined and expected category information 62. Then, the determination device 10 of this embodiment calculates the category score information 74 from the image 50 and the expected category information 62 using an NNW 70 that outputs category score information 74 so that the score of the expected category 62A included in the expected category information 62 is high. That is, the determination device 10 of this embodiment calculates the category score information 74 using the NNW 70 that outputs the category score information 74 so that the score of the expected category 62A is high. Then, the determination device 10 of this embodiment determines the category 60 to which the character string 52 included in the character string area 54 belongs, based on the category score information 74.

[0117] Therefore, the determination device 10 of this embodiment can determine the category 60 to which each of the character strings 52 included in the image 50 belongs so as to increase the likelihood that the character strings 52 belong to the acquired expected category 62A.

[0118] 12A and 12B are explanatory diagrams of an example of the effect of the determination device 10 of this embodiment.

[0119] As shown in FIG. 12A , for example, assume a situation in which the acquisition unit 20A acquires an image 50B including the character string 52 "Kawasaki" and expected category information 62 including the expected category 62A "Address." The image 50B is an example of an image 50. In this case, the determination device 10 of this embodiment inputs the image 50B and the expected category information 62 to the NNW 70, and obtains category score information 74 with a high score for the expected category 62A "Address" as output from the NNW 70. The determination device 10 then uses the category score information 74 to determine the category 60 to which the character string 52 "Kawasaki" in the character string area 54 belongs, thereby determining, for example, the category 60 "Address."

[0120] On the other hand, as shown in FIG. 12B , assume a situation in which the acquisition unit 20A acquires an image 50B including the character string 52 "Kawasaki" and expected category information 62 including the expected category 62A "Name." The image 50B is the same image 50 as in FIG. 12A . In this case, the determination device 10 of this embodiment inputs the image 50B and the expected category information 62 to the NNW 70, and obtains category score information 74 with a high score for the expected category 62A "Name" as output from the NNW 70. The determination device 10 then uses the category score information 74 to determine the category 60 to which the character string 52 "Kawasaki" in the character string area 54 belongs, thereby determining, for example, the category 60 "Name."

[0121] In this way, the determination device 10 of this embodiment inputs expected category information 62 including the expected category 62A, in addition to the image 50, to the NNW 70, and thereby obtains category score information 74 predicted so as to increase the likelihood score of the expected category 62A. Then, the determination device 10 determines the category 60 of the character string 52 using the category score information 74.

[0122] Therefore, in the determination device 10 of this embodiment, even for an image 50 containing a character string 52 whose meaning varies across multiple types and which is difficult to narrow down to a single category 60, the category 60 of the character string 52 can be determined with high accuracy.

[0123] Therefore, the determination device 10 of this embodiment can determine the category 60 to which the character string 52 included in the image 50 belongs with high accuracy.

[0124] Furthermore, the determination device 10 of this embodiment determines the category 60 to which the character string 52 belongs without performing character recognition on the character string 52 included in the image 50. Therefore, in addition to the above-mentioned effects, the determination device 10 of this embodiment can determine the category 60 of the character string 52 with a small amount of calculation, i.e., a low load.

[0125] (Variation 1) In the above embodiment, the expected category information 62 is described as being represented by a K-dimensional vector representing the expected category 62A and the expected output degree 62B for the expected category 62A. However, as described above, the data format of the expected category information 62 is not limited. For example, the expected category information 62 may be a map representing the expected category 62A or the expected category 62A and the expected output degree 62B for the expected category 62A for each small region of the image 50.

[0126] FIG. 13 is an explanatory diagram of an example of expected category information 62 represented by a map.

[0127] 13, expected category information 62 may be a map representing expected categories 62A or expected categories 62A and output expectations 62B for expected categories 62A for each small region 76 of image 50. The map is represented by, for example, a W×H×K tensor, where W, H, and K are as described above.

[0128] 13 shows, as an example, small regions 76A and 76B as small regions 76. The position, size, and shape of the small regions 76 are specified, for example, by a user operating the UI unit 16. Furthermore, the CNN used to derive the expected category information 62 may adjust and output the small regions 76 included in the image 50 using a known method.

[0129] In this case, calculation unit 20B receives, from acquisition unit 20A, images 50 acquired by acquisition unit 20A and expected category information 62 represented by a map. Then, similar to the above embodiment, calculation unit 20B generates a W×H×(K+1) tensor 64 by bundling the expected category information 62 represented as a W×H×K tensor into K images. Then, similar to the above embodiment, calculation unit 20B inputs tensor 64 to NNW 70.

[0130] (Variation 2) In the above embodiment, the extraction unit 20C extracts the character string region 54 from the image 50 using a known method. However, the extraction unit 20C may extract the character string region 54 from the image 50 by acquiring the character string 52B output from the NNW 70.

[0131] FIG. 14 is an explanatory diagram of an example of this modified example.

[0132] For example, as in the above embodiment, assume a situation in which the calculation unit 20B generates a W×H×(K+1) tensor 64 by duplicating the expected category information 62 W×H times, which is the number of element regions 72 included in the image 50, and bundling them into K images. Then, as in the above embodiment, assume a situation in which the calculation unit 20B inputs the tensor 64 to the NNW 70. In this case, the NNW 70 outputs category score information 74 and character string regions 54.

[0133] For example, the NNW 70 outputs a W′×H′×(K+1) tensor that represents the score of each category 60 for each element region 72 and has connected components other than the background region as each character string region 54. W′ and H′ are the same as above.

[0134] In this case, the extraction unit 20C may extract the character string region 54 by acquiring the character string region 54 output from the NNW 70.

[0135] In this way, the determination device 10 may use one NNW 70 to calculate the category score information 74 and extract the character string region 54.

[0136] By using one NNW 70 to calculate the category score information 74 and extract the character string region 54, it is possible to reduce the calculation cost.

[0137] (Variation 3) Furthermore, the NNW 70 may be configured from a plurality of NNWs, and the calculation unit 20B and the extraction unit 20C may share and use some of the NNWs.

[0138] FIG. 15 is an explanatory diagram of an example of this modified example.

[0139] For example, the NNW 70 includes a first NNW 70A, a second NNW 70B, and a third NNW 70C.

[0140] Then, the calculation unit 20B uses the first NNW 70A to calculate a feature map 82 representing the feature amount for each element region 72, from the image 50 or from the image 50 and the expected category information 62. Then, the calculation unit 20B further uses the second NNW 70B to calculate category score information 74 from the feature map 82 or from the feature map 82 and the expected category information 62.

[0141] On the other hand, the extraction unit 20C extracts the character string region 54 from the feature map 82 using the third NNW 70C.

[0142] In this way, in the determination device 10, the calculation unit 20B and the extraction unit 20C may share and use a part of the NNW 70.

[0143] Next, the hardware configuration of the determination device 10 of the above embodiment and modified example will be described.

[0144] FIG. 16 is a hardware configuration diagram of an example of the above embodiment and modified example.

[0145] The determination device 10 of the above embodiment and modified example includes a control device such as a CPU 91, a storage device such as a ROM (Read Only Memory) 92 and a RAM (Random Access Memory) 93, a communication I / F 94 that connects to a network and communicates, and a bus 95 that connects each part.

[0146] The programs executed by the determination device 10 of the above-described embodiment and modified example are provided in advance in the ROM 92 or the like.

[0147] The programs executed in the above embodiments and variations may be configured to be provided as a computer program product by being recorded in an installable or executable file format on a computer-readable recording medium such as a CD-ROM (Compact Disk Read Only Memory), a flexible disk (FD), a CD-R (Compact Disk Recordable), or a DVD (Digital Versatile Disk).

[0148] Furthermore, the programs executed in the above-described embodiments and modifications may be stored on a computer connected to a network such as the Internet and provided by being downloaded via the network. Also, the programs executed in the above-described embodiments and modifications may be provided or distributed via a network such as the Internet.

[0149] The programs executed in the above-described embodiment and modified examples can cause a computer to function as each unit of the determination device 10 of this embodiment. In this computer, the CPU 91 can read the programs from a computer-readable storage medium onto a main storage device and execute the programs.

[0150] Although the embodiments and modifications of the present invention have been described above, these embodiments and modifications are presented as examples and are not intended to limit the scope of the invention. The novel embodiments and modifications can be embodied in various other forms, and various omissions, substitutions, and modifications can be made without departing from the spirit of the invention. The present embodiments and their modifications are included within the scope and spirit of the invention, and are also included in the inventions and their equivalents as set forth in the claims.

[0151] For example, the determination device 10 of this embodiment may be configured without at least one of the character recognition unit 20E and the output control unit 20F. In this case, for example, at least one of the character recognition unit 20E and the output control unit 20F may be mounted on an external information processing device communicatively connected to the determination device 10 via the communication unit 14. [Explanation of symbols]

[0152] 10 Judgment device 20A Acquisition Department 20B Calculation part 20C extraction part 20D Judgment section 20E Character recognition section 20F Output control section

Claims

1. an acquisition unit that acquires an image including a character string and expected category information including an expected category that is an expected category among predetermined categories that are expected as an output; a calculation unit that calculates category score information that indicates a score of likelihood of belonging to the category for each of one or more element regions of the image; an extraction unit that extracts a character string region included in the image; a determination unit that determines the category to which the character string included in the character string area belongs based on the category score information; Equipped with The calculation unit calculating the category score information using a neural network that receives the image and the expected category information as input and outputs the category score information; The neural network outputting the category score information so that the score of the expected category included in the expected category information becomes higher; Judgment device.

2. The acquisition unit acquiring the expected category information including the expected category and an expected output degree for the expected category; The neural network outputting the category score information so that the score of the expected category included in the expected category information becomes higher as the output expectation becomes higher; The determination device according to claim 1 .

3. The acquisition unit acquiring the expected category information input by the user; The determination device according to claim 1 or 2.

4. The acquisition unit obtaining the expected category information using a model that derives the expected category information from the image; The determination device according to any one of claims 1 to 3.

5. The expected category information is a vector representing the expected category and the expected output; or a map representing the expected category or the expected category and the expected output degree for each small region of the image; Represented by The determination device according to claim 2 .

6. The calculation unit calculating the category score information using a convolutional neural network, which is the neural network to which at least one of downsampling by pooling and a stride of two or more of convolution operations is applied; The determination device according to any one of claims 1 to 5.

7. The element region is an area of one or more pixels in size; The determination device according to any one of claims 1 to 6.

8. The determination unit for each character string region that is a region of a character string corresponding to the expected category included in the expected category information, determining the category to which the character string included in the character string region belongs based on the category score information; The determination device according to any one of claims 1 to 7.

9. The neural network outputting the category score information and the character string region from the image and the expected category information; The extraction unit extracting the character string region by obtaining the character string region output from the neural network; The determination device according to any one of claims 1 to 8.

10. The neural network a first neural network, a second neural network, and a third neural network; The calculation unit calculating a feature map representing a feature amount for each of the element regions from the image or the image and the expected category information using the first neural network; calculating the category score information from the feature map or the feature map and the expected category information using the second neural network; The extraction unit extracting the character string region from the feature map using the third neural network; The determination device according to any one of claims 1 to 9.

11. The determination unit determining a category having the largest average value of the scores for each category of the element regions included in the character string region as the category to which the character string included in the character string region belongs; The determination device according to any one of claims 1 to 10.

12. The determination unit identifying the category corresponding to the maximum score for each of the element regions included in the character string region; determining the category most frequently identified in the character string region as the category to which the character string included in the character string region belongs; The determination device according to any one of claims 1 to 11.

13. The determination unit If the score of the category of each of the element regions included in the character string region changes by a threshold or more within the character string region, the character string region is divided according to the threshold, and the category to which each of the divided character string regions belongs is determined. The determination device according to any one of claims 1 to 12.

14. a character recognition unit that recognizes characters included in the character string area according to the determined category; The determination device according to any one of claims 1 to 13, comprising:

15. an output control unit that outputs recognition result information that associates the recognized character with the category determined for the character string area including the character; Equipped with The determination device according to claim 14.

16. The neural network the image and the expected category information are input, and the category score information is output so as to increase the score of the expected category included in the expected category information; The determination device according to any one of claims 1 to 15.

17. an acquisition step of acquiring an image including a character string and expected category information including an expected category that is a category expected as an output from among predetermined categories; a calculation step of calculating category score information representing a score of likelihood of belonging to the category for each of one or more element regions of the image; an extraction step of extracting a character string region included in the image; a determining step of determining the category to which the character string included in the character string area belongs based on the category score information; Including, The calculation step calculating the category score information using a neural network that receives the image and the expected category information as input and outputs the category score information; The neural network outputting the category score information so that the score of the expected category included in the expected category information becomes higher; Judgment method.

18. A determination program to be executed by a computer, an acquisition step of acquiring an image including a character string and expected category information including an expected category that is a category expected as an output from among predetermined categories; a calculation step of calculating category score information representing a score of likelihood of belonging to the category for each of one or more element regions of the image; an extraction step of extracting a character string region included in the image; a determining step of determining the category to which the character string included in the character string area belongs based on the category score information; Including, The calculation step calculating the category score information using a neural network that receives the image and the expected category information as input and outputs the category score information; The neural network outputting the category score information so that the score of the expected category included in the expected category information becomes higher; Judging program.

Images