Information processing apparatus, and information processing program

Abstract

To provide an information processing apparatus which can output a key character string and a value character string, as corresponding character strings, even when a key character string cannot be extracted by character recognition processing or when a key character string corresponding to a value character string is not included in a document, etc., and an information processing program.SOLUTION: An information processing apparatus includes a processor. The processor acquires attribute information indicating attributes of key character strings, which are character strings designated in advance as keys, and value character strings, which are character strings indicating values corresponding to the key character strings, by using a result of executing character recognition on a target image including character strings, acquires a key character string corresponding to a value character string extracted from the result of executing the character recognition by using the attribute information, and outputs the acquired key character string and the corresponding value character string.SELECTED DRAWING: Figure 2

Description

Technical Field

[0001] The present invention relates to an information processing apparatus and an information processing program.

Background Art

[0002] Patent Document 1 discloses a form recognition apparatus that reads a group of forms in which various layouts are mixed without form definition, and determines reading of a character string to be read and attributes of the character string. The form recognition apparatus includes a character string detection unit that detects a character string region from the form image, a character string recognition unit that recognizes each character in the character string region, an item name likelihood calculation unit that calculates an item name likelihood representing a probability that the character string in the form image is an item name, an item value likelihood calculation unit that calculates an item value likelihood representing a probability that the character string in the form image matches a word or a grammar notation rule of a character string registered in a notation dictionary, an arrangement likelihood calculation unit that calculates an arrangement likelihood representing whether an arrangement relationship of the character string pair is appropriate as an item name-item value relationship based on a frame of the character string pair or a character string rectangle, an item name-item value relationship evaluation value calculation unit that calculates an evaluation value representing plausibility of the character string pair as an item name-item value based on the item name likelihood, the item value likelihood, and the arrangement likelihood, and an item name-item value relationship determination unit that determines an association of the item name-item value relationship in the form image based on the evaluation value output by the item name-item value relationship evaluation value calculation unit.

[0003] Patent Document 2 discloses a technique of causing a computer of a form processing apparatus that processes forms to function as follows: an image reading unit that causes an image reading apparatus to read the form to obtain a form image; a character string recognition unit that executes character recognition processing on the form image obtained by the image reading unit to recognize a character string; a same-line character string group information acquisition unit that acquires same-line character string group information composed of a group of character strings arranged within the same line among the character strings recognized by the character string recognition unit; a specific character string determination unit that determines whether or not each same-line character string group information obtained by the same-line character string group information acquisition unit includes a predetermined specific character string; a specific image determination unit that determines whether or not a predetermined specific image exists in the vicinity of the same-line character string group information determined by the specific character string determination unit to include the specific character string; and a content item acquisition unit that, when the specific image determination unit determines that the specific image exists, acquires an item character string included in the same-line character string group information in the vicinity of the specific image as a specific content item described in the form.

Prior Art Documents

Patent Documents

[0004]

Patent Document 1

Patent Document 2

Summary of the Invention

Problems to be Solved by the Invention

[0005] There is a technique of extracting a character string included in an image by executing character recognition (OCR: Optical Character Recognition) processing on an image obtained by reading a document or the like. When extracting a character string from an image by character recognition processing, key-value extraction may be performed to extract a character string (hereinafter referred to as a "value character string") that becomes a value corresponding to a character string specified in advance as a key (hereinafter referred to as a "key character string").

[0006] Even when a value string can be extracted from the execution result of character recognition, if the key string cannot be extracted because the execution result contains misrecognition or the like, or if the document or the like does not contain the key string corresponding to the value string, the key string and the value string cannot be output as the corresponding strings.

[0007] An object of the present invention is to provide an information processing apparatus and an information processing program capable of outputting a key string and a value string as corresponding strings even when the key string cannot be extracted by character recognition processing or when the document or the like does not contain the key string corresponding to the value string.

Means for Solving the Problems

[0008] The information processing apparatus according to the first aspect has a processor. The processor acquires attribute information, which is information indicating an attribute to which a key string, which is a string specified in advance as a key, and a value string, which is a string indicating a value corresponding to the key string, belong, from the execution result of character recognition for a target image including a string. Using the attribute information, the processor acquires a key string corresponding to the value string extracted from the execution result of character recognition, and outputs the acquired key string and the corresponding value string.

[0009] The information processing apparatus according to the second aspect is the information processing apparatus according to the first aspect, wherein the processor further includes an estimation model that has learned to estimate the attribute to which a string belongs, and uses the estimation model to estimate the attribute of the string included in the target image.

[0010] The information processing apparatus according to the third aspect is the information processing apparatus according to the second aspect, wherein the processor causes the estimation model to learn in advance a string and the attribute of the string, and estimates the attribute using the string extracted from the result of character recognition.

[0011] The information processing apparatus according to the fourth aspect is the information processing apparatus according to the second aspect, wherein the processor further acquires position information which is information indicating the position of a character string in the target image, and causes the estimation model to learn in advance the position of the character string in the image and the attribute of the character string, and estimates the attribute using the position of the character string extracted from the result of character recognition.

[0012] The information processing apparatus according to the fifth aspect is the information processing apparatus according to the fourth aspect, wherein the processor causes the estimation model to learn the positional relationship between an object located at a predetermined location in the image and the character string as the position of the character string in the image, and estimates the attribute from the relationship between the position of the object included in the target image and the position of the character string in the target image.

[0013] The information processing apparatus according to the sixth aspect is the information processing apparatus according to any one of the first to fifth aspects, wherein the processor further includes a detection model that has been learned to detect a character string included in an image, and when a key character string cannot be extracted from the result of character recognition for the target image, uses the detection model to extract the key character string.

[0014] The information processing apparatus according to the seventh aspect is the information processing apparatus according to the sixth aspect, wherein the processor further acquires position relationship information which is information representing the positional relationship between the key character string and the value character string, and uses the detection model to detect the key character string based on the position relationship information and the position of the value character string extracted from the result of character recognition.

[0015] The information processing apparatus according to the eighth aspect is the information processing apparatus according to any one of the first to seventh aspects, wherein the processor corrects and outputs the value character string extracted from the result of character recognition using a value character string stored in advance or a value character string corrected in the past.

[0016] The information processing apparatus according to the ninth aspect is the information processing apparatus according to the eighth aspect, wherein the processor corrects the value string extracted from the result of character recognition using a value string stored in advance or a correction model learned from a value string corrected in the past.

[0017] The information processing apparatus according to the tenth aspect is the information processing apparatus according to any one of the first to ninth aspects, wherein when the key string corresponding to the value string cannot be extracted, the processor sets the attribute information as the key string and outputs the key string and the corresponding value string.

[0018] The information processing apparatus according to the eleventh aspect is the information processing apparatus according to any one of the first to ninth aspects, wherein the processor further acquires association information which is information associating position relationship information which is information representing the positional relationship between the key string and the value string, and the attribute information, and for the attribute information indicating the attribute of the value string, acquires the key string corresponding to the value string extracted based on the position relationship information associated in the association information and the position of the value string extracted from the result of character recognition.

[0019] The information processing program according to the twelfth aspect causes a computer to acquire attribute information which is information indicating an attribute to which a key string which is a string specified in advance as a key and a value string which is a string indicating a value corresponding to the key string belong, from the execution result of character recognition for a target image including a string, and using the attribute information, acquire the key string corresponding to the value string extracted from the execution result of character recognition, and output the acquired key string and the corresponding value string.

Advantages of the Invention

[0020] According to the information processing apparatus of the first aspect and the information processing program of the twelfth aspect, even when a key string cannot be extracted by character recognition processing or when a document or the like does not contain a key string corresponding to a value string, the key string and the value string can be output as corresponding strings.

[0021] According to the information processing apparatus of the second aspect, attributes can be estimated by reflecting learning.

[0022] According to the information processing apparatus of the third aspect, the attributes of the target image and the type of the string can be estimated from the content of the extracted string.

[0023] According to the information processing apparatus of the fourth aspect, even when there are deficiencies in the extracted string, the attributes and type of the string can be estimated.

[0024] According to the information processing apparatus of the fifth aspect, the accuracy of estimating attributes and the type of string can be further improved as compared with the case where the search direction is not determined.

[0025] According to the information processing apparatus of the sixth aspect, even when a key string cannot be extracted by the extraction process, the key string can be obtained.

[0026] According to the information processing apparatus of the seventh aspect, the key string can be detected in consideration of the positional relationship of the key string with respect to the value string.

[0027] According to the information processing apparatus of the eighth aspect, the user's load can be reduced as compared with the case where the user corrects a string.

[0028] According to the information processing apparatus of the ninth aspect, a string can be corrected more accurately as compared with the case of correcting using one correction candidate.

[0029] According to the information processing apparatus of the tenth aspect, even when there is no string corresponding to the key string in the target image, the key string can be output.

[0030] According to the information processing apparatus of the eleventh aspect, the key string can be extracted in consideration of the positional relationship of the key string with respect to the value string.

Brief Description of the Drawings

[0031]

Figure 1

Figure 2

Figure 3

Figure 4

Figure 5

Figure 6

Figure 7

Figure 8

Figure 9

Mode for Carrying Out the Invention

[0032] [First Embodiment] Hereinafter, with reference to the drawings, an example of a mode for carrying out the present invention will be described in detail.

[0033] Referring to FIG. 1, the configuration of the information processing apparatus 10 will be described. FIG. 1 is a block diagram showing an example of the hardware configuration of the information processing apparatus 10 according to the present embodiment. As an example, the form in which the information processing apparatus 10 according to the present embodiment is a terminal such as a personal computer or a server will be described. However, it is not limited thereto. The information processing apparatus 10 may be incorporated into other apparatuses such as an image forming apparatus.

[0034] As shown in FIG. 1, the information processing apparatus 10 according to the present embodiment includes a CPU (Central Processing Unit) 11, a ROM (Read Only Memory) 12, a RAM (Random Access Memory) 13, a storage 14, an input unit 15, a monitor 16, and a communication interface (communication I / F) 17. Each of the CPU 11, the ROM 12, the RAM 13, the storage 14, the input unit 15, the monitor 16, and the communication I / F 17 is interconnected by a bus 18. Here, the CPU 11 is an example of a processor.

[0035] The CPU 11 overall controls and manages the information processing apparatus 10. The ROM 12 stores various programs and data including the information processing program used in the present embodiment. The RAM 13 is a memory used as a work area during the execution of various programs. The CPU 11 extracts a character string by expanding and executing the program stored in the ROM 12 in the RAM 13. The storage 14 is, for example, an HDD (Hard Disk Drive), an SSD (Solid State Drive), or a flash memory. Note that the storage 14 may store an information processing program or the like. The input unit 15 is a mouse, a keyboard, or the like that accepts input of characters and the like. The monitor 16 displays the extracted character string and the like. The communication I / F 17 transmits and receives data.

[0036] Next, referring to FIG. 2, the functional configuration of the information processing apparatus 10 will be described. FIG. 2 is a block diagram showing an example of the functional configuration of the information processing apparatus 10 according to the present embodiment.

[0037] As shown in FIG. 2, the information processing apparatus 10 includes an acquisition unit 21, a recognition unit 22, an estimation unit 23, an extraction unit 24, a confirmation / correction unit 25, an output unit 26, a storage unit 27, and a learning unit 28A. By executing an information processing program, the CPU 11 functions as the acquisition unit 21, the recognition unit 22, the estimation unit 23, the extraction unit 24, the confirmation / correction unit 25, the output unit 26, the storage unit 27, and the learning unit 28A.

[0038] The acquisition unit 21 acquires an image (hereinafter referred to as a "target image") 31 that is a target for extracting a character string. Note that the target image 31 according to the present embodiment is an image of a document including items to be described, character strings described by a user for the items, and an object 32 such as a stamp, as shown in FIG. 3. However, the present invention is not limited to this. The target image 31 may be an image of a form or slip separated by ruled lines or the like, may be an image of a mechanically printed receipt or the like, or may be any image including a character string. Further, the object 32 will be described as being located at a position predetermined for each type of document.

[0039] The recognition unit 22 uses optical character recognition (OCR) processing to acquire a character string and an object 32 included in the document from the target image 31, and the positions (coordinates) of the character string and the object 32 in the target image 31, and outputs them as a recognition result 33.

[0040] Note that the recognition result 33 according to the present embodiment, as shown in FIG. 4 as an example, includes a character string and an object 32 acquired from the target image 31, attributes, types, and positions (coordinates) for each character string and object 32. Here, the attribute for each character string related to the recognition result 33 is information (hereinafter referred to as "attribute information") indicating an attribute to which a character string such as "date" and "address" belongs. Also, the type is a type of character string representing a specified key (hereinafter referred to as a "key character string") or a character string representing a value corresponding to the key character string (hereinafter referred to as a "value character string").

[0041] In addition, in this embodiment, the form of recognizing the character string included in the target image 31 in the recognition unit 22 has been described. However, the recognition unit 22 may analyze the target image 31 to identify the type of document and output it as a recognition result. For example, as the analysis, the position of a specific character string and ruled lines, etc. may be recognized and compared with the characteristics of the document stored in advance to identify the type of document, or an identifier for identifying the document included in the target image 31 may be recognized to identify the type of document. By identifying the type of document, the key character string included in each document is specified. That is, the recognition unit 22 may identify the type of document and specify the key character string to be extracted.

[0042] The estimation unit 23 estimates the attribute information for each character string in the recognition result 33 and the type of the character string in the recognition result. The estimation unit 23 outputs the attribute estimated for each character string and the type of the character string to the recognition result 33. For example, the estimation unit 23 is a learning model that has performed learning for estimating the attributes and types of character strings such as RNN (Recurrent Neural Network) and SVM (Support Vector Machine). The estimation unit 23 pre-learns the character string, the attribute of the character string, and the type of the character string, and uses the character string output by the recognition unit 22 to estimate the attribute and type of the character string and output them to the recognition result 33.

[0043] Note that the estimation unit 23 according to this embodiment has been described in a form that is a learning model that has learned to estimate the attributes and types of character strings. However, it is not limited to this. The attributes and types of the character string may be estimated using the character strings stored in advance in the storage unit 27 described later. For example, the estimation unit 23 derives the degree of similarity (hereinafter referred to as "similarity") between the character string output by the recognition unit 22 and the character strings stored in the storage unit 27. The estimation unit 23 may estimate the attributes and types of the character string with the highest similarity among the character strings stored in the storage unit 27 as the attributes and types of the character string. Further, the similarity may be derived using the Levenshtein distance. Here, the Levenshtein distance is a distance derived by counting the number of times of character replacement, addition, and deletion when changing an arbitrary character string to another character string.

[0044] The extraction unit 24 acquires the position of the value character string from the recognition result 33, and extracts the key character string corresponding to the value character string from the recognition result 33 using the acquired position. Specifically, the extraction unit 24 extracts, as the key character string, a character string having the same attribute as the value character string and located in the vicinity of the value character string. Here, the character string located in the vicinity is, for example, a character string located at a predetermined distance from the position of the value character string, or a character string located at the shortest distance from the position of the value character string. For example, when "Fuji Taro" shown in FIG. 3 is acquired as the value character string, the extraction unit 24 extracts "Applicant Name", which has the same attribute as the value character string, as the corresponding key character string.

[0045] In addition, in this embodiment, a form of extracting a string located in the vicinity of a value string as a key string has been described. However, the present invention is not limited to this. A string located in a predetermined direction may be extracted as a key string. For example, a string located in a predetermined direction starting from a value string, such as the left side of the value string, may be extracted as a key string. Further, as shown in FIG. 5, a storage unit 27 to be described later may store a positional relationship database (hereinafter referred to as "positional relationship DB") 34 that stores attributes, key names, and positional relationships in association with each other. The attribute is an attribute of the string, the key name is the name of the key string described in the document in the target image 31, and the positional relationship is information indicating the direction in which the corresponding value string is located with each key string as a reference point. Here, the positional relationship DB 34 is an example of related information.

[0046] As an example, the positional relationship "K - right - V" of "application date" shown in FIG. 5 represents that in the target image 31, the value string "XX year XX month XX day" is located on the right side of the key string "application date". In other words, the positional relationship represents that the key string "application date" is located on the left side of the value string "XX year XX month XX day".

[0047] That is, the extraction unit 24 acquires the positional relationship related to the value string from the positional relationship DB 34 using the attribute of the value string extracted from the recognition result 33. The extraction unit 24 may extract a string located in the direction opposite to the direction indicated by the acquired positional relationship with the position of the value string as a reference point as a key string.

[0048] The confirmation and correction unit 25 displays the character string extracted from the target image 31, the attributes of the character string, the type of the character string, and the position of the character string, and accepts corrections to the attributes, type, and position of the character string. As an example, as shown in FIG. 6, the confirmation and correction unit 25 displays a confirmation and correction screen 40. The confirmation and correction screen 40 includes an extracted character string display area 41 and a target image display area 42. The confirmation and correction unit 25 displays the character string extracted by the extraction unit 24 as an extracted character string in the extracted character string display area 41, and highlights the position of the character string corresponding to the displayed character string in the target image 31 in the target image display area 42.

[0049] Also, after the extracted character string displayed in the extracted character string display area 41 is selected, the confirmation and correction screen 40 accepts corrections to the character string extracted by designating the position corresponding to the extracted character string in the target image display area 42 and the position of the character string in the target image 31. For example, after "application date" is selected in the extracted character string display area 41, when the user designates the area where "application date" is described in the target image display area 42, corrections to the key character string and the position of the key character string are accepted. At this time, the same color highlight is displayed in the color column for "application date" in the extracted character string display area 41 and in the area where "application date" is described in the target image display area 42.

[0050] The output unit 26 outputs the key character string and the value character string extracted from the target image 31.

[0051] The storage unit 27 stores the target image 31 in association with the character string extracted from the target image 31 and the position of the character string in the target image 31. The storage unit 27 also stores the position relationship DB 34, the target images 31 from which extraction has been performed in the past, and the character strings in the target images 31 from which extraction has been performed in the past.

[0052] The learning unit 28A performs learning of the estimation unit 23 which is a learning model. The learning unit 28A uses the target image 31 and the character string as input data, and the attributes and types of the character string as teacher data to cause the estimation unit 23 to perform learning.

[0053] Note that the estimation unit 23 according to this embodiment has been described in a form that learns a character string as input data and estimates the attribute and type of the character string. However, it is not limited to this. The position of the character string in the target image 31 may be used as input data. For example, the estimation unit 23 may learn and estimate the attribute and type of the character string using the position of the character string included in the recognition result 33 in the target image 31 as input data.

[0054] Further, the estimation unit 23 may learn the relationship between the position of the character string in the target image 31 and the position of the object 32 included in the target image 31, and estimate the attribute and type of the character string. For example, even in a document, a character string indicating "address" with the same attribute may indicate different "addresses" such as "applicant's address" and "originator's address", and may not include a key character string in the target image 31 like "claimant's address". However, since the items described and the position of the object 32 included in the document are predetermined for each document, it is possible to specify the attribute and type of the character string from the positional relationship between the object 32 included in the target image 31 and the character string. That is, the estimation unit 23 may learn the relationship between the position of the object 32 included in the target image 31 and the position of the character string in the target image 31 as input data, and estimate the attribute and type of the character string.

[0055] Further, the estimation unit 23 may learn the positional relationship of each character string as input data and estimate the attribute and type of the character string, or may learn the positional relationship included in the positional relationship DB 32 and the position of the character string as input data and estimate the attribute and type of the character string. Also, when the key character string corresponding to the value character string is not included in the target image 31, the estimation unit 23 may set the estimated attribute as the key character string, or may estimate the key character string corresponding to the value character string.

[0056] Next, with reference to FIG. 7, the operation of the information processing apparatus 10 according to the present embodiment will be described. FIG. 7 is a flowchart showing an example of an extraction process for extracting a character string according to the present embodiment. When the CPU 11 reads out and executes an information processing program from the ROM 12 or the storage 14, the information processing program shown in FIG. 7 is executed. The information processing program shown in FIG. 7 is executed, for example, when an instruction to execute the extraction process is input from the user for the target image 31.

[0057] In step S101, the CPU 11 acquires the target image 31 input by the user.

[0058] In step S102, the CPU 11 executes character recognition processing on the acquired target image 31 and outputs the character string and the position of the character string as the recognition result 33.

[0059] In step S103, the CPU 11 estimates the attribute and type of the character string using the recognition result 33 and outputs them to the recognition result 33.

[0060] In step S104, the CPU 11 searches for and extracts a value character string from the recognition result 33.

[0061] In step S105, the CPU 11 specifies and extracts a key character string corresponding to the value character string using the position and attribute of the extracted value character string.

[0062] In step S106, the CPU 11 determines whether a key character string can be extracted. If a key character string can be extracted (step S106: YES), the CPU 11 proceeds to step S107. On the other hand, if a key character string cannot be extracted (step S106: NO), the CPU 11 proceeds to step S107.

[0063] In step S107, the CPU 11 sets the attribute of the value character string as the key character string.

[0064] In step S108, the CPU 11 associates the key character string and the value character string and outputs the extracted result.

[0065] In step S109, the CPU 11 determines whether there is another value character string. If there is another value character string (step S109: YES), the CPU 11 proceeds to step S104. On the other hand, if there is no other value character string (step S109: NO), the CPU 11 proceeds to step S110.

[0066] In step S110, the CPU 11 displays a confirmation and correction screen and accepts correction of the attributes, types, and positions of the character strings by the user.

[0067] In step S111, the CPU 11 stores the target image 31 in association with the attributes, types, and positions of the character strings.

[0068] In step S112, the CPU 11 outputs the key character string and the value character string using the extracted result.

[0069] As described above, according to the present embodiment, the key character string corresponding to the value character string is extracted using the attribute of the character string, and the key-value extraction is executed. Therefore, even when the key character string cannot be extracted by the character recognition process or when the document or the like does not include the key character string corresponding to the value character string, the key character string and the value character string are output as the corresponding character strings.

[0070] [Second Embodiment] In the first embodiment, when the key character string corresponding to the value character string could be extracted from the recognition result 33, the form of associating and outputting the key character string and the value character string was described. In this embodiment, when the key character string corresponding to the value character string could not be extracted from the recognition result 33, the form of detecting the image corresponding to the key character string from the target image 31 and associating and outputting the key character string and the value character string will be described.

[0071] Note that since the hardware configuration of the information processing apparatus 10 according to the present embodiment (see FIG. 1), the example of the target image 31 (see FIG. 3), the example of the recognition result 33 (see FIG. 4), and the example of the position relationship DB 34 (see FIG. 5) are the same as those in the first embodiment, the description thereof is omitted. Further, since the example of the confirmation / correction screen 40 according to the present embodiment (see FIG. 6) is the same as that in the first embodiment, the description thereof is omitted.

[0072] Next, with reference to FIG. 8, the functional configuration of the information processing apparatus 10 will be described. FIG. 8 is a block diagram showing an example of the functional configuration of the information processing apparatus 10 according to the present embodiment. Note that the same functions as those of the information processing apparatus 10 shown in FIG. 2 in FIG. 8 are denoted by the same reference numerals as in FIG. 8, and the description thereof is omitted.

[0073] As shown in FIG. 8, the information processing apparatus 10 includes an acquisition unit 21, a recognition unit 22, an estimation unit 23, an extraction unit 24, a confirmation / correction unit 25, an output unit 26, a storage unit 27, a learning unit 28B, and a detection unit 29. When the CPU 11 executes an information processing program, it functions as the acquisition unit 21, the recognition unit 22, the estimation unit 23, the extraction unit 24, the confirmation / correction unit 25, the output unit 26, the storage unit 27, the learning unit 28B, and the detection unit 29.

[0074] The learning unit 28B performs learning of the estimation unit 23 which is a learning model and the detection unit 29 described later. The learning unit 28B uses the target image 31 and the character string as input data, and the attributes and types of the character string as teacher data to cause the estimation unit 23 to perform learning. Further, the learning unit 28B uses the target image 31 and the position of the character string as input data, and the key string corresponding to the value string as teacher data to cause the detection unit 29 described later to perform learning.

[0075] The detection unit 29 detects a key string located near the value string specified from the target image 31 using object detection processing. Specifically, the detection unit 29 is a learning model that has performed machine learning for detecting a string located near a specified string such as a CNN (Convolution Neural Network) and YOLO (You Only Look Once). The detection unit 29 detects an image of the key string located near the value string from the target image 31, identifies the detected image to obtain the key string, and outputs it to the estimation unit 23.

[0076] Note that the detection unit 29 according to the present embodiment is a learning model using machine learning, and the form of detecting an image of a key string located near the position of the value string from the target image 31 has been described. However, it is not limited to this. The image of the key string located near the position of the value string may be detected using pattern matching processing. For example, each image corresponding to the key string is stored in advance, and the image of the key string is detected by pattern matching processing such as shape detection and template matching. The detection unit 29 may detect the key string from the target image 31 using the image corresponding to the key string and specify the key string located near the value string.

[0077] Next, with reference to FIG. 9, the operation of the information processing apparatus 10 according to the present embodiment will be described. FIG. 9 is a flowchart showing an example of extraction processing for extracting a string according to the present embodiment. When the CPU 11 reads and executes an information processing program from the ROM 12 or the storage 14, the information processing program shown in FIG. 9 is executed. The information processing program shown in FIG. 9 is executed, for example, when an instruction to execute the target image 31 and extraction processing is input from the user. Note that the same steps as the extraction processing shown in FIG. 7 in FIG. 9 are denoted by the same reference numerals as in FIG. 7, and the description thereof is omitted.

[0078] In step S113, the CPU 11 determines whether a key string has been extracted. If a key string has been extracted (step S113: YES), the CPU 11 proceeds to step S108. On the other hand, if a key string has not been extracted (step S113: NO), the CPU 11 proceeds to step S114.

[0079] In step S114, the CPU 11 executes a detection process for detecting a key string from the target image 31 using the position of the value string, and outputs the key string as a detection result. Here, the detection process is a process of detecting an image of the key string corresponding to the value string from the target image 31 and acquiring the key string.

[0080] In step S115, the CPU 11 estimates the attribute and type of the string using the detection result and outputs them to the detection result.

[0081] In step S116, the CPU 11 determines whether the detected key string corresponds to the value string. If it corresponds to the value string (step S116: YES), the CPU 11 proceeds to step S108. On the other hand, if it does not correspond to the value string (step S116: NO), the CPU 11 proceeds to step S117.

[0082] In step S117, the CPU 11 sets the attribute of the value string as the key string.

[0083] As described above, according to the present embodiment, a key value extraction is executed by detecting a key string corresponding to a value string using a detection process. Therefore, even when a key string cannot be extracted by a character recognition process, a key string and a value string are output as corresponding strings.

[0084] Note that, in the present embodiment, a form of detecting a key string by a detection process has been described. However, the present invention is not limited to this. A value string may be detected.

[0085] Also, in the present embodiment, the form of accepting corrections to the attributes, types, and positions of character strings on the confirmation correction screen 40 has been described. However, it is not limited to this. Accepting corrections to the extracted character strings may also be acceptable. Further, when accepting corrections to the extracted character strings, the extracted character strings may be uniformly displayed and corrections may be accepted. Also, a confidence level indicating the reliability of the character string may be derived, and when the confidence level is less than a predetermined threshold value, only the character strings with a confidence level less than the predetermined threshold value may be displayed.

[0086] Also, in the present embodiment, the form of accepting corrections to the positions of character strings obtained from the recognition result 33 on the confirmation correction screen 40 has been described. However, it is not limited to this. Accepting specifications of the positions of character strings in the target image 31 may also be acceptable.

[0087] Also, in the present embodiment, the form of accepting corrections to the positions of character strings when performing confirmation corrections has been described. However, it is not limited to this. When the target image 31 is input to the information processing apparatus 10, specifications of the positions of character strings may be accepted in advance, or when the character strings are output, the target image 31 stored in the storage unit 27 may be displayed at an arbitrary opportunity, and corrections to the positions of the character strings may be accepted.

[0088] Also, in the present embodiment, the form of accepting corrections to character strings on the confirmation correction screen 40 has been described. However, it is not limited to this. The extracted character strings may be corrected using the character strings stored in the storage unit 27 or the character strings extracted in the past. Also, a plurality of value strings may be stored in the storage unit 27 in association with each other, and other value strings corresponding to the extracted value strings may be retrieved from the storage unit 27 and presented. For example, the storage unit 27 stores, in association with each other, a value string of "name" extracted in the past and a value string of "address". When the extraction unit 24 extracts a value string of "name", the value string of "address" corresponding to the value string of "name" may be obtained from the storage unit 27 and presented as a correction candidate for correction.

[0089] Also, the extracted character string may be corrected using a learning model that uses machine learning for correcting the extracted character string. For example, the character string extracted from the target image 31 for which extraction processing has been performed in the past and the character string corrected in the past are stored in the storage unit 27, and a correction unit (not shown) learns the target image 31 stored in the storage unit 27, and the character string and the character string corrected in the past in the target image 31. The correction unit may present a correction candidate for the extracted character string and correct the character string.

[0090] As described above, the present invention has been described using each embodiment, but the present invention is not limited to the scope described in each embodiment. Various changes or improvements can be made to each embodiment without departing from the gist of the present invention, and the forms to which such changes or improvements are made are also included in the technical scope of the present invention.

[0091] In the above embodiments, the processor refers to a processor in a broad sense, and includes, for example, a general-purpose processor (for example, CPU: Central Processing Unit) and a dedicated processor (for example, GPU: Graphics Processing Unit, ASIC: Application Specific Integrated Circuit, FPGA: Field Programmable Gate Array, programmable logic device, etc.).

[0092] Also, the operations of the processor in the above embodiments may be performed not only by one processor but also by a plurality of physically separated processors cooperating with each other. Also, the order of each operation of the processor is not limited to the order described in the above embodiments, and may be changed as appropriate.

[0093] In addition, in this embodiment, although the form in which the information processing program is installed in the storage has been described, it is not limited to this. The information processing program according to this embodiment may be provided in a form recorded on a computer-readable storage medium. For example, the information processing program according to the present invention may be provided in a form recorded on an optical disk such as a CD (Compact Disc)-ROM and a DVD (Digital Versatile Disc)-ROM. The information processing program according to the present invention may be provided in a form recorded on a semiconductor memory such as a USB (Universal Serial Bus) memory and a memory card. Further, the information processing program according to this embodiment may be acquired from an external device via a communication line connected to the communication I / F 17.

Explanation of Signs

[0094] 10 Information processing apparatus 11 CPU 12 ROM 13 RAM 14 Storage 15 Input unit 16 Monitor 17 Communication interface 18 Bus 21 Acquisition unit 22 Recognition unit 23 Estimation unit 24 Extraction unit 25 Confirmation / correction unit 26 Output unit 27 Storage unit 28A, 28B Learning unit 29 Detection unit 31 Target image 32 Object 33 Recognition result 34 Position relationship database 40 Confirmation / correction screen 41 Extracted character string display area 42 Target image display area

Claims

1. An information processing apparatus having a processor, wherein the processor: obtains attribute information, which is information indicating an attribute to which a key string, which is a string specified in advance as a key, and a value string, which is a string indicating a value corresponding to the key string, belong, from an execution result of character recognition on a target image including a string; obtains the key string corresponding to the value string extracted from the execution result of the character recognition by using the attribute information; outputs the obtained key string and the corresponding value string; The processor: further includes an estimation model that has been learned to estimate an attribute to which a string belongs; estimates an attribute of a string included in the target image by using the estimation model; The processor: further obtains position information, which is information indicating a position of the string in the target image; pre-learns the estimation model with a position of a string in an image and an attribute of the string; estimates the attribute by using a position of the string extracted from the result of the character recognition; The processor: pre-learns the estimation model with a positional relationship between an object located at a predetermined location in an image and a string as a position of the string in the image; estimates an attribute from a relationship between a position of an object included in the target image and a position of the string in the target image An information processing apparatus.

2. An information processing apparatus having a processor, wherein the processor: obtains attribute information, which is information indicating an attribute to which a key string, which is a string specified in advance as a key, and a value string, which is a string indicating a value corresponding to the key string, belong, from an execution result of character recognition on a target image including a string; obtains the key string corresponding to the value string extracted from the execution result of the character recognition by using the attribute information; outputs the obtained key string and the corresponding value string; The processor: further includes a detection model that has been learned to detect a string included in an image; extracts the key string by using the detection model when the key string cannot be extracted from the result of the character recognition on the target image An information processing apparatus.

3. The processor: further obtains position relationship information, which is information representing a positional relationship between the key string and the value string Using the detection model, detect the key string based on the positional relationship information and the position of the value string extracted from the result of the character recognition. The information processing apparatus according to claim 2. **Claim 4** A processor, wherein the processor Obtain attribute information, which is information indicating an attribute to which a key string, which is a string specified in advance as a key, and a value string, which is a string indicating a value corresponding to the key string, belong, from the execution result of character recognition on a target image including a string. Using the attribute information, obtain the key string corresponding to the value string extracted from the execution result of the character recognition. Output the obtained key string and the corresponding value string. The processor Using a pre-stored value string or a value string modified in the past, modify and output the value string extracted from the result of the character recognition. An information processing apparatus. **Claim 5** The processor Using a correction model learned from a pre-stored value string or a value string modified in the past, correct the value string extracted from the result of the character recognition. The information processing apparatus according to claim 4. **Claim 6** A processor, wherein the processor Obtain attribute information, which is information indicating an attribute to which a key string, which is a string specified in advance as a key, and a value string, which is a string indicating a value corresponding to the key string, belong, from the execution result of character recognition on a target image including a string. Using the attribute information, obtain the key string corresponding to the value string extracted from the execution result of the character recognition. Output the obtained key string and the corresponding value string. The processor When the key string corresponding to the value string cannot be extracted, set the attribute information to the key string and output the key string and the corresponding value string. An information processing apparatus. **Claim 7** A processor, wherein the processor Obtain attribute information, which is information indicating an attribute to which a key string, which is a string specified in advance as a key, and a value string, which is a string indicating a value corresponding to the key string, belong, from the execution result of character recognition on a target image including a string. Using the attribute information, obtain the key string corresponding to the value string extracted from the execution result of the character recognition. Output the obtained key string and the corresponding value string. The processor further obtains associated information which is information associating position relationship information, which is information representing the positional relationship between the key string and the value string, and the attribute information. For the attribute information indicating the attribute of the value string, obtain the key string corresponding to the value string extracted based on the position relationship information associated in the associated information and the position of the value string extracted from the result of the character recognition. An information processing apparatus.

8. Cause a computer to obtain attribute information, which is information indicating an attribute to which a key string, which is a string specified in advance as a key, and a value string, which is a string indicating a value corresponding to the key string, belong, from the execution result of character recognition on a target image including a string; use the attribute information to obtain the key string corresponding to the value string extracted from the execution result of the character recognition; output the obtained key string and the corresponding value string; and cause it to further include an estimation model that has been trained to estimate the attribute to which a string belongs, use the estimation model to estimate the attribute of the string included in the target image; and cause it to further obtain position information, which is information indicating the position of the string in the target image; pre-train the estimation model to associate the position of a string in an image with the attribute of the string; cause it to estimate the attribute using the position of the string extracted from the result of the character recognition; and cause it to train the estimation model to associate, as the position of a string in an image, the position relationship between an object located at a predetermined location in the image and the string; estimate the attribute from the relationship between the position of the object included in the target image and the position of the string in the target image. An information processing program for causing the above to be executed.

Images