Information processor, program, and method for processing program

Abstract

To provide an information processor, a program, and a method for processing information which can easily relate a letter string read from a document image to an item name.SOLUTION: In an information processing system, operations of a control unit of an information processor include: acquiring image data of a document; recognizing a letter string having a region overlapping with at least one specified color of the image or a letter string surrounded by at least one specification line of the image data; and relating the recognized letter to the item name related to the specified color overlapping with the letter string or to the item name related to the specification line surrounding the letter string.SELECTED DRAWING: Figure 8

Description

Technical Field

[0001] The present disclosure relates to an information processing apparatus, a program, and an information processing method.

Background Art

[0002] Conventionally, a technique for reading a character string from document image data and associating the read character string with an item name is known.

[0003] For example, Patent Document 1 discloses a technique of obtaining a character string by character recognition from a document image, determining whether the character string is an item name or an item value, and when the character string determined to be an item name is continuous in the horizontal or vertical direction by one or more, regarding the one or more continuous character strings as one group, combining a region where one or more groups are continuous on the document image and a region where the character strings determined to be one or more item values adjacent to the region are continuous as one search region, and extracting an item value from each search region based on the relationship between the item name and the item value.

Prior Art Documents

Patent Documents

[0004]

Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0005] A method of identifying an item name for a character string obtained by character recognition from a document image, as in Patent Document 1, takes resources for the process of identifying the item name. In addition, since there are various document layout patterns, it is difficult to identify the item name without error and to associate the item name and the item value without error.

[0006] An object of the present disclosure is to easily associate a character string read from a document image with an item name.

Means for Solving the Problems

[0007] The information processing apparatus according to the present disclosure acquires image data of a document, performs character recognition on a character string having an area overlapping with at least one designated color in the image data, or a character string surrounded by at least one designated line in the image data, and associates the character string obtained by the character recognition with an item name associated with the designated color overlapping with the character string, or an item name associated with the designated line surrounding the character string, a control unit. It is provided with.

[0008] The program according to the present disclosure causes a computer to execute operations including: a step of acquiring image data of a document, a step of performing character recognition on a character string having an area overlapping with at least one designated color in the image data, or a character string surrounded by at least one designated line in the image data, and a step of associating the character string obtained by the character recognition with an item name associated with the designated color overlapping with the character string, or an item name associated with the designated line surrounding the character string.

[0009] The information processing method according to the present disclosure is an information processing method in an information processing apparatus, including: a step of acquiring image data of a document, a step of performing character recognition on a character string having an area overlapping with at least one designated color in the image data, or a character string surrounded by at least one designated line in the image data, and a step of associating the character string obtained by the character recognition with an item name associated with the designated color overlapping with the character string, or an item name associated with the designated line surrounding the character string. It includes.

Effect of the Invention

[0010] According to the present disclosure, a character string read from a document image can be easily associated with an item name.

Brief Description of Drawings

[0011]

Figure 1

Figure 2

Figure 3

Figure 4

Figure 5

Figure 6

Figure 7A

Figure 7B

Figure 8

Modes for Carrying Out the Invention

[0012] Hereinafter, embodiments of the present disclosure will be described with reference to the drawings.

[0013] FIG. 1 is a diagram showing the configuration of an information processing system 1 according to an embodiment of the present disclosure. With reference to FIG. 1, the configuration and outline of the information processing system 1 according to an embodiment of the present disclosure will be described.

[0014] The information processing system 1 is a system that reads a character string from image data of a document and associates the read character string with an item name.

[0015] The document to be read for the character string can be any document. The document to be read for the character string may be, for example, a receipt.

[0016] Hereinafter, the case where the document to be read for the character string is a receipt will be described as an example. Various items are described in the receipt. For example, the receipt describes, as items, the amount, the company name of the company that issued the receipt, the address of the company that issued the receipt, and the like.

[0017] Hereinafter, in the present embodiment, the names of items such as the amount, company name, address, etc. described in a document such as a receipt may be referred to as "item names". Also, the content described as the details of items such as the amount, company name, address, etc. may be referred to as "item values". For example, in the column where the "item name" is "amount", a value of an amount such as "2500 yen" is described as the "item value".

[0018] The information processing system 1 includes an information processing device 10 and a terminal device 20. The information processing device 10 and the terminal device 20 are communicably connected via a network 30. The network 30 may be a network including a mobile communication network and the Internet, etc.

[0019] The information processing device 10 may be, for example, a general-purpose PC (Personal Computer). The information processing device 10 may also be a dedicated computer configured to function as the information processing device 10 of the information processing system 1.

[0020] The information processing device 10 acquires image data of a document photographed by the terminal device 20 from the terminal device 20. Details of the processing executed by the information processing device 10 on the acquired image data will be described later.

[0021] The terminal device 20 may be, for example, a smartphone or a tablet. The terminal device 20 may also be a dedicated computer configured to function as the terminal device 20 of the information processing system 1.

[0022] The terminal device 20 captures a document to be read and generates image data of the document. The terminal device 20 transmits the generated image data to the information processing device 10.

[0023] Referring to FIG. 2, the configuration of the information processing device 10 according to an embodiment of the present disclosure will be described.

[0024] The information processing device 10 includes a communication unit 11, a storage unit 12, an input unit 13, an output unit 14, and a control unit 15.

[0025] The communication unit 11 includes a communication module that connects to the network 30. For example, the communication unit 11 may include a communication module corresponding to a LAN (Local Area Network). In one embodiment, the information processing device 10 is connected to the network 30 via the communication unit 11. The communication unit 11 transmits and receives various information via the network 30. The communication unit 11 can communicate with the terminal device 20 via the network 30.

[0026] The storage unit 12 is, for example, a semiconductor memory, a magnetic memory, an optical memory, etc., but is not limited thereto. The storage unit 12 may function as, for example, a main memory device, an auxiliary memory device, or a cache memory. The storage unit 12 stores any information used for the operation of the information processing device 10. For example, the storage unit 12 may store a system program, an application program, and various information received by the communication unit 11. The information stored in the storage unit 12 may be updated, for example, with information received from the network 30 via the communication unit 11. A part of the storage unit 12 may be installed outside the information processing device 10. In that case, a part of the storage unit 12 installed outside may be connected to the information processing device 10 via an arbitrary interface.

[0027] The input unit 13 includes one or more input interfaces that detect user input and obtain input information based on the user's operations. For example, the input unit 13 includes, but is not limited to, physical keys, capacitive keys, a touch screen provided integrally with the display of the output unit 14, or a microphone that accepts voice input.

[0028] The output unit 14 includes one or more output interfaces that output information and notify the user. For example, the output unit 14 includes, but is not limited to, a display that outputs information as an image, or a speaker that outputs information as voice.

[0029] The control unit 15 includes at least one processor, at least one dedicated circuit, or a combination thereof. The processor is a general-purpose processor such as a CPU (Central Processing Unit) or a GPU (Graphics Processing Unit), or a dedicated processor specialized for specific processing. The dedicated circuit is, for example, an FPGA (Field-Programmable Gate Array) or an ASIC (Application Specific Integrated Circuit). While controlling each part of the information processing apparatus 10, the control unit 15 executes processing related to the operation of the information processing apparatus 10. Referring to FIG. 3, the configuration of the terminal device 20 according to an embodiment of the present disclosure will be described.

[0030] Referring to FIG. 3, the configuration of the terminal device 20 according to an embodiment of the present disclosure will be described.

[0031] The terminal device 20 includes a communication unit 21, a storage unit 22, an input unit 23, an output unit 24, a camera 25, and a control unit 26.

[0032] The communication unit 21 includes a communication module that connects to the network 30. For example, the communication unit 21 may include a communication module compatible with mobile communication standards such as LTE, 4G, and 5G. In one embodiment, the terminal device 20 is connected to the network 30 via the communication unit 21. The communication unit 21 transmits and receives various information via the network 30. The communication unit 21 can communicate with the information processing device 10 via the network 30.

[0033] The storage unit 22 is, for example, a semiconductor memory, a magnetic memory, an optical memory, etc., but is not limited thereto. The storage unit 22 may function as, for example, a main storage device, an auxiliary storage device, or a cache memory. The storage unit 22 stores any information used for the operation of the terminal device 20. For example, the storage unit 22 may store a system program, an application program, and various information received by the communication unit 21, etc. The information stored in the storage unit 22 may be updatable with information received from the network 30 via the communication unit 21, for example. A part of the storage unit 22 may be installed outside the terminal device 20. In that case, a part of the storage unit 22 installed outside may be connected to the terminal device 20 via an arbitrary interface.

[0034] The input unit 23 includes one or more input interfaces that detect user input and acquire input information based on the user's operation. For example, the input unit 23 is a physical key, a capacitance key, a touch screen provided integrally with the display of the output unit 24, a microphone that accepts voice input, etc., but is not limited thereto.

[0035] The output unit 24 includes one or more output interfaces that output information and notify the user. For example, the output unit 24 includes a display that outputs information as an image, a speaker that outputs information as voice, etc., but is not limited thereto.

[0036] The camera 25 may be a camera with any configuration. The camera 25 can photograph documents.

[0037] The control unit 26 includes at least one processor, at least one dedicated circuit, or a combination thereof. The processor is a general-purpose processor such as a CPU or GPU, or a dedicated processor specialized for specific processing. The dedicated circuit is, for example, an FPGA or an ASIC. The control unit 26 executes processing related to the operation of the terminal device 20 while controlling each part of the terminal device 20.

[0038] (Operation of the information processing system) Subsequently, the operation of the information processing system 1 shown in FIG. 1 will be described.

[0039] The camera 25 of the terminal device 20 photographs a document that is the target for reading the character string. Hereinafter, the case where the document that is the target for reading the character string is a receipt will be described as an example. The case where the document that is the target for reading the character string is a receipt will be described as an example.

[0040] FIG. 4 shows an example of a receipt. The receipt shown in FIG. 4 describes an amount 101, a company name 102 of the company that issued the receipt, an address 103 of the company that issued the receipt, and the like.

[0041] Before photographing the receipt with the camera 25 of the terminal device 20, the user colors the character string for which character recognition is desired among the items described in the receipt with a predetermined designated color. The user can describe the designated color on the receipt with a writing instrument such as a ballpoint pen and a fluorescent pen, for example.

[0042] FIG. 5 shows an example of a state where the receipt is colored. In the example shown in FIG. 5, the area 201 overlapping with the amount 101 is colored. Also, the area 202 overlapping with the company name 102 is colored. Also, the area 203 overlapping with the address 103 is colored.

[0043] When coloring the receipt, the user colors it according to the specified color associated with the item name. FIG. 6 shows an example of the association between the item name and the specified color. In the example shown in FIG. 6, the color "yellow" is associated with the item name "Amount", the color "green" is associated with the item name "Company Name", and the color "blue" is associated with the item name "Address". Note that the association shown in FIG. 6 is just an example and is not limited to this.

[0044] The association between the specified color and the item name as shown as an example in FIG. 6 may be stored in advance in the storage unit 12 of the information processing apparatus 10.

[0045] When the association as shown in FIG. 6 is made, in the example of the receipt shown in FIG. 5, the user describes the area 201 that overlaps with the amount 101 in yellow. Also, the user describes the area 202 that overlaps with the company name 102 in green. Also, the user describes the area 203 that overlaps with the address 103 in blue.

[0046] Note that in the example shown in FIG. 5, the user colors the amount 101, the company name 102, and the address 103, but this is the case when character recognition is desired for the amount 101, the company name 102, and the address 103. It is not limited to this, and the user may color only the items for which character recognition is desired. For example, if only the amount 101 is desired to be recognized, the user may color only the amount 101.

[0047] Also, when coloring the character string of the receipt, the user may color it so that the area where the character string and the specified color overlap. FIG. 7A shows an example of the state where the amount 101 is colored. Also, FIG. 7B shows another example of the state where the amount 101 is colored.

[0048] In the example shown in FIG. 7A, the amount 101 is colored in the area 201, but the area 201 is linearly described so as to overlap with the amount 101. FIG. 7A corresponds to a case where, for example, the amount 101 is colored with a ballpoint pen. When coloring linearly as shown in FIG. 7A, the user can easily color with a writing instrument such as a ballpoint pen.

[0049] In the example shown in FIG. 7B, the amount 101 is colored in the area 201, but the area 201 is described so as to surround the entire amount 101. FIG. 7B corresponds to a case where, for example, the entire amount 101 is colored with a fluorescent pen.

[0050] When the user colors the character string of the receipt, the user may color it in the format as shown in FIG. 7A. Or the user may color it in the format as shown in FIG. 7B.

[0051] When the camera 25 of the terminal device 20 captures an image of a document in which a character string for which character recognition is desired is colored with a predetermined specified color, the control unit 26 generates image data of the captured document.

[0052] The control unit 26 transmits the image data of the captured document to the information processing device 10 via the communication unit 21.

[0053] The control unit 15 of the information processing device 10 acquires the image data transmitted by the terminal device 20 via the communication unit 11.

[0054] The control unit 15 recognizes an area with a color corresponding to at least one specified color in the acquired image data. For example, when the specified colors are yellow, green, and blue, the control unit 15 recognizes areas with yellow, green, and blue in the image data.

[0055] When the control unit 15 recognizes an area with a color corresponding to at least one specified color, it performs character recognition on a character string having an area overlapping with the at least one specified color. For example, the control unit 15 performs character recognition on a character string having an area overlapping with any one of yellow, green, and blue, which are specified colors.

[0056] At this time, the control unit 15 only needs to perform character recognition on a character string having an area overlapping with any one of yellow, green, and blue, which are specified colors. That is, for a character string that does not overlap with any of yellow, green, and blue, character recognition does not need to be performed. Thereby, since the control unit 15 does not perform character recognition on a character string that does not need to be recognized, the processing load when performing character recognition can be reduced.

[0057] The control unit 15 is capable of performing character recognition in multiple languages. Therefore, for example, even if the character string is a character string described in various languages such as Japanese, English, and French, character recognition can be performed.

[0058] When the control unit 15 performs character recognition on a character string having an area overlapping with the specified color, it associates the character string obtained by character recognition with the item name associated with the specified color overlapping with the character string. When the control unit 15 associates the character string with the item name, it may store the associated content in the storage unit 22.

[0059] For example, in FIG. 5, a yellow line, which is one of the specified colors, is drawn in the area 201. In this case, the control unit 15 performs character recognition on the character string “￥2,500” that overlaps with the area 201. Thereby, the control unit 15 recognizes the image of the portion corresponding to the character string “￥2,500” as the text “￥2,500”. Also, since the specified color yellow is associated with the item name “Amount” as shown in FIG. 6, the control unit 15 associates the character string “￥2,500” obtained by character recognition with the item name “Amount”.

[0060] The control unit 15 performs the same processing for the area 202 with the green line drawn and the area 203 with the blue line drawn.

[0061] When the control unit 15 associates the recognized character string with the item name, the control unit 15 may cause the output unit 14 to display the character string together with the item name. For example, the control unit 15 may cause the output unit 14 to display "Amount: ¥2,500". Thereby, the user can confirm the item name and the corresponding item value.

[0062] With reference to the flowchart shown in FIG. 8, the operation of the information processing system 1 will be described.

[0063] Step S101: The control unit 15 of the information processing apparatus 10 acquires the image data of the document photographed by the terminal device 20 from the terminal device 20.

[0064] Step S102: When the control unit 15 recognizes an area with a color corresponding to the specified color among the acquired image data, the control unit 15 performs character recognition on a character string having an area overlapping with the specified color.

[0065] Step S103: The control unit 15 associates the character string obtained by character recognition with an item name associated with the specified color overlapping with the character string.

[0066] <Method of surrounding with a specified line> In the above-described embodiment, the user has colored a character string for which character recognition is desired among the items described in the receipt with a predetermined specified color. However, instead of coloring, the user may surround the character string for which character recognition is desired with a predetermined specified line. In this case, the user may describe the specified line on the receipt with a writing instrument such as a black ballpoint pen.

[0067] The designated line is a specific type of line that is pre-specified as a line for enclosing the string for which character recognition is to be performed. The designated line may be, for example, a solid line, a dotted line, a dashed line, etc. When enclosing the string for which character recognition is to be performed with the designated line, the user encloses the string with the designated line associated with the item name. For example, a "solid line" may be associated as the designated line with the item name "amount". Also, a "dotted line" may be associated as the designated line with the item name "company name". Further, a "dashed line" may be associated as the designated line with the item name "address".

[0068] The association between the designated line and the item name may be pre-stored in the storage unit 12 of the information processing apparatus 10.

[0069] The control unit 15 performs character recognition on the string enclosed by at least one designated line. The control unit 15 performs character recognition on, for example, a string enclosed by any one of a solid line, a dotted line, and a dashed line which are designated lines.

[0070] At this time, the control unit 15 only needs to perform character recognition on the string enclosed by any one of the solid line, the dotted line, and the dashed line which are designated lines. That is, for a string not enclosed by any of the solid line, the dotted line, and the dashed line, character recognition does not need to be performed. Thereby, since the control unit 15 does not perform character recognition on strings that do not need to be recognized, the processing load when performing character recognition can be reduced.

[0071] When the control unit 15 performs character recognition on the string enclosed by the designated line, the control unit 15 associates the character-recognized string with the item name associated with the designated line enclosing the string. When the control unit 15 associates the string and the item name, the associated content may be stored in the storage unit 22.

[0072] <Printing of the Designated Color or Designated Line on the Paper> In the above-described embodiment, the case where the user colors the document has been described as an example, but the designated color may be pre-printed at a predetermined location on the paper of the document.

[0073] For example, on the receipt paper, the background of the amount column part is printed in yellow, the background of the company name column part is printed in green, and the background of the address column part is printed in blue. This may be the case. By using the paper with the specified color pre-printed in this way, the user can omit the operation of coloring the specified color with a writing instrument or the like.

[0074] Also, in the above-described embodiment, the case where the user describes a specified line in the document has been described as an example, but the specified line may be pre-printed at a predetermined position on the document paper.

[0075] For example, on the receipt paper, the amount column part may be printed so as to be surrounded by a solid line, the company name column part may be printed so as to be surrounded by a dotted line, and the address column part may be printed so as to be surrounded by a dashed line. By using the paper with the specified line pre-printed in this way, the user can omit the operation of describing the specified line with a writing instrument or the like. Also, since the specified line may be black, the paper with the specified line pre-printed may be a paper printed in black and white.

[0076] As described above, in the information processing apparatus 10 according to the present embodiment, the control unit 15 performs character recognition on a character string having an area overlapping at least one specified color among the image data of a document such as a receipt, and associates the character string obtained by the character recognition with an item name associated with the specified color overlapping with the character string. Thereby, the user can easily associate the read character string with the item name only by coloring the character string in the document for which the user wants to read the item value. That is, the information processing apparatus 10 according to the present embodiment can easily associate the character string read from the document image with the item name.

[0077] Alternatively, as described above, in the information processing apparatus 10 according to the present embodiment, the control unit 15 performs character recognition on a character string surrounded by at least one designated line among the image data of a document such as a receipt, and associates the character string obtained by the character recognition with an item name associated with the designated line surrounding the character string. As a result, the user can easily associate the read character string with the item name simply by surrounding with a designated line the character string in the document for which the item value is to be read. That is, the information processing apparatus 10 according to the present embodiment can easily associate the character string read from the document image with the item name.

[0078] The present disclosure is not limited to the above-described embodiments. For example, a plurality of blocks described in the block diagram may be integrated, or one block may be divided. Instead of executing a plurality of steps described in the flowchart in time series according to the description, each step may be executed in parallel or in a different order according to the processing capabilities of the device executing the steps, or as necessary. In addition, modifications can be made without departing from the spirit of the present disclosure.

[0079] For example, some of the processing operations executed by the information processing apparatus 10 in the above-described embodiment may be executed by the terminal device 20. In this case, for example, the terminal device 20 may execute a process of performing character recognition.

[0080] For example, some of the processing operations executed by the terminal device 20 in the above-described embodiment may be executed by the information processing apparatus 10. In this case, for example, the information processing apparatus 10 may execute a process of photographing a document.

[0081] For example, it is also possible to configure a general-purpose electronic device such as a smartphone or a computer to function as the information processing apparatus 10 according to the above-described embodiment. Specifically, a program describing the processing content for realizing each function of the information processing apparatus 10 or the like according to the embodiment is stored in the memory of the electronic device, and the program is read and executed by the processor of the electronic device. Therefore, the disclosure according to one embodiment can also be realized as a program executable by a processor.

Explanation of Signs

[0082] 1 Information processing system 10 Information processing apparatus 11 Communication unit 12 Storage unit 13 Input unit 14 Output unit 15 Control unit 20 Terminal device 21 Communication unit 22 Storage unit 23 Input unit 24 Output unit 25 Camera 26 Control unit 30 Network 101 Amount of money 102 Company name 103 Address 201~203 Regions

Claims

1. Obtain image data of a document, Perform character recognition on a character string having an area overlapping with at least one specified color in the image data, or a character string surrounded by at least one specified line in the image data, A control unit that associates the character string obtained by character recognition with an item name associated with the specified color overlapping with the character string, or an item name associated with the specified line surrounding the character string, An information processing apparatus comprising the same.

2. In the information processing apparatus according to Claim 1, Further comprising an output unit, The control unit causes the output unit to display the character string obtained by character recognition together with the associated item name. An information processing apparatus.

3. In the information processing apparatus according to Claim 1, Further comprising a storage unit that stores each specified color of the at least one specified color in association with the item name. An information processing apparatus.

4. In the information processing apparatus according to Claim 1, Further comprising a storage unit that stores each specified line of the at least one specified line in association with the item name. An information processing apparatus.

5. In the information processing apparatus according to Claim 1, The specified color is described in the document by a writing instrument. An information processing apparatus.

6. In the information processing apparatus according to Claim 1, The specified color is pre-printed at a predetermined location on the paper of the document. An information processing apparatus.

7. In the information processing apparatus according to Claim 1, The specified line is described in the document by a writing instrument. An information processing apparatus.

8. In the information processing apparatus according to Claim 1, The specified line is pre-printed at a predetermined location on the paper of the document. An information processing apparatus.

9. In the information processing apparatus according to Claim 1, The control unit is capable of performing character recognition on the character string in multiple languages. An information processing apparatus.

10. A step of obtaining image data of a document, A step of performing character recognition on a character string having an area overlapping with at least one specified color in the image data, or a character string surrounded by at least one specified line in the image data, A step of associating the character string obtained by character recognition with an item name associated with the specified color overlapping with the character string, or an item name associated with the specified line surrounding the character string, A program that causes a computer to execute operations including the above.

11. An information processing method in an information processing apparatus, A step of acquiring image data of a document, and a character string having an area overlapping with at least one designated color in the image data, or a step of performing character recognition on a character string surrounded by at least one designated line in the image data, and a step of associating the character string obtained by character recognition with an item name associated with the designated color overlapping with the character string, or an item name associated with the designated line surrounding the character string, and An information processing method including the above steps.

Images