Information processing apparatus, information processing method, and information processing program

Abstract

To improve operability when a user performs a setting operation in a reading system.SOLUTION: An image processing apparatus includes an extraction unit that extracts a candidate region of a data conversion target from a difference between first image data based on an unwritten document and second image data based on a written document, a recognition processing unit that executes character recognition processing or figure recognition processing on the second image data and acquires a recognition result, and a setting unit that sets a region of the data conversion target on the basis of the candidate region and the recognition result.SELECTED DRAWING: Figure 1

Description

[Technical Field]

[0001] The present disclosure relates to an information processing device, an information processing method, and an information processing program. [Background technology]

[0002] A reading system is known that generates image data by reading documents such as slips using a reading device (scanner) and converts the information in the document into electronic data by performing a recognition process on the generated image data. With this reading system, specific information can be converted into electronic data by performing a recognition process on the area to be converted into data (for example, the area where the amount of a transaction statement is written) among the areas in the image data. Summary of the Invention [Problem to be solved by the invention]

[0003] On the other hand, when converting information contained in a region to be digitized into electronic data, the region to be digitized needs to be set in advance by the user. Therefore, in the case of the above-mentioned reading system, the operability when the user performs the setting operation is low for image data with a large number of regions to be digitized.

[0004] The present disclosure aims to improve the operability when a user performs setting operations in a reading system. [Means for solving the problem]

[0005] According to one aspect, an information processing device includes: an extraction unit that extracts a candidate area to be digitized from a difference between first image data based on an unwritten document and second image data based on a written document; a recognition processing unit that performs character recognition processing or graphic recognition processing on the second image data and obtains a recognition result; The apparatus further includes a setting unit that sets an area to be converted into data based on the candidate area and the recognition result. [Effects of the Invention]

[0006] According to the present disclosure, it is possible to improve the operability when a user performs setting operations in a reading system. [Brief explanation of the drawings]

[0007] [Figure 1] 1 is a diagram illustrating an example of a system configuration of a reading system and an example of a functional configuration of an information processing apparatus. [Figure 2] FIG. 10 is a diagram illustrating an example of use of the reading system. [Figure 3] FIG. 10 is a diagram illustrating an example of a processing flow of the reading system. [Figure 4A] FIG. 1 is a first diagram showing an example of an unfilled document and a filled document. [Figure 4B] FIG. 2 is a second diagram showing an example of an unfilled document and a filled document. [Figure 5] FIG. 2 illustrates an example of a hardware configuration of an information processing device. [Figure 6] FIG. 2 is a diagram illustrating an example of a functional configuration of an area setting unit of the information processing device. [Figure 7] FIG. 10 is a diagram illustrating a specific example of processing by a matching unit. [Figure 8A] FIG. 10 is a first diagram showing a specific example of processing by the differential image generating unit. [Figure 8B] FIG. 10 is a second diagram showing a specific example of processing by the differential image generating unit. [Figure 9] 10A and 10B are diagrams illustrating a specific example of processing by a character recognition processing unit. [Figure 10] 10A and 10B are diagrams illustrating a specific example of processing by a pattern recognition processing unit. [Figure 11] FIG. 10 is a first diagram showing a specific example of processing by a setting unit. [Figure 12] 10 is an example of a first flowchart showing a flow of processing by a setting unit. [Figure 13] FIG. 2 is a second diagram showing a specific example of processing by the setting unit. [Figure 14] 10 is an example of a second flowchart showing the flow of processing by the setting unit. [Figure 15] 10A and 10B are diagrams illustrating a specific example of processing by an area correction unit. [Figure 16] FIG. 10 is a first diagram showing a specific example of processing by a correction unit. [Figure 17] FIG. 10 is a second diagram showing a specific example of processing by the correction unit. [Figure 18] FIG. 10 is a third diagram showing a specific example of processing by the correction unit. [Figure 19] 10 is an example of a flowchart showing a flow of processing by an area correction unit. [Figure 20] 10 is a flowchart illustrating an example of a flow of an area setting process in the reading system. [Figure 21] FIG. 10 is a diagram showing an example of a setting screen during region setting processing. [Figure 22] FIG. 4 is a fourth diagram showing a specific example of processing by the correction unit. DETAILED DESCRIPTION OF THE INVENTION

[0008] Hereinafter, each embodiment will be described with reference to the accompanying drawings. In this specification and drawings, components having substantially the same functional configurations are designated by the same reference numerals, and redundant description will be omitted.

[0009] [First embodiment] <Outline of the reading system> First, an outline of a reading system including an information processing device according to Embodiment 1 will be described. Fig. 1 is a diagram showing an example of the system configuration of the reading system and an example of the functional configuration of the information processing device.

[0010] 1, the reading system 100 includes a reading device 110 (also called a scanner) and an information processing device 120. In the reading system 100, the reading device 110 and the information processing device 120 are connected so as to be able to communicate with each other.

[0011] The reading device 110 reads a document such as a slip and generates image data. The reading device 110 transmits the generated image data to the information processing device 120.

[0012] The information processing device 120 is a device that converts specific information into electronic data by executing a recognition process on a predetermined area to be converted into data, out of the image data transmitted from the reading device 110.

[0013] An information processing program is installed in the information processing device 120. When the information processing program is executed, the information processing device 120 functions as a user interface unit 121, an area setting unit 122, an image data recognition unit 123, and a data file generation unit 124.

[0014] The user interface unit 121 provides the user with a user interface for executing a series of processes from reading a document to converting information written in a preset area to electronic data.

[0015] In the setting phase, the area setting unit 122 sets an area to be digitized, in which information to be converted into electronic data is described, from the image data transmitted from the reading device 110. When setting the area to be digitized, the area setting unit 122 also sets items indicated by the information described in the area, attributes of the information described in the area, etc.

[0016] In the recognition phase, the image data recognition unit 123 extracts the area to be digitized from the image data transmitted from the reading device 110, which area was set by the area setting unit 122 in the setting phase, and performs recognition processing on the extracted area.

[0017] The data file generation unit 124 generates a data file by arranging in a predetermined format the recognition results (electronic data) obtained by the recognition process performed by the image data recognition unit 123. The data file generation unit 124 stores the generated data file in the data file storage unit 125.

[0018] <Example of use of the reading system> Next, a description will be given of an example of use of the reading system 100. Fig. 2 is a diagram showing an example of use of the reading system.

[0019] 2, reference numeral 210 indicates an example of image data generated by reading a document with the reading device 110. The image data indicated by reference numeral 210 is an example of image data generated by reading a completed document with the zip code, address, name, gender, telephone number, etc., written by the writer with the reading device 110.

[0020] Reference numeral 211 indicates that, from the image data indicated by reference numeral 210, at least each area containing a postal code, address, and name is extracted as a target for digitization, and a recognition process is performed on each extracted area to generate a data file. As indicated by reference numeral 211, the data file includes "items" and "item values," and the recognition results are arranged in the corresponding columns according to the format.

[0021] 2, reference numeral 220 indicates an example of image data generated by reading a document with the reading device 110. The image data indicated by reference numeral 220 is an example of image data generated by reading a completed document on which the writer has written their name, date, names of executives, etc. with the reading device 110.

[0022] Reference numeral 221 shows how, from the image data shown by reference numeral 220, at least the areas containing names, dates, and executive names are extracted as targets for digitization, and a recognition process is performed on each extracted area to generate a data file. As shown by reference numeral 221, the data file includes "items" and "item values," and the recognition results are arranged in the corresponding columns according to the format.

[0023] <Reading system processing flow> Next, a description will be given of the processing flow of the reading system 100. Here, the processing flow of the reading system 100 will be described in comparison with a comparative example. Fig. 3 is a diagram showing an example of the processing flow of the reading system.

[0024] Of these, Figure 3(a) shows the processing flow of the reading system of the comparative example. As shown in Figure 3(a), in the case of the reading system of the comparative example, in the setting phase, a document that has been written by any of the writers is used.

[0025] Specifically, the document on which the information has been written is read by a reading device, and image data is generated. The read image data is displayed to the user, and the user manually inputs an area of ​​the displayed image data to be converted into data and subjected to recognition processing. For example, the user manually inputs a circumscribing rectangular area that surrounds the character information to be recognized. The user manually inputs a circumscribing rectangular area for all character information to be recognized. The user then sets the input circumscribing rectangular area as the area to be converted into data.

[0026] Furthermore, as shown in FIG. 3(a), in the case of the reading system of the comparative example, in the recognition phase, all written documents on which writing has been performed by the writer are targeted.

[0027] Specifically, all of the written documents are read in sequence by a reading device, and multiple pieces of image data are generated. Recognition processing is performed on each of the multiple pieces of image data generated for the area to be digitized that was previously set in the setting phase. The recognition results (electronic data) obtained from the area to be digitized by the recognition processing are arranged in a predetermined format, and a data file is generated. The generated data file is stored in a data file storage unit.

[0028] On the other hand, Fig. 3(b) shows a processing flow of the reading system 100. As shown in Fig. 3(b), in the case of the reading system 100, in the setting phase, an unwritten document that has not been written by a writer and a written document that has been written by any writer are used.

[0029] Specifically, the reading device 110 reads the unwritten document and the written document, and generates image data for each. The read image data is subjected to a matching process, a difference process, and a recognition process to extract an area to be converted into data. The extracted area to be converted into data is then corrected as necessary, and the corrected area to be converted into data is automatically set.

[0030] In this way, the information processing device 120 according to the first embodiment makes it possible to automatically set the areas to be digitized. As a result, according to the first embodiment, even for image data with a large number of areas to be digitized, the user can improve the operability when performing the setting operation compared to when the areas to be digitized are manually set.

[0031] The processing flow in the recognition phase of FIG. 3(b) is the same as the processing flow in the recognition phase of FIG. 3(a), and therefore will not be described here.

[0032] <Examples of documents> Next, specific examples of unwritten documents and written documents used by the reading system 100 in the setting phase will be described. FIG. 4A is a first diagram showing examples of unwritten documents and written documents. Of these, reference numeral 210 is an example of image data generated by reading a written document with the reading device 110, and is the same as the image data shown with reference numeral 210 in FIG. 2. On the other hand, reference numeral 410 is image data corresponding to the image data shown with reference numeral 210, and is an example of image data generated by reading an unwritten document with the reading device 110.

[0033] 4B is a second diagram showing an example of an unfilled document and a filled document. Of these, reference numeral 220 is an example of image data generated by reading a filled document with the reading device 110, and is the same as the image data shown with reference numeral 220 in FIG. 2. On the other hand, reference numeral 420 is image data corresponding to the image data shown with reference numeral 220, and is an example of image data generated by reading an unfilled document with the reading device 110.

[0034] <Hardware configuration of information processing device> Next, the hardware configuration of the information processing device 120 will be described. Fig. 5 is a diagram showing an example of the hardware configuration of the information processing device. As shown in Fig. 5, the information processing device 120 has a processor 501, a memory 502, an auxiliary storage device 503, a connection device 504, a communication device 505, and a drive device 506. The hardware components included in the information processing device 120 are connected to each other via a bus 507.

[0035] The processor 501 has various arithmetic devices such as a CPU (Central Processing Unit), a GPU (Graphics Processing Unit), etc. The processor 501 reads various programs (for example, information processing programs, etc.) into the memory 502 and executes them.

[0036] The memory 502 has a main storage device such as a ROM (Read Only Memory), a RAM (Random Access Memory), etc. The processor 501 and the memory 502 form a so-called computer, and the processor 501 executes various programs read onto the memory 502, causing the computer to realize various functions.

[0037] The auxiliary storage device 503 stores various programs and various information used when the processor 501 executes the various programs.

[0038] The connection device 504 connects the information processing device 120 to external devices (such as an operation device 511 and a display device 512).

[0039] The communication device 505 transmits and receives various information between the reading device 110 and the information processing device 120 .

[0040] The drive device 506 is a device for setting the recording medium 513. The recording medium 513 here includes media that record information optically, electrically, or magnetically, such as a CD-ROM, a flexible disk, a magneto-optical disk, etc. The recording medium 513 may also include semiconductor memories that record information electrically, such as ROMs, flash memories, etc.

[0041] The various programs to be installed in the auxiliary storage device 503 are installed, for example, by setting the distributed recording medium 513 in the drive device 506 and reading the various programs recorded on the recording medium 513 by the drive device 506. Alternatively, the various programs to be installed in the auxiliary storage device 503 may be installed by being downloaded from a network (not shown) via the communication device 505.

[0042] <Functional configuration of area setting unit of information processing device> Next, a description will be given of the functional configuration of the area setting unit 122 of the information processing device 120, which operates in the setting phase. Fig. 6 is a diagram showing an example of the functional configuration of the area setting unit of the information processing device.

[0043] As shown in FIG. 6, the area setting unit 122 includes an image data acquisition unit 610, a matching unit 620, a difference image generation unit 630, a character recognition processing unit 640, a graphic recognition processing unit 650, an extraction unit 660, a setting unit 670, and an area correction unit 680.

[0044] The image data acquisition unit 610 acquires from the reading device 110 image data (first image data) generated by reading an unwritten document and image data (second image data) generated by reading a written document.

[0045] The comparison unit 620 compares the first image data and the second image data acquired by the image data acquisition unit 610, and compares whether the unwritten document read by the reading device 110 in the setting phase and the written document are documents of the same format.

[0046] The matching unit 620 may use any matching method. For example, the matching unit 620 may perform matching based on the similarity between the feature extracted from the first image data and the feature extracted from the second image data. For example, AKAZE features, ORB features, etc. may be used as the feature. Furthermore, the similarity between the feature may be calculated using the knn nearest neighbor method using the L1 norm, the ε nearest neighbor method using the Hamming distance, etc.

[0047] In the first embodiment, in order to improve matching accuracy, the matching unit 620 performs matching using edges, frames, lines, and pixel density within the first image data and second image data, and also performs positioning processing by matching the edges of the first image data and second image data.

[0048] If the collation unit 620 determines that the documents are not of the same format (if the collation fails), it displays a message indicating the failure of the collation to the user via the user interface unit 121. In this case, the area setting process by the area setting unit 122 ends.

[0049] On the other hand, if it is determined that the documents have the same format (if the collation is successful), the collation unit 620 displays a message of successful collation to the user via the user interface unit 121. In this case, the area setting process by the area setting unit 122 continues.

[0050] The differential image generation unit 630 calculates the difference between the first image data and the second image data acquired by the image data acquisition unit 610, and generates differential image data. The differential image generation unit 630 notifies the generated differential image data to the extraction unit 660 and the character recognition processing unit 640.

[0051] The character recognition processing unit 640 is a functional unit that performs character recognition processing on the second image data or differential image data and acquires information necessary to set the area to be digitized, etc., and further has a recognition unit 641, an identification unit 642, and an estimation unit 643.

[0052] The recognition unit 641 executes character recognition processing on the second image data acquired by the image data acquisition unit 610, and when acquiring character information as a result of the character recognition, notifies the identification unit 642 of the acquired character information.

[0053] Furthermore, the recognition unit 641 executes character recognition processing on the differential image data generated by the differential image generation unit 630, and when character information is acquired as a result of the character recognition, it notifies the specification unit 642 of the acquired character information.

[0054] The specification unit 642 determines whether the attribute of the character information acquired as a character recognition result by executing the character recognition process on the second image data is mark information. Mark information refers to a character string indicating a mark, for example,

[0055]

number

[0056] Furthermore, the specifying unit 642 determines whether the attribute of the character information acquired as a character recognition result by executing character recognition processing on the differential image data is text. If the specifying unit 642 determines that the attribute of the character information is not mark information, the specifying unit 642 determines that the attribute of the character information is text, and notifies the estimation unit 643 of the character information whose attribute has been determined to be text.

[0057] When the estimation unit 643 is notified by the identification unit 642 of character information whose attribute is determined to be text, the estimation unit 643 estimates the item indicated by the character information and notifies the setting unit 670 of the estimated item. The estimation unit 643 may estimate the item by any method, and for example, a classifier that identifies the item to be estimated (such as name, address, etc.) may be prepared in advance, and the item indicated by the character information may be estimated by inputting the character information to the classifier. For example, a machine-learned model that has been trained in advance using a natural language library such as FastText may be used as the classifier.

[0058] Furthermore, for character information identified as "other" by the classifier, the estimation unit 643 may further estimate the item based on a predetermined notation rule. For example, when estimating whether the item indicated by the character information is a telephone number, the estimation may be based on notation rules such as that the number is 10 digits and that the number is an actual area code. Alternatively, the estimation may be based on notation rules such as that the number is 11 digits and that the first three digits are either 070, 080, or 090.

[0059] Furthermore, for example, when estimating whether or not the item indicated by the character information is a postal code, the estimation may be based on notation rules such as the number of digits being seven and the fact that it is an actual postal code.

[0060] Furthermore, for example, when estimating whether an item indicated by character information is a social security number, it may be estimated based on the notation rule that the number is 12 digits and the value calculated from the left 11 digits matches the check digit at the end. The check digit is "x" in the 12-digit number "abcdefghijkx", where "x" is x=11-(6a+5b+4c+3d+2e+7f+6g+5h+4i+3j+2k)mod11 (However, if x≧10, then x=0) It is calculated as follows.

[0061] The graphic recognition processing unit 650 is a functional unit that executes graphic recognition processing on the second image data and acquires information necessary to set the area to be converted into data.

[0062] The recognition unit 651 executes graphic recognition processing on the second image data acquired by the image data acquisition unit 610. When the recognition unit 651 acquires code information as a result of graphic recognition, it notifies the setting unit 670 of the area of ​​the acquired code information and also notifies the identification unit 652 of the acquired code information. Furthermore, when the recognition unit 651 acquires image information other than code information as a result of graphic recognition, it notifies the setting unit 670 of the area of ​​the acquired image information.

[0063] When the identification unit 652 receives code information as a graphic recognition result from the recognition unit 651, the identification unit 652 identifies the type of the code information. For example, if the code information is a barcode, the type of code information refers to UPC-A / EAN / JAN, Code3of9, CODE128 / EAN128, PDF417, etc. Also, if the code information is a QR Code (registered trademark), the type of code information refers to Model 1, Model 2, Micro QR, etc.

[0064] The extraction unit 660 extracts candidate areas to be converted into data based on the differential image data notified by the differential image generation unit 630, and notifies the setting unit 670 of the extracted candidate areas to be converted into data.

[0065] The setting unit 670 sets the area to be converted into data based on the candidate area to be converted into data notified by the extraction unit 660 and the character recognition result notified by the character recognition processing unit 640 or the graphic recognition result notified by the graphic recognition processing unit 650.

[0066] Specifically, the setting section 670 sets the candidate area to be converted into data notified by the extraction section 660 (the candidate area to be converted into data extracted based on the differential image data) as the area to be converted into data.

[0067] Furthermore, the setting unit 670 sets the area of ​​the character information, which has been notified by the specifying unit 642 of the character recognition processing unit 640 and which has been determined to have the attribute of mark information, as the area to be converted into data.

[0068] Furthermore, the setting unit 670 sets the area of ​​the code information and image information notified by the recognition unit 651 of the graphic recognition processing unit 650 as the area to be converted into data.

[0069] Furthermore, the setting unit 670 sets the items notified by the estimation unit 643 of the character recognition processing unit 640 in association with the area to be converted into data. The setting unit 670 sets the type of code information notified by the identification unit 652 of the graphic recognition processing unit 650 in association with the area to be converted into data.

[0070] When a correction instruction is received from the area correction section 680, the setting section 670 sets the area to be converted into data after correction.

[0071] When the area correction unit 680 detects a line segment at a corresponding position in the area to be converted into data set by the setting unit 670, the area correction unit 680 corrects the set area to be converted into data based on the detected line segment. Furthermore, the area correction unit 680 issues a correction instruction to the setting unit 670 so as to set the area to be converted into data after the correction. The area correction unit 680 has a detection unit 681, a determination unit 682, and a correction unit 683.

[0072] The detection unit 681 detects line segments from the second image data by executing a line segment detection process on the second image data acquired by the image data acquisition unit 610. The line segment detection method used by the detection unit 681 is arbitrary, and the detection unit 681 may detect line segments by, for example, performing a Hough transform on the second image data.

[0073] The determining unit 682 determines the type of the detected line segment according to the positional relationship between the line segment detected by the detecting unit 681 and the set area to be converted into data.

[0074] The correction unit 683 corrects the shape of the set area to be converted into data in accordance with the determination result by the determination unit 682. Specifically, if it is determined that the detected line segment is a rectangular frame surrounding the set area to be converted into data, the correction unit 683 corrects the shape of the set area to be converted into data based on the rectangular frame. Furthermore, if it is determined that the detected line segment is an underline of the set area to be converted into data, the correction unit 683 corrects the shape of the set area to be converted into data based on the underline. The correction unit 683 notifies the setting unit 670 of the corrected area to be converted into data, and issues a correction instruction to set the corrected area to be converted into data.

[0075] <Specific examples of processing by each unit included in the area setting unit> Next, we will explain specific examples of processing by each unit included in the area setting unit 122 (here, the matching unit 620, the differential image generation unit 630, the character recognition processing unit 640, the graphic recognition processing unit 650, the extraction unit 660, the setting unit 670, and the area correction unit 680).

[0076] (1) Specific example of processing by the collation unit 620 First, a specific example of processing by the matching unit 620 will be described. Fig. 7 is a diagram showing a specific example of processing by the matching unit. As shown in Fig. 7, the matching unit 620 further includes a feature extraction unit 701, a feature extraction unit 702, and a feature matching unit 703.

[0077] The feature extraction unit 701 extracts features from the first image data acquired by the image data acquisition unit 610 (image data generated by reading an unwritten document using the reading device 110) and notifies the feature matching unit 703.

[0078] The feature extraction unit 702 extracts features from the second image data acquired by the image data acquisition unit 610 (image data generated by reading a written document using the reading device 110) and notifies the feature matching unit 703.

[0079] The feature matching unit 703 calculates the similarity between the feature notified from the feature extraction unit 701 and the feature notified from the feature extraction unit 702, and compares the unfilled document and the filled document to determine whether they are documents of the same format based on the calculated similarity. If the feature matching unit 703 determines that the documents are not of the same format (if the comparison fails), it displays a message indicating the comparison failed to the user via the user interface unit 121. In this case, the area setting process by the area setting unit 122 ends.

[0080] On the other hand, if it is determined that the documents have the same format (if the matching is successful), the feature matching unit 703 displays a message of successful matching to the user via the user interface unit 121. In this case, the area setting process by the area setting unit 122 continues.

[0081] (2) Specific example of processing by the differential image generating unit 630 Next, a description will be given of a specific example of processing by the difference image generating section 630. Fig. 8A is a first diagram showing a specific example of processing by the difference image generating section.

[0082] 8A, reference numeral 410 indicates first image data (image data generated by reading an unwritten document with the reading device 110) acquired by the image data acquisition unit 610. Reference numeral 210 indicates second image data (image data generated by reading a written document with the reading device 110) acquired by the image data acquisition unit 610.

[0083] The first image data indicated by the reference numeral 410 and the second image data indicated by the reference numeral 210 are input to the differential image generating unit 630, whereby the differential image generating unit 630 calculates the difference and generates differential image data. In Fig. 8A, reference numeral 810 indicates differential image data generated based on the first image data indicated by the reference numeral 410 and the second image data indicated by the reference numeral 210.

[0084] 8B is a second diagram showing a specific example of processing by the differential image generation unit. In FIG. 8B, reference numeral 420 indicates the first image data (image data generated by reading an unwritten document with the reading device 110) acquired by the image data acquisition unit 610. Reference numeral 220 indicates the second image data (image data generated by reading a written document with the reading device 110) acquired by the image data acquisition unit 610.

[0085] The first image data indicated by the reference numeral 420 and the second image data indicated by the reference numeral 220 are input to the differential image generation unit 630, whereby the differential image generation unit 630 calculates the difference and generates differential image data. In Fig. 8B, reference numeral 820 indicates differential image data generated based on the first image data indicated by the reference numeral 420 and the second image data indicated by the reference numeral 220.

[0086] (3) Specific example of processing by the character recognition processing unit 640 Next, a specific example of processing by the character recognition processing unit 640 will be described. Fig. 9 is a diagram showing a specific example of processing by the character recognition processing unit.

[0087] As described above, the recognition unit 641 of the character recognition processing unit 640 performs character recognition processing on the second image data. In Fig. 9, the reference numeral 210 indicates the second image data, which is the same as the second image data indicated by the reference numeral 210 in Fig. 8A.

[0088] The example in Figure 9 shows that the recognition unit 641 performs character recognition processing on the area indicated by the symbol 901 of the second image data indicated by the symbol 210, and obtains character information="checked mark" and character information="unchecked mark" as the character recognition results.

[0089] 9 shows a state in which the specification unit 642 has determined that the attributes of the character information="checked mark" and character information="unchecked mark" acquired as the character recognition results are mark information. In FIG. 9, reference numeral 902 indicates a state in which the setting unit 670 has been notified of the area of ​​the character information whose attribute has been determined to be mark information.

[0090] As described above, the recognition unit 641 of the character recognition processing unit 640 performs character recognition processing on the differential image data. In Fig. 9, reference numeral 810 denotes differential image data, which is the same as the differential image data denoted by reference numeral 810 in Fig. 8A.

[0091] The example of Fig. 9 shows a state in which the recognition unit 641 has performed character recognition processing on the area indicated by reference numeral 911 of the differential image data indicated by reference numeral 810. Furthermore, in the example of Fig. 9, reference numeral 912 shows a state in which the recognition unit 641 has performed character recognition processing on the area indicated by reference numeral 911, and has acquired character information="Ta", "Ka", "Mo", "To", "Ta", "Ro", and "U" as the character recognition result.

[0092] Also, in the example of Figure 9, symbol 913 indicates that the identification unit 642 has determined that the attributes of the character information="Ta", "Ka", "Mo", "To", "Ta", "Ro", and "U" obtained as the character recognition result are text (not mark information).

[0093] Also, in the example of Figure 9, symbol 914 indicates that the estimation unit 643 has estimated that the item indicated by the character information="Ta", "Ka", "Mo", "To", "Ta", "Ro", and "U" is a name, and has notified the setting unit 670 of the estimated item.

[0094] (4) Specific example of processing by the graphic recognition processing unit 650 Next, a specific example of the processing by the pattern recognition processing unit 650 will be described. Fig. 10 is a diagram showing a specific example of the processing by the pattern recognition processing unit.

[0095] As described above, the recognition unit 651 of the graphic recognition processing unit 650 performs graphic recognition processing on the second image data. In the example of Fig. 10, the reference numeral 1010 denotes second image data generated by reading a written document with the reading device 110, and is an example of second image data containing multiple pieces of information other than character information. For ease of explanation, the following description will be given taking as an example image data generated by reading a document containing multiple images and codes with the reading device 110.

[0096] As shown in FIG. 10, the recognition unit 651 performs a graphic recognition process on the second image data indicated by the symbol 1010, whereby the recognition unit 651 recognizes the barcode indicated by the symbol 1011, the QR code (registered trademark) indicated by the symbol 1012, and the graphic indicated by the symbol 1013.

[0097] 10, reference numeral 1021 indicates a state in which the recognition unit 651 acquires code information and image information (the barcode indicated by reference numeral 1011, the QR code (registered trademark) indicated by reference numeral 1012, and the figure indicated by reference numeral 1013) by executing a figure recognition process. Reference numeral 1021 also indicates a state in which the setting unit 670 is notified of the areas of the acquired code information and image information (the barcode indicated by reference numeral 1011, the QR code (registered trademark) indicated by reference numeral 1012, and the figure indicated by reference numeral 1013).

[0098] Furthermore, the example in Figure 10 shows that the identification unit 652 has determined that the type of barcode indicated by symbol 1011 acquired by the recognition unit 651 as a result of the graphic recognition process being executed is one of the types indicated by symbol 1014.

[0099] Similarly, the example in Figure 10 shows that the identification unit 652 has determined that the type of QR code (registered trademark) indicated by symbol 1012 obtained by the recognition unit 651 through execution of a graphic recognition process is one of the types indicated by symbol 1015.

[0100] (5) Specific Example of Processing by the Setting Unit 670 (Text Information) Next, a specific example of processing relating to text information by setting unit 670 will be described. Fig. 11 is a first diagram showing a specific example of processing by the setting unit.

[0101] As described above, when the setting unit 670 performs processing related to character information, the extraction unit 660 extracts candidate areas to be converted into data based on the differential image data notified by the differential image generation unit 630, and notifies the setting unit 670. Furthermore, the identification unit 642 of the character recognition processing unit 640 notifies the setting unit 670 of the areas of character information whose attribute has been determined to be mark information. Furthermore, the estimation unit 643 of the character recognition processing unit 640 notifies the setting unit 670 of the items indicated by the character information whose attribute has been determined to be text.

[0102] In Fig. 11, reference numeral 810 denotes differential image data, which is the same as the differential image data indicated by reference numeral 810 in Fig. 8A. In Fig. 11, reference numeral 902 denotes character information whose attribute has been determined to be mark information, which is the same as the character information indicated by reference numeral 902 in Fig. 9.

[0103] In the example of FIG. 11, each rectangle in the reference numeral 1110 indicates a candidate area to be converted into data, extracted based on the differential image data indicated by the reference numeral 810 .

[0104] In the example of FIG. 11, each rectangle in the reference numeral 1120 is A candidate area to be converted into data extracted based on the differential image data indicated by the reference numeral 810; An area of ​​character information (code 902) whose attribute is mark information; 6 shows how the setting unit 670 sets the area to be converted into data based on the above.

[0105] The rectangles indicated by reference numerals 1121 and 1122 in reference numeral 1120 indicate areas to be converted into data whose attributes are set based on the character information of the mark information (reference numeral 902). Of these, the rectangle indicated by reference numeral 1121 is also included in the candidate areas to be converted into data extracted by the extraction unit 660 based on the differential image data indicated by reference numeral 810. In other words, the rectangle indicated by reference numeral 1121 is set by overwriting by the setting unit 670. On the other hand, the rectangle indicated by reference numeral 1122 is not included in the candidate areas to be converted into data extracted by the extraction unit 660 based on the differential image data indicated by reference numeral 810. In other words, if the area whose attribute is the character information of the mark information (reference numeral 902) had not been set as the area to be converted into data, this area would not have been set as the area to be converted into data.

[0106] In this way, the setting unit 670 sets the areas to be converted into data related to the character information based on the candidate areas to be converted into data extracted based on the differential image data and the character recognition result for the second image data. In this way, the setting unit 670 can set all the areas to be converted into data.

[0107] 11, reference numeral 1130 denotes an item indicated by character information determined to have a text attribute. In FIG. 11, reference numeral 1140 denotes a state in which the setting unit 670 sets an item indicated by character information determined to have a text attribute in association with the area to be converted into data.

[0108] FIG. 12 is an example of a first flowchart illustrating the flow of processing by the setting unit.

[0109] In step S1201, the extraction unit 660 extracts candidate areas to be converted into data based on the acquired differential image data, and notifies the setting unit 670 of the extraction.

[0110] In step S1202, the specifying unit 642 of the character recognition processing unit 640 notifies the setting unit 670 of the area of ​​the character information whose attribute has been determined to be mark information.

[0111] In step S1203, the estimation unit 643 of the character recognition processing unit 640 notifies the setting unit 670 of the item indicated by the character information whose attribute has been determined to be text.

[0112] In step S1204, the setting unit 670 sets the candidate area to be converted into data notified in step S1201 and the area of ​​character information notified in step S1202 as areas to be converted into data. The setting unit 670 also sets the items notified in step S1203 in association with the areas to be converted into data.

[0113] (6) Specific Examples of Processing by the Setting Unit 670 (Code Information and Image Information) Next, a description will be given of a specific example of processing relating to code information and image information by the setting unit 670. Fig. 13 is a second diagram showing a specific example of processing by the setting unit.

[0114] As described above, the recognition unit 651 of the graphic recognition processing unit 650 notifies the setting unit 670 of the areas of the code information and image information. In addition, the specification unit 652 of the graphic recognition processing unit 650 notifies the setting unit 670 of the type of code information.

[0115] In Fig. 13, reference numeral 1021 indicates an area of ​​code information and image information, which is the same as the area of ​​code information and image information indicated by reference numeral 1021 in Fig. 10. In the example of Fig. 13, each rectangle within reference numeral 1310 indicates that the setting unit 670 has set the area of ​​code information and image information indicated by reference numeral 1021 as the area to be converted into data.

[0116] In this way, the setting unit 670 sets the area of ​​the code information and the image information as the area to be converted into data, based on the graphic recognition result for the second image data.

[0117] In Fig. 13, the reference numeral 1320 indicates the type of code information. In the example of Fig. 13, the reference numeral 1330 indicates how the setting unit 670 sets the type of code information in association with the area to be converted into data.

[0118] FIG. 14 is an example of a second flowchart showing the flow of processing by the setting unit.

[0119] In step S1401, the recognition unit 651 of the graphic recognition processing unit 650 notifies the setting unit 670 of the areas of the code information and image information.

[0120] In step S1402, the specifying unit 652 of the graphic recognition processing unit 650 notifies the setting unit 670 of the type of code information.

[0121] In step S1403, the setting unit 670 sets the area of ​​the code information and image information notified in step S1401 as the area to be converted into data. The setting unit 670 also sets the type of code information notified in step S1402 in association with the area to be converted into data.

[0122] (7) Specific example of processing by the area correction unit 680 Next, a specific example of processing by area correction section 680 will be described. Here, a specific example of processing related to character information by area correction section 680 will be described with reference to Fig. 15 to Fig. 19. Fig. 15 is a diagram showing a specific example of processing by the area correction section.

[0123] As described above, with regard to character information, the setting unit 670 sets, as the area to be converted into data, the candidate area to be converted into data notified by the extraction unit 660 and the area of ​​character information notified by the character recognition processing unit 640 and determined to have the attribute of mark information. Meanwhile, the detection unit 681 detects line segments from the second image data by executing a line segment detection process on the second image data. Furthermore, the correction unit 683 determines the type of the detected line segment according to the positional relationship between the detected line segment and the set area to be converted into data, and corrects the shape of the area to be converted into data based on the determined type.

[0124] In the example of Fig. 15, reference numeral 210 indicates second image data, which is the same as the second image data indicated by reference numeral 210 in Fig. 8A. Also, in the example of Fig. 15, reference numeral 1510 indicates a line segment detected by executing a line segment detection process on the second image data indicated by reference numeral 210.

[0125] Similarly, in the example of Fig. 15, reference numeral 220 indicates second image data, which is the same as the second image data indicated by reference numeral 220 in Fig. 8B. Also, in the example of Fig. 15, reference numeral 1520 indicates a line segment detected by executing a line segment detection process on the second image data indicated by reference numeral 220.

[0126] Fig. 16 is the first diagram showing a specific example of processing by the correction unit, and shows a specific example of processing by the correction unit 683 when the determination unit 682 determines that the type of the detected line segment is a rectangular frame. In the example of Fig. 16, reference numeral 1610 indicates a part of the line segment indicated by reference numeral 1510 in Fig. 15, and reference numeral 1620 indicates an area to be converted into data at a position corresponding to the line segment indicated by reference numeral 1610. Note that the example of Fig. 16 also shows differential image data of the area indicated by reference numeral 1620 when the area is set as the area to be converted into data (see reference numeral 1621).

[0127] The example in Figure 16 shows how the determination unit 682 has determined that the line segment indicated by reference numeral 1610 is a rectangular frame that contains the area to be converted into data, based on the positional relationship between the line segment indicated by reference numeral 1610 and the area to be converted into data, indicated by reference numeral 1620 (see reference numeral 1630).

[0128] Here, when the correction unit 683 determines that the detected line segment is a rectangular frame that contains the area to be converted into data, it corrects the area to be converted into data so that it expands to at least the area inside the smallest rectangular frame that contains the area to be converted into data. In Fig. 16, reference numeral 1640 indicates a state in which the area to be converted into data (reference numeral 1620) has been corrected to expand to the area inside the smallest rectangular frame (reference numeral 1610) that contains the area to be converted into data (reference numeral 1620).

[0129] Furthermore, the correction unit 683 determines whether or not adjacent areas of the post-correction data conversion target area (1640) satisfy a predetermined linking condition as a result of correcting the area to be converted to data (1620) to enlarge it. The correction unit 683 determines that the adjacent areas satisfy the predetermined linking condition if they satisfy any of the following: The distance between adjacent regions is less than a predetermined threshold. Adjacent areas are in contact with each other. Adjacent areas partially overlap.

[0130] If the correction unit 683 determines that the adjacent regions satisfy a predetermined connection condition, it corrects the region shapes of the adjacent regions by connecting the adjacent regions.

[0131] In the example of FIG. 16, reference numeral 1650 indicates an area to be converted into data after correction, in which adjacent areas are connected together.

[0132] Fig. 17 is a second diagram showing a specific example of processing by the correction unit, and shows a specific example of another process by the correction unit 683 when the determination unit 682 determines that the type of the detected line segment is a rectangular frame. In the example of Fig. 17, reference numeral 1710 indicates a portion of the line segment indicated by reference numeral 1510 in Fig. 15, and reference numeral 1720 indicates an area to be converted into data at a position corresponding to the line segment indicated by reference numeral 1710. Note that the example of Fig. 17 also shows the character information of the area indicated by reference numeral 1720 (character information whose attribute is determined to be mark information) when the area indicated by reference numeral 1720 is set as the area to be converted into data (see reference numeral 1721).

[0133] The example in Figure 17 shows how the determination unit 682 has determined that the line segment indicated by reference numeral 1710 is a rectangular frame that contains the area to be converted into data, based on the positional relationship between the line segment indicated by reference numeral 1710 and the area to be converted into data, indicated by reference numeral 1720 (see reference numeral 1730).

[0134] Here, if it is determined that the detected line segment is a rectangular frame that includes the area to be converted into data, the correction unit 683 Whether the size of the area to be digitized is below a predetermined threshold, Whether the ratio of the area to be digitized to the rectangular frame is equal to or less than a predetermined threshold, If the result of the determination is that the size of the area to be digitized is equal to or smaller than a predetermined threshold and the ratio of the area to be digitized to the rectangular frame is equal to or smaller than a predetermined threshold, the correction unit 683 determines not to perform correction on the area to be digitized. Note that if the result of the determination is that the size of the area to be digitized is not equal to or smaller than the predetermined threshold or the ratio of the area to be digitized to the rectangular frame is not equal to or smaller than a predetermined threshold, the correction unit 683 performs correction similar to the example shown in FIG.

[0135] The example in Figure 17 is The size of the area to be digitized (reference numeral 1720) is equal to or smaller than a predetermined threshold, and The ratio of the area to be converted to data (reference numeral 1720) to the rectangular frame (reference numeral 1720) is equal to or less than a predetermined threshold value. 17 shows that the correction unit 683 did not issue a correction instruction for the area to be converted into data (reference numeral 1720) by determining that the area is not correct.

[0136] Fig. 18 is a third diagram showing a specific example of processing by the correction unit, and shows a specific example of processing by the correction unit 683 when the determination unit 682 determines that the type of the detected line segment is an underline. In the example of Fig. 18, reference numeral 1810 indicates a portion of the line segment indicated by reference numeral 1520 in Fig. 15, and reference numeral 1820 indicates an area to be converted into data at a position corresponding to the line segment indicated by reference numeral 1810. Note that the example of Fig. 18 also shows differential image data of the area indicated by reference numeral 1820 when the area indicated by reference numeral 1820 is set as the area to be converted into data (see reference numeral 1821).

[0137] The example in Figure 18 shows how the determination unit 682 has determined that the area to be converted into data is an underlined area based on the positional relationship (see reference numeral 1830) between the line segment shown at reference numeral 1810 and the area to be converted into data shown at reference numeral 1820.

[0138] Here, when the correction unit 683 determines that the area to be converted into data is an area having an underline, it corrects the area to be converted into data by expanding it in the length direction of the underline according to the length of the underline, and by expanding it in the height direction according to the height of other areas to be converted into data that are within the range of the underline.

[0139] In FIG. 18, reference numeral 1840 denotes the area to be converted into data (reference numeral 1820), The underline (code 1810) expands in the length direction according to its length. - Expand in height according to the height of other data target areas within the underlined range (code 1810), The figure shows the correction results.

[0140] FIG. 19 is an example of a flowchart showing the flow of processing by the area correction unit.

[0141] In step S1901, the area correction unit 680 executes line segment detection processing on the second image data to detect line segments from the second image data.

[0142] In step S1902, the area correction unit 680 reads, from the setting unit 670, the area to be converted into data that corresponds to the position of the line segment of interest.

[0143] In step S1903, the area correction unit 680 determines the type of the target line segment from the positional relationship between the target line segment and the read area to be converted into data.

[0144] In step S1904, area correction unit 680 determines whether the type of the target line segment is a rectangular frame or an underline. If it is determined in step S1904 that the type is a rectangular frame, the process proceeds to step S1905.

[0145] In step S1905, area correction unit 680 determines whether the size of the area to be digitized read in step S1902 is equal to or smaller than a predetermined threshold, and whether the proportion of the area to be digitized in the rectangular frame is equal to or smaller than a predetermined threshold. If it is determined in step S1905 that the size of the area to be digitized is equal to or smaller than the predetermined threshold, and the proportion of the area to be digitized in the rectangular frame is equal to or smaller than a predetermined threshold (YES in step S1905), the process proceeds to step S1910.

[0146] On the other hand, if it is determined in step S1905 that the size of the area to be digitized is not equal to or less than a predetermined threshold, or that the proportion of the area to be digitized in the rectangular frame is not equal to or less than a predetermined threshold (NO in step S1905), proceed to step S1906.

[0147] In step S1906, the area correction unit 680 performs correction to expand the area to be converted into data to at least the area inside the rectangular frame.

[0148] In step S1907, area correction unit 680 determines whether adjacent areas satisfy a predetermined linking condition for the area to be converted into data after the enlargement correction in step S1906. If it is determined in step S1907 that the predetermined linking condition is not satisfied (NO in step S1907), the process proceeds to step S1910. On the other hand, if it is determined in step S1907 that the predetermined linking condition is satisfied (YES in step S1907), the process proceeds to step S1908.

[0149] In step S1908, the area correction unit 680 performs correction to connect the areas to be converted into data after correction that have been determined to satisfy the predetermined connection condition.

[0150] On the other hand, if it is determined in step S1904 that the area is an underline, the process proceeds to step S1909. In step S1909, area correction unit 680 corrects the area to be converted into data by expanding it in the length direction according to the length of the underline, and by expanding it in the height direction according to the height of other areas to be converted into data that are within the range of the underline.

[0151] In step S1910, area correction unit 680 determines whether the processes of steps S1902 to S1909 have been executed for all line segments detected in step S1901. If it is determined in step S1910 that there are line segments for which processing has not been executed (NO in step S1910), the line segments for which processing has not been executed are treated as target line segments and the process returns to step S1902.

[0152] On the other hand, if it is determined in step S1910 that the processing has been executed for all line segments (YES in step S1910), the process proceeds to step S1911.

[0153] In step S1911, the area correction unit 680 sets the area to be converted into data after correction, and ends the correction process.

[0154] <Area setting process flow in the reading system> Next, a description will be given of the flow of area setting processing in the reading system 100. Fig. 20 is an example of a flowchart showing the flow of area setting processing in the reading system.

[0155] In step S2001, the reading device 110 reads an unwritten document and a written document, generates first image data and second image data, and transmits them to the information processing device 120.

[0156] In step S2002, the information processing device 120 executes a matching process on the first image data and the second image data transmitted from the reading device 110.

[0157] In step S2003, the information processing device 120 determines whether the matching process in step S2002 was successful. If it is determined that the matching process in step S2002 was unsuccessful (NO in step S2003), the area setting process ends. On the other hand, if it is determined that the matching process in step S2002 was successful (YES in step S2003), the process proceeds to steps S2011 to S2041.

[0158] In step S2011, the information processing device 120 executes a difference image generation process to generate difference image data based on the first image data and the second image data.

[0159] In step S2012, the information processing device 120 executes extraction processing to extract candidate regions to be converted into data from the differential image data.

[0160] In step S2013, the information processing device 120 executes character recognition processing on the differential image data and estimates the item indicated by the character information whose attribute is text.

[0161] In step S2021, the information processing device 120 executes character recognition processing on the second image data to determine character information whose attribute is mark information.

[0162] In step S2031, the information processing device 120 executes a graphic recognition process on the second image data to recognize code information and image information, and to determine the type of code information.

[0163] In step S2041, the information processing device 120 executes line segment detection processing on the second image data to detect line segments.

[0164] In step S2051, the information processing device 120 The candidate area to be converted into data extracted in step S2012; The region of character information determined in step S2021; The area of ​​the code information and image information recognized in step S2031; As a region to be converted into data, the information processing device 120 also sets the item indicated by the character information estimated in step S2013 and the type of code information determined in step S2031 in association with the region to be converted into data.

[0165] In step S2052, the information processing device 120 corrects the area to be converted into data based on the line segments detected in step S2041, and ends the area setting process.

[0166] <Example of setting screen during area setting process> Next, a description will be given of a setting screen displayed on the display device 512 connected to the information processing device 120 by the user interface unit 121 during the area setting process in the reading system 100. Fig. 21 is a diagram showing an example of the setting screen during the area setting process.

[0167] As shown in FIG. 21, a setting screen 2100 has a second image data display area 2110, a region display area 2120, a region detailed information display area 2130, and a recognition result display area 2140.

[0168] The second image data display range 2110 displays the second image data generated by reading the written document with the reading device 110. Also, the second image data display range 2110 displays an area frame superimposed on the second image data to clearly indicate the area designated by the user among the areas to be digitized.

[0169] The area display range 2120 displays the areas to be converted into data that have been set by the reading system 100 executing an area setting process. When the user designates one of the areas to be converted into data that are displayed in the area display range 2120, an area frame corresponding to that area is displayed superimposed on the second image data in the second image data display range 2110. The example in Fig. 21 shows how the user has designated "area 3" from among the areas to be converted into data that are displayed in the area display range 2120, and an area frame 2111 is displayed superimposed on the second image data in the second image data display range 2110.

[0170] The area detailed information display range 2130 displays detailed information about an area designated by the user among the areas to be converted into data. The example of Fig. 21 shows how, among the areas to be converted into data displayed in the area display range 2120, "area 3" has been designated by the user, and information set in association with "area 3" is displayed in the area detailed information display range 2130. The example of Fig. 21 shows - Character recognition processing was performed on the area, and the text information written in the area was determined to be Japanese, and the language was set to Japanese in association with the area. - Character recognition processing was performed on the area, and the attribute of the character information written in the area was determined to be text, and attribute = text was set in association with the area. - Character recognition processing was performed on the area, and the item indicated by the character information written in the area was estimated to be a name, and the item = name was set in association with the area. -Handwriting is set as the method for writing the text information in the area. The position of the area is set to (x, y), and the size of the area is set to width XX mm x height YY mm. This shows:

[0171] In the area detail information display range 2130, among the information set in association with "area 3", the language, attribute, item, and area size are automatically set by the setting unit 670. On the other hand, the description method is set by the user. Furthermore, the area size is automatically set by the setting unit 670 and can be changed as desired by the user.

[0172] The recognition result for the area designated by the user among the areas to be converted into data is displayed in the recognition result display range 2140. The example in Fig. 21 shows that the user designated "Area 3" among the areas to be converted into data displayed in the area display range 2120, and the recognition result of the character recognition process for that area, "Takamoto", is displayed.

[0173] By checking that the correct recognition result is displayed in the recognition result display area 2140, the user can recognize that the position and size of the area to be digitized that has been set are appropriate.

[0174] <Summary> As is clear from the above description, the information processing device 120 according to the first embodiment: A candidate area to be digitized is extracted from the difference between the first image data based on an unfilled document and the second image data based on a filled document. Character recognition processing or graphic recognition processing is performed on the second image data, and the recognition result is obtained. The area to be digitized is set based on the candidate area to be digitized and the recognition results.

[0175] As a result, the information processing device 120 according to the first embodiment can reduce the amount of user operation required when setting an area to be digitized. In other words, the information processing device 120 according to the first embodiment can improve the operability of the reading system 100 when the user performs setting operations.

[0176] [Second embodiment] In the first embodiment, the area to be converted into data set by the setting unit 670 is: -A region to be digitized that is set by processing related to character information; A data target area set by processing mark information or image information; The case where the two do not overlap has been described.

[0177] On the other hand, as described in the first embodiment, in the case of an area to be converted into data that is set by processing related to character information, the area may be corrected by the correction unit 683 based on the detected line segments. In this case, the area to be converted into data after being enlarged by the correction may overlap with other areas to be converted into data.

[0178] Fig. 22 is a fourth diagram showing a specific example of processing by the correction unit. Of these, Fig. 22(a) shows an excerpt of a portion of the second image data generated by reading, with the reading device 110, a document in which the position where the name should be written (the position indicated by the underline) and the position of the marker information are relatively close to each other.

[0179] Also, Figure 22(b) shows that when an area setting process is performed on the second image data shown in Figure 22(a), the setting unit 670 sets the area indicated by the symbol 2201 and the area indicated by the symbol 2202 as areas to be converted into data.

[0180] Also, Figure 22(c) shows that when an underline is detected by the detection unit 681, the area indicated by the symbol 2201 set by the setting unit 670 is corrected, and the area indicated by the symbol 2201' is set as the area to be converted into data after the correction.

[0181] Here, it is assumed that a part of the area to be converted into data after correction, indicated by reference numeral 2201′, overlaps with a part of the area to be converted into data, indicated by reference numeral 2202. In such a case, the setting unit 670 determines the information to be set in association with the area to be converted into data, based on the proportion of the overlapping part in the area to be converted into data.

[0182] Fig. 22(d-1) is a diagram showing the relationship between the area to be converted into data after correction, indicated by reference numeral 2201', and the overlapping portion. In the example of Fig. 22(d-1), the proportion of the overlapping portion in the area to be converted into data after correction, indicated by reference numeral 2201', is equal to or less than a predetermined threshold, so the attribute set in association with the area to be converted into data after correction, indicated by reference numeral 2201', is determined to be text.

[0183] Fig. 22(d-2) is a diagram showing the relationship between the area to be converted into data and the overlapping portion indicated by the reference numeral 2202. In the example of Fig. 22(d-2), the proportion of the overlapping portion in the area to be converted into data indicated by the reference numeral 2202 is equal to or less than a predetermined threshold, so the information set in association with the area to be converted into data indicated by the reference numeral 2202 is determined to be code information.

[0184] In this way, when the areas to be converted into data overlap, the information to be set in association with each area to be converted into data is determined based on the proportion of the overlapping portion.

[0185] [Other embodiments] In the above first embodiment, the information processing device 120 and the reading device 110 are described as being configured as separate entities. However, some or all of the functional units of the information processing device 120 may be realized in the reading device 110. In other words, the information processing device 120 may refer to a device separate from the reading device 110, or may refer to a device incorporated into the reading device 110.

[0186] In the first embodiment, the information processing device 120 executes the information processing program by itself. However, the information processing device 120 may be configured with, for example, multiple computers, and the information processing program may be installed in each of the computers, so that the information processing program is executed in a distributed computing format.

[0187] The present invention is not limited to the configurations described in the above embodiments, but may be combined with other elements, etc. These aspects can be changed without departing from the spirit of the present invention, and can be appropriately determined depending on the application form. [Explanation of symbols]

[0188] 100: Reading system 110: Reading device 120: Information processing device 121: User interface section 122: Area setting section 123: Image data recognition unit 124: Data file generation unit 610: Image data acquisition unit 620: Matching section 630: differential image generation unit 640: Character recognition processing unit 641 :Recognition part 642: Specific part 643: Estimation section 650: Graphic recognition processing unit 651: Recognition part 652: Specific part 660:Extraction part 670: Settings section 680: Area correction section 681:Detection unit 682: Judgment section 683: Correction section 701, 702: Feature extraction unit 703: Feature matching unit 2100: Setting screen [Prior art documents] [Patent documents]

[0189] [Patent Document 1] Patent Publication No. 2021-56659

Claims

1. an extraction unit that extracts a candidate area to be converted into data from a difference between first image data based on an unwritten document and second image data based on a written document; a recognition processing unit that performs character recognition processing or graphic recognition processing on the second image data and obtains a recognition result; a setting unit that sets an area to be converted into data based on the candidate area and the recognition result; An information processing device having the above.

2. an identification unit that identifies an attribute of character information when the character recognition process is executed and character information is acquired as the recognition result; an estimation unit that estimates an item indicated by the character information when the attribute of the character information is identified as text; The information processing device according to claim 1 , further comprising:

3. When the attribute of the character information acquired by executing the character recognition process is mark information, the extraction unit extracts an area of ​​the character information as a candidate area to be converted into data, and the setting unit sets the extracted area of ​​the character information as the area to be converted into data. The information processing device according to claim 2 .

4. the estimation unit estimates the item using a machine-learned classifier. The information processing device according to claim 2 .

5. the estimation unit estimates the item based on a notation rule corresponding to a predetermined item; The information processing device according to claim 2 .

6. the setting unit sets an item indicated by character information included in the area to be converted into data. The information processing device according to claim 2 .

7. an identification unit that identifies a type of code information when the graphic recognition process is executed and code information is acquired as the recognition result; The information processing device according to claim 1 ,

8. the setting unit sets the type of code information included in the area to be converted into data. The information processing device according to claim 7 .

9. When the graphic recognition process is executed and image information is acquired as the recognition result, the setting unit sets an area of ​​the candidate area corresponding to the image information as the area to be converted into data. The information processing device according to claim 1 .

10. a detection unit that detects line segments from the second image data; a determination unit that determines whether the area to be converted into data is an area having a rectangular frame or an area having an underline, based on the detected line segments; a correction unit that corrects the shape of the area to be converted into data according to the determination result; The information processing device according to claim 2 , further comprising:

11. The correction unit If it is determined that the area to be converted into data has a rectangular frame, the area to be converted into data is corrected so as to be enlarged to at least the area inside the smallest rectangular frame that contains the area to be converted into data. The information processing device according to claim 10.

12. The correction unit When the adjacent areas to be converted into data satisfy a predetermined condition as a result of the correction to enlarge the areas to be converted into data, the adjacent areas to be converted into data are connected to each other. The information processing device according to claim 11.

13. The correction unit If it is determined that the area to be converted into data is an area having an underline, the area to be converted into data is enlarged in the length direction of the underline according to the length of the underline, and is enlarged in the height direction according to the height of other areas to be converted into data that are within the range of the underline. The information processing device according to claim 10.

14. a collating unit that collates the first image data and the second image data to determine whether they are image data based on documents of the same format; When the matching by the matching unit is successful, the extraction unit extracts the candidate region. The information processing device according to claim 1 .

15. extracting a candidate area to be converted into data from a difference between first image data based on an unwritten document and second image data based on a written document; performing character recognition processing or graphic recognition processing on the second image data and obtaining a recognition result; a step of setting an area to be converted into data based on the candidate area and the recognition result; An information processing method performed by a computer.

16. extracting a candidate area to be converted into data from a difference between first image data based on an unwritten document and second image data based on a written document; performing character recognition processing or graphic recognition processing on the second image data and obtaining a recognition result; a step of setting an area to be converted into data based on the candidate area and the recognition result; An information processing program that causes a computer to execute the above.

Images