Form processing apparatus and form processing method

Abstract

To provide a form processing device capable of realizing selection of a template image with high versatility.SOLUTION: A template selection processing unit 112 calculates a plurality of feature points in a form image and a feature quantity descriptor of each feature point, and selects, for each template image, a plurality of corresponding points corresponding to the plurality of feature points in the template image based on the calculation result. The template selection processing unit 112 selects a corresponding template image based on each feature point and each corresponding point of each template image.SELECTED DRAWING: Figure 1

Description

Technical Field

[0001] The present disclosure relates to a form processing apparatus and a form processing method.

Background Art

[0002] A form processing apparatus that recognizes the contents described in a form image showing a form and performs various processes usually recognizes the contents of the form using a template image corresponding to the format of the form. However, since the format of the form varies depending on the business or the like and there are a large number of types thereof, it is difficult to select an appropriate template image for the form image.

[0003] On the other hand, Patent Document 1 describes a form processing apparatus that extracts characteristics such as the number, distribution, and interval of ruled lines for each table described in a form image and groups the form images according to the characteristics. This form processing apparatus selects an appropriate template image by selecting a template image corresponding to the group.

Prior Art Documents

Patent Documents

[0004]

Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0005] However, the technique described in Patent Document 1 has a problem that it cannot cope with a case where there are a plurality of types of template images for a form image without ruled lines. Further, it cannot cope with a case where there are a plurality of types of template images in which description portions other than the table in the form image (for example, printed portions) are different. Therefore, the technique described in Patent Document 1 has a problem of low versatility.

[0006] An object of the present disclosure is to provide a form processing apparatus and a form processing method capable of realizing selection of a highly versatile template image.

Means for Solving the Problems

[0007] A form processing apparatus according to an aspect of the present disclosure is a form processing apparatus that selects a corresponding template image corresponding to a form image from a plurality of template images, calculates a plurality of feature points in the form image and a feature amount descriptor of each feature point, and based on the calculation result, for each of the template images, selects a plurality of corresponding points corresponding to each feature point in the template image, and has a selection unit that selects the corresponding template image based on each feature point and each corresponding point of each template image.

Effects of the Invention

[0008] According to the present invention, it becomes possible to realize selection of a highly versatile template image.

Brief Description of the Drawings

[0009]

Figure 1

Figure 2

Figure 3

Figure 4

Figure 5

Figure 6

Figure 7

Figure 8

Figure 9

Figure 10

Figure 11

Embodiments for Carrying Out the Invention

[0010] Hereinafter, embodiments of the present disclosure will be described with reference to the drawings.

[0011] In the following, various information may be described using tables, but the data structure of the information is not limited to tables. Also, when describing identification information, expressions such as "identification information", "name", and "number" may be used, but these are mutually replaceable.

[0012] FIG. 1 is a block diagram showing a functional configuration of a form recognition system according to an embodiment of the present disclosure. The form recognition system 100 shown in FIG. 1 is a system that provides a service related to form recognition processing for recognizing the content of a form image indicating a form, and is mutually connected to a user terminal 200 and an external system 300 via the Internet 500.

[0013] The user terminal 200 is a terminal used by a user who receives the service provided by the form recognition system 100, and is, for example, a PC (personal computer) or the like. The user terminal 200 transmits a form image that is the target of form recognition processing to the form recognition system 100, and receives a character recognition result that is the processing result of the form recognition processing from the form recognition system 100.

[0014] The external system 300 is a form recognition engine that executes character recognition processing for recognizing characters described in a form image according to a request from the form recognition system 100, and transmits the processing result to the form recognition system 100 as a character recognition result.

[0015] Note that the connection forms of the form recognition system 100, the user terminal 200, and the external system 300 are not limited to the example in FIG. 1. For example, the form recognition system 100 may be connected to the user terminal 200 and the external system 300 via a network other than the Internet 500, or may be directly connected without using a network. Also, at least a part of the functions of the user terminal 200 and the external system 300 may be provided in the form recognition system 100.

[0016] The form recognition system 100 includes a Web server 101, a cooperation server 102, and a service server 103.

[0017] The Web server 101 is a server connected to the user terminal 200, and has a function of receiving a form image as an input image from the user terminal 200 and a function of transmitting a character recognition result to the user terminal 200. Note that in this embodiment, the Web server 101 is realized by httpd (Hyper Text Transfer Protocol Daemon). Also, the Web server 101 is provided in the DMZ (Demilitarized Zone). However, the Web server 101 is not limited to these examples.

[0018] The cooperation server 102 is a server that connects the form recognition system 100 and the external system 300, and is realized by a proxy server in this embodiment.

[0019] The service server 103 is a form processing device that performs form recognition processing on the input image received by the Web server 101. Specifically, the service server 103 includes a service control unit 111, a template selection processing unit 112, an image preprocessing unit 113, a form recognition processing unit 114, and a form recognition result postprocessing unit 115.

[0020] The service control unit 111 is a control unit that controls the entire service server 103.

[0021] The template selection processing unit 112 is a selection unit that executes a template selection process of selecting, as a corresponding template image, a template image corresponding to the input image received by the Web server 101 from among a plurality of pre-registered template images. The template selection processing unit 112 holds a plurality of template images (not shown) and template feature information 121 regarding the plurality of template images. The template feature information 121 is calculated, for example, by the service server 103 when registering the template image. Note that the input image may include margins. Also, the orientation of the input image is not particularly limited. On the other hand, the presence or absence and orientation of the margins of the template image are determined according to the specifications of the external system 300. In the present embodiment, the template image does not include margins, and the orientation of the template image is a predetermined orientation (specifically, the orientation in which the characters in the template image face the normal direction). However, the presence or absence and orientation of the margins of the template image are not limited to this example. For example, when the external system 300 has an image rotation correction function, the orientation of the template image is not particularly limited.

[0022] The image preprocessing unit 113 is a preprocessing unit that executes preprocessing on the input image before the form recognition process. The preprocessing includes a fitting process for fitting the input image and the template image, and a pre-selection process performed according to the corresponding template image. The image preprocessing unit 113 holds template preprocessing information 122 regarding the pre-selection process, and determines the pre-selection process to be executed based on the template preprocessing information 122.

[0023] The form recognition processing unit 114 is a recognition unit that executes form recognition processing on the form preprocessing result image, which is the input image preprocessed by the image preprocessing unit 113. In the present embodiment, the form recognition processing unit 114 executes form recognition processing in cooperation with the external system 300 via the cooperation server 102. Specifically, the form recognition processing unit 114 holds template reading position information 123 indicating positions for performing character recognition in each template image, and generates character reading position information indicating positions for performing character recognition in the form preprocessing result image based on the template reading position information 123. The form recognition processing unit 114 passes the form preprocessing result image and the character reading position information to the external system 300 to execute character recognition processing. Further, the form recognition processing unit 114 may pass a cut-out image obtained by cutting out a portion corresponding to the character reading position information from the form preprocessing result image to the external system 300 to execute character recognition processing.

[0024] The form recognition result post-processing unit 115 is a post-processing unit that executes post-processing on the character recognition result, which is the processing result of the form recognition processing unit 114. The post-processing is, for example, processing for adjusting the confidence level of the character recognition result.

[0025] The Web server 101, the cooperation server 102, and the service server 103 of the form recognition system 100 described above can be realized with a hardware configuration equivalent to a general computer device equipped with a processor such as a CPU (Central Processing Unit), a memory, an auxiliary storage device, an input device, an output device, and a communication device such as a network card (none of which are shown). Further, at least a part of each function of the form recognition system 100 may be realized by a processor reading a program recorded in a memory or the like and executing the read program. At least a part of the functions realized by the execution of the program may be realized by a hardware circuit (for example, an ASIC (Application Specific Integrated Circuit) or an FPGA (Field-Programmable Gate Array)). Further, the program can also be recorded on a recording medium that non-temporarily stores data, such as a semiconductor memory, a magnetic disk, an optical disk, a magnetic tape, or a magneto-optical disk. Further, the Web server 101, the cooperation server 102, and the service server 103 of the form recognition system 100 may be physically configured as separate devices or may be configured as the same device.

[0026] FIG. 2 is a diagram showing an example of the template feature information 121. The template feature information 121 shown in FIG. 2 is a table having records including fields 201 to 207.

[0027] Field 201 stores a template ID which is identification information for identifying a template image. Fields 202 to 206 store feature amounts indicating features of the template image. Specifically, fields 202 and 203 are fields for storing size feature amounts which are feature amounts related to the size of the template image. More specifically, field 202 stores the horizontal width of the template image, and field 203 stores the vertical height of the template image. Field 204 stores a color feature amount which is a feature amount related to the color of the template image. Field 205 stores a global feature amount which is a global feature amount of the template image. Field 206 stores a local feature amount which is a local feature amount of the template image. Field 207 stores additional information regarding the template image.

[0028] The local feature amount is composed of one or more feature points and a feature amount descriptor for each feature point. Also, in the example of FIG. 2, the color feature amount, the global feature amount, and the local feature amount are stored as binary data, but they do not have to be binary data. Also, in the example of FIG. 2, the additional information regarding the template image indicates a group ID which is identification information for identifying the group to which the template image belongs, but it is not limited to this example.

[0029] FIG. 3 is a diagram showing an example of template preprocessing information 122. The template preprocessing information 122 shown in FIG. 3 is a table having records including fields 301 to 304.

[0030] Field 301 stores the template ID. Fields 302 to 304 store necessity information indicating whether to perform pre-selection processing on the input image when the template image specified by the template ID stored in Field 301 is selected as the corresponding template image. Specifically, Field 302 stores ground texture removal necessity information indicating whether to perform ground texture removal processing for removing the ground texture of the input image as pre-processing. Field 303 stores noise removal necessity information indicating whether to perform noise removal processing for removing the noise of the input image as pre-processing. Field 304 stores binarization necessity information indicating whether to perform binarization processing for converting the input image into a two-color (for example, black and white) image as pre-processing. In the example of FIG. 3, each necessity information indicates "TRUE" when pre-selection processing is performed, and indicates "FALSE" when pre-selection processing is not performed.

[0031] Note that the ground texture removal processing, the noise removal processing, and the binarization processing are examples of pre-selection processing, and the pre-selection processing is not limited to these examples, and may include other processes, for example.

[0032] FIG. 4 is a diagram showing an example of the template reading position information 123. The template reading position information 123 shown in FIG. 4 is a table having records including fields 401 to 407.

[0033] Field 401 stores the template ID. Field 402 stores an attribute name which is identification information for identifying the attribute of the reading area for reading characters in the template image. Fields 403 to 406 store template reading position information indicating the position of the reading area on the template image. Specifically, Field 403 stores the X coordinate of the position of a predetermined location of the reading area, and Field 404 stores the Y coordinate of the position of a predetermined location of the reading area. Field 405 stores the width which is the length of the reading area in the X direction, and Field 403 stores the height which is the length of the reading area in the Y direction. Field 407 stores additional information regarding the reading area.

[0034] In the example of FIG. 4, the attribute of the reading area is the attribute of the characters described in the reading area, and indicates, for example, "bank name" and "account number". Also, although the reading area is rectangular, other shapes may be used. Further, the predetermined location is, for example, the upper left location of the reading area. The X direction is the horizontal direction (lateral direction) of the image, and the Y direction is the vertical direction of the image. The additional information for the reading area is information specified for the character recognition process by the external system 300, and in the example of FIG. 4, indicates the character type of the characters that may be described in the reading area.

[0035] FIG. 5 is a sequence diagram for explaining an example of the operation of the form recognition system 100.

[0036] First, the user terminal 200 transmits an execution request for requesting execution of the form recognition process to the Web server 101 of the form recognition system 100 in accordance with an instruction from the user (step S101). The execution request includes the form image that is the target of the form recognition process. Also, additional information regarding the form image may be attached to the form image. The additional information is, in this embodiment, information for narrowing down template candidates that are candidates for the corresponding template image corresponding to the form image, and indicates, for example, a group ID for identifying a group of template images or a template ID for identifying a template image. Also, there may be a plurality of form images.

[0037] When the Web server 101 receives the execution request, it transmits the execution request to the service control unit 111 of the service server 103 (step S102).

[0038] When the service control unit 111 receives the execution request, it transmits the form image included in the execution request to the template selection processing unit 112 as an input image (step S103). When the template selection processing unit 112 receives the input image, it executes a template selection process (see FIG. 6) for selecting the corresponding template image corresponding to the input image, and transmits a template selection result as the result of the process to the service control unit 111 (step S104).

[0039] When the service control unit 111 receives the template selection result, it transmits the input image, the coordinate transformation matrix estimated in the template selection process, and the template ID of the corresponding template image that is the template selection result to the image preprocessing unit 113 (step S105). When the image preprocessing unit 113 receives the input image, the coordinate transformation matrix, and the template ID of the corresponding template image, it performs preprocessing including fitting processing using the coordinate transformation matrix on the input image and selection preprocessing according to the template ID of the corresponding template image, and transmits the input image after the preprocessing to the service control unit 111 as an image preprocessing result image (step S106).

[0040] Note that the image preprocessing unit 113 performs the fitting processing and the selection preprocessing in this order as preprocessing. The fitting processing is a process of adjusting the character reading area in the input image to match the reading area of the corresponding template image by converting the input image using the coordinate transformation matrix estimated in the template selection process. The coordinate transformation matrix will be described later. Also, the image preprocessing unit 113 executes the selection preprocessing for which the necessity information corresponding to the template ID of the corresponding template image is "TRUE" in the template preprocessing information 122.

[0041] When the service control unit 111 receives the image pre - processing result image, it transmits the image pre - processing result image and the template ID of the corresponding template image to the form recognition processing unit 114 (step S107). When the form recognition processing unit 114 receives the image pre - processing result image and the template ID, it obtains, from the template reading position information 123, the template reading position information corresponding to the received template ID as the character reading position information indicating the position of the reading area of the image pre - processing result image. The form recognition processing unit 114 transmits a character recognition request including the image pre - processing result image and the character reading position information to the cooperation server 102 (step S108). Note that the character recognition request may include an attribute name and additional information corresponding to the received template ID. Also, the form recognition processing unit 114 may transmit a character recognition request including a cut - out image obtained by cutting out a portion corresponding to the character reading position information from the image pre - processing result image to the cooperation server 102.

[0042] When the cooperation server 102 receives the character recognition request, it transmits the character recognition request to the external system 300 (step S109). When the external system 300 receives the character recognition request, it performs character recognition processing on the image pre - processing result image included in the character recognition request using the character reading position information included in the character recognition request, and transmits the processing result of the character recognition processing to the cooperation server 102 as the character recognition result (step S110). The character recognition result includes, for example, the image pre - processing result image, the character reading position information, the character information recognized from the image pre - processing result image, and the confidence level indicating the accuracy of the character information.

[0043] When the cooperation server 102 receives the character recognition result, it transmits the character recognition result to the form recognition processing unit 114 (step S111). When the form recognition processing unit 114 receives the character recognition result, it transmits the character recognition result to the service control unit 111 (step S112). When the service control unit 111 receives the character recognition result, it transmits the character recognition result to the form recognition result post - processing unit 115 (step S113).

[0044] When the form recognition result post - processing unit 115 receives the character recognition result, it corrects the confidence level included in the character recognition result according to a predetermined correction rule, and transmits the corrected recognition result, which is the character recognition result with the confidence level corrected, to the service control unit 111 (step S114). The correction rule is set for each attribute of the reading area where the character information is read. For example, when the attribute name is "account number" and the reading result of the account number is not 7 digits, the confidence level is decreased, etc.

[0045] When the service control unit 111 receives the corrected recognition result, it transmits the corrected recognition result to the Web server 101 (step S115). When the Web server 101 receives the corrected recognition result, it transmits the corrected recognition result to the user terminal 200. When the user terminal 200 receives the corrected recognition result, it displays or stores the corrected recognition result, etc. (step S116) and ends the process.

[0046] FIG. 6 is a flowchart for explaining an example of the template selection process executed in step S104 of FIG. 5.

[0047] In the template selection process, the template selection processing unit 112 determines whether additional information is attached to the input image (step S201).

[0048] When additional information is attached, the template selection processing unit 112 determines whether the number of template candidates, which are candidates for the corresponding template image narrowed down by the additional information, is one (step S202). For example, when the additional information indicates a template ID, or when the additional information indicates a group ID and there is only one template image included in the group identified by the group ID, the template selection processing unit 112 determines that the number of template candidates narrowed down by the additional information is one.

[0049] When there is one template candidate, the template selection processing unit 112 selects the template candidate as the corresponding template image and ends the process.

[0050] On the other hand, when no additional information is given and there is more than one template candidate, the template selection processing unit 112 executes a size narrowing process (Fig. 7) to narrow down the template candidates based on the size of the input image (step S203).

[0051] Subsequently, the template selection processing unit 112 executes a color narrowing process (Fig. 8) to narrow down the template candidates based on the color of the input image (step S204).

[0052] Thereafter, the template selection processing unit 112 executes a global feature amount narrowing process (Fig. 9) to narrow down the template candidates based on the global feature amount of the input image (step S205).

[0053] Furthermore, the template selection processing unit 112 calculates local feature amounts (a plurality of feature points and feature amount descriptors for each feature point) in the input image, and for each template candidate narrowed down by the global feature amount narrowing process, selects a plurality of corresponding points corresponding to the plurality of feature points in the template candidate, and based on each feature point and each corresponding point of each template image, executes a local feature amount narrowing process (Fig. 10) to select a corresponding template image (step S206).

[0054] Then, the template selection processing unit 112 executes an appropriateness determination process (see Fig. 11) to determine whether the corresponding template image selected by the local feature amount narrowing process is appropriate (step S207), and ends the process.

[0055] Note that the processes of steps S203 to S205 may be skipped according to the characteristics of the input image. For example, when there are both form images with margins and form images without margins in the input image, the size narrowing process in step S203 and the global feature amount narrowing process in step S205 may be skipped. Also, when the input image is not an image read by a scanner but an image taken by a tablet terminal or a digital camera, etc., the color narrowing process in step S204 may be skipped because the color tone of the image changes depending on the shooting environment. The processes to be skipped are set by the user, for example.

[0056] FIG. 7 is a flowchart for explaining an example of the size narrowing process executed in step S203 of FIG. 6.

[0057] In the size narrowing process, the template selection processing unit 112 acquires the horizontal width and vertical height of the input image as the size of the input image from the input image (step S301).

[0058] Thereafter, the template selection processing unit 112 executes a loop process (A) that repeats the processes of steps S302 to S305 for each template image. Note that when additional information is added to the input image in step S201 of FIG. 6, the template selection processing unit 112 repeats the processes of steps S302 to S305 for each template image (template candidate) narrowed down by the additional information.

[0059] In the loop process (A), first, the template selection processing unit 112 acquires the horizontal width and vertical height of the template image from the template feature information 121 as the size of the template image to be processed (step S302).

[0060] The template selection processing unit 112 determines whether the input image and the template image to be processed are similar in terms of size (step S303). Specifically, the template selection processing unit 112 determines whether both of the following size-related conditional expressions (1) and (2) are satisfied. If both conditional expressions (1) and (2) are satisfied, it is determined that the input image and the template image to be processed are similar. w×(1−X)≦W≦w×(1+X) ···(1) h×(1−Y)≦H≦h×(1+Y) ···(2)

[0061] In conditional expressions (1) and (2), the horizontal width of the template image is W, the vertical height of the template image is H, the horizontal width of the input image is w, the vertical height of the input image is h, and X and Y are values that are 0 or more and less than 1. Note that X and Y may be the same value.

[0062] When the template selection processing unit 112 determines that the input image and the template image to be processed are similar in terms of size, it selects the template image to be processed as a template candidate (step S304). When the input image and the template image to be processed are not similar in terms of size, it excludes the template image to be processed from the template candidates (step S305).

[0063] Then, when the template selection processing unit 112 executes the processing of steps S302 to S305 for all the template images, it exits the loop processing (A) and ends the size narrowing-down processing.

[0064] In the above description, the horizontal width and the vertical height are used as the sizes of the input image and the template image, but other indicators such as the aspect ratio may be used. For example, when there is a possibility that the resolutions of the input image and the template image are different, it is desirable to use the aspect ratio as the size of the input image and the template image. Note that the indicator used for the sizes of the input image and the template image is set by the user, for example.

[0065] FIG. 8 is a flowchart for explaining an example of the color narrowing process executed in step S204 of FIG. 6.

[0066] In the color narrowing process, first, the template selection processing unit 112 analyzes the input image to obtain the color feature amount of the input image (step S401). The color feature amount is, in this embodiment, a color histogram related to color, and more specifically, an HSV histogram with the color system being the HSV color system. However, the color feature amount is not limited to the above example, and may be a color histogram of other color systems (for example, the RGB color system or the L*a*b color system, etc.).

[0067] Thereafter, the template selection processing unit 112 executes a loop process (B) that repeats the processes of steps S402 to S403 for each template candidate narrowed down in the size narrowing process of FIG. 7.

[0068] In the loop process (B), first, the template selection processing unit 112 obtains the color histogram, which is the color feature amount of the template candidate to be processed, from the template feature information 121 (step S402). The template selection processing unit 112 calculates the color similarity, which is the similarity between the color histogram of the input image and the color histogram of the template image to be processed (step S403). The color similarity is, in this embodiment, the Bhattacharyya distance, but other values such as the chi-square distance may also be used.

[0069] Then, when the template selection processing unit 112 executes the processes of steps S402 to S403 for all template candidates, it exits the loop process (B) and calculates the variance of the plurality of color similarities calculated for each template candidate (step S404).

[0070] The template selection processing unit 112 narrows down the template candidates based on the variance of the color similarity (step S405) and ends the color narrowing process. Specifically, the template selection processing unit 112 obtains the narrowing rate of the template candidates according to the variance of the color similarity, and selects the template candidates in order from the ones with higher color similarity by the number corresponding to the narrowing rate, thereby narrowing down the template candidates. Note that, since it is considered that the larger the variance of the color similarity is, the more significant the narrowing down of the template image by the color feature amount is, the template selection processing unit 112 increases the narrowing rate of the template image as the variance of the color similarity increases. For example, when the variance of the color similarity is equal to or greater than the threshold value, the template selection processing unit 112 sets the narrowing rate to 0.5, and when the variance of the color similarity is less than the threshold value, the template selection processing unit 112 sets the narrowing rate to 0.125. In this case, assuming that there are 8000 original template candidates, if the variance of the color similarity is equal to or greater than the threshold value, 4000 (8000×(1−0.5)) template candidates are selected, and if the variance of the color similarity is less than the threshold value, 7000 (8000×(1−0.125)) template candidates are selected.

[0071] FIG. 9 is a flowchart for explaining an example of the global feature amount narrowing process executed in step S205 of FIG. 6.

[0072] In the global feature amount narrowing process, first, the template selection processing unit 112 analyzes the input image to obtain the global feature amount of the input image (step S501). The global feature amount is a hash value in the present embodiment, and more specifically, three hash values calculated from three hash algorithms of aHash, dHash, and pHash. Also, each hash value is 64 bits. However, the hash algorithm is not limited to the above example, and other algorithms (for example, Block·Hash and Wavelet·Hash, etc.) may be used. Also, the number of hash values does not have to be three, and the number of bits of the hash value does not have to be 64 bits.

[0073] After that, the template selection processing unit 112 executes a loop process (C) that repeats the processes of steps S502 to S503 for each template candidate narrowed down by the color narrowing process in FIG. 8.

[0074] In the loop process (C), first, the template selection processing unit 112 acquires, from the template feature information 121, the hash value that is the global feature amount of the template image to be processed (step S502). The template selection processing unit 112 calculates the global similarity, which is the similarity between the hash value of the input image and the hash value of the template image to be processed (step S503). In this embodiment, it is assumed that the global similarity is the Hamming distance. Specifically, for each of the three hash values, the template selection processing unit 112 obtains the Hamming distance as the similarity and calculates the statistical value of each similarity as the global similarity. The statistical value is, for example, a weighted sum or an average value. Note that the global similarity is not limited to the Hamming distance.

[0075] Then, when the template selection processing unit 112 executes the processes of steps S502 to S503 for all the template images, it exits the loop process (C) and calculates the variance of the global similarity between the input image and each template image (step S504).

[0076] Based on the variance of the global similarity, the template selection processing unit 112 narrows down the template candidates (step S505) and ends the global narrowing process. Here, it is considered that the larger the variance of the global similarity, the more significant the narrowing of the template image by the global feature amount. Therefore, the template selection processing unit 112 increases the narrowing rate of the template image as the variance of the global similarity increases.

[0077] FIG. 10 is a flowchart for explaining an example of the local feature amount narrowing process executed in step S206 of FIG. 6.

[0078] In the local feature amount narrowing process, first, the template selection processing unit 112 analyzes the input image to obtain the local feature amount of the input image (step S501). The local feature amount is calculated using a predetermined local feature amount calculation method. In this embodiment, the local feature amount calculation method is ORB (Oriented FAST and Rotated BRIEF (Binary Robust Independent Elementary Features)), but other methods (such as AKAZE (Accelerated KAZE), BRISK (Binary Robust Invariant Scalable Keypoints), and SIFT (Scale-invariant feature transform)) may also be used. Also, the number of feature points of the local feature amount may be changed according to the type of form.

[0079] Thereafter, the template selection processing unit 112 executes a loop process (D) that repeats the processes of steps S602 to S606 for each template candidate narrowed down in the global narrowing process of FIG. 9.

[0080] In the loop process (D), first, the template selection processing unit 112 obtains the local feature amount of the template candidate to be processed from the template feature information 121 (step S602).

[0081] The template selection processing unit 112 specifies, for each feature point of the input image, the feature point of the template candidate to be processed corresponding to that feature point as a corresponding point (step S603). Here, the template selection processing unit 112 uses the k-nearest neighbor method to specify, for each feature point of the input image, from the feature points of the target template candidate, the feature points up to the predetermined number from the ones with higher similarity to the feature point of the input image as corresponding points. The predetermined number is 2 in this embodiment.

[0082] The template selection processing unit 112 performs narrowing of the corresponding points for each feature point of the input image using the following ratio test method (step S604). The ratio test method is for the feature point m of the input image iAmong the corresponding points of the template candidate, the point n that is the most (first) similar j1 and the point n that is the second most similar j2 Let the similarity degrees to them be s 1 , s 2 respectively. Then, when the following relational expression (3) is satisfied, the point n j1 is adopted as the corresponding point to the feature point m i . In the relational expression (3), ratio is a predetermined ratio. s 1 < s 2 * ratio ··· (3)

[0083] Furthermore, the template selection processing unit 112 performs further narrowing down on the narrowed-down corresponding points using the RANSAC (Random Sample Consensus) algorithm (step S604).

[0084] Specifically, the template selection processing unit 112 repeats the following processes 1 and 2 a predetermined number of times or until the error rate described later falls below a specified value, and based on the processing results, selects the valid corresponding points of the template candidate to be processed. Process 1: The template selection processing unit 112 randomly selects one or more feature points of the input image having corresponding points, and estimates a coordinate transformation matrix in which the selected coordinates are transformed into the coordinates of the corresponding points. In the present embodiment, the template selection processing unit 112 selects 4 feature points of the input image. The coordinate transformation matrix is, for example, an affine matrix or a homography matrix. Process 2: The template selection processing unit 112 performs coordinate transformation on other feature points (feature points other than the selected feature points) of the input image using the coordinate transformation matrix estimated in Process 1, and for each transformed feature point after the coordinate transformation, calculates the distance between the transformed feature point and its corresponding point (the corresponding point corresponding to the feature point (other feature point) before the coordinate transformation of the transformed feature point). Then, the template selection processing unit 112 extracts other feature points whose distance is equal to or less than a predetermined distance and the feature points selected in Process 1. Furthermore, the template selection processing unit 112 calculates an error rate that evaluates the error between the transformed feature point and its corresponding point (Process 2).

[0085] When processes 1 and 2 are repeated a predetermined number of times or until the error rate falls below a specified value, the template selection processing unit 112 specifies, as valid corresponding points, the corresponding points corresponding to the feature points extracted by the process with the lowest error rate.

[0086] Thereafter, the template selection processing unit 112 calculates the number of valid corresponding points (step S606).

[0087] Then, when the template selection processing unit 112 executes the processes of steps S602 to S606 for all template candidates, it exits the loop processing (D), and determines whether or not the number of valid corresponding points in the template candidate with the largest number of valid corresponding points among the template candidates is equal to or greater than a specified number (step S607).

[0088] If the number of valid corresponding points is equal to or greater than the specified number, the template selection processing unit 112 selects the template candidate with the largest number of valid corresponding points as the corresponding template image and ends the local feature amount narrowing process.

[0089] On the other hand, if the number of valid corresponding points is less than the specified number, the template selection processing unit 112 determines that there is no corresponding template image and ends the local feature amount narrowing process.

[0090] FIG. 11 is a flowchart for explaining an example of the appropriateness determination process executed in step S207 of FIG. 6.

[0091] In the appropriateness determination process, first, the template selection processing unit 112 determines whether or not the magnification indicated by the target transformation matrix, which is the coordinate transformation matrix corresponding to the corresponding template image, is within the allowable range (step S701). For example, when the template selection processing unit 112 sets Z to a value of 0 or more, it determines whether or not the magnification is within 100 ± Z [%]. Note that the corresponding transformation matrix is the coordinate transformation matrix used when selecting the valid corresponding points of the corresponding template image.

[0092] When the magnification ratio is within the allowable range, the template selection processing unit 112 determines whether a first ratio, which is the ratio of the number of pixels where the deformed image obtained by deforming (coordinate transformation) the input image using the target transformation matrix overlaps with the corresponding template image to the total number of pixels of the corresponding template image, is equal to or greater than a first constant ratio (step S702).

[0093] When the first ratio is equal to or greater than the first constant value, the template selection processing unit 112 determines whether a second ratio, which is the ratio of the number of pixels that are black pixels at the same position and are black pixels in both the image obtained by binarizing the deformed image and the image obtained by binarizing the corresponding template image, to the total number of black pixels in the image obtained by binarizing the corresponding template image, is equal to or greater than a second constant ratio (step S703).

[0094] When the second ratio is equal to or greater than the second constant ratio, the template selection processing unit 112 determines that the corresponding template image is appropriate and ends the appropriate determination processing.

[0095] Also, when the magnification ratio is not within the allowable range, when the first ratio is less than the first constant value, and when the second ratio is less than the second constant value, the template selection processing unit 112 determines that there is no corresponding template image and ends the local feature amount narrowing processing.

[0096] In the above processing, when it is determined that there is no corresponding template image in the local feature amount narrowing processing of FIG. 10 or the appropriate determination processing of FIG. 11, the template selection processing unit 112 may output notification information indicating that there is no corresponding template image to the user terminal 200 instead of performing the processing after step S104 of FIG. 5.

[0097] As described above, according to the present embodiment, the template selection processing unit 112 calculates a plurality of feature points in the form image and a feature amount descriptor for each feature point, and based on the calculation result, for each template image, selects a plurality of corresponding points corresponding to the plurality of feature points in the template image. The template selection processing unit 112 selects the corresponding template image based on each feature point and each corresponding point of each template image. Therefore, since it is possible to select a template image even without ruled lines and tables, etc., it is possible to select a template image with higher versatility.

[0098] Also, in the present embodiment, the template selection processing unit 112, based on the feature amount descriptor, for each feature point of the form image, selects, as corresponding points, the template feature points, which are the feature points of the template image, from the ones with higher similarity to the feature point of the form image up to a predetermined number. In particular, when the similarity between the feature point of the form image and the most similar template feature point is smaller than the value obtained by multiplying the similarity between the feature point of the form image and the second most similar template feature point by a predetermined ratio, the most similar template feature point is selected as the corresponding point. In these cases, since it is possible to select more appropriate corresponding points, it is possible to more appropriately select the corresponding template image.

[0099] Also, in the present embodiment, the template selection processing unit 112 estimates, for each template image, a coordinate transformation matrix that transforms the coordinates of one or more feature points selected from the plurality of feature points into the coordinates of one or more corresponding points corresponding to the one or more feature points, calculates the distance between the transformed feature points obtained by performing coordinate transformation on the coordinates of other feature points using the coordinate transformation matrix and the corresponding points corresponding to the other feature points, repeatedly performs the process of extracting the selected feature points and other feature points whose distance is equal to or less than a predetermined distance, and selects the corresponding template image based on the processing result of each process. In this case, since it is possible to select the corresponding template image based on the feature points having appropriate corresponding points, it is possible to more appropriately select the corresponding template image.

[0100] Also, in the present embodiment, the template selection processing unit 112 specifies, for each template image, the corresponding points corresponding to the feature points extracted by the process with the lowest error rate as valid corresponding points, and selects the template image with the largest number of valid corresponding points as the corresponding template image. In this case, it becomes possible to more appropriately select the corresponding template image.

[0101] Also, in the present embodiment, when the number of the valid corresponding points in the template image with the lowest error rate is equal to or more than a specified number, the template selection processing unit 112 selects the template image as the corresponding template image. In this case, it becomes possible to more appropriately select the corresponding template image.

[0102] Also, in the present embodiment, the template selection processing unit 112 determines whether the corresponding template image is appropriate based on the coordinate transformation matrix corresponding to the corresponding template image. In this case, since the appropriateness of the corresponding template image is determined, it becomes possible to more appropriately select the corresponding template image.

[0103] Also, in the present embodiment, when the magnification factor indicated by the coordinate transformation matrix is within the allowable range, the template selection processing unit 112 determines that the corresponding template image is appropriate. In this case, it becomes possible to appropriately determine the appropriateness of the corresponding template image.

[0104] Also, in the present embodiment, when the ratio of the number of pixels where the deformed image obtained by deforming the form image using the coordinate transformation matrix and the template image overlap to the total number of pixels of the corresponding template image is equal to or more than a certain value, the template selection processing unit 112 determines that the corresponding template image is appropriate. In this case, it becomes possible to appropriately determine the appropriateness of the corresponding template image.

[0105] Also, in the present embodiment, the template selection processing unit 112 determines that the corresponding template image is appropriate when the ratio of the number of pixels that are both at the same position and black pixels in the binary image of the deformed image obtained by deforming the form image using the coordinate transformation matrix to the number of all black pixels in the binary image of the corresponding template image is equal to or greater than a certain value. In this case, it becomes possible to appropriately determine the appropriateness of the corresponding template image.

[0106] Also, in the present embodiment, when the corresponding template image is not appropriate, the template selection processing unit 112 outputs a message indicating that the corresponding template image does not exist. In this case, since it becomes possible to notify the user that the corresponding template image does not exist, it becomes possible to cause the user to perform processing such as adding a new template image.

[0107] The above-described embodiments of the present disclosure are examples for explaining the present disclosure, and are not intended to limit the scope of the present disclosure only to those embodiments. Those skilled in the art can implement the present disclosure in various other modes without departing from the scope of the present disclosure.

[0108] For example, in the above-described embodiment, the character recognition process is performed after the corresponding template image is selected, but the process after the corresponding template image is selected is not particularly limited. For example, a process of notifying the user terminal 200 of the corresponding template image may be performed.

Explanation of Reference Numerals

[0109] 100: Form recognition system 101: Web server 102: Cooperation server 103: Service server 111: Service control unit 112: Template selection processing unit 113: Image preprocessing unit 114: Form recognition processing unit 115: Form recognition result post-processing unit

Claims

1. An invoice processing apparatus that selects a corresponding template image corresponding to an invoice image from a plurality of template images, calculates a plurality of feature points and feature quantity descriptors of each feature point in the invoice image, and based on the calculation results, for each template image, selects a plurality of corresponding points corresponding to each feature point in the template image, and has a selection unit that selects the corresponding template image based on each feature point and each corresponding point of each template image, the selection unit estimates a coordinate transformation matrix that transforms the coordinates of one or more feature points selected from the plurality of feature points into the coordinates of one or more corresponding points corresponding to the one or more feature points for each template image, calculates the distance between the transformed feature points obtained by performing coordinate transformation on the coordinates of the other feature points using the coordinate transformation matrix and the corresponding points corresponding to the other feature points, and repeatedly performs the process of extracting the selected feature points and the other feature points whose distance is equal to or less than a predetermined distance, and selects the corresponding template image based on the processing results of each process, the selection unit determines whether the corresponding template image is appropriate based on the coordinate transformation matrix corresponding to the corresponding template image, the selection unit determines that the corresponding template image is appropriate when the ratio of the number of pixels that are the same position and black pixels in the binary image of the deformed image obtained by deforming the invoice image using the coordinate transformation matrix and the binary image of the corresponding template image to the number of all black pixels in the binary image of the corresponding template image is equal to or greater than a certain value. An invoice processing apparatus.

2. The invoice processing apparatus according to claim 1, wherein the selection unit selects, as the corresponding points, template feature points up to a predetermined number from the template feature points that are feature points of the template image and have a high similarity to the feature points of the invoice image for each feature point of the invoice image based on the feature quantity descriptor.

3. The invoice processing apparatus according to claim 1, wherein the selection unit specifies, as valid corresponding points, the corresponding points corresponding to the feature points extracted in the process with the lowest error rate evaluated for the distance between the transformed feature points and the corresponding points for each template image, and selects the template image with the largest number of valid corresponding points as the corresponding template image.

4. The form processing apparatus according to claim 3, wherein the selection unit selects the template image as the corresponding template image when the number of effective corresponding points in the template image having the largest number of effective corresponding points is equal to or greater than a specified number.

5. The form processing apparatus according to claim 1, wherein the selection unit determines that the corresponding template image is appropriate when the magnification factor indicated by the coordinate transformation matrix is within an allowable range.

6. The form processing apparatus according to claim 1, wherein the selection unit determines that the corresponding template image is appropriate when the ratio of the number of pixels where the deformed image obtained by deforming the form image using the coordinate transformation matrix and the template image overlap to the total number of pixels of the corresponding template image is equal to or greater than a certain value.

7. The form processing apparatus according to claim 1, wherein the selection unit outputs a message indicating that the corresponding template image does not exist when the corresponding template image is not appropriate.

8. A form processing method by a form processing apparatus that selects a corresponding template image corresponding to a form image from a plurality of template images, calculating a plurality of feature points in the form image and a feature quantity descriptor for each feature point, and based on the calculation result, for each template image, selecting a plurality of corresponding points corresponding to each feature point in the template image, and selecting the corresponding template image based on each feature point and each corresponding point of each template image, in the selection unit of the corresponding template image, for each template image, estimating a coordinate transformation matrix that transforms the coordinates of one or more feature points selected from the plurality of feature points into the coordinates of one or more corresponding points corresponding to the one or more feature points, calculating the distance between the transformed feature points obtained by performing coordinate transformation on the coordinates of the other feature points using the coordinate transformation matrix and the corresponding points corresponding to the other feature points, and repeatedly performing a process of extracting the selected feature points and the other feature points whose distance is equal to or less than a predetermined distance, and selecting the corresponding template image based on the processing result of each process, determining whether the corresponding template image is appropriate based on the coordinate transformation matrix corresponding to the corresponding template image. In the determination, when the ratio of the number of pixels that are pixels at the same position and black pixels in both the binarized image of the deformed image obtained by deforming the form image using the coordinate transformation matrix and the binarized image of the corresponding template image to the number of all black pixels in the binarized image of the corresponding template image is equal to or greater than a certain value, the corresponding template image is determined to be appropriate. A form processing method.

Images