Information processing apparatus, information processing system, and program

Abstract

To provide an information processing apparatus configured to improve accuracy while suppressing an amount of processing in alignment of images relative to prior arts, an information processing system, and a program.SOLUTION: In an information processing system configured by connecting a user terminal and a management server over a network, the management server 30, serving as an information processing apparatus, for managing the whole of the information processing system includes a control unit. The control unit includes an image correction unit 304 which decides an amount of correction required for second alignment for two images, on the basis of a result of first alignment for the acquired two images, and corrects one of the two images with the decided amount of correction; and an alignment unit 302 which performs the second alignment.SELECTED DRAWING: Figure 4

Description

Technical Field

[0001] The present invention relates to an information processing apparatus, an information processing system, and a program.

Background Art

[0002] Image alignment for character recognition, form discrimination, and defect detection of printed matter has been conventionally performed, and as related techniques, techniques using feature points, techniques using patches, etc. are known (for example, Patent Documents 1 to 3).

Prior Art Documents

Patent Documents

[0003]

Patent Document 1

Patent Document 2

Patent Document 3

Summary of the Invention

Problems to be Solved by the Invention

[0004] Among such conventional techniques, for example, in the technique using feature points, only the portion regarded as the feature point becomes the target of alignment, and there is a risk of a decrease in the alignment accuracy of portions other than the feature point and false determination. Further, in the technique using patches, in order to detect the pixel difference in the movement area of the patch, if the number of patches, the width of the movement area, and the number of pixels in the movement area are increased, the processing amount increases accordingly.

[0005] An object of the present invention is to improve the accuracy while suppressing the processing amount in image alignment more than in the prior art.

Means for Solving the Problems

[0006] The invention according to claim 1 comprises a processor, which determines a correction amount required for a second alignment of two images based on a result of a first alignment of the two acquired images, corrects one of the two images by the determined correction amount, and performs the second alignment. The information processing apparatus is characterized by this. The invention according to claim 2 is the information processing apparatus according to claim 1, characterized in that the processor performs an alignment for the whole of the two images as the first alignment, and performs an alignment for a local part of the two images as the second alignment. The invention according to claim 3 is the information processing apparatus according to claim 2, characterized in that the processor performs an alignment using feature points as the alignment for the whole of the two images, and performs an alignment using patches as the alignment for a local part of the two images. The invention according to claim 4 is the information processing apparatus according to claim 3, characterized in that the processor arranges the patches at one or more positions having predetermined features when performing the second alignment. The invention according to claim 5 is the information processing apparatus according to claim 4, characterized in that the processor arranges the one or more patches based on a relationship with the positions of the feature points. The invention according to claim 6 is the information processing apparatus according to claim 3, characterized in that the processor arranges the one or more patches based on input information by a user's input operation. The invention according to claim 7 is the information processing apparatus according to claim 6, characterized in that the processor determines whether to perform a correction of a condition of coincidence of positions used in the test and a correction of a condition of arrangement of the patches based on a result of a test of alignment of the two images by the first alignment and the second alignment. The invention described in claim 8 is an information processing apparatus according to claim 7, characterized in that the processor modifies at least one of the first threshold used for the first alignment and the second threshold used for the second alignment as a modification of the position matching conditions, and modifies at least one of the number of patches used for the second alignment, the position where they are placed, and the movement area as a modification of the patch placement conditions. The invention described in claim 9 is an information processing device according to claim 8, characterized in that, if the processor incorrectly determines that the positions of the two images do not match in the test results, it makes it easier to determine that the positions of the two images match by modifying the conditions for matching positions and modifying the conditions for placing the patches. The invention described in claim 10 is an information processing device according to claim 8, characterized in that, if the processor incorrectly determines that the positions of the two images in the test result match, it is easier to determine that the positions of the two images do not match, by modifying the conditions for matching positions and modifying the conditions for placing the patches. The invention described in claim 11 is an information processing apparatus according to claim 7, characterized in that the processor controls the presentation of the test results to the user and determines, based on the user's input, whether to modify the position matching conditions and the patch placement conditions. The invention described in claim 12 is an information processing apparatus according to claim 11, characterized in that the processor performs a control to present the results of the test to the user by superimposing the two images and presenting them to the user in a comparable manner. The invention described in claim 13 is an information processing device according to claim 1, characterized in that, of the two images, the first image is a predetermined template image, and the second image is an image corrected by the correction amount. The invention described in claim 14 is an information processing device according to claim 13, characterized in that the template image is a template image for character recognition, and the corrected image is an image corrected for said character recognition. The invention described in claim 15 is an information processing device according to claim 13, characterized in that the template image is a template image for document discrimination, and the corrected image is an image that has been corrected for the purpose of document discrimination. The invention described in claim 16 is an information processing apparatus according to claim 15, characterized in that, in the second alignment, the processor extracts from a group of pre-prepared template images for document discrimination the image that most closely resembles the image corrected for document discrimination. The invention described in claim 17 is an information processing device comprising a processor, wherein the processor controls the transmission of an image input as the object to be aligned to another information processing device, and controls the acquisition and output of the results of a first alignment process performed by the other information processing device on two images including the image, and a second alignment process performed by correcting the image. The invention described in claim 18 is an information processing system characterized by comprising: acquisition means for acquiring two images to be aligned; first alignment means for performing a first alignment on the two acquired images; determination means for determining a correction amount necessary for a second alignment on the two images based on the result of the first alignment; conversion means for correcting one of the two images by the determined correction amount; and second alignment means for performing a second alignment on the two images, one of which has been corrected. The invention described in claim 19 is a program for a computer that enables the following functions: determining a correction amount necessary for a second alignment of two acquired images based on the result of a first alignment of the two acquired images; and correcting one of the two images by the determined correction amount to perform the second alignment. [Effects of the Invention]

[0007] According to the present invention of claim 1, it is possible to provide an information processing device that improves accuracy while suppressing the amount of processing required for aligning two images compared to conventional methods. According to the present invention as of claim 2, approximate alignment is performed by a first alignment targeting the entire image, and more detailed alignment is performed by a second alignment targeting a local area. This makes it possible to improve accuracy while suppressing the processing load of the second alignment compared to conventional methods. According to the present invention of claim 3, approximate alignment is performed by a first alignment using feature points of the image and a second alignment using patches, after which patch-based alignment is performed. This limits the amount of patch movement, making it possible to improve accuracy while reducing the amount of alignment processing compared to conventional methods. According to the present invention of claim 4, the position where the patch is placed is taken into consideration during the second alignment. As a result, the accuracy of the alignment can be improved compared to the conventional method. According to the present invention of claim 5, during the second alignment, the patch is placed at a position that takes into account its relationship with the position of the feature point. As a result, the accuracy of the alignment can be improved compared to the conventional method. According to the present invention of claim 6, during the second alignment, the placement position of the patch is determined based on the user's judgment. As a result, the accuracy of the alignment can be improved compared to the conventional method. According to the present invention of claim 7, it is possible to improve the accuracy of alignment by determining whether to modify the positional matching conditions and the patch placement conditions based on the results of the alignment test. According to the present invention of claim 8, it is possible to improve the accuracy of alignment by determining whether to modify the positional matching conditions and the patch placement conditions based on the results of the alignment test. According to the present invention of claim 9, the threshold used for alignment is modified, thereby improving the accuracy of subsequent alignment. According to the present invention of claim 10, the threshold used for alignment is modified, thereby improving the accuracy of subsequent alignment. According to the present invention of claim 11, since it is determined whether to modify the position matching conditions and the patch placement conditions based on the user's judgment, the amount of processing required for alignment can be reduced to the necessary amount. According to the present invention of claim 12, it is possible to determine at a glance whether the results of the alignment test are satisfactory to the user. According to the present invention of claim 13, it is possible to improve accuracy while suppressing the amount of processing required for aligning a predetermined template image with a corrected image compared to conventional methods. According to the present invention of claim 14, it is possible to improve accuracy while suppressing the amount of processing required for aligning a template image and an image corrected for character recognition compared to conventional methods. According to the present invention of claim 15, it is possible to improve accuracy while suppressing the amount of processing required for aligning the template image and the image corrected for document discrimination compared to conventional methods. According to the present invention of claim 16, the alignment of a template image for document discrimination, which has been prepared in advance, with an image corrected for document discrimination is performed, so that the accuracy of the alignment can be improved while reducing the amount of processing compared to the conventional method. According to the present invention of claim 17, it is possible to provide an information processing device that improves accuracy while suppressing the amount of processing required for aligning two images compared to conventional methods. According to the present invention of claim 18, it is possible to provide an information processing system that improves accuracy while suppressing the amount of processing required for aligning two images compared to conventional methods. According to the present invention of claim 19, it is possible to provide a program that improves accuracy while suppressing the amount of processing required for aligning two images compared to conventional methods. [Brief explanation of the drawing]

[0008] [Figure 1] This figure shows an example of the overall configuration of an information processing system to which this embodiment is applied. [Figure 2] This figure shows the hardware configuration of a user terminal, which serves as an information processing device to which this embodiment is applied. [Figure 3] This diagram shows the functional configuration of the control unit of a user terminal when the user terminal performs the following processes: the first alignment, the determination of the correction amount based on the result, and the second alignment after correction. [Figure 4] This diagram shows the functional configuration of the control unit of the management server when the management server performs the following processes: the first alignment, the determination of the correction amount based on the result, and the second alignment after correction. [Figure 5] This flowchart shows the processing flow of a user terminal when it performs a series of operations from the first alignment to the second alignment. [Figure 6] This flowchart shows the processing flow of the management server when it performs the series of operations from the first alignment to the second alignment described above. [Figure 7] This figure shows a concrete example of alignment using feature points. [Figure 8] This figure shows a concrete example of alignment using patches. [Figure 9] This figure shows a specific example of a conventional technique where alignment using feature points and alignment using patches are performed simultaneously. [Figure 10] This figure shows a specific example of this embodiment in which alignment is performed using a patch after alignment is performed using feature points. [Figure 11] This figure shows a specific example of this embodiment in which alignment is performed using a patch after alignment is performed using feature points. [Figure 12]This figure shows a specific example of correction performed after alignment using feature points but before alignment using patches. [Figure 13] This figure shows a specific example of correction performed after alignment using patches. [Figure 14] This figure shows a specific example of correction performed after alignment using patches. [Figure 15] This figure shows a specific example of correction performed after alignment using patches. [Figure 16] This figure shows a specific example of a user interface displayed on the user terminal's display. [Figure 17] This figure shows a specific example of a user interface displayed on the user terminal's display. [Figure 18] This figure shows a specific example of a user interface displayed on the user terminal's display. [Figure 19] This figure shows a specific example of a user interface displayed on the user terminal's display. [Figure 20] This figure shows a concrete example of automatic patch configuration. [Figure 21] This figure shows a concrete example of automatic patch configuration. [Figure 22] This figure shows a concrete example of automatic patch configuration. [Figure 23] This figure shows a concrete example of automatic patch configuration. [Figure 24] This figure shows a concrete example of automatic patch configuration. [Figure 25] This figure shows a concrete example of automatic setting of the first threshold. [Figure 26] This figure shows a concrete example of automatic setting of the first threshold. [Figure 27] This figure shows a concrete example of automatic setting of the first threshold. [Modes for carrying out the invention]

[0009] Embodiments of the present invention will be described in detail below with reference to the attached drawings. (Configuration of the information processing system) Figure 1 shows an example of the overall configuration of the information processing system 1 to which this embodiment is applied. The information processing system 1 is configured by connecting a user terminal 10 and a management server 30 via a network 90. ​​The network 90 is, for example, a LAN (Local Area Network), the Internet, etc.

[0010] The user terminal 10 is an information processing device such as a personal computer, smartphone, or tablet that is operated by the user. The user terminal 10 receives various information entered by the user through user input operations and aligns a predetermined template image with the image entered by the user (hereinafter referred to as the "input image"). The "alignment" process is necessary to perform processes such as character recognition, form discrimination, and print defect detection on the input image with high accuracy.

[0011] The user terminal 10 performs the alignment process of the two images (template image and input image) by first performing a first alignment on the entire two images, and then determining the amount of correction required for the second alignment based on the result. After applying the determined correction amount, it performs a second alignment on local areas of the two images. Details of these processes performed by the user terminal 10 will be described later.

[0012] The management server 30 is an information processing device that acts as a server for managing the entire information processing system 1. The management server 30 can perform at least a portion of the above-described processing performed by the user terminal 10. Specifically, the management server 30 can perform at least a portion of the following processes: the first alignment of a predetermined template image and an input image, the determination of the correction amount based on the result, and the second alignment after correction. In this case, information entered into the user terminal 10 by the user's input operation, information output as a result of processing by the user terminal 10 and the management server 30, and information acquired from external sources are exchanged between the management server 30 and the user terminal 10. Details of these processes will be described later.

[0013] The configuration of the information processing system 1 described above is merely an example; the information processing system 1 as a whole only needs to have the functionality to implement the processing described above. For this reason, some or all of the functions that implement the processing described above may be shared or collaborated within the information processing system 1. That is, some or all of the functions of the user terminal 10 may be functions of the management server 30, or some or all of the functions of the management server 30 may be functions of the user terminal 10. Furthermore, some or all of the functions of the user terminal 10 and the management server 30 that constitute the information processing system 1 may be transferred to other servers, etc., not shown in the diagram. This will facilitate processing for the information processing system 1 as a whole, and also allow for mutual complementarity of processing.

[0014] (User terminal hardware configuration) Figure 2 shows the hardware configuration of the user terminal 10, which is an information processing device to which this embodiment is applied. The user terminal 10 includes a control unit 11, a memory 12, a storage unit 13, a communication unit 14, an operation unit 15, and a display unit 16. These units are connected by a data bus, an address bus, a PCI (Peripheral Component Interconnect) bus, etc.

[0015] The control unit 11 is a processor that controls the functions of the user terminal 10 through the execution of various software such as the OS (operating system) and application software. The control unit 11 is composed of, for example, a CPU (Central Processing Unit). The memory 12 is a storage area that stores various software and data used for its execution, and is used as a work area during calculations. The memory 12 is composed of, for example, RAM (Random Access Memory).

[0016] The storage unit 13 is a storage area that stores input data for various software and output data from various software. The storage unit 13 is composed of, for example, an HDD (Hard Disk Drive), an SSD (Solid State Drive), or semiconductor memory used to store programs and various setting data. The storage unit 13 stores a database for storing various information, such as a template DB 131 which stores template images that are the target of alignment.

[0017] The communication unit 14 transmits and receives data between the management server 30 and the outside world via the network 90. ​​The operation unit 15 consists of, for example, a keyboard, mouse, mechanical buttons, and switches, and accepts input operations. The operation unit 15 also has operation images displayed on the display unit 16. The operation unit 15 also includes a touch sensor that forms a touch panel integrally with the display unit 16. The display unit 16 consists of, for example, a liquid crystal display or an organic EL (=Electro Luminescence) display used for displaying information, and displays image and text data. The user interface is displayed on the display unit 16.

[0018] (Management server hardware configuration) The hardware configuration of the management server 30 is the same as that of the user terminal 10 shown in Figure 2. That is, the management server 30 is equipped with a control unit, memory, storage unit, communication unit, operation unit, and display unit, each having the same functions as the control unit 11, memory 12, storage unit 13, communication unit 14, operation unit 15, and display unit 16 in Figure 2, and their illustration and description are omitted.

[0019] (Functional configuration of the user terminal's control unit) Figure 3 shows the functional configuration of the control unit 11 of the user terminal 10 when the user terminal 10 performs the first alignment, determines the correction amount based on the result, and performs the second alignment after correction. When the user terminal 10 performs the above-mentioned first alignment, determination of the correction amount based on the result, and second alignment after correction, the control unit 11 of the user terminal 10 functions as follows: input operation reception unit 101, image acquisition unit 102, alignment unit 103, result acquisition unit 104, image correction unit 105, patch placement unit 106, display control unit 107, condition correction unit 108, and extraction unit 109.

[0020] The input operation reception unit 101 receives user input operations. Examples of user input operations include touch operations with a finger and mouse operations. For example, the input operation reception unit 101 receives input operations to launch application software and display the user interface on the display unit 16 (see Figure 2). The input operation reception unit 101 also receives image information input via the user interface. For example, it receives input images to be aligned, which are input via the user interface.

[0021] The image acquisition unit 102 acquires two images that are to be aligned. For example, the image acquisition unit 102 acquires an input image specified as the target for alignment by user input and a template image corresponding to the input image stored in the template DB 131. Examples of template images stored in the template DB 131 include template images used for alignment in character recognition processing, template images used for alignment in form discrimination processing, and template images used for alignment in print defect detection processing.

[0022] The alignment unit 103 aligns the two images (template image and input image) acquired by the image acquisition unit 102. Specifically, the alignment unit 103 performs a first alignment using feature points, targeting the entirety of both images, and a second alignment using patches, targeting local areas of both images. Since the alignment using feature points is performed on the entirety of both images, approximate alignment can be achieved. Since the alignment using patches is performed after the approximate alignment using feature points is completed, the area of ​​patch movement can be narrowed, reducing the processing load. Furthermore, by placing patches at important locations on the image, the alignment using patches can achieve higher accuracy than the alignment using feature points.

[0023] If the alignment performed by the alignment unit 103 is for character recognition processing, the accuracy of character recognition will be improved. If the alignment is for form discrimination processing, the accuracy of form discrimination will be improved. Furthermore, if the alignment is for defect detection processing of printed materials, the accuracy of defect detection will be improved. The alignment method performed by the alignment unit 103 is not particularly limited, and known methods such as SIFT, SURF, and AKAZE can be used.

[0024] The result acquisition unit 104 acquires the results of the alignment performed by the alignment unit 103. Specifically, the result acquisition unit 104 acquires the results of the first alignment using feature points and the results of the second alignment using patches.

[0025] The image correction unit 105 determines the correction amount required for the second alignment using patches for at least one of the two images, based on the results of the first alignment using feature points obtained by the result acquisition unit 104, and performs the correction. The image correction unit 105 also determines the correction amount based on the results of the alignment using patches obtained by the result acquisition unit 104, and performs the correction. For example, the image correction unit 105 performs correction of the input image targeted for alignment using feature points and the input image targeted for alignment using patches in order to perform alignment in character recognition processing, form discrimination processing, and printed defect detection processing. The correction method by the image correction unit 105 is not particularly limited, and examples include methods that correct geometric transformation coefficients such as projection transformation coefficients and affine transformation coefficients.

[0026] The patch placement unit 106 places patches to be used for the second alignment at one or more predetermined locations on each of the two images. For example, the patch placement unit 106 places patches at one or more locations on the image specified by user input. Alternatively, for example, the patch placement unit 106 automatically determines important locations on the image and places patches at those locations.

[0027] The display control unit 107, as a presentation control means, controls the display of various information on the display unit 16 (see Figure 2). For example, the display control unit 107 controls the display of a user interface on the display unit 16. The user interface displayed on the display unit 16 can be displayed by launching dedicated application software for the user that is pre-installed on the user terminal 10, or by accessing a dedicated website for the user.

[0028] The display control unit 107, as a presentation control means, performs a test (hereinafter referred to as the "alignment test") to determine whether the alignment result of the two images is satisfactory. The display control unit 107 also controls the display of the alignment test result on the user interface. In the alignment test, a series of processes from the first alignment to the second alignment described above are performed, and the display control unit 107 controls the display of the result on the user interface.

[0029] Specifically, the display control unit 107 controls the superimposition of the two images targeted for alignment as a result of the alignment test and displays them on the user interface in a comparable manner. The "comparable manner" is not particularly limited; for example, the upper of the two superimposed images may be displayed semi-transparently, or only the edges of each of the two images may be color-coded. A specific example of the alignment test results displayed on the user interface will be described later with reference to Figure 19.

[0030] The condition modification unit 108 determines whether to modify the conditions used in the alignment test based on the results of the alignment test. Specifically, the condition modification unit 108 determines whether to modify the conditions used in the test based on the results of the alignment test, which consist of the results of the first alignment using feature points and the results of the second alignment using patches, obtained by the result acquisition unit 104. Examples of conditions used in the alignment test include conditions for positional matching and conditions for patch placement. The decision on whether to modify the conditions may also be made based on input information entered by the user.

[0031] Furthermore, when the condition modification unit 108 modifies the positional matching conditions as a modification of the conditions used in the alignment test, it modifies at least one of the thresholds: the first threshold adopted during the first alignment using feature points and the second threshold adopted during the second alignment using patches. Also, when the condition modification unit 108 modifies the patch placement conditions as a modification of the conditions used in the alignment test, it modifies at least one of the following that were adopted as the second threshold during the patch alignment: the number of patches, the placement position of the patches, and the movement area of ​​the patches.

[0032] Furthermore, if the condition modification unit 108 incorrectly determines that the positions of the two images do not match based on the results of the alignment test obtained by the result acquisition unit 104, it will modify at least one of the position matching conditions or the patch placement conditions to make it easier to determine that the positions of the two images do match.

[0033] The extraction unit 109 extracts the image that most closely resembles the image corrected for form discrimination from among the group of template images used for form discrimination alignment in patch-based alignment. Specifically, the extraction unit 109 extracts the template image that most closely resembles the input image corrected for form discrimination alignment from among the template images stored in the template DB 131 of the storage unit 13 (see Figure 2) as template images used for form discrimination alignment.

[0034] (Functional configuration of the control unit of the management server) Figure 4 shows the functional configuration of the control unit of the management server 30 when the management server 30 performs the first alignment, determines the correction amount based on the result, and performs the second alignment after correction. When the management server 30 performs the above-mentioned first alignment, determination of the correction amount based on the result, and second alignment after correction, the control unit 11 of the management server 30 functions as follows: image acquisition unit 301, alignment unit 302, result acquisition unit 303, image correction unit 304, patch placement unit 305, condition correction unit 306, and transmission control unit 307. Of these, the functions of the image acquisition unit 301, alignment unit 302, result acquisition unit 303, image correction unit 304, patch placement unit 305, and condition correction unit 306 are the same as the functions of the image acquisition unit 102, alignment unit 103, result acquisition unit 104, image correction unit 105, patch placement unit 106, and condition correction unit 108 in Figure 3, and therefore their explanation is omitted.

[0035] The transmission control unit 307 controls the transmission of various information to the user terminal 10 or an external source via the communication unit. For example, the transmission control unit 307 controls the transmission of the results of a test to align two images to the user terminal 10. Specifically, the transmission control unit 307 controls the transmission of an image to the user terminal 10 in which the two images targeted for alignment are superimposed in a manner that allows for comparison, as a result of the alignment test.

[0036] (Processing flow at the user terminal) Figure 5 is a flowchart showing the processing flow of the user terminal 10 when it performs a series of operations from the first alignment to the second alignment. The user terminal 10 stores and manages the template images to be aligned in a database (step 501). Specifically, the user terminal 10 stores and manages the template images to be aligned in the template DB 131 (see Figure 2) of the storage unit 13.

[0037] When an input operation to launch the application software is performed on the user terminal 10 (YES in step 502), the user terminal 10 accepts the input operation (step 503) and displays the user interface on the display unit 16 (see Figure 2) (step 504). Conversely, if no input operation to launch the application software is performed (NO in step 502), the user terminal 10 repeats step 502 until an input operation to launch the application software is performed.

[0038] When the user terminal 10 receives an input operation via the user interface to input an image to be aligned (YES in step 505), it accepts the input operation (step 506) and retrieves the input image and a template image corresponding to that input image stored in the database (step 507). Conversely, if no input operation has been performed to input an image to be aligned (NO in step 505), the user terminal 10 repeats step 505 until an input operation to input an image to be aligned is performed.

[0039] The user terminal 10 performs a first alignment using feature points on the entirety of the two images acquired in step 507 (step 508), obtains the result (step 509), determines the amount of correction required for a second alignment using patches on local areas of the two images based on the result (step 510), and performs the correction (step 511).

[0040] The user terminal 10 places a patch at one or more locations having predetermined characteristics (step 512), performs a second alignment (step 513), obtains the result (step 514), determines a correction amount based on the result (step 515), corrects the correction amount (step 516), and outputs the alignment result (step 517).

[0041] (Processing flow of the management server) Figure 6 is a flowchart showing the processing flow of the management server 30 when it performs the series of operations from the first alignment to the second alignment described above. The management server 30 stores and manages the template images to be aligned in a database (step 601).

[0042] When the management server 30 receives an input image to be aligned, which has been entered into the user terminal 10 by a user's input operation (YES in step 602), it retrieves that input image and the corresponding template image stored in the database (step 603). However, if no input image to be aligned has been received (NO in step 602), the management server 30 repeats step 602 until an input image to be aligned is received.

[0043] The management server 30 performs a first alignment using feature points on the entirety of the two images acquired in step 603 (step 604), obtains the result (step 605), determines the amount of correction required for a second alignment using patches on local areas of the two images based on the result (step 606), and performs the correction (step 607).

[0044] The management server 30 places the patch at one or more locations having predetermined characteristics (step 608), performs a second alignment (step 609), obtains the result (step 610), determines a correction amount based on the obtained alignment result (step 611), applies the correction amount (step 612), and transmits the alignment result to the user terminal 10 (step 613).

[0045] (Specific example) Figure 7 shows a specific example of alignment using feature points. Figure 7(A) shows the template image Gt, which is one of the two images to be aligned. In the template image Gt, objects J1 and J2 are placed in the upper left region of the image. Figure 7(B) shows the input image Gi, which is the other of the two images to be aligned. In the input image Gi, objects J1 and J2 are placed in the upper left region of the image, and object J3 is placed in the lower right region of the image.

[0046] When aligning a template image Gt with an input image Gi using only feature points, feature points are automatically extracted based on the image analysis results, and a comparison is performed on the extracted feature points and the surrounding areas. In this case, it is not possible for the user to set the feature points to be extracted from their intended locations. As a result, for example, as shown in Figures 7(A) and (B), if the feature points indicated by the "×" marks are concentrated in the upper left area of ​​the image, the accuracy of the alignment of areas that were not extracted as feature points (for example, the lower right area of ​​the image) will decrease.

[0047] As a result, for example, in character recognition processing and print defect detection processing, the accuracy of the processing decreases because it becomes impossible to align the area intended by the user. Also, for example, in form discrimination processing, object J3, which is located in the lower right area of ​​the input image Gi, is not detected, and it is incorrectly identified as the same form with only objects J1 and J2.

[0048] Figure 8 shows a specific example of alignment using patches. Figure 8(A) shows the template image Gt. Figure 8(B) shows the input image Gi. Both the template image Gt and the input image Gi have objects J1 and J2 positioned in the upper left region of the image, with a patch P for alignment placed at the lower right edge of object J2.

[0049] When aligning a template image Gt with an input image Gi using only patch P, pixels from the entire area of ​​patch P are extracted from each of the two images, and the difference in pixel values ​​of the extracted pixels is obtained. Then, while patch P is moved to various positions within the image, or transformed by scaling, rotation, etc., the extraction of pixels from the entire area of ​​patch P and the acquisition of the difference in pixel values ​​of the extracted pixels are performed again.

[0050] However, if alignment is performed using only patch P, it is necessary to perform a difference calculation of the pixel values ​​of all pixels in the entire movement area A to which patch P moves. Therefore, the processing load increases with the increase in the number of pixels in patch P, the number of patches P themselves, and the expansion of the movement area A to which patch P moves. For example, if the size of patch P is 100 x 100 pixels and the upper and lower range of movement area A is plus or minus 100 pixels, then 100 x 100 x 200 x 200 = 400 million difference calculations are required for one patch P. Furthermore, if the number of patches P themselves is n (where n is an integer value of 1 or more), then 400 million x n00 million difference calculations are required. As a result, the time required for the entire process in character recognition processing, print defect detection processing, form discrimination processing, etc., which include some alignment processing, increases significantly.

[0051] Furthermore, if the distortion of the input image Gi is greater than expected, causing the movement of patch P to exceed a predetermined amount, it becomes impossible to perform alignment with high accuracy. This problem can be solved by widening the movement area A of patch P and increasing the maximum movement amount of patch P. However, as mentioned above, the processing load increases with the expansion of the movement area A of patch P, so the time required for character recognition processing, which includes alignment processing in part, increases significantly.

[0052] Figure 9 shows a specific example of the conventional technology, illustrating a case where alignment using feature points and alignment using patches are performed simultaneously. When performing alignment using feature points and alignment using patch P simultaneously, alignment using patch P is performed directly, separate from alignment using feature points. In this case, patch P is moved to various positions within the image, and transformations such as scaling, rotating, etc., are performed on patch P while searching for the transformation that best matches the pixel values ​​of the entire area of ​​patch P placed in each of the two images. At this time, if the positions of objects J1 and J2 are as shown in Figures 9(A) and (B), the rotation angle of patch P becomes larger, and the size of the movement area A of patch P is expanded. As a result, the processing load increases.

[0053] Figures 10 and 11 show a specific example of this embodiment in which alignment is performed using patches after alignment is performed using feature points. Figure 10(A) shows the template image Gt. Figure 10(B) shows the input image Gi after transforming the input image Gi shown in Figure 9(B) based on the result of alignment using feature points as shown in the example in Figure 9 above. Since alignment using feature points covering the entire image is performed before alignment using patch P is performed, approximate alignment is completed by the time alignment using patch P is started. As a result, as shown in Figure 10(B), the patch positions of the two images almost coincide. This suppresses the expansion of the size of the movement area A of patch P, thereby limiting the amount of movement of patch P and reducing the processing load. As a result, the overall processing time required for character recognition processing and other processes that include alignment processing is shortened.

[0054] Figure 11 shows a specific example where the placement of the patch is determined by user input. Figure 11(A) shows the template image Gt. Figure 11(B) shows the input image Gi. In both the template image Gt and the input image Gi, objects J1 and J2 are located in the upper left region of the image, and object J3 is located in the lower right region of the image. In the examples in Figures 11(A) and (B), the feature points indicated by the "×" marks are concentrated in the upper left region of the image. In this case, although it is a region with few feature points, patch P can be placed in the lower right position of the image, which is an important location where object J3 is located.

[0055] Specifically, the template image Gt and input image Gi shown in Figures 11(A) and (B) are displayed in the user interface, and the user places the patch P precisely at the location shown in Figure 11(B) by performing input operations on the user interface. This improves the alignment accuracy and makes it possible to improve the accuracy of discrimination in, for example, the form discrimination process.

[0056] Figure 12 shows a specific example of correction performed after alignment using feature points but before alignment using patches. When determining the correction amount for the input image Gi based on the results of alignment using feature points, geometric transformation coefficients such as projection transformation coefficients and affine transformation coefficients can be used as the correction amount. When using projection transformation coefficients as the correction amount for the input image Gi, the following process is performed: Feature points indicated by "×" marks are extracted from the two images, and the feature vector of the feature point is calculated from the image region surrounding that feature point. Then, the feature points of each of the two images are matched. Specifically, pairs of feature points with close feature vector distances are extracted. The dashed lines connecting the feature points of the two images in Figure 12 are dashed lines connecting pairs of feature points. When multiple pairs of feature points are extracted, projection transformation coefficients are calculated using those pairs of feature points. Note that there are no particular limitations on the method for calculating projection transformation coefficients, and general methods can be used.

[0057] Figures 13 to 15 show specific examples of corrections performed after alignment using patches. Assume that a patch P is placed on a template image Gt shown in Figure 13(A) at the position shown in Figure 13(B). In this case, first, patch P is placed on the input image Gi shown in Figure 13(C) at the position corresponding to the template image Gt. If distortion exists as shown in Figure 13(C), then when comparing patch P placed on the template image Gt with patch P placed on the input image Gi, the position and rotation angle will differ. Therefore, for example, the following correction is performed.

[0058] In other words, the difference between the pixel values ​​of all pixels in the entire region of patch P placed in the template image Gt and the pixel values ​​of all pixels in the entire region of patch P placed in the input image Gi is calculated. Specifically, for example, the difference in pixel values ​​is calculated using the formula shown in the upper part of Figure 14. In addition, the difference between the pixel values ​​of the area of ​​patch P moved when patch P is moved within the image and the pixel values ​​of all pixels in the entire region of patch P when patch P is moved is calculated.

[0059] Specifically, for example, the difference in pixel values ​​is calculated using the formula shown in the middle of Figure 14. Note that the formula in the middle of Figure 14 represents the sum of the absolute values ​​of the differences in pixel values ​​of all pixels in the entire area of ​​patch P, but it could also be the sum of squares, or the square root of the sum of squares, etc. Furthermore, the amount of movement of patch P may include rotation, etc. For example, the amount of movement that minimizes the difference in pixel values ​​of all pixels in the entire area of ​​patch P can be searched using the formula shown in the lower part of Figure 14.

[0060] Furthermore, as shown in Figure 15, the dashed lines forming pairs of patch P placed in the two images allow for matching between patch P placed in the input image Gi and patch P placed in the template image Gt. Once such pairs of patch P are formed, geometric transformation coefficients can be calculated in the same way as the feature point alignment described above. Alternatively, geometric transformation coefficients may be calculated by combining pairs of feature points and pairs of patch P. Also, geometric transformation coefficients may be calculated locally rather than for the entire image. For the image region surrounding patch P, the amount of patch P movement calculated using that patch P may be used directly as the transformation coefficient. Furthermore, the pixel positions between patch P may be calculated using linear interpolation or the like.

[0061] Figures 16 to 19 show specific examples of user interfaces displayed on the display unit 16 of the user terminal 10. Figure 16 shows a specific example of a "report design tool" as an example of a user interface. The "report design tool" is a user interface displayed when a user inputs a set of test input images and a template image, and outputs the result of a report design. The result of the report design output by the "report design tool" includes a template image with patches placed on it.

[0062] The "Report Design Tool" contains four buttons, B1 through B4. Specifically, button B1 is labeled "Patch Addition Tool," button B2 is labeled "Image Alignment Test Tool," button B3 is labeled "Report Discrimination Test Tool," and button B4 is labeled "Feature Point Alignment Design Tool."

[0063] Users can add patches by pressing button B1, labeled "Add Patch Tool." When adding a patch, users can set the patch's position on the template image, its size, and its movement range through input operations. Set patches can also be deleted through user input operations. Furthermore, users can set a threshold value used to determine whether patches in two images match.

[0064] Figure 17 shows a concrete example of the "Patch Addition Tool." When button B1 in Figure 16 is pressed, a user interface labeled "Patch Addition Tool" is displayed, as shown in Figure 17. The "Patch Design Tool" allows you to set the position, size, rotation angle, etc., of patch P using touch or mouse operations. You can also set the range of patch P's movement area A using touch or mouse operations. Although not shown in the illustration, you can also set a threshold used to determine whether the positions of patch P in two images match. Note that the threshold can be set for each patch P or for all placed patch P at once.

[0065] Figure 18 shows another specific example of the "Patch Addition Tool." In the example in Figure 17 above, the position, size, rotation angle, etc. of patch P, and the movement area A of patch P are set by user input, but these settings can also be done automatically. In this case, the setting values ​​for the position, size, rotation angle, etc. of patch P, and the setting value for the movement area A of patch P are calculated, and the calculated results are displayed in the user interface, for example, as "recommended values." Alternatively, predetermined values ​​may be displayed in the user interface as "default values."

[0066] The user checks the displayed calculation results and, if there are no problems, presses button B11 labeled "OK". This sets the position of patch P, etc. Furthermore, if the user determines that there is a problem with the displayed calculation results for the position of patch P, etc., they can correct it themselves through input operations, as shown in the example in Figure 17 above. For specific methods of automatically setting the position of patch P, etc., please refer to Figures 20 to 24 and refer to them later.

[0067] Returning to Figure 16, the user can perform an alignment test by pressing button B2, labeled "Image Alignment Test Tool." When performing the alignment test, the user can specify the input image to be used for the test through input operations. In the alignment test, the initial number of patches is set to "0 (zero)." When the user runs the alignment test, alignment using feature points and alignment using patches are performed on the template image and the input image. The alignment results are then displayed as test results in the user interface.

[0068] Figure 19 shows an example of the results of an alignment test displayed in the user interface. In the example in Figure 19, the results of the alignment test are displayed with the template image Gt, shown by a dashed line, and the input image Gi, shown by a solid line, superimposed. This allows the user to grasp the alignment results at a glance. Note that while Figure 19 shows an example of distinguishing the template image and the input image by solid and dashed lines, it is not limited to this example. Any method that allows the template image and the input image to be distinguished at a glance is acceptable; for example, they could be distinguished by color coding, or by changing the thickness of the solid line indicating the outer edge of the image.

[0069] The user interface can also display the results of character recognition processing on the input image Gi used for the alignment test. If the alignment test results are unsatisfactory, the results of the character recognition processing, which includes some alignment processing, will also be unsatisfactory. Therefore, users can use the character recognition processing results as an indicator when judging the results of the alignment test.

[0070] Furthermore, the user can perform the following actions while viewing the alignment test results displayed on the user interface: If the alignment test results are satisfactory, the user exits the "Form Design Tool." If the alignment results are unsatisfactory, the user performs the following actions: If the alignment test results indicate that the input image Gi used in the test and the template image Gt do not match, the user modifies the first or second threshold. Specifically, the threshold is modified to a looser value that makes it easier to determine that the input image Gi and the template image Gt match.

[0071] Furthermore, if there is a discrepancy in the alignment results between the input image Gi used in the test and the template image Gt, perform the following input operations: either perform an input operation to add a patch, or perform an input operation to re-test the alignment. If you re-test the alignment, you may specify multiple input image Gi to use in the test and perform the above test for each of the specified input image Gi.

[0072] Returning to Figure 16, the user can test the document recognition process by pressing button B3, labeled "Document Recognition Test Tool." When testing the document recognition process, the user can specify the input image to be used for the test through input operations. In the initial stage of the document recognition test, the number of patches is set to "0 (zero)." In the document recognition test, the template image and the input image are aligned using feature points and aligned using patches to recognize the document. The recognition result is then displayed in the user interface.

[0073] The user interface displays the judgment results for each of the 0 (zero) or 1 or more forms. The user can perform the following operations while viewing the form judgment results: If the form judgment results are satisfactory, the "Form Design Tool" is terminated. Specifically, if template image 1 matches all input images specified for the form judgment test, or if template image 1 is determined to be the image that best matches all input images specified for the form judgment test, the "Form Design Tool" is terminated. If the form judgment results are not satisfactory, the following operations are performed: If different template images match the input images specified for the form judgment test, the first or second threshold is modified, or a patch is added. When modifying the first or second threshold, the threshold is made stricter so that it is more likely to be determined that the input image Gi and template image Gt used in the test do not match.

[0074] Furthermore, if a template image Gt corresponding to the input image Gi used in the test does not exist, the following input operation is performed: that is, the first or second threshold is modified, or the patch is deleted. If the first or second threshold is modified, it is modified to a looser threshold so that it is easier to determine that the input image Gi used in the test and the template image Gt match. Also, if multiple template images Gt are determined to match the input image Gi used in the test, the following input operation is performed: that is, the first or second threshold is modified, or a patch is added. If the first or second threshold is modified, it is modified to a stricter threshold so that it is easier to determine that the input image Gi used in the test and the template image Gt do not match.

[0075] Returning to Figure 16, the user can, for example, set a first threshold indicating whether feature point alignment was successful by pressing button B4, which is labeled "Feature Point Alignment Design Tool."

[0076] Figures 20 to 24 show specific examples of automatic patch configuration. Automatic patch configuration automates the user's judgment and input operations that occur after button B1 of the "Patch Addition Tool" in Figure 16 is pressed. Here, the output report ID is an identifier that uniquely identifies the report, i is a variable that represents the output report ID, and N is the number of report types, so i can be between 1 and N. In addition, a set of setting images is prepared in order to set thresholds and patches. In the example in Figure 20, the patch is configured by inputting the prepared set of setting images.

[0077] The set of images provided for configuration is as follows: That is, the input image belonging to form i is M i Assuming there are multiple input images, each input image is i-1 to iM i Let's assume that the j-th image belonging to form i is denoted as ij. Also, the set of images belonging to form i is called "Form i Image Set A". i In this case, form i image set Ai ={i-1,i-2,···,iM i}. Also, the set of images for evaluation includes a group of images that do not belong to any form. Let's call this "Form 0 Image Set A0". In this case, Form 0 Image Set A0 = {0-1, 0-2, ..., 0-M0}. Note that Form 0 Image Set A0 is optional. Also, let iT be the template image for form i. iT is Form i Image Set A i It may or may not be included.

[0078] When a set of images for configuration is input, the patch is automatically configured. In the automatic patch configuration, the patch is automatically configured so that images belonging to form i are classified only as form i. Here, it is assumed that the matching form has already been selected based on the result of alignment using feature points, and that the alignment using feature points is pre-set so that the output form ID of all input images for form i is i. In other words, the alignment using feature points is set loosely to match a wider range of forms. The table in Figure 21 shows how this setting works.

[0079] In the table in Figure 21, "Input Image Setting Image Set" refers to the images entered for report identification, and "Output Report ID" indicates the result of report identification. "〇" indicates the report that matched each input image as a result of alignment using feature points. That is, if an image belonging to report i matches report i, it is judged as OK and marked with "〇". Conversely, "●" indicates that an image belonging to report i was judged as NG because it matched report k (k≠i). In automatic patch configuration, the patch is configured to remove "●" from the table in Figure 21. Figure 22 shows how this configuration works.

[0080] In the example of FIG. 22, independent processing is performed for each output form ID. That is, patches are automatically set for the form i template, but at that time, patches for other form k (k≠i) templates are not considered. Also, when patches are automatically set for the form i template, independent processing is performed for each output form ID of the input image. That is, patches that do not match the input image k-j (j = 1 to M k )(where the output form ID is k) are automatically set, but at that time, patches for other input images (output form ID is m (m≠k)) are not considered.

[0081] As shown in the table of FIG. 22, when the output form ID is i and the output form ID of the input image is k, the information required for automatic patch setting is the area where masking is performed, and patches are automatically set so that "〇" is judged as OK and "●" is judged as NG. Here, for the input image k-j (j = 1 to M k ), the number of forms that match the alignment using the feature amount of form i (that is, the number of "●") is p i-k , and the input images that match are set as k-q i-k-m (however, m = 1 to p i-k ). In this case, in the automatic patch setting, the positions of candidate patches are searched. Regarding the method of searching for the positions of candidate patches, it will be described later with reference to FIG. 23.

[0082] As shown in the table of FIG. 22, if patches match form i for all of the input images i-j (j = 1 to M i ) (condition 1), and at least one patch of the input image k-q i-k-m (m = 1 to p i-k ) does not match form i (condition 2), then that patch is added as a patch for form i. Also, if all "●" have been removed, the process proceeds to the next process as "k→k+1". On the contrary, if "●" has not been removed, a patch is added. That is, in the example of FIG. 22, p i-k = 2, q i-k-1 = 2, q i-k-2 = j, and the "●" in the input image k-2 is k-qi-k-1 =k-2, and the "●" in the input image kj is kq i-k-2 =kj. And p i-k = 2.

[0083] Figure 23 shows a specific example of a method for searching for candidate patch locations in automatic patch placement. In automatic patch placement, patch locations are set using a raster scan. A "raster scan" is a method of obtaining scan lines by scanning one-dimensionally at points, and then obtaining a two-dimensional surface image by scanning perpendicular to those scan lines. That is, as shown in Figure 23(A), the location of patch P is scanned pixel by pixel from the top left to the bottom right of the input image Gi. The raster scan can be performed pixel by pixel, or it can be performed by W pixels (W is an integer value of 2 or more) at once.

[0084] In automatic patch setting, as shown in Figure 23(B), the patch location with the largest sum of absolute differences in pixel values ​​between template image i and template image k is set as the patch candidate. Specifically, pixels of patches in template image i and template image k are extracted from the raster scan, and the sum of absolute differences in the pixel values ​​of these two sets of pixels is calculated. The patch location with the largest calculated value is then selected. Note that the calculated value is not limited to the sum of absolute values; for example, it could be the sum of squares or the square root of the sum of squares.

[0085] Figure 24 shows a specific example of how the placement of a patch is determined in automatic patch placement. In automatic patch placement, when the placement of a patch is determined, the location of a writing field is automatically excluded from the candidates. In other words, in the case of a form image, there are locations where some kind of handwritten addition is made, such as the writing field J11 for writing a name as shown in Figure 24(A). Such locations are automatically excluded from the candidates for patch placement. Specifically, as shown in Figure 24(B), patch P that partially overlaps with the writing field J11 is automatically excluded from the candidates for placement.

[0086] Figures 25 to 27 show specific examples of automatic setting of the first threshold. The variable names in Figures 25 to 27 are the same as those in Figures 20 to 24 described above. In the automatic setting of the first threshold, the set of images that do not belong to form i is referred to as "Image set B other than form i". i "

[0087] In the first automatic threshold setting, the following processing is performed on form i (i=1 to N): namely, the form i template image and form i image set A. i The evaluation value is calculated for the images within the set A. i The minimum value of the evaluation value within is A i_min The "evaluation value" to be calculated can be, for example, the distance between feature vectors or image similarity. Next, the template image of form i and the set of images other than form i, Y. i The evaluation value is calculated for the images within the set B. i The maximum value within the evaluation is B i_max Therefore, the evaluation value of the same form will be higher, so set A i The values ​​of the internal elements (evaluation values) are in set B. i It will be higher than the value of the internal element (evaluation value). However, the opposite can also occur.

[0088] Figure 26 shows a histogram of the results of the above processing. The leftmost histogram shows the image similarity of the Form 1 template image. The middle histogram shows the image similarity of the Form i template image. The rightmost histogram shows the image similarity of the Form N template image. In the leftmost histogram, "A1" represents the "Form 1 image" and "B1" represents the "image other than Form 1". Also, in the middle histogram, "A i " indicates "Form i image", and "B i " indicates "images other than form i". Also, in the histogram on the far right, "A N " indicates "Form N image", and "B N This indicates "images other than form N".

[0089] The first threshold is automatically set for each form. For example, for form i, the threshold is TH. i =A i_min Let's assume that. Or, A i_min Instead of using it directly as the threshold, a margin is set. In this case, the threshold TH i =A i_min Let -α (where α is a predetermined margin). Next, B i_max The value will be adjusted to take this into account. For example, the threshold TH i =( A i_min +B i_max ) / 2, for example, the value of "A i_min " may be used as the first threshold value for form i.

[0090] The first threshold is not limited to being automatically set for each form; it is also possible to automatically set a threshold common to all forms. Figure 27 shows a specific example of the formula used when automatically setting the first threshold common to all forms. When the first threshold common to all forms is automatically set, the number of forms increases, and statistical stability improves. First, the minimum evaluation value C of all forms i min Calculate A. i_min Without seeking C min You can also directly calculate the minimum value C of the evaluation value of all reports i. min The formula F11 shown in Figure 27 can be used to calculate this.

[0091] Next, the maximum value D of the evaluation values ​​other than all reports i. max Calculate B. i_max Without seeking D max You may also directly calculate the maximum value D of all evaluation values ​​other than all reports i. max For the calculation of C, the formula F12 shown in Figure 27 can be used. For example, C min and D max The midpoint is the threshold candidate TH mid In this case, the formula F13 shown in Figure 27 can be used. Alternatively, for example, a margin coefficient α can be introduced to obtain a smaller threshold candidate TH. marginWhen setting this, you can use formula F14 shown in Figure 27. Alternatively, for example, the margin may be used as a multiplication, as shown in formula F15 in Figure 27.

[0092] Although this embodiment has been described above, the present invention is not limited to this embodiment. Furthermore, the effects of the present invention are not limited to those described in this embodiment. For example, the configuration of the information processing system 1 shown in Figure 1 and the hardware configuration of the user terminal 10 shown in Figure 2 are merely examples for achieving the objectives of the present invention and are not particularly limited. Similarly, the functional configuration of the user terminal 10 shown in Figure 3 and the functional configuration of the management server 30 shown in Figure 4 are also merely examples and are not particularly limited. It is sufficient that the information processing system 1 in Figure 1 is equipped with a function that can execute the above-described process as a whole, and the functional configuration used to realize this function is not limited to the examples in Figures 3 and 4.

[0093] Furthermore, the order of the processing steps for the user terminal 10 shown in Figure 5 and the processing steps for the management server 30 shown in Figure 6 are merely illustrative and not particularly limiting. Processing is not limited to being performed chronologically according to the illustrated steps; it may also be performed in parallel or individually. Similarly, the specific examples shown in Figures 7 to 27 are merely examples and not particularly limiting. [Explanation of symbols]

[0094] 1...Information processing system, 10...User terminal, 11...Control unit, 30...Management server, 90...Network, 101...Input operation reception unit, 102...Image acquisition unit, 103...Alignment unit, 104...Result acquisition unit, 105...Image correction unit, 106...Patch placement unit, 107...Display control unit, 108...Condition correction unit, 109...Extraction unit, 301...Image acquisition unit, 302...Alignment unit, 303...Result acquisition unit, 304...Image correction unit, 305...Patch placement unit, 306...Condition correction unit, 307...Transmission control unit

Claims

1. A processor is provided. The processor, determining a correction amount required for a second alignment of the two acquired images based on a result of the first alignment of the two acquired images; one of the two images is corrected by the determined correction amount, and the second alignment is performed. Information processing device.

2. The processor performs, as the first alignment, alignment targeting the entirety of the two images, and performs, as the second alignment, alignment targeting local areas of the two images. The information processing device according to claim 1 .

3. The processor performs registration using feature points as a registration target for the entirety of the two images, and performs registration using patches as a registration target for local areas of the two images. The information processing device according to claim 2 .

4. The processor, when performing the second alignment, places the patch at one or more positions having a predetermined characteristic. The information processing device according to claim 3 .

5. The processor is configured to place the one or more patches based on a relationship with a position of the feature point. The information processing device according to claim 4.

6. The processor arranges the one or more patches based on input information input by a user's input operation. The information processing device according to claim 3 .

7. the processor determines whether to modify the conditions for matching the positions and the conditions for disposing the patches used in the test based on a result of the test of the alignment of the two images using the first alignment and the second alignment. The information processing device according to claim 6.

8. The processor, as a modification of the condition for matching the positions, modifying at least one of a first threshold value used in the first alignment and a second threshold value used in the second alignment; As the correction of the patch placement conditions, at least one of the number of patches used in the second alignment, the placement positions, and the movement area is corrected. The information processing device according to claim 7.

9. When the result of the test indicates that the positions of the two images do not match, the processor modifies at least one of the conditions for the match and the conditions for the placement of the patches so that the positions of the two images can be more easily determined to match. The information processing device according to claim 8.

10. When the result of the test indicates that the positions of the two images match, the processor modifies at least one of the conditions for the match and the conditions for the placement of the patches so that the processor can more easily determine that the positions of the two images do not match. The information processing device according to claim 8.

11. The processor, Controlling the presentation of results of the test to the user; determining whether to modify the position matching condition and the patch placement condition based on the input operation of the user; The information processing device according to claim 7.

12. The processor controls the display of the test result to the user by superimposing the two images and displaying the results to the user in a comparative manner. The information processing device according to claim 11.

13. Of the two images, a first image is an image of a predetermined template, and a second image is an image corrected by the correction amount. The information processing device according to claim 1 .

14. The template image is an image of a template in character recognition, and the corrected image is an image corrected for the character recognition. The information processing device according to claim 13.

15. The template image is an image of a template for document discrimination, and the corrected image is an image corrected for the document discrimination. The information processing device according to claim 13.

16. The processor is characterized in that, in the second alignment, an image most similar to the image corrected for the form discrimination is extracted from a group of images of templates for the form discrimination prepared in advance. The information processing device according to claim 15.

17. A processor is provided. The processor, Controlling transmission of the input image to be aligned to another information processing device; The method is characterized in that the method controls the other information processing device to obtain and output results of a first alignment process for two images including the image and a second alignment process for correcting the images. Information processing device.

18. An acquisition means for acquiring two images to be aligned; A first alignment means for performing a first alignment on the two acquired images; a determination unit that determines a correction amount required for a second alignment of the two images based on a result of the first alignment; a conversion means for correcting one of the two images by the determined correction amount; a second alignment means for performing a second alignment on the two images, one of which has been corrected; Characterized in that it has Information processing system.

19. On the computer, A function of determining, based on a result of a first alignment performed on the two acquired images, a correction amount required for a second alignment performed on the two acquired images; a function of correcting one of the two images using the determined correction amount and performing the second alignment; A program to achieve this.

20. A computer comprising: A function of performing a first alignment on the two acquired images; a function of controlling displaying one and the other of the two images corrected by the first alignment on a user interface; a function of arranging a patch at a position on an image designated by a user's input operation on the user interface; performing a second alignment of the corrected one image with the other image using the patch; and A program to achieve this.

Images