Information processing device, information processing method, and computer-readable medium

Abstract

The present invention provides an information processing device, an information processing method, and a computer-readable medium. The information processing device includes: an input unit that receives an input image to be recognized; and a processor that, by executing a program, performs the following processing: aligning the input image with a template image so that a recognition area in the input image matches a recognition area defined in the template image; performing recognition processing on the recognition area of the aligned input image; generating a confirmation image containing the input image aligned with the template image; and displaying the confirmation image and the result of the recognition processing in a manner that allows identification of the correspondence between them.

Description

Technical Field

[0001] The present disclosure relates to an information processing device, an information processing method, and a computer-readable medium. Background Art

[0002] Conventionally, technologies for automatically recognizing characters entered in forms and the like have been proposed.

[0003] Japanese Patent Application Laid-Open No. 2006-129311 describes a position alignment device that aligns standardized documents with minimal computational effort. The device includes an image rotation unit that uprights an input image and stores the rotation angle of the input image; and a position offset correction unit that calculates the position offset of the image based on the rotation angle or input device information and performs correction based on the resulting position offset. Summary of the Invention

[0004] When recognizing a character string based on an input image such as a form, it is desirable to display the input image and the recognition result simultaneously so that the user can compare and confirm the input image and the recognition result and make corrections as needed.

[0005] However, if the input image is tilted, displaying it in this tilted state can create a sense of discomfort for the user. While it is possible to display the image after correcting the tilt, this may result in the loss of certain information. More specifically, if a template image is used to define the recognition area, the template image's recognition area is aligned with the input image's recognition area, and only the template image area is displayed, information outside the template image area, particularly information relevant to confirming the recognition result, may be lost and not displayed.

[0006] The purpose of the present disclosure is to provide a technology that can prevent information loss when using a template image to define the recognition area of an input image and aligning the recognition area of the template image with the recognition area of the input image to recognize a character string, so that the user can compare the input image and the recognition result and easily confirm it.

[0007] According to the first embodiment of the present disclosure, there is provided an information processing device comprising: an input unit which inputs an input image to be recognized; and a processor which, by executing a program, performs the following processing: aligning the input image with the template image so that a recognition area in the input image is consistent with a recognition area defined in the template image; performing recognition processing of the recognition area of the input image after the position alignment; generating a confirmation image including the input image after the position alignment with the template image; and displaying the confirmation image and the result of the recognition processing in a manner such that the correspondence between them can be identified.

[0008] According to the second aspect of the present invention, the processor generates a rectangular image as the confirmation image.

[0009] According to the third aspect of the present invention, the processor generates, as the confirmation image, a circumscribed rectangular image of the input image aligned with the template image.

[0010] According to a fourth aspect of the present disclosure, the processor annotates, as the confirmation image, information indicating that the image is the template image in a region of the circumscribed rectangular image where the positionally aligned input image does not exist and only the template image exists.

[0011] According to the fifth aspect of the present disclosure, the processor includes information that exists in the input image after position alignment but does not exist in the template image in the confirmation image and outputs the information.

[0012] According to the sixth aspect of the present disclosure, the template image defines a recognition area in the input image and defines a character string to be recognized.

[0013] According to a seventh aspect of the present invention, the input image is a scanned image obtained by scanning a document in an oblique state.

[0014] According to an eighth aspect of the present disclosure, the document is a form.

[0015] According to the 9th embodiment of the present invention, there is provided a computer-readable medium storing a program for causing a computer to execute processing, the processing comprising the following steps: aligning the input image with the template image so that a recognition area in the input image is consistent with a recognition area defined in the template image; performing recognition processing on the recognition area of the input image after position alignment; generating a confirmation image including the input image after position alignment with the template image; and displaying the confirmation image and the result of the recognition processing in a manner that enables identification of the correspondence between them.

[0016] According to the tenth aspect of the present disclosure, there is provided an information processing method, comprising the following steps: aligning the input image with the template image in order to make the recognition area in the input image consistent with the recognition area defined in the template image; performing recognition processing of the recognition area of the input image after the position alignment; generating a confirmation image containing the input image after the position alignment with the template image; and displaying the confirmation image and the result of the recognition processing in a manner that enables identification of the correspondence between them.

[0017] (Effect)

[0018] According to the first, ninth or tenth scheme, when a template image is used to define the recognition area of an input image and the recognition area of the template image is aligned with the recognition area of the input image to recognize a character string, information loss can be prevented and the user can easily confirm the input image and the recognition result by comparing them.

[0019] According to the second or third aspect, the user can also use the rectangular area for confirmation.

[0020] According to the fourth aspect, the user can also easily confirm the template image.

[0021] According to the fifth aspect, the user can also confirm the related information present in the input image.

[0022] According to the sixth aspect, the recognition target can also be specified using a template image.

[0023] According to the seventh aspect, it is possible to cope with the situation where the input image is tilted due to an error during scanning or the like.

[0024] According to the eighth aspect, the form can also be recognized and confirmed and corrected. BRIEF DESCRIPTION OF THE DRAWINGS

[0025] Figure 1 This is a conceptual structural diagram of an implementation method.

[0026] Figure 2 This is a diagram illustrating the calculation of the transformation matrix in the implementation method.

[0027] Figure 3 This is a diagram for explaining alignment of an embodiment (part 1).

[0028] Figure 4 This is a position alignment explanation diagram (part 2) of the embodiment.

[0029] Figure 5 1 and 2 are diagrams illustrating the circumscribed rectangle of an input image after position alignment according to an embodiment.

[0030] Figure 6It is an explanatory diagram of the circumscribed rectangles of the input image and the template image after position alignment according to the embodiment.

[0031] Figure 7 A supplementary explanatory diagram of a template image according to an embodiment.

[0032] Figure 8 It is a functional block diagram of an embodiment.

[0033] Figure 9 It is a structural block diagram of an implementation method.

[0034] Figure 10 It is a process flow chart of an implementation method.

[0035] Figure 11A It is a diagram illustrating the coordinate system of the input image according to the embodiment.

[0036] Figure 11B It is a diagram illustrating the coordinate system of the template image according to the embodiment.

[0037] Figure 11C It is an explanatory diagram of a coordinate system for confirming and correcting an image according to an embodiment.

[0038] Figure 12 This is a functional block diagram of a modified example.

[0039] Figure 13 It is a rectangular explanatory diagram of a modified example. DETAILED DESCRIPTION

[0040] Hereinafter, embodiments of the present disclosure will be described with reference to the accompanying drawings.

[0041] <Basic Principles>

[0042] First, the basic principle of this embodiment will be described.

[0043] In the information processing device of this embodiment, when an image such as a form is input and the character string is recognized, a template image is used to define the recognition area of the input image. The character string is recognized by aligning the recognition area of the template image with the recognition area of the input image. Here, a "template image" is a fixed pattern that defines the recognition object in the input image and includes the recognition area and recognition method. Specifically, it may be "recognize the character string at position XX using a Romanization dictionary." The recognition area can be defined, for example, by the vertex coordinates of a rectangular area, the coordinates of the top left vertex, and its size.

[0044] Then, the recognition area of the template image and the recognition area of the input image are aligned to recognize the character string, and the input image and the recognition result are displayed together on a display device, etc., for the user to confirm the recognition result.

[0045] Figure 1 This is a conceptual structural diagram of this embodiment.

[0046] An input image 10 such as a form includes a recognition area 12 to be recognized. In the figure, a character string “ABC” is written in the recognition area. The input image 10 is supplied to a recognition unit 14 .

[0047] A template image 16 defining a recognition area and a recognition method is prepared in advance and supplied to the recognition unit 14. The recognition area defined in the template image 16 is shown as an area (recognition area) 15 in the figure.

[0048] The recognition unit 14 recognizes the character string by aligning the recognition area of the template image with the recognition area of the input image. Alignment involves aligning the position of the input image with that of the template image, and known alignment techniques can be used. Specifically, for example, feature points are extracted from the input image and the template image, and pairs of feature points in the input image and corresponding feature points in the template image are generated. Using these pairs of feature points, a transformation matrix is calculated to transform the coordinates of the input image into the coordinates of the template image.

[0049] When the recognition unit 14 recognizes the recognition area 12 in the input image 10 using the template image, the recognition result 22 is displayed on the confirmation screen 18. Simultaneously, the input image 10 itself is displayed side by side on the confirmation screen 18. The user can easily confirm the accuracy of the recognition result by visually recognizing the input image 20 and the recognition result 22 displayed side by side on the confirmation screen 18.

[0050] Figure 2 The following diagram schematically illustrates the positional alignment of input image 10 and template image 16. Pairs of feature points in recognition area 12 of input image 10 and feature points in recognition area 15 defined in template image 16 are generated. These feature point pairs are then used to calculate a transformation matrix that transforms the coordinates of input image 10 into those of template image 16. The transformation matrix is either an affine transformation matrix or a projective transformation matrix. This transformation matrix is then applied to input image 10, aligning the positions of input image 10 with those of template image 16, and recognizing the character string in recognition area 12 of input image 10.

[0051] Figure 3 The following diagram schematically illustrates how the position of input image 10 is aligned with the position of template image 16 using a transformation matrix. Input image 10 is translated parallely to align recognition area 12 of input image 10 with recognition area 15 of template image 16. This allows the character string "ABC" written in recognition area 12 to be correctly recognized.

[0052] On the other hand, when the input image 10 is tilted, in principle it is sufficient to rotate the input image 10, correct the tilt, and align it with the template image 16. However, simply rotating the input image 10 may cause a sense of discomfort to the user.

[0053] Specifically, when scanning and reading a form, if an error occurs and the input image 10 is tilted, the recognition result 22 displayed on the confirmation screen 18 is also displayed tilted, so the user feels uncomfortable and it becomes difficult to confirm and correct it compared to when the recognition result is upright.

[0054] Therefore, it is preferred that after the input image 10 is rotated, tilted, and aligned with the template image 16 for recognition, the recognition result 22 is displayed on the confirmation screen 18 in a manner aligned with the template image 16, that is, upright. However, other problems may arise in this case.

[0055] Figure 4 The diagram schematically shows a state where the input image 10 is rotated to correct its tilt and displayed upright so as to be aligned with the template image 16. A scanning range 17 of the input image 10 is indicated by a dashed line. Figure 5 The figure shows a state in which the input image 10 is rotated using a transformation matrix and corrected to be upright in the form of alignment with the template image 16. At this time, although the recognition area 12 of the input image 10 is upright, the scanning range 17 and the template image 16 do not overlap, so if only the image consisting of the area overlapping with the template image 16 is displayed upright in the form of alignment with the template image 16, there will be an area outside the area of the template image 16 despite being scanned. If the detached area is completely unrelated to the recognition area 12, it will have no effect on the recognition result 22. However, if there is information related to the recognition area 12 in the detached area, more specifically, if there is some associated information related to the character string of the recognition area 12, it will have a greater impact when confirming and correcting the recognition result 22. For example, if Figure 4 As shown, when related information related to the character string in the recognition area 12 such as "C error" is written in the separation area 10a, the user cannot correctly confirm and correct the recognition result due to the lack of such related information.

[0056] Furthermore, when scanning and reading a form or the like, if an error occurs and the input image 10 is tilted, or if a portion of the input image 10 is missing, other problems may also occur.

[0057] Figure 5The state in this case is schematically shown. Even if the input image 10 is rotated using a transformation matrix and corrected for upright position alignment with the template image, the portion 16a missing during scanning is not displayed. Therefore, the information intended for data input, i.e., the entire template image information, cannot be displayed based on the recognition results, resulting in a sense of dissonance (the display is a non-rectangular area that should be displayed as a rectangular area).

[0058] Thus, in this embodiment, in order to eliminate the sense of incongruity that may arise when the input image 10 is simply rotated and aligned with the template image 16 due to the loss of information associated with the recognition area 12 and the inability to display the missing part during scanning, the transformation result of the input image 10 using the transformation matrix is overlapped with the template image 16, and a circumscribed rectangular area circumscribed to the overlapping area is set, and the circumscribed rectangular area is displayed on the confirmation screen 18.

[0059] Figure 6 The transformation result of the input image 10 and the template image 16 are overlapped, and the circumscribed rectangular area 30 circumscribes the overlapped area. Figure 1 The input image 20 is displayed on the confirmation screen 18 in the image. The circumscribed rectangular area 30 includes the outlier area outside the template image 16 and also includes the portion 16a that was lost during scanning. Therefore, by displaying the circumscribed rectangular area 30 as the input image 20, the user can visually identify and confirm the associated information of the recognition area 12 while visually recognizing the input image 20 as a rectangular area, eliminating any sense of incongruity.

[0060] Attention, when paying attention Figure 6 When the portion 16a is missing during scanning, since the portion is not scanned, there is no image information, and even if the circumscribed rectangular area 30 is displayed as the input image 20 on the confirmation screen 18, the portion 16a is displayed as blank. Therefore, for the portion 16a, it is also possible to Figure 7 As shown, a portion corresponding to portion 16a is cut out from template image 16 and displayed as portion 16a of template image 16 instead of being displayed blank. Thus, portion 16a is displayed as part of template image 16 instead of being blank, further eliminating the user's sense of discomfort.

[0061] Here, the portion 16a may be displayed in a specific color, such as light gray or light blue, in order to emphasize that it is a part of the template image 16.

[0062] Next, this embodiment will be described in more detail.

[0063] <Structure>

[0064] Figure 8 The functional block diagram of the information processing device in this embodiment is shown. The information processing device includes a positioning unit 50 , a recognition unit 52 , a coordinate conversion unit 54 , a confirmation correction image generation unit 56 , and a display unit 58 .

[0065] The alignment unit 50 receives the input image 10 and the template image 16 as input and uses a transformation matrix to align the input image 10 with the template image 16. The alignment unit 50 generates a transformation matrix based on the input image 10 and the template image 16 and uses this transformation matrix to align the two images. Specifically, the alignment unit 50 extracts feature points from each of the input image 10 and the template image 16, generates pairs of feature points in the input image 10 and corresponding feature points in the template image 16, and calculates the transformation matrix using these pairs of feature points. Alternatively, this alignment process may involve transforming the coordinate system of the input image 10 into the coordinate system of the template image 16. U.S. Patent No. 6,711,293 describes a technique for projectively transforming the input image 10 to align it with the template image 16, which may also be used. During alignment, the alignment unit 50 also calculates the shift between the coordinate system of the template image 16 and the coordinate system of the confirmation correction image. The alignment unit 50 outputs the input image 10, transformed into the coordinate system of the template image 16, to the recognition unit 52. In addition, the amount of shift from the coordinate system of the template image 16 to the coordinate system of the confirmation correction image is output to the coordinate conversion unit 54 .

[0066] The recognition unit 52 performs character recognition on the input image 10 from the position alignment unit 50 using the recognition area and recognition method defined by the template image 16. The recognition area is provided as character frame coordinates in the coordinate system of the template image 16. If the recognition method is, for example, "recognition using a Romanization dictionary," the character string is recognized as Roman characters (letters) using a Romanization dictionary. The recognition unit 52 outputs the character frame coordinates to the coordinate conversion unit 54 and the recognition results (text data) to the display unit 58.

[0067] The coordinate conversion unit 54 uses the shift amount from the coordinate system of the template image 16 to the coordinate system of the confirmation correction image calculated by the position alignment unit 50 to convert the character frame coordinates input from the recognition unit 52 (which are the character frame coordinates of the template image 16) into the coordinate system of the confirmation correction image, and outputs the converted character frame coordinates to the display unit 58. The character frame is a frame that defines the recognition area 15 defined in the template image 16, and the character frame coordinates are the position coordinates of the frame.

[0068] The confirmation correction image generation unit 56 generates a confirmation correction image using the input image 10 and the template image 16 from the position alignment unit 50. That is, the confirmation correction image generation unit 56 overlaps the input image 10 and the template image 16 after the position alignment by the position alignment unit 50, and generates a circumscribed rectangular area of the overlapped area as a confirmation correction image. This confirmation correction image is equivalent to Figure 1 In addition, the correction image generation unit 56 confirms that the correction image generation unit 56 is necessary, such as Figure 7 As shown, a process is performed to interpolate the missing portion 16a with a portion of the template image 16. The technique of interpolating or synthesizing a portion of an image with another image is well known. The confirmation correction image generation unit 56 outputs the generated confirmation correction image to the display unit 58.

[0069] The display unit 58 corresponds to Figure 1 The confirmation screen 18 in , displays the recognition result from the recognition unit 52 , the character frame coordinates from the coordinate conversion unit 54 , and the confirmation correction image from the confirmation correction image generation unit 56 .

[0070] Figure 9 The information processing device is a block diagram showing the configuration of the information processing device in this embodiment. The information processing device is composed of a computer and includes a processor 60 , a ROM 62 , a RAM 64 , an input unit 66 , an output unit 68 , and a storage unit 70 .

[0071] The processor 60 reads the processing program stored in the ROM 62 or other program memory and executes the processing program using the RAM 64 as a working memory, thereby realizing Figure 8 The positioning unit 50, the recognition unit 52, the coordinate conversion unit 54 and the confirmation correction image generation unit 56 in the processor 60 are listed as follows.

[0072] ·Calculation and processing of transformation matrix

[0073] Position alignment processing of the input image 10 using the transformation matrix

[0074] Recognition processing of input image 10

[0075] Character frame coordinate transformation

[0076] Confirm the correction image generation process

[0077] Confirm the display processing of the correction image, recognition results and character frame coordinates

[0078] The processor 60 refers to a processor in a broad sense, including general-purpose processors (such as a CPU (Central Processing Unit), etc.) and specialized processors (such as a GPU (Graphics Processing Unit), an ASIC (Application Specific Integrated Circuit), an FPGA (Field Programmable Gate Array), and programmable logic devices). Furthermore, the actions of a processor can be implemented not only by a single processor but also by the collaboration of multiple processors located in physically separate locations.

[0079] The input unit 66 is composed of a keyboard, mouse, scanner, touch panel, communication interface, etc., and inputs an input image 10 such as a form and a template image 16. The template image 16 may be included in advance as part of the program or may be stored in the storage unit 70. Furthermore, the user confirms the recognition results using a confirmation correction image. If correction is determined to be necessary, the user operates the input unit 66 to correct the recognition results.

[0080] The output unit 68 is composed of a display device including a touch panel, and displays the confirmation correction image, recognition result, and character frame generated by the processor 60. Figure 8 The display unit 58 in the image processing apparatus can be configured such that the output unit 68 includes a communication interface, and the confirmation and correction image, the recognition result, and the character frame are output to an external display device via the Internet for display. This allows the user to confirm and correct the recognition result on a screen of a tablet terminal or PC, etc., physically remote from the information processing apparatus.

[0081] The storage unit 70 stores the input image 10 , template images, recognition results, corrected recognition results, and the like.

[0082] <Processing Flowchart>

[0083] Figure 10 A flowchart showing the processing of this embodiment.

[0084] The processor 60 inputs a document, such as a form, to be recognized as an input image 10 (S101). Specifically, for example, a form placed on an automatic document feeder is read by a scanner and input as digital image data. Alternatively, the form can be read by a scanner of a multifunction peripheral. Alternatively, digital image data already read by a scanner and stored in the storage unit 70 can be read from the storage unit 70. The user can specify the digital image data to be recognized from a plurality of digital image data sets by operating a keyboard, for example.

[0085] Next, the processor 60 inputs the template image 16 (S102). A template image 16 is prepared in advance for each input image 10. If the input image 10 is a form, since the format of the form is known in advance, a template image 16 corresponding to the position and character type of the entry column of the recognition object is prepared. The processor 60 can also identify the type of the input image 10 and select the template image 16 corresponding to the identified type. Alternatively, the user can manually select the template image 16. The template image 16 is embedded in the program or stored in the storage unit 70. The processor 60 reads the template image 16 at a predetermined timing after the input image 10 is input.

[0086] Next, the processor 60 calculates a transformation matrix for aligning the position of the input image 10 to the template image 16 based on the input image 10 and the template image 16 (S103). The transformation matrix generates pairs of feature points of the recognition area 12 in the input image 10 and feature points of the recognition area 15 defined by the template image 16, and uses these pairs of feature points for calculation. The transformation matrix is an affine transformation matrix or a projective transformation matrix, but is not limited thereto. In addition, the positional offset between the input image 10 and the template image 16 can also be measured. If the positional offset is less than a predetermined threshold, it is considered that it does not interfere with the recognition process, and the process of calculating the transformation matrix is omitted. In addition, if the transformation matrix has already been calculated and stored in the storage unit 70, etc., and the transformation matrix can be directly referenced in the positional alignment of the input image 10 and the template image 16, the process of calculating the transformation matrix can also be omitted.

[0087] Next, processor 60 uses the transformation matrix to align the input image 10 with the template image 16 (S104). This alignment allows the positions of recognition area 12 of input image 10 and recognition area 15 of template image 16 to coincide. This correspondence may be within the range necessary to ensure recognition accuracy, and does not necessarily mean perfect correspondence.

[0088] Next, the processor 60 identifies the character string of the recognition area 12 of the input image 10 after the position alignment according to the recognition area 15 and the recognition method defined in the template image 16 (S105). The character string recognition process can use any well-known technology, for example, the method described in Japanese Patent Laid-Open No. 5-40853 can be used, but it is not limited to this. In addition, in this embodiment, Roman character recognition using a Roman character dictionary is illustrated, but of course it can also be number, hiragana, katakana, kanji and other character string recognition. In addition, the processor 60 transforms the character frame coordinates (S106). That is, the character frame coordinates that define the recognition area 15 defined in the template image 16 (if the recognition area 15 is a rectangle, for example, the position coordinates of the four vertices, or the dimensions of the two sides perpendicular to the position coordinates of the upper left) are transformed into a coordinate system for confirming the corrected image (S106). This transformation will be described in more detail later.

[0089] Next, the processor 60 generates a confirmation correction image (S107). The confirmation correction image is generated based on the aligned input image 10 and template image 16. The processor 60 overlaps the aligned input image 10 and template image 16 and generates the confirmation correction image as a circumscribed rectangle of the overlapped area. Figure 6 The circumscribed rectangular area 30 shown corresponds to the confirmation correction image.

[0090] Next, the processor 60 outputs the confirmation correction image generated in S107, the transformed character frame coordinates generated in S106, and the recognition result in S105 to the output unit 68 for display (S108). The display format is arbitrary, but it is preferable to display the confirmation correction image and the recognition result in a manner that allows identification of their correspondence. For example, Figure 1 As shown, a confirmation correction image is displayed on the left side of the screen and the character frame coordinates are displayed superimposed on it, and the recognition result is displayed on the right side of the screen in a comparative manner. The confirmation correction image can also be annotated with "input image" and a label and displayed, and the recognition result can be annotated with "recognition result of input image" and a label and displayed. Alternatively, a mark such as an arrow can be added between the confirmation correction image and the recognition result. Of course, the confirmation correction image can also be displayed on the upper part and the recognition result can be displayed on the lower part. Alternatively, it can be configured so that when the confirmation correction image is clicked, its recognition result is displayed. Since the entire area of the scanned input image 10 is included in the confirmation correction image, even if it includes information associated with the recognition area 12, the information can be reliably displayed and the user can refer to it appropriately. In addition, since the confirmation correction image is rectangular, the user will not feel a sense of disobedience and can compare and confirm the confirmation correction image and the recognition result.

[0091] The processor 60 then determines whether the recognition result needs to be corrected (S109). Specifically, the user visually confirms the correction image. If correction is required ("Yes" in S109), a correction instruction is input from the input unit 66. The processor 60 receives the correction instruction from the user and corrects the recognition result (S110). For example, if the recognition result is "ABC" but the user determines that it is "ABD" by mistake, an instruction to correct "C" to "D" is input from the input unit 66. The processor 60 corrects the recognition result to "ABD" in accordance with the instruction and displays it.

[0092] After the user's confirmation and correction are completed, the processor 60 stores the recognition result in the storage unit 70 according to the instruction from the user, or outputs it to another computer (including a server computer) via the output unit 68 .

[0093] Figure 11A 、 Figure 11B as well as Figure 11C The coordinate system conversion process performed by the processor 60 is schematically shown.

[0094] Figure 11A The coordinate system of the input image 10 is shown. The coordinate system of the input image 10 is defined as an xy coordinate system with the upper left corner of the input image 10 as the origin, the upper side of the circumscribed rectangle of the input image 10 as the x-axis, and the left side of the circumscribed rectangle as the y-axis. When the input image 10 is tilted, the recognition area 12 in the input image 10 is tilted accordingly.

[0095] Figure 11B This represents the coordinate system of template image 16. This is defined as an XY coordinate system with the upper left corner of template image 16 as the origin, the upper edge of template image 16 as the X-axis, and the left side of template image 16 as the Y-axis. Recognition area 15 in template image 16 is in an upright position. The frame of recognition area 15 is the character frame, and the coordinates of this character frame are the character frame coordinates.

[0096] exist Figure 10 In S104, the input image 10 is aligned with the template image 16. That is, the xy coordinate system of the input image 10 is transformed into the XY coordinate system of the template image 16. As a result, the input image 10 is placed on the XY coordinate system of the template image 16.

[0097] Then, in Figure 10In S107, a circumscribed rectangle of the overlap of input image 10 and template image 16 is set in the XY coordinate system of template image 16. A UV coordinate system is assumed, with the upper left position of this circumscribed rectangle as the origin, and is parallel to the XY coordinate system of template image 16. Specifically, the U axis and V axis are parallel to the X axis and Y axis, respectively. This UV coordinate system serves as the coordinate system for confirming the corrected image.

[0098] Figure 11C Indicates confirmation of the UV coordinate system of the corrected image. Figure 11B The character frame coordinates shown are the position coordinates in the XY coordinate system of the template image 16, so they need to be transformed from the XY coordinate system to the UV coordinate system. As mentioned above, the U axis and the V axis are parallel to the X axis and the Y axis respectively, so the transformation from the XY coordinate system to the UV axis coordinate system is just a parallel movement, which is provided by the shift amount. Figure 10 In S106, the shift amount is calculated and the character frame coordinates are transformed.

[0099] Thus, in this embodiment, the input image 10 is aligned with the template image 16, and a circumscribed rectangular area of the image obtained by overlapping the aligned input image 10 and the template image is generated as a confirmation correction image, and character frame coordinates are generated. The confirmation correction image, character frame coordinates and recognition results are displayed together on the confirmation screen 18, thereby being able to display an image without deformation (without rotation, etc.), and being able to display related information recorded on the end of the input image 10, and furthermore, by setting the confirmation correction image as a rectangle, the user's sense of disharmony can be eliminated.

[0100] <Variation 1>

[0101] In an embodiment, the recognition unit 52 recognizes a character string from the input image 10 after position alignment by the position alignment unit 50. However, the input image 10 after position alignment by the position alignment unit 50 is provided to the confirmation correction image generation unit 56, and a confirmation correction image is generated by the confirmation correction image generation unit 56, so the recognition unit 52 can use the confirmation correction image to recognize the character string.

[0102] Figure 12 A functional block diagram of an information processing device in this modification is shown.

[0103] The position alignment unit 50 receives the input image 10 and the template image 16 as input and uses a transformation matrix to positionally align the input image 10 with the template image 16. The position alignment unit 50 generates a transformation matrix based on the input image 10 and the template image 16 and uses this generated transformation matrix to positionally align the two images. This position alignment process may also involve transforming the coordinate system of the input image 10 into the coordinate system of the template image 16. During the position alignment process, the position alignment unit 50 also calculates the amount of displacement between the coordinate system of the template image 16 and the coordinate system of the confirmation correction image.

[0104] The position alignment unit 50 outputs the input image 10 transformed into the coordinate system of the template image 16 to the confirmation correction image generation unit 56. Shift amount Output to the coordinate conversion unit 54.

[0105] The coordinate transformation unit 54 uses the shift amount from the coordinate system of the template image 16 to the coordinate system of the confirmation correction image calculated by the position alignment unit 50 to transform the character frame coordinates (which are the character frame coordinates of the template image 16) into the coordinate system of the confirmation correction image and output it to the recognition unit 52.

[0106] The confirmation correction image generation unit 56 generates a confirmation correction image using the input image 10 and the template image 16 from the position alignment unit 50. Specifically, the confirmation correction image generation unit 56 overlaps the input image 10 and the template image 16 aligned by the position alignment unit 50, and generates a circumscribed rectangular area of the overlapped area as the confirmation correction image. Figure 7 As shown, a process is executed to interpolate the missing portion 16 a using a portion of the template image 16 . The confirmation correction image generation unit 56 outputs the generated confirmation correction image to the recognition unit 52 and the display unit 58 .

[0107] The recognition unit 52 performs character recognition on the confirmation and correction image from the confirmation and correction image generation unit 56 using the recognition area and recognition method defined in the template image 16. The recognition area is provided as character frame coordinates from the coordinate conversion unit 54. The recognition unit 52 outputs the recognition result (text data) and the character frame coordinates to the display unit 58.

[0108] The display unit 58 corresponds to Figure 1 The confirmation screen 18 in , displays the recognition result and character frame coordinates from the recognition unit 52 , and the confirmation correction image from the confirmation correction image generation unit 56 .

[0109] In this modification, the character frame coordinates are output from the recognition unit 52 to the display unit 58 , but may be output from the coordinate conversion unit 54 to the display unit 58 .

[0110] <Variation 2>

[0111] In this embodiment, the input image 10 and the template image 16 after position alignment are overlapped, and the circumscribed rectangle of the overlap is set as the confirmation correction image, but an area of any shape containing the input image 10 and the template image 16 after position alignment can also be used as the confirmation correction image, or a rectangular area containing the input image 10 and the template image 16 after position alignment can also be used as the confirmation correction image.

[0112] Figure 13 An example of a confirmation correction image in this case is shown. The transformation result of the input image 10 and the rectangular area 32 containing the template image 16 are shown. The rectangular area 32 is used as the confirmation correction image. It should be noted that Figure 6 Differently, the rectangular area 32 does not circumscribe the input image 10 and the template image 16 .

[0113] In this case, you can also Figure 7 As shown, the portion 16a is highlighted as being a part of the template image 16. For example, it is displayed in light gray or light blue.

[0114] The user can select any of the following:

[0115] (1) An area of arbitrary shape including the aligned input image 10 and template image 16

[0116] (2) A rectangular area containing the aligned input image 10 and template image 16

[0117] (3) The circumscribed rectangular area of the aligned input image 10 and template image 16

[0118] Furthermore, it is also possible to configure the default state to be (3), and the user can change it to (1) or (2) as needed.

Claims

1. An information processing device comprising: an input unit that receives an input image to be recognized; and processor, The processor performs the following processing by executing the program: In order to make the recognition area in the input image consistent with the recognition area defined in the template image, the input image is aligned with the template image; executing recognition processing of the recognition area of the input image after position alignment; generating a confirmation image including the input image aligned with the template image; as well as displaying the confirmation image and the result of the recognition process in such a manner that the correspondence between them can be identified; Furthermore, the processor generates a circumscribed rectangular image of the input image after position alignment with the template image as the confirmation image, and annotates information indicating that it is the template image in the area of the circumscribed rectangular image where the input image after position alignment does not exist and only the template image exists.

2. The information processing device according to claim 1, wherein The processor includes information that exists in the input image after position alignment but does not exist in the template image in the confirmation image and outputs the information.

3. The information processing device according to claim 1, wherein The template image defines a recognition area in the input image and defines a character string to be recognized.

4. The information processing device according to claim 1, wherein: The input image is a scanned image obtained by scanning a document in a tilted state.

5. The information processing apparatus according to claim 4, wherein: The document is a form.

6. A computer-readable medium storing a program for causing a computer to execute a process comprising the following steps: In order to make the recognition area in the input image consistent with the recognition area defined in the template image, the input image is aligned with the template image; executing recognition processing of the recognition area of the input image after position alignment; generating a confirmation image including the input image aligned with the template image; displaying the confirmation image and the result of the recognition processing in a manner that enables identification of a correspondence between them; as well as A circumscribed rectangular image of the input image aligned with the template image is generated as the confirmation image, and information indicating that it is the template image is marked in the area of the circumscribed rectangular image where the input image after alignment does not exist but only the template image exists.

7. An information processing method comprising the following steps: In order to make the recognition area in the input image consistent with the recognition area defined in the template image, the input image is aligned with the template image; executing recognition processing of the recognition area of the input image after position alignment; generating a confirmation image including the input image aligned with the template image; displaying the confirmation image and the result of the recognition processing in a manner that enables identification of a correspondence between them; as well as A circumscribed rectangular image of the input image aligned with the template image is generated as the confirmation image, and information indicating that it is the template image is marked in the area of the circumscribed rectangular image where the input image after alignment does not exist but only the template image exists.

Images