Image processing device and program

Abstract

To provide an image processing apparatus which processes an image of a document in a fixed format, and a program.SOLUTION: In an image processing system, an image processing unit includes: a target image acquisition unit which acquires an image to be processed, which is a captured image of a document in a fixed format; a template image acquisition unit which acquires a template image of the fixed format; a feature point pair extraction unit which extracts multiple pairs of feature points having similar image information between the image to be processed and the template image, from the image to be processed and the template image; and an image transformation unit which performs projective transformation on the image to be processed on the basis of positions of the extracted pairs of feature points, to form a front image which is to be obtained when the document is seen from the front. In the fixed format, a common area is defined where common information is to be shared with a plurality of documents. The template image does not include image information of areas other than the common area, and includes at least a part of a plurality of character images in the common area.SELECTED DRAWING: Figure 2

Description

[Technical field]

[0001] The present invention relates to an image processing apparatus and a program. [Background technology]

[0002] Patent Document 1 describes a method of calculating a projective transformation matrix expressing the relative positional relationship between local feature point information of a product image pre-registered in a dictionary and local feature point information extracted from an input image based on the correspondence between the two, and performing projective transformation. Patent Document 2 describes a method of recognizing characters detected in an image and determining the font. [Prior art document] [Patent documents] [Patent Document 1] JP 2017-187971 A [Patent Document 2] JP 2013-188935 A Summary of the Invention

[0003] In a first aspect of the present invention, an image processing device is provided. The image processing device includes a target image acquisition unit that acquires a processing target image, which is an image of a photographed document in a fixed format. The image processing device includes a template image acquisition unit that acquires a template image in the fixed format. The image processing device includes a feature point pair extraction unit that extracts a plurality of feature point pairs having similar image information between the processing target image and the template image from the processing target image and the template image. The image processing device includes an image conversion unit that performs projective conversion of the processing target image into a front image that should be obtained when the document is viewed from the front, based on the positions of the plurality of feature point pairs extracted by the feature point pair extraction unit. The fixed format defines a common area that is an area that should have common information among a plurality of documents. The template image does not include image information of areas other than the common area, and includes at least some character images among a plurality of character images in the common area.

[0004] The template image may include at least a part of the image of the frame line included in the common area.

[0005] The feature points pair extraction unit may extract a plurality of feature points pairs by extracting feature points having similar image information from the processing target image for each of a plurality of feature points set in each of a plurality of image regions included in the template image. The image processing device may include a region selection unit that counts, for each of the plurality of image regions, the number of feature points having similar image information extracted from the processing target image for the plurality of feature points set in each image region, and selects a portion of image regions to be used for projective transformation from among the plurality of image regions based on the number of feature points counted for each of the plurality of image regions. The image transformation unit may projectively transform the processing target image into a front image based on positions of the plurality of feature points pairs extracted for the plurality of feature points set in the portion of image regions selected by the region selection unit.

[0006] The region selection section may preferentially select, from among the plurality of image regions, an image region for which a greater number of feature points have been counted as an image region to be used in projective transformation.

[0007] The feature points pair extraction unit may extract feature points from the image to be processed that have similar image information to each of the multiple feature points set in each of the multiple image regions, and may preferentially select, from among the multiple feature points extracted from the image to be processed, some feature points that are closer in positional relationship to each other than the feature points extracted as feature points pairs, as feature points that constitute the multiple feature points pairs.

[0008] A storage unit may be provided for storing the template image.

[0009] The fixed format document may be a document including a photographic image of a person at a predetermined position. The image processing device may include an identification unit that identifies an area including a photographic image of a person from the processing target image using a trained model that has been machine-learned using the photographic image of the person included in the fixed format document as training data. The image conversion unit may perform projective conversion of the processing target image into a front image based on the position of the area including the photographic image identified by the identification unit.

[0010] The standard format document may be a driver's license for an automobile or motorbike, a passport, or a health insurance card.

[0011] The image processing device may further include a character image extraction unit that extracts a character image from the front image, and a character processing unit that outputs information indicating the difference between the character image extracted by the character image extraction unit and an image of a reference character having a predetermined character shape.

[0012] The character processing unit may include a processing target character acquisition unit that acquires an image of a character extracted by the character image extraction unit as an image of a processing target character. The character processing unit may include a storage unit that stores a trained model that is generated by machine learning using an image of a reference character and images of a plurality of characters having mutually different character shapes, and generates an image of a character adapted to a predetermined character shape from an input image of a character. The character processing unit may include a character image generation unit that uses the trained model to generate an image of a processing target character adapted to a predetermined character shape from an image of a processing target character. The character processing unit may include a difference information output unit that outputs information indicating a difference between the processing target character and the reference character based on a result of comparing the image generated by the character image generation unit with the image of the reference character.

[0013] In a second aspect, a program is provided, the program causing a computer to function as the image processing device described above.

[0014] The above summary of the invention does not list all of the necessary features of the present invention. Also, subcombinations of these features may also be inventions. [Brief description of the drawings]

[0015] [Figure 1] 1 shows a schematic diagram of an overall configuration of an image processing system 10 according to an embodiment. [Diagram 2] 2 shows functional blocks of the image processing unit 40. [Diagram 3]11 is a diagram for explaining an image cutout process performed by an image cutout unit 43. [Figure 4] A standard format 50 of a driver's license is shown diagrammatically together with an example of a driver's license 52 . [Diagram 5] A template image 60 used by the feature points pair extraction unit 44 is shown diagrammatically. [Figure 6] 13A and 13B show schematic diagrams of the results of feature point matching processing by the feature point pair extraction unit 44. [Figure 7] 11 is a table for explaining a process in which the area selection unit 46 selects an area. [Figure 8] In the template image 60, an image region 90 is shown which is to contain a photographic image. [Figure 9] 13 is a diagram for explaining a process in which projective transformation is performed further using the position of a photographic image of a person. FIG. [Figure 10] 2 shows an outline of the flow of processing executed by the image processing system 10. [Figure 11] Functional blocks of a character processing unit 200, a learning unit 202, and a character analysis unit 280 are shown. [Figure 12] The configuration of the learning data is shown below. [Figure 13] 2 shows a conceptual configuration of a learning device that executes machine learning in the model generation unit 206. [Figure 14] 13 shows an example of difference information output by difference information output section 240. [Figure 15] Character image 620-1, character image 630-1, character image 640-1, and comparative image 650-1 are shown enlarged. [Figure 16] As a reference example, a state in which a character image 640-1 is superimposed on a character image 620-1 is shown. [Figure 17] 2 shows an example of a data structure of peculiar character information 160. [Figure 18] 2 shows an example of a computer 2000 according to the present embodiment. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0016] The present invention will be described below through the embodiments of the invention, but the following embodiments do not limit the invention according to the claims. In addition, not all of the combinations of features described in the embodiments are necessarily essential to the solution of the invention. In the drawings, the same reference numbers are used for the same or similar parts, and duplicate explanations may be omitted.

[0017] 1 shows a schematic diagram of an overall configuration of an image processing system 10 according to an embodiment. The image processing system 10 includes an image processing device 12, a learning device 202, a character analysis device 280, a storage device 290, and a display device 88. The image processing device 12 includes an image processing unit 40 and a character processing unit 200. The storage device 290 is a storage unit that stores font data 100, a model 120, and peculiar character information 160.

[0018] The processing target image 20 and the analysis target image 30 are images of photographed standard format documents. The processing target image 20 and the analysis target image 30 are images of identity verification documents. As an example, the processing target image 20 and the analysis target image 30 are images generated by photographing cards such as a driver's license for an automobile or motorcycle, or a health insurance card. The processing target image 20 is an image to be processed by the image processing unit 40 and the character processing unit 200. In this embodiment, a driver's license will be described as an example of a standard format document.

[0019] The image processing unit 40 has a function of extracting characters used on a driver's license from the processing target image 20. The processing target image 20 is, for example, an image of the driver's license taken by the owner of the driver's license using a camera function of a smartphone or the like. Therefore, the processing target image 20 may contain background noise such as the pattern of the desk or floor on which the driver's license is placed. In addition, the processing target image 20 may contain distortion of the image of the driver's license due to the driver's license not being photographed from the front. In addition, the processing target image 20 may contain the shadow of the photographer, changes in shading due to a light source, and the like. Therefore, if an image of characters is directly extracted from the processing target image 20, there is a possibility that the characters used on the driver's license cannot be extracted in their exact shape.

[0020] Therefore, the image processing unit 40 performs feature point matching between the processing target image 20 and the template image, and calculates a projective transformation matrix for converting the processing target image 20 into a front image of the driver's license photographed from the front. The image processing unit 40 generates a front image of the processing target image 20 by projectively transforming the processing target image 20 using the calculated projective transformation matrix. The image processing unit 40 extracts an image of characters used on the driver's license from the front image of the processing target image 20 generated by projective transformation. For example, the image processing unit 40 extracts an image of the portion of the driver's license where the name of the owner is printed and characters of the portion of the driver's license where the address of the owner is printed.

[0021] The template image used by the image processing unit 40 includes only images of the parts that are necessarily included in the standard format of a driver's license. For example, the template image does not include images of the parts where personal information such as the name and address of the owner of the driver's license is printed, but includes images of the part where the character "name" is printed in the name column and the part where the character "address" is printed in the address column. The template image also includes images of the parts where the characters "drive", "transfer", "license", "permission" and "certificate" of the "driver's license" are printed, for example. Note that the characters of the year printed in the date of birth column of the driver's license and the part where the driver's license number is printed may use characters in different fonts or different character styles depending on the issuer. It is desirable that the template image does not include images of characters in parts where different fonts or different character styles may be used depending on the issuer. In this way, since the template image includes only images of the parts that are necessarily included in the standard format of a driver's license, an accurate projective transformation matrix can be calculated. This makes it possible to extract an image that accurately represents the shapes of characters used in the name and address fields of a driver's license.

[0022] The image of the characters extracted by the image processing unit 40 is input to the character processing unit 200. The character processing unit 200 has a function of generating information indicating characters having characteristic shapes among the characters used on the driver's license. An overview of the function of the character processing unit 200 will be described.

[0023] The learning device 202 generates a trained model 120 that adapts the character shape of an input character to that of a reference font through machine learning using character images of multiple fonts included in the font data 100. As an example, the model 120 is a neural network model for adapting decorative character shape elements such as "stops," "strokes," and "stroke strokes" to character shape elements of a reference font. The process by which the learning device 202 generates the model 120 will be described later.

[0024] The character processing unit 200 uses the model 120 to adapt the characters extracted from the processing target image 20 to a reference character shape, and then generates information indicating which characters among the characters included in the processing target image 20 have a distinctive character shape, and records the information in the peculiar character information 160. The information generated by the character processing unit 200 is displayed on the display device 88, and the judge 80 may finally judge whether or not the character has a distinctive character shape. In this embodiment, a character with a distinctive character shape may be called a "peculiar character".

[0025] The processing target image 20 and the analysis target image 30 may be images of driver's licenses issued by the same issuing agency. As an example, the processing target image 20 and the analysis target image 30 may be images of driver's licenses n issued by the same issuer. As described above, driver's licenses may be created using fonts unique to each issuer. The character analysis device 280 analyzes the analysis target image 30 using information from the specific character information 160, and therefore can recognize characters with characteristic character shapes used in creating driver's licenses and appropriately analyze the characters included in the analysis target image 30. The functions of the character processing unit 200 will be described in relation to FIG. 9 to FIG. 17 and the like.

[0026] 2 shows functional blocks of the image processing unit 40. The image processing unit 40 includes a template image acquisition unit 41, a target image acquisition unit 42, an image clipping unit 43, a feature point pair extraction unit 44, a specification unit 45, a region selection unit 46, an image conversion unit 47, a character image extraction unit 48, and a storage unit 49.

[0027] The image processing unit 40 is realized by a computer. The character processing unit 200, the learning device 202, and the character analysis device 280 may be realized by any number of computers greater than or equal to one. The storage unit 49 may be realized by a non-volatile storage medium or a volatile storage medium. The storage unit 49 may be realized by an external storage medium accessible via a communication line such as the Internet.

[0028] The target image acquisition unit 42 acquires a processing target image, which is an image of a photographed document in a fixed format. In this embodiment, the document in the fixed format is a driver's license.

[0029] The template image acquiring unit 41 acquires a template image in a standard format. Specifically, the storage unit 49 stores the template image, and the template image acquiring unit 41 acquires the template image from the storage unit 49.

[0030] The fixed format defines a common area that is an area that should have common information among a plurality of documents. The template image does not include image information of areas other than the common area, and includes at least some of the character images among the plurality of character images in the common area. The template image may include at least some of the images of the frame lines included in the common area.

[0031] The feature points pair extraction unit 44 extracts a plurality of feature points pairs having similar image information between the processing target image and the template image from the processing target image and the template image. For example, the feature points pair extraction unit 44 extracts local features from each region of the processing target image, selects regions from which local features similar to those extracted from pre-set feature points set in the template image are extracted as feature points, and extracts the selected feature points and the feature points set in the template image as feature points pairs. The image conversion unit 47 performs projective transformation of the processing target image into a front image that should be obtained when the document is viewed from the front, based on the positions of the plurality of feature points pairs extracted by the feature points pair extraction unit 44.

[0032] The feature point pair extraction unit 44 extracts a plurality of feature point pairs by extracting feature points having similar image information from the processing target image for each of a plurality of feature points set in each of a plurality of image regions included in the template image. The region selection unit 46 counts the number of feature points having similar image information extracted from the processing target image for each of the plurality of image regions for each of the plurality of image regions, and selects a portion of the image regions to be used for projective transformation from among the plurality of image regions based on the number of feature points counted for each of the plurality of image regions. For example, the region selection unit 46 preferentially selects an image region having a larger number of feature points counted for each of the plurality of image regions as an image region to be used for projective transformation from among the plurality of image regions. The image transformation unit 47 performs projective transformation of the processing target image into a front image based on the positions of the plurality of feature point pairs extracted for the plurality of feature points set in the portion of the image regions selected by the region selection unit 46.

[0033] For example, the image conversion unit 47 calculates a projection transformation matrix for projectively transforming the processing target image into a front image based on the positions on the image of the feature points that make up the feature point pair, and generates the front image by applying the calculated projection transformation matrix to the processing target image. For example, when the pixel position after the projection transformation is (x1, y1) and the pixel position before the projection transformation is (x2, y2), the image conversion unit 47 calculates the projection transformation matrix H expressed by the following formula (1).

number

[0034] For example, the image conversion unit 47 defines the position of the feature point set in the template image as (x1, y1), and the position of the feature point extracted from the processing target image 20 as (x2, y2), and calculates parameters h11 to h33 of the projection transformation matrix H by numerical calculation using the least squares method or the like.

[0035] The feature points pair extraction unit 44 may extract feature points from the processing target image that have similar image information to each of the multiple feature points set in each of the multiple image regions, and may preferentially select, from the multiple feature points extracted from the processing target image, some feature points that are closer in positional relationship to each other as feature points that constitute the multiple feature points pairs.

[0036] The standard format document may be a document including a photographic image of a person. For example, the standard format document may be a driver's license or a passport. When the standard format document is a document including a photographic image of a person, the identification unit 45 identifies an area including a photographic image of the person from the processing target image using a trained model that has been machine-learned using the photographic image of the person included in the driver's license as training data. The image conversion unit 47 performs projective conversion of the processing target image into a frontal image based on the position of the area including the photographic image identified by the identification unit 45.

[0037] The character image extraction unit 48 extracts a character image from the front image generated by the image conversion unit 47. The character processing unit 200 outputs information indicating a difference between the image of the character extracted by the character image extraction unit 48 and an image of a reference character having a predetermined character shape.

[0038] FIG. 3 is a diagram for explaining the image cutout process by the image cutout unit 43. The processing target image 20 includes a background image 21 and an image 22 in which the driver's license is captured. The background image 21 is, for example, an image of a part of the image of a desk, floor, etc. behind the driver's license. The image cutout unit 43 identifies the part in which the driver's license is captured by performing object recognition from the processing target image 20. The image cutout unit 43 may identify the part in which the driver's license is captured by detecting, from the processing target image 20, a characteristic image feature that the image of the driver's license should have, and identifying a frame 23 that surrounds the part in which the characteristic image feature of the driver's license is detected. The image cutout unit 43 cuts out the image 22 of the part in which the driver's license is captured from the processing target image 20. The image 22 cut out by the image cutout unit 43 is supplied to the feature point pair extraction unit 44 and the image conversion unit 47. This makes it possible to prevent background noise from being included in the image processed by the feature points pair extraction unit 44 and the image conversion unit 47.

[0039] If the driver's license is not photographed from the front, distortion from a rectangular shape may occur in the driver's license image 22. The feature point pair extraction unit 44, the area selection unit 46, and the image transformation unit 47 perform projective transformation on the image 22 to virtually generate a front image that would be obtained if the driver's license were photographed from the front.

[0040] FIG. 4 shows a standard format 50 for a driver's license, along with an example of a driver's license 52. The standard format 50 includes the characters "Name" in the name field, the characters "Year Month Day Birth" in the date of birth field, the characters "Address" in the address field, the characters "License conditions, etc." in the driver's license conditions field, and the characters "Driver's License". The information included in the standard format 50 is common to all driver's licenses. Note that the dotted lines in FIG. 4 are illustrated for the purpose of clearly showing the outer frame of the driver's license, and are not included in the standard format as printed information.

[0041] In addition to the common information, the driver's license 52 includes information such as the name "Taro XX", the date of birth "Heisei 21 1", the address "1-1-1 XX City, XX Prefecture", etc. Since this information is printed by the issuer of the driver's license, the typeface and font may differ depending on the issuer.

[0042] Fig. 5 is a schematic diagram of a template image 60 used by the feature points pair extraction unit 44. The dotted lines in Fig. 5 are illustrated for the purpose of clearly showing the outer frame of the driver's license, similar to the dotted lines in Fig. 4, and are not included as image information of the template image 60.

[0043] The template image 60 is an image consisting of an image area 61 including the characters "Name" in the name field, an image area 62 including the characters "Address" in the address field, an image area 63 including the character "Un", an image area 64 including the character "Ten", an image area 65 including the character "Men", an image area 66 including the character "Kou", and an image area 67 including the character "Shou". The image of the image area 62 includes an image of a frame 68 surrounding the characters of "Name". The image of the image area 63 includes an image of a frame 69 surrounding the characters of "Address". Note that the template image 60 may include only images of characters without including the images of the frame 68 and the frame 69.

[0044] 5, the template image 60 includes image information of information included in the fixed format 50, but does not include image information of information not included in the fixed format 50. Therefore, the template image 60 is composed of image information of information that should be common to all driver's licenses, regardless of the issuer of the driver's license. This enables the feature points pair extraction unit 44 to extract feature points from the processing target image 20 more accurately.

[0045] FIG. 6 is a schematic diagram showing a result of the feature point matching process performed by the feature point pair extraction unit 44. The feature point pair extraction unit 44 sets a plurality of feature points in an image region 61, an image region 62, an image region 63, an image region 64, an image region 65, an image region 66, and an image region 67 of a template image 60. Note that FIG. 6 shows a case where two feature points are set in each of the image regions 61 to 67 in order to clearly show the correspondence between the feature points. The number of feature points set in each of the image regions 61 to 67 is not limited to two. The number of feature points set in each region may be three or more. The number of feature points set in each region may be one or more.

[0046] As an example, a plurality of feature points including feature point 71A and feature point 72B are set in image region 61. The feature point pair extraction unit 44 extracts, by feature point matching, feature point 71a having image information similar to feature point 71A and feature point 71b having image information similar to feature point 72B from image 22. The feature point pair extraction unit 44 extracts feature point 71A and feature point 71a as one feature point pair, and extracts feature point 71B and feature point 71b as one feature point pair.

[0047] Also, as an example, a plurality of feature points including feature point 73A and feature point 73B are set in image region 63. The feature point pair extraction unit 44 extracts, by feature point matching, feature point 73a having image information similar to feature point 73A and feature point 73b having image information similar to feature point 73B from image 22. The feature point pair extraction unit 44 extracts feature point 73A and feature point 73a as one feature point pair, and extracts feature point 73B and feature point 73b as one feature point pair.

[0048] In addition, when feature point 73c is extracted as a feature point having image information similar to feature point 73B in addition to feature point 73b, the feature points pair extraction unit 44 excludes feature point 73c without selecting it as a feature points pair based on the positional relationship between the other feature points set in the template image 60 and the corresponding feature points extracted from the image 22. For example, the line segment connecting feature point 73B and feature point 73c has a significantly different direction from the line segment connecting the other feature points set in the template image 60 and the feature points extracted from the image 22. Therefore, the feature points pair extraction unit 44 determines that the positional relationship between feature point 73B and feature point 73c is different from the positional relationship between the other feature points, and excludes feature point 73B and feature point 73c as a feature points pair of feature point 73B. As a result, the feature points pair extraction unit 44 extracts only feature point 73B and feature point 73b as a feature points pair of feature point 73B.

[0049] Fig. 7 is a table for explaining the process of selecting an area by area selection unit 46. Here, as shown in the "Area" column of Fig. 7, an image area 61 including the characters "Name", an image area 62 including the characters "Address", and an image area 63 including the characters "Luck" will be explained. Also, as shown in the "Feature Point" column of Fig. 7, five feature points 71A to 71E are set in image area 61, five feature points 72A to 72E are set in image area 62, and five feature points 73A to 73E are set in image area 63.

[0050] In the table of FIG. 7, the column "Results of feature points pair extraction" indicates whether or not a feature points pair has been extracted by the feature points pair extraction unit 44. In the column "Results of feature points pair extraction", "◯" indicates that a feature points pair has been extracted, and "×" indicates that a feature points pair has not been extracted. Note that, as examples of cases in which a feature points pair is not extracted, such as when a shadow of the photographer is caught or when a shadow change caused by a light source occurs, or when a part of the driver's license is covered by the photographer's finger, a feature points pair may not be extracted. In addition, as described in relation to feature point 73B of image region 63 in FIG. 6, there may be cases in which one or more feature points are extracted from image 22, but no feature points pair is extracted because the feature points are excluded from the feature points constituting the feature points pair based on the positional relationship of the feature points.

[0051] In the table of FIG. 7, the column "Use" indicates whether or not the position information of the feature points pair is used for projective transformation. In the "Use" column, "○" indicates that the position information of the feature points pair is used for projective transformation, and "×" indicates that the position information of the feature points pair is not used for projective transformation. For example, as shown in the column "Feature Points Pair Extraction Results" in the table of FIG. 7, in image region 61 and image region 62, feature points pairs are extracted for four of the five set feature points. On the other hand, in image region 63, feature points pairs are not extracted for three of the five set feature points. In this case, region selection unit 46 selects image region 61 and image region 62 as image regions to be used for projective transformation, and does not select image region 63 as an image region to be used for projective transformation.

[0052] In this way, the region selection unit 46 preferentially selects an image region from which a larger number of feature points pairs are extracted as an image region for selecting a feature points pair to be used in projective transformation. The region selection unit 46 may select an image region from which the number of extracted feature points pairs is equal to or greater than a predetermined threshold as an image region for selecting a feature points pair to be used in projective transformation. The region selection unit 46 may select an image region from which the ratio of the number of extracted feature points pairs to the number of feature points set in the image region is equal to or greater than a predetermined threshold as an image region for selecting a feature points pair to be used in projective transformation. In this way, the region selection unit 46 may select some image regions from the multiple image regions set in the template image 60 as image regions for selecting a feature points pair to be used in projective transformation based on the number of extracted feature points pairs. For example, it can be determined that a larger number of extracted feature points pairs is more reliable as an extraction result of feature points pairs than a smaller number of feature points pairs extracted for feature points set in a specific image region. This makes it possible to exclude image regions for which the reliability of the extraction results of feature points pairs is highly likely to be low from image regions for selecting feature points pairs to be used in projective transformation.

[0053] The image conversion unit 47 calculates a projective transformation matrix H based on the positions of the feature point pairs extracted for the feature points set in the image region selected by the region selection unit 46. The image conversion unit 47 then applies the calculated projective transformation matrix H to the image 22 to generate a front image. This allows a front image with reduced distortion to be generated from the image 22 of the driver's license that has distortion in the processing target image 20. The character image extraction unit 48 then extracts a character image from the front image generated by the image conversion unit 47. For example, the character image extraction unit 48 extracts a character image of "XX Taro" from the area where the name of the driver's license owner is printed in the name field of the fixed format 50 from the front image generated by the image conversion unit 47. The character image extraction unit 48 also extracts a character image of "XX City, XX Prefecture, △△ 1-1-1" from the area where the address of the driver's license owner is printed in the address field of the fixed format 50.

[0054] 8 and 9 are diagrams for explaining a process of performing projective transformation further using the position of a photographic image of a person. FIG. 8 shows an image area 90 in the template image 60 that should contain a photographic image. The image area 90 is a specific area that is predefined in the fixed format 50. In FIGS. 8 and 9, a case will be explained in which feature points 91A and 91B are set in the image area 90. The image area 90 has a rectangular shape. In the example shown in FIG. 8, feature points 91A and 91B are two vertices of the rectangle of the image area 90.

[0055] 9 is a schematic diagram showing feature points pairs extracted from an image region of a photographic image. The identification unit 45 identifies an image region 900 including a photographic image of a person from the image 22. The feature points pair extraction unit 44 extracts feature points 91a and feature points 91b from the image region 900 identified by the identification unit 45, thereby identifying the feature points pair.

[0056] For example, the identification unit 45 uses a trained model that has been machine-learned using a photographic image of a person included in a driver's license as training data. For example, the first trained model is generated by supervised machine learning using a machine learning algorithm such as a convolutional neural network (CNN) to learn the relationship between the image feature amount extracted from the image area included in the driver's license and the image area of ​​the person, using as training data an image feature amount extracted from the image area of ​​the person among the image areas of the photographic image included in the image obtained by imaging the driver's license. The second trained model is generated by supervised machine learning using a machine learning algorithm such as a convolutional neural network (CNN) to learn the relationship between the image feature amount extracted from the image area included in the driver's license and the image area of ​​the person's background, using as training data an image feature amount extracted from the image area of ​​the person among the image areas of the photographic image included in the image obtained by imaging the driver's license.

[0057] The identification unit 45 uses the image features extracted from each image region of the image 22 and the first trained model to determine whether each image region is an image region of a person included in the driver's license, and uses the image features extracted from each image region of the image 22 and the second trained model to determine whether each image region is an image region of the background of the person included in the driver's license, thereby identifying the image region 900 from the image 22.

[0058] The feature point pair extraction unit 44 extracts a feature point 91A set in the template image 60 and a feature point 91a, which is one vertex of the image region 900 identified by the identification unit 45, as a feature point pair. The feature point pair extraction unit 44 also extracts a feature point 91B set in the template image 60 and a feature point 91b, which is one vertex of the image region 900 identified by the identification unit 45, as a feature point pair. The feature point pair extraction unit 44 may select, from among the two vertices that represent the contour of the image region 900 on the side determined by the identification unit 45 as the image region of the background, the vertex on the right side of the person as the feature point 91a. The feature point pair extraction unit 44 may select, from among the two vertices that represent the contour of the image region 900 on the side determined by the identification unit 45 as the image region of the person, the vertex on the right side of the person as the feature point 91b.

[0059] As described in relation to FIG. 5 etc., the feature points pair extraction unit 44 may determine whether or not to select feature point 91a and feature point 91b as feature points constituting a feature points pair, based on the positional relationships between other feature points set in template image 60 and the corresponding feature points extracted from image 22.

[0060] 8, a case where two feature points are set in the image area 90 has been described as an example, but the number of feature points set in the image area 90 is not limited to two. The number of feature points set in the image area 90 may be three or more. The number of feature points set in the image area 90 may be one.

[0061] The image conversion unit 47 performs projective transformation using feature points pairs extracted for feature points 91A and 91B set in the image area 900 in addition to feature points pairs extracted for a plurality of feature points set in the image area 61, the image area 62, the image area 63, the image area 64, the image area 65, the image area 66, and the image area 67 of the template image 60. At this time, as described in relation to FIG. 7 and the like, the area selection unit 46 may determine whether or not to select the image area 900 as an image area to be used for projective transformation, based on the number of feature points pairs extracted from the image area 900. When the image area 900 is selected as the image area to be used for projective transformation, the image conversion unit 47 may perform projective transformation further using feature points pairs extracted for feature points 91A and feature points 91B set in the image area 900.

[0062] As described above, the processing of the image processing unit 40 can make the image of characters extracted from a standard format document less susceptible to the influence of differences in character shapes and fonts used by the issuer of the standard format document. This makes it possible to accurately determine the characters printed by the issuer of the document. In the present embodiment, a driver's license has been described as an example of a standard format document. In addition to a driver's license, card-like documents including personal information, such as health insurance certificates and credit cards, can be used as standard format documents. The standard format document may be a page in a passport that includes personal information of the owner. The standard format document may be a page in a passport that includes a photo image of the owner. The standard format document may be any document that includes an area in which characters are printed or written as a standard format.

[0063] Next, with reference to Figs. 10 to 17, a process related to the character processing unit 200 will be specifically described as one mode of using the image of characters extracted by the image processing unit 40.

[0064] FIG. 10 shows an outline of the flow of processing executed by the image processing system 10. The learning device 202 generates a model 120 by machine learning using font data 100. The font data 100 includes data of fonts A, B1, B2, etc. Fonts A, B1, B2, etc. each have a different character shape. Among fonts A, B1, B2, etc., a specific font A is called a reference font A. The reference font A is a reference font used as a comparison target when identifying a unique character. Note that multiple fonts B1, B2, etc. may be collectively referred to as "font Bi" using the symbol "i" that identifies each font.

[0065] The data of font A and font Bi are used as font data for machine learning. The learning device 202 performs machine learning using the font data of font A and font Bi to generate a model 120 that adapts a character of each font of font Bi to the glyph of a character of font A. The model 120 is a model constructed by, for example, a neural network. The learning device 202 performs machine learning on {a character of font Bi and a character of font A} as one character pair, and generates a model 120 that generates an image of a character adapted to the glyph of a character of font A when an image of a character of font Bi is input.

[0066] Once model 120 is generated, character processing unit 200 inputs an image of a character extracted from image 20 to model 120 to generate adaptive character 140. Based on a comparison result between adaptive character 140 and a character in reference font A, character processing unit 200 determines whether the character extracted from image 20 is a unique character or not, and records information on the character determined to be a unique character in unique character information 160.

[0067] The character analysis device 280 uses the unique character information 160 to perform character analysis on the image to be analyzed 30. For example, if the character "fuji" is determined to be a unique character, the unique character information 160 performs character recognition using an algorithm constructed to correctly recognize the character "fuji" in the image to be analyzed 30.

[0068] FIG. 11 shows functional blocks of the character processing unit 200, the learning unit 202, and the character analysis unit 280.

[0069] Learning device 202 includes a font selection unit 204 and a model generation unit 206. Character processing unit 200 includes a processing target character acquisition unit 210, a character image generation unit 220, a character image selection unit 230, a difference information output unit 240, and a determination result acquisition unit 250. Character analysis device 280 includes an analysis target image acquisition unit 282, a character image extraction unit 284, and a character analysis unit 286.

[0070] Character processing unit 200, learning device 202, and character analysis device 280 are realized by computers. Character processing unit 200, learning device 202, and character analysis device 280 may be realized by one or more arbitrary number of computers. Storage device 290 is realized by a non-volatile storage medium or a volatile storage medium.

[0071] The storage device 290 stores the trained model 120 generated by machine learning using an image of a reference character having a predetermined character shape and images of a plurality of characters having different character shapes. The model 120 is a model that generates an image of a character adapted to a predetermined character shape from an input character image.

[0072] In the learning device 202, the model generation unit 206 generates a model 120 using a generative adversarial network (GAN) that uses as training data multiple pairs of an image of a reference character having a predetermined character shape and images of characters having mutually different character shapes.

[0073] The model generating unit 206 may use the font data 100 as learning data. In this case, the "reference character having a predetermined character shape" is a character belonging to a predetermined first font, and the "multiple characters having mutually different character shapes" are characters belonging to multiple mutually different second fonts different from the first font. Specifically, the first font corresponds to the above-mentioned reference font A, and the second font corresponds to the above-mentioned font Bi.

[0074] The font selection unit 204 selects a first font and a plurality of second fonts. The font selection unit 204 may select a font designated by the judge 80 as the first font. The model generation unit 206 performs machine learning using an image of the first font and images of the plurality of second fonts to generate a model 120. The model 120 is a model generated by machine learning to generate a model 120 that generates an image of a character adapted to the character shape of a character belonging to the first font from images of characters belonging to the plurality of second fonts.

[0075] In the character processing unit 200, the processing target character acquisition unit 210 acquires an image of a processing target character. The processing target character is, for example, an image of a character included in the processing target image 20. Specifically, the processing target character image is an image of a character extracted by the image processing unit 40 from the front image of the processing target image 20.

[0076] The character image generating unit 220 generates an image of a character to be processed adapted to a predetermined character shape from an image of the character to be processed by using the trained model 120. The difference information output unit 240 outputs information indicating the difference between the character to be processed and the reference character based on a comparison result between the image generated by the character image generating unit 220 and the image of the reference character.

[0077] The difference information output unit 240 displays the image generated by the character image generating unit 220 and the image of the reference character in a superimposed manner. For example, the difference information output unit 240 may display the image generated by the character image generating unit 220 and the image of the reference character in different colors in a superimposed manner. The judgment result acquisition unit 250 acquires information indicating whether or not the image generated by the character image generating unit 220 and the image of the reference character differ from the judge 80 as a user.

[0078] For example, the character image selection unit 230 selects an image of a character similar to the image of a character generated by the character image generation unit 220 from among a plurality of reference character images. Then, the difference information output unit 240 displays the image selected by the character image selection unit 230 and the image generated by the character image generation unit 220 in a superimposed manner. For example, the difference information output unit 240 displays the image selected by the character image selection unit 230 and the image generated by the character image generation unit 220 in a superimposed manner on the display device 88. Since the image generated by the character image generation unit 220 and the image of the reference character are displayed in a superimposed manner, the assessor 80 can appropriately determine the portion where the character, the character shape of which has been adapted to the reference font, differs from the reference character.

[0079] When it is determined that the character to be processed differs from the reference character, the difference information output unit 240 may record information indicating that the character to be processed is a character having a distinctive character shape in the peculiar character information 160. Note that "information indicating that the character to be processed is a character having a distinctive character shape" is an example of "information indicating a difference between the character to be processed and the reference character." When recording in the peculiar character information 160, the difference information output unit 240 may record the information together with issuer information of the image to be processed 20.

[0080] The difference information output unit 240 compares the image generated by the character image generation unit 220 with the image of a reference character, and if there is a part in the image generated by the character image generation unit 220 where the character skeleton differs from that in the image of the reference character, it records information indicating the part where the character skeleton differs in association with identification information of the character to be processed.

[0081] In the character analysis device 280, the analysis target image acquisition unit 282 acquires image data of the analysis target image 30 as a document to be analyzed for characters. The analysis target image 30 is an example of a "document to be analyzed for characters". The character image extraction unit 284 extracts an image including characters from the image data of the analysis target image 30. The character analysis unit 286 analyzes characters included in the image extracted by the character image extraction unit 284 using information recorded by the difference information output unit 240. The character analysis unit 286 may receive issuer information of the analysis target image 30 from the judge 80 and analyze characters using peculiar character information 160 selected based on the issuer information. In this way, the character analysis unit 286 analyzes characters included in the image extracted by the character image extraction unit 284 using the peculiar character information 160. The processing target image 20 is a sample image of a document of the same type as the analysis target image 30. Therefore, the analysis target image 30 uses characters in the same font as the characters used in the processing target image 20. The character analysis unit 286 recognizes the specific characters identified from the processing target image 20 and analyzes the analysis target image 30, so that the content written in the analysis target image 30 can be analyzed more accurately.

[0082] 12 shows the configuration of the learning data. The learning data includes character pair 400-11, character pair 400-12, character pair 400-13... and character pair 400-21, character pair 400-22, character pair 400-23...

[0083] Character pair 400-11 is a pair of the character “” in font B1 and the character “” in the reference font A. Character pair 400-12 is a pair of the character “” in font B1 and the character “” in the reference font A. Character pair 400-13 is a pair of the character “” in font B1 and the character “” in the reference font A.

[0084] The character pair 400-21 is a pair of the character "藤" in font B2 and the character "藤" in the reference font A. The character pair 400-22 is a pair of the character "研" in font B2 and the character "研" in the reference font A. The character pair 400-23 is a pair of the character "あ" in font B2 and the character "あ" in the reference font A.

[0085] Generally, if the code for identifying font Bi is "i" and the code for identifying the font style (typeface) is "j", the model generation unit 206 performs machine learning using the character pair 400-ij to generate the model 120. For example, when an image of the character "藤" in font B1 is input, the model generation unit 206 performs machine learning using the character pair 400-ij to generate an image of the character "藤" with a glyph that is as adapted as possible to the glyph of the character "藤" in the reference font A.

[0086] Thereby, for example, when an image of the character "藤" in font B1 is input, the model 120 is generated that generates an image of the character "藤" with a glyph adapted to the glyph of the reference font A. Thus, the model 120 is a model that generates an image of the character j with a glyph adapted to the glyph of the character j in font A from an image of the character j in font Bi.

[0087] FIG. 13 shows a conceptual configuration of a learning device that executes machine learning in the model generation unit 206. FIG. 13 shows a case where learning is performed using, as learning data, a character pair of an image 500b of the character "藤" in font Bi and an image 500a of the character "藤" in the reference font A. The learning device shown in FIG. 13 is a learning device that uses a conditional GAN, which is a type of GAN.

[0088] The learning device includes a generation network G and a discrimination network D. The generation network G and the discrimination network D are each a neural network. The generation network G generates a fake image 500b' from the input image 500b.

[0089] A combination of image 500b and image 500a is input to the discrimination network D. A combination of image 500b and fake image 500b' generated by the generation network G is also input to the discrimination network D. The discrimination network D outputs a discrimination result for the combination of input images. For example, the discrimination network D outputs a probability indicating the degree of correctness of the combination of input images as a numerical value ranging from 0 to 1. For example, the discrimination network D outputs "1" if it determines that the combination of input images is correct, and outputs "0" if it determines that the combination of input images is incorrect.

[0090] The discrimination network D learns to output a value close to "1" when a pair of the image 500b and the image 500a is input, and to output a value close to "0" when a pair of the image 500b and the fake image 500b' is input. In other words, the discrimination network D learns to be able to determine that the fake image 500b' generated by the generation network G is a fake. On the other hand, the generation network G learns to generate a fake image 500b' such that a value close to "1" is output from the discrimination network D when the pair of the image 500b and the fake image 500b' is input to the discrimination network D. When the output of the discrimination network D when the pair of the image 500b and the fake image 500b' is input becomes sufficiently close to 1 / 2, it is determined that learning has been achieved.

[0091] The model generation unit 206 performs machine learning using a large number of character pairs with different combinations of fonts and character styles (character types) as shown in Fig. 12. When the model generation unit 206 determines that learning has been achieved for each character pair, it outputs a generation network G as a model 120.

[0092] Fig. 14 shows an example of the difference information output by the difference information output section 240. A screen 600 in Fig. 14 is a display example of the difference information output by the difference information output section 240 to the display device 88.

[0093] Character image 620-1 and character image 620-2 are images of characters in the processing target image 20. On the screen 600, character image 620-1 and character image 620-2 are displayed as “characters to be inspected” in the character processing unit 200.

[0094] Character image 630-1 is a character image generated by character image generation unit 220 by inputting character image 620-1 to model 120. Character image 630-2 is a character generated by character image generation unit 220 by inputting character image 620-2 to model 120. On screen 600, character image 630-1 and character image 630-2 are displayed as "converted characters" having characters that match the character shapes of the reference font.

[0095] Character image 640-1 is an image of a character in reference font A corresponding to the character in character image 620-1. Character image 640-2 is an image of a character in reference font A corresponding to the character in character image 620-2. Character image 640-1 and character image 640-2 are images selected by character image selection unit 230. For example, character image selection unit 230 selects a character having a degree of matching with character image 630-1 higher than a predetermined value from among the characters in reference font A. On screen 600, character image 640-1 and character image 640-2 are displayed as "reference characters" to be compared with converted characters.

[0096] Comparative image 650-1 is an image in which character image 640-1 is superimposed on character image 630-1. Comparative image 650-2 is an image in which character image 640-2 is superimposed on character image 630-2.

[0097] The difference information output unit 240 causes the display device 88 to display the character images 620-1, 630-1, 640-1, and 650-1 in association with the button 610-1. The difference information output unit 240 also causes the display device 88 to display the character images 620-2, 630-2, 640-2, and 650-2 in association with the button 610-2.

[0098] FIG. 14 shows a state in which information about characters of two typefaces, "Fuji" and "Ken", is displayed. Generally, assuming that the symbol indicating the character type to be inspected is "i", the difference information output unit 240 associates the character image 620-i, the character image 630-i, the character image 640-i, and the comparison image 650-i with the button 610-i and causes them to be displayed on the display device 88.

[0099] Note that the difference information output unit 240 may compare the character image 630-i and the character image 640-i and calculate the degree of coincidence between the character image 630-i and the character image 640-i. The difference information output unit 240 may display character images including the character image 620-i, the character image 630-i, the character image 640-i, and the comparison image 650-i and the button 610-i in ascending order of the calculated low degree of coincidence of the images.

[0100] The determiner 80 determines, for example, by referring to the comparison image 650-i whether the character image 630-i has a part that is different from the character image 640-i. Specifically, the determiner 80 determines whether the character image 630-i has a skeletal part that is different from the character image 640-i. For example, when the determiner 80 determines that the character image 630-i has a skeletal part that is different from the character image 640-i, the determiner 80 presses the button 610-i.

[0101] In response to the button 610-i being pressed, the determination result acquisition unit 250 acquires a determination result that the character image 620-i is a special character. In this case, the difference information output unit 240 records the character information of the character image 620-i as special character information 160 as a special character. For example, the difference information output unit 240 records the identification information of the character type of the character image 620-i in the special character information 160.

[0102] The determiner 80 may input information indicating a portion where the character image 630-i differs from the character image 640-i. For example, the determiner 80 inputs information indicating a range including a skeleton part where the character image 630-i differs from the character image 640-i. In this case, the difference information output unit 240 records the input information indicating the range including the skeleton part in the peculiar character information 160 in association with the identification information of the character type of the character image 620-i.

[0103] 15 shows enlarged views of character image 620-1, character image 630-1, character image 640-1, and comparative image 650-1. D1 indicates the skeletal distance between sides 701 and 702 constituting the character "fuji" in character image 620-1. D2 is the skeletal distance between sides 701 and 702 in character image 630-1, and D3 is the skeletal distance between sides 701 and 702 in character image 640-1. It can be clearly seen from comparative image 650-1 that both D1 and D2 are shorter than D3.

[0104] The character in character image 630-1 is obtained by adapting the character shape of the character in character image 620-1 to the character shape of "Fuji" in the reference font A. As a result, the difference between character image 630-1 and character image 640-1 caused by differences in decorative designs such as "stops," "strokes," and "strokes" and differences in character thickness is reduced. Therefore, the decorative design and thickness of the character in character image 630-1 are close to the decorative design and thickness of the character in character image 640-1 of the reference font. Most of the fonts in general circulation have different decorative designs and thicknesses. Therefore, by training model 120 using fonts with different decorative designs and thicknesses, model 120 that generates images of characters with decorative designs close to the decorative design of the reference font A is obtained. As a result, the difference in decorative designs between character image 630-1 and character image 640-1 is reduced. Therefore, for example, through the comparison image 650-1, it is possible to clearly present a characteristic difference from the font used for learning, that is, the short skeleton interval between the side 701 and the side 702 of the character image 630-1.

[0105] In addition, in the font data used in the machine learning for generating the model 120, it is assumed that the difference between fonts is small with respect to the skeleton spacing between the sides 701 and 702. In this case, in the learning process of the model 120 using the font data, the skeleton spacing between the sides 701 and 702 is learned as having a universal characteristic. The model 120 generated in this way does not greatly adapt the skeleton spacing between the sides 701 and 702 to the reference font for the input image of the character "fuji". Therefore, when the character image 620-1 is converted by the model 120, an image of a character with a relatively narrow skeleton spacing between the sides 701 and 702, such as the character image 630-1, is obtained.

[0106] Thus, according to the image processing system 10, for character elements having various differences such as decorative designs among the character elements of the characters input to the model 120, the character image generating unit 220 generates an image of the character adapted to the reference font A. On the other hand, for character elements having characteristic character elements different from character elements common to the font data used in the machine learning among the character elements of the characters input to the model 120, the character image generating unit 220 generates an image of the character that substantially maintains the characteristic character elements.

[0107] Therefore, the judge 80 can easily judge whether or not the character image 620-1 is a unique character by comparing the character image 630-1 and the character image 640-1. In particular, since the difference information output unit 240 displays the comparison image 650-1 on the display device 88, the judge 80 can judge at a glance that the skeleton interval between the side 701 and the side 702 of the character "fuji" in the processing target image 20 is short.

[0108] FIG. 16 shows a state where character image 640-1 is superimposed on character image 620-1 as a reference example. Unlike character image 630-1, character image 620-1 is not adapted to the character shape of "fuji" in reference font A. Therefore, there are many errors between character image 620-1 and character image 640-1 caused by differences in the design of the character shapes. Thus, even when character image 620-1 and character image 640-1 are compared, errors caused by differences in the decorative design of the character shapes are noticeable. Therefore, whether a judge 80 judges by sight or by a computer or the like, it is not easy to judge whether the character is a peculiar character or not.

[0109] In contrast, as shown in FIGS. 16 and 17, the image processing system 10 makes it relatively easy to determine which characters are peculiar characters.

[0110] 17 shows an example of the data structure of the peculiar character information 160. The peculiar character information 160 stores character identification information, character images, characteristic parts, and characteristic amounts in association with each other.

[0111] "Character identification information" stores identification information of a character determined as a peculiar character. The character identification information may be, for example, information indicating a character type. The character identification information may be a character code. "Character image" stores image data of a character determined as a peculiar character in binary format.

[0112] The "characteristic portion" is information indicating the range of a portion having a characteristic character shape. For example, when an area having a characteristic character shape is indicated by a rectangular area, the "characteristic portion" may include information indicating the coordinates of the diagonal corners of the rectangle.

[0113] The "feature amount" is information indicating a feature amount extracted from a character image. The feature amount may include a feature amount extracted from the entire character image. The feature amount may include a feature amount extracted from a characteristic portion of the character image.

[0114] The peculiar character information 160 is used when the character analysis device 280 analyzes the analysis target image 30. For example, when analyzing the analysis target image 30, the analysis target image acquisition unit 282 acquires image data of the analysis target image 30, and the character image extraction unit 284 extracts character images from the image data of the analysis target image 30. Then, the character analysis unit 286 uses the information stored in the peculiar character information 160 to analyze the character images extracted by the character image extraction unit 284.

[0115] For example, the character analysis unit 286 detects a feature from a character image extracted from the analysis target image 30. When the detected feature matches a feature stored in the peculiar character information 160, the character analysis unit 286 reads out character identification information stored in the peculiar character information 160 in association with the feature. Then, the character analysis unit 286 recognizes that a character in the character identification information is a character in the character image extracted from the analysis target image 30.

[0116] According to the image processing system 10 described above, it is possible to appropriately determine whether a character input as an image is a unique character or not. Therefore, it is possible to recognize a unique character and analyze the character by character recognition or the like.

[0117] It is preferable that the font style used in machine learning matches the font style used in the processing target image 20 and the analysis target image 30. For example, if the font used in the processing target image 20 and the analysis target image 30 is a Gothic typeface font, it is preferable to use a Gothic typeface font for machine learning.

[0118] Character recognition is an example of character analysis, such as determining the validity of a document.

[0119] As an application example of character processing in the image processing system 10, it may be used for evaluation of handwritten characters. For example, it may be used for evaluation when a student practices a standard style of writing such as a regular script in penmanship. For example, the model 120 is generated by using handwritten characters of a specific instructor as a reference character and learning using handwritten characters of many other instructors. Then, a document written by a student is input as a document to be input to the character processing unit 200. As a result, it is determined whether or not the character written by the student has a unique feature based on the comparison result between the character adapted to the handwritten character written by the specific instructor and the reference character written by the specific instructor. Also, as a character analysis process in the image processing system 10, it may be used for analysis of handwritten characters such as handwriting analysis.

[0120] 18 shows an example of a computer 2000 according to the present embodiment. A program installed in the computer 2000 can cause the computer 2000 to function as an apparatus or system, such as the image processing unit 40, the character processing unit 200, the learning device 202, and the character analysis device 280, or the image processing system 10 according to the present embodiment, or each part of the apparatus or system, execute operations associated with the apparatus or each part of the apparatus, and / or execute a process or steps of the process according to the present embodiment. Such a program can be executed by the CPU 2012 to cause the computer 2000 to execute specific operations associated with some or all of the blocks of the processing procedures and block diagrams described herein.

[0121] The computer 2000 according to this embodiment includes a CPU 2012 and a RAM 2014, which are connected to each other by a host controller 2010. The computer 2000 also includes a ROM 2026, a flash memory 2024, a communication interface 2022, and an input / output chip 2040. The ROM 2026, the flash memory 2024, the communication interface 2022, and the input / output chip 2040 are connected to the host controller 2010 via the input / output controller 2020.

[0122] The CPU 2012 operates according to the programs stored in the ROM 2026 and RAM 2014, thereby controlling each unit.

[0123] The communication interface 2022 communicates with other electronic devices via a network. The flash memory 2024 stores programs and data used by the CPU 2012 in the computer 2000. The ROM 2026 stores a boot program and the like executed by the computer 2000 upon activation, and / or a program that depends on the hardware of the computer 2000. The input / output chip 2040 may also connect various input / output units, such as a keyboard, mouse, and monitor, to the input / output controller 2020 via input / output ports, such as a serial port, a parallel port, a keyboard port, a mouse port, a monitor port, a USB port, an HDMI port, and the like.

[0124] The programs are provided via a computer-readable medium such as a CD-ROM, a DVD-ROM, or a memory card, or a network. The RAM 2014, the ROM 2026, or the flash memory 2024 are examples of computer-readable media. The programs are installed in the flash memory 2024, the RAM 2014, or the ROM 2026, and executed by the CPU 2012. Information processing described in these programs is read by the computer 2000, and brings about cooperation between the programs and the various types of hardware resources described above. An apparatus or method may be constructed by implementing an operation or processing of information according to the use of the computer 2000.

[0125] For example, when communication is performed between the computer 2000 and an external device, the CPU 2012 may execute a communication program loaded in the RAM 2014 and instruct the communication interface 2022 to perform communication processing based on the processing described in the communication program. Under the control of the CPU 2012, the communication interface 2022 reads transmission data stored in a transmission buffer processing area provided in a recording medium such as the RAM 2014 and the flash memory 2024, transmits the read transmission data to a network, and writes reception data received from the network to a reception buffer processing area or the like provided on the recording medium.

[0126] Furthermore, the CPU 2012 may cause all or a necessary portion of a file or database stored on a recording medium such as the flash memory 2024 to be read into the RAM 2014, and perform various types of processing on the data on the RAM 2014. The CPU 2012 then writes back the processed data to the recording medium.

[0127] Various types of information, such as various types of programs, data, tables, and databases, may be stored in the recording medium and subjected to information processing. The CPU 2012 may execute various types of processing, including various types of operations, information processing, conditional judgment, conditional branching, unconditional branching, information search / replacement, etc., described herein and specified by the instruction sequence of the program, on the data read from the RAM 2014, and write back the results to the RAM 2014. The CPU 2012 may also search for information in a file, database, etc. in the recording medium. For example, when a plurality of entries each having an attribute value of a first attribute associated with an attribute value of a second attribute are stored in the recording medium, the CPU 2012 may search for an entry that matches a condition in which an attribute value of the first attribute is specified from among the plurality of entries, read the attribute value of the second attribute stored in the entry, and thereby obtain the attribute value of the second attribute associated with the first attribute that satisfies a predetermined condition.

[0128] The above-described programs or software modules may be stored in a computer-readable medium on the computer 2000 or in the vicinity of the computer 2000. A recording medium such as a hard disk or a RAM provided in a server system connected to a dedicated communication network or the Internet can be used as the computer-readable medium. The programs stored in the computer-readable medium may be provided to the computer 2000 via a network.

[0129] A program installed in computer 2000 and causing computer 2000 to function as image processing system 10 may act on CPU 2012 and the like to cause computer 2000 to function as each part of image processing system 10, including image processing unit 40 and character processing unit 200. When information processing described in these programs is read into computer 2000, it functions as each part of image processing system 10, which is a specific means in which software and the above-mentioned various hardware resources work together. These specific means then realize calculation or processing of information according to the intended use of computer 2000 in this embodiment, thereby constructing a specific image processing system 10 according to the intended use.

[0130] Various embodiments have been described with reference to block diagrams, etc. In the block diagrams, each block may represent (1) a stage of a process where an operation is performed or (2) a portion of an apparatus responsible for performing an operation. Particular stages and portions may be implemented by dedicated circuitry, programmable circuitry provided with computer readable instructions stored on a computer readable medium, and / or a processor provided with computer readable instructions stored on a computer readable medium. The dedicated circuitry may include digital and / or analog hardware circuitry, and may include integrated circuits (ICs) and / or discrete circuits. The programmable circuitry may include reconfigurable hardware circuitry, including logical AND, logical OR, logical XOR, logical NAND, logical NOR, and other logical operations, memory elements such as flip-flops, registers, field programmable gate arrays (FPGAs), programmable logic arrays (PLAs), etc.

[0131] A computer-readable medium may include any tangible device capable of storing instructions that are executed by a suitable device, such that the computer-readable medium having instructions stored thereon constitutes at least a portion of an article of manufacture that includes instructions that can be executed to provide a means for performing the operations specified in a process or block diagram. Examples of computer-readable media may include electronic storage media, magnetic storage media, optical storage media, electromagnetic storage media, semiconductor storage media, and the like. More specific examples of computer-readable media may include floppy disks, diskettes, hard disks, random access memories (RAMs), read-only memories (ROMs), erasable programmable read-only memories (EPROMs or flash memories), electrically erasable programmable read-only memories (EEPROMs), static random access memories (SRAMs), compact disk read-only memories (CD-ROMs), digital versatile disks (DVDs), Blu-ray disks, memory sticks, integrated circuit cards, and the like.

[0132] The computer readable instructions may include either assembler instructions, instruction set architecture (ISA) instructions, machine instructions, machine-dependent instructions, microcode, firmware instructions, state setting data, or source or object code written in any combination of one or more programming languages, including object-oriented programming languages ​​such as Smalltalk®, JAVA®, C++, etc., and conventional procedural programming languages ​​such as the “C” programming language or similar programming languages.

[0133] The computer readable instructions may be provided to a processor or programmable circuitry of a general purpose computer, special purpose computer, or other programmable data processing apparatus, either locally or over a wide area network (WAN) such as a local area network (LAN), the Internet, etc., to cause the computer readable instructions to execute means for performing operations specified in the process steps or block diagrams described. Examples of processors include computer processors, processing units, microprocessors, digital signal processors, controllers, microcontrollers, etc.

[0134] Although the present invention has been described above using the embodiments, the technical scope of the present invention is not limited to the scope described in the above embodiments. It is clear to those skilled in the art that various modifications or improvements can be made to the above embodiments. In addition, the matters described for a specific embodiment can be applied to other embodiments to the extent that they are not technically inconsistent. It is clear from the claims that such modified or improved forms can also be included in the technical scope of the present invention.

[0135] It should be noted that the order of execution of each process, such as operations, procedures, steps, and stages, in the devices, systems, programs, and methods shown in the claims, specifications, and drawings is not specifically stated as "before," "prior to," etc., and may be realized in any order unless the output of a previous process is used in a later process. Even if the operational flow in the claims, specifications, and drawings is explained using "first," "next," etc. for convenience, it does not mean that it is essential to perform the process in this order. [Explanation of symbols]

[0136] 10 Image Processing System 12 Image processing device 20 Processing target image 21 Background Images 22 images 23 Border 30 Images to be analyzed 40 Image processing section 41 Template image acquisition unit 42 Target image acquisition unit 43 Image Extraction Section 44 Feature point pair extraction unit 45 Specific part 46 Area Selection Section 47 Image conversion section 48 Character Image Extraction Unit 49 Storage area 50 Standard Format 52 Driver's License 60 template images 61, 62, 63, 64, 65, 66, 67 Image Area 68, 69 Border 71, 72, 73 Feature points 80 Judge 88 Display device 90 Image Area 91 feature points 91A, 91B Features 100 Font Data 120 Model 140 Adaptation characters 160 Unique Character Information 200 Character Processing Unit 202 Learning Device 204 Font selection section 206 Model Generation Unit 210 Processing target character acquisition unit 220 Character image generation unit 230 Text image selection section 240 Difference information output unit 250 Judgment result acquisition section 280 Character analyzer 282 Analysis target image acquisition unit 284 Character Image Extraction Unit 286 Character analysis section 290 Storage device 400 character pairs 500 images 600 screens 610 Button 620, 630, 640 Text images 650 Comparison Images 701, 702 sides 900 Image Area 2000 Computer 2010 Host Controller 2012 CPU 2014 RAM 2020 Input / Output Controller 2022 Communication Interface 2024 Flash Memory 2026 ROM 2040 Input / Output Chip

Claims

1. An image processing device, a target image acquisition unit that acquires a processing target image which is an image of a photographed document in a fixed format; a template image acquisition unit for acquiring a template image of the fixed format; a feature points pair extraction unit that extracts a plurality of feature points pairs having similar image information between the processing target image and the template image from the processing target image and the template image; an image transformation unit that performs projective transformation on the processing target image into a front image that should be obtained when the document is viewed from the front, based on the positions of the plurality of feature point pairs extracted by the feature point pair extraction unit; Equipped with The standard format defines a common area that is an area that should have information common to a plurality of documents, the template image does not include image information of an area other than the common area, and includes at least some character images among the plurality of character images in the common area; the feature points pair extraction unit extracts feature points having similar image information from the processing target image for each of a plurality of feature points set in a plurality of image regions included in the template image, thereby extracting the plurality of feature points pairs; The image processing device includes: a region selection unit that counts, for each of the plurality of image regions, a number of feature points whose image information extracted from the processing target image is similar to the plurality of feature points set in each of the plurality of image regions, and selects a portion of the image regions to be used for the projective transformation from among the plurality of image regions based on the number of feature points counted for each of the plurality of image regions. Further equipped with The image conversion unit performs projective conversion of the processing target image onto the front image based on positions of a plurality of feature point pairs extracted for the plurality of feature points set in the partial image region selected by the region selection unit. Image processing device.

2. The region selection unit preferentially selects, from among the plurality of image regions, an image region having a greater number of counted feature points as an image region to be used in the projective transformation. The image processing device according to claim 1 .

3. The feature points pair extraction unit extracts feature points from the processing target image having similar image information to each of a plurality of feature points set in each of the plurality of image regions, and selects, from among the plurality of feature points extracted from the processing target image, some feature points having a closer positional relationship to each other as feature points constituting the plurality of feature points pairs. The image processing device according to claim 2 .

4. A storage unit for storing the template image The image processing device according to claim 1 , further comprising:

5. An image processing device, a target image acquisition unit that acquires a processing target image which is an image of a photographed document in a fixed format; a template image acquisition unit for acquiring a template image of the fixed format; a feature points pair extraction unit that extracts a plurality of feature points pairs having similar image information between the processing target image and the template image from the processing target image and the template image; an image transformation unit that performs projective transformation on the processing target image into a front image that should be obtained when the document is viewed from the front, based on the positions of the plurality of feature point pairs extracted by the feature point pair extraction unit; Equipped with The standard format defines a common area that is an area that should have information common to a plurality of documents, the template image does not include image information of an area other than the common area, and includes at least some character images among the plurality of character images in the common area; the standard format document is a document including a photographic image of a person at a predetermined position; The image processing device includes: an identifying unit that identifies an area including a photographic image of a person from the processing target image by using a trained model that has been machine-learned using a photographic image of a person included in the document of the standard format as training data; Further equipped with The image conversion unit performs projective conversion of the processing target image onto the front image, further based on the position of the area including the photographic image identified by the identification unit. Image processing device.

6. The standard format document is a driver's license for a car or a motorbike, a passport, or a health insurance card. The image processing device according to claim 1 .

7. a character image extraction unit that extracts a character image from the front image; a character processing unit that outputs information indicating a difference between the image of the character extracted by the character image extraction unit and an image of a reference character having a predetermined character shape; The image processing device according to claim 1 , further comprising:

8. a target image acquisition unit that acquires a processing target image which is an image of a photographed document in a fixed format; a template image acquisition unit for acquiring a template image of the fixed format; a feature points pair extraction unit that extracts a plurality of feature points pairs having similar image information between the processing target image and the template image from the processing target image and the template image; an image transformation unit that performs projective transformation on the processing target image based on the positions of the plurality of feature point pairs extracted by the feature point pair extraction unit, into a front image that should be obtained when the document is viewed from the front; a character image extraction unit that extracts a character image from the front image; a character processing unit that outputs information indicating a difference between the image of the character extracted by the character image extraction unit and an image of a reference character having a predetermined character shape; Equipped with The standard format defines a common area that is an area that should have information common to a plurality of documents, the template image does not include image information of an area other than the common area, and includes at least some character images among the plurality of character images in the common area; The character processing unit includes: a processing target character acquisition unit that acquires the image of the character extracted by the character image extraction unit as an image of a processing target character; a storage unit for storing a trained model that is generated by machine learning using the reference character image and a plurality of character images having different character shapes, and that generates a character image adapted to the predetermined character shape from an input character image; a character image generating unit that generates an image of the processing target character adapted to the predetermined character shape from the image of the processing target character by using the trained model; a difference information output unit that outputs information indicating a difference between the target character and the reference character based on a comparison result between the image generated by the character image generating unit and the image of the reference character; An image processing device comprising:

9. A program for causing a computer to function as the image processing device according to any one of claims 1 to 8.

Images