Comparison system, comparison method and information storage medium

Abstract

The present invention provides a comparison system, a comparison method and an information storage medium capable of strictly performing electronic identity verification. A transformation image generation unit (32) generates a transformation image by transforming an oblique image. An object area determination unit (36) determines an object area in which an object appears in a front image based on features extracted from a fixed-form portion of a document recorded on the object in a sample image and features extracted from at least a portion of a front image. A comparison unit (38) determines whether an object appearing in the oblique image is the same as an object appearing in the front image by comparing the object area in the front image with an area in a transformation image corresponding to the object area.

Description

Technical Field

[0001] The invention relates to a comparison system, a comparison method and an information storage medium. Background Art

[0002] Non-Patent Document 1 describes SIFT (Scale Invariant Feature Transform) as one type of local image feature quantity.

[0003] In addition, there is known a technology for electronic Know Your Customer (eKYC) that performs online identification of a user based on an image of an identification document such as a driver's license sent from the user via the Internet. In addition, Patent Document 1 describes a technology for speeding up image processing in electronic identification, etc., using local image features such as SIFT. In addition, Patent Document 2 describes a technology that can improve the accuracy of image processing in electronic identification, etc.

[0004] Patent Document 1: International Publication No. 2020 / 008628

[0005] Patent Document 2: International Publication No. 2020 / 008629

[0006] Non-patent literature 1: David G. Lowe "Distinctive Image Features from Scale-Invariant Keypoints" International Journal of Computer Vision, 2004, January 5, 2004

[0007] In electronic identity verification, character recognition is performed on an image of an original object such as an identity verification document to extract character strings such as the name, address, date of birth, etc. recorded on the object. The user's identity is then verified based on the extracted character strings.

[0008] However, even if a user provides an illegal image such as a photocopy or scanned image of an object, a forged or tampered image, or other illegal image other than the original image of the object, the provided image may not be found to be an illegal image in electronic identity verification. Summary of the invention

[0009] The present invention has been made in view of the above-mentioned problems, and one of its objects is to provide a verification system, a verification method and an information storage medium capable of strictly performing electronic identity verification.

[0010] The comparison system of the present invention comprises: a front image acquisition unit, which acquires a front image, which represents the situation of observing the recording surface of a document recorded in an object from the front direction; an oblique image acquisition unit, which acquires an oblique image, which represents the situation of observing the recording surface from the oblique direction; a transformation image generation unit, which generates a transformation image by transforming the oblique image based on features extracted from at least a part of the front image and features extracted from at least a part of the oblique image, and the transformation image represents the situation of observing the object appearing in the oblique image from the front direction; an area determination unit, which determines the object area where the object appears in the front image based on features extracted from a fixed form part of the document recorded on the object in a sample image in which a given object appears in a known area and features extracted from at least a part of the front image; and a comparison unit, which determines whether the object appearing in the oblique image is the same as the object appearing in the front image by comparing the object area in the front image with the area in the transformation image corresponding to the object area.

[0011] In one aspect of the present invention, the region specifying unit specifies the object region after the transformed image generating unit generates the transformed image.

[0012] Alternatively, the transformed image generating unit generates the transformed image after the object region is determined by the region determining unit.

[0013] In this aspect, the transformed image generating unit may generate the transformed image by transforming the oblique image based on features extracted from the object region determined by the region determining unit and features extracted from at least a portion of the oblique image.

[0014] In one aspect of the present invention, the device further includes a thickness determination unit configured to determine, based on the oblique image, whether a thickness of an object appearing in the oblique image is thicker than a predetermined thickness.

[0015] In one aspect of the present invention, the device further includes a character recognition unit that recognizes characters included in a document written in the target object area.

[0016] In addition, the comparison method of the present invention includes the following steps: obtaining a frontal image, which represents the situation of observing the recording surface of the document recorded in the object from the front direction; obtaining an oblique image, which represents the situation of observing the recording surface from the oblique direction; generating a transformed image by transforming the oblique image based on features extracted from at least a part of the frontal image and features extracted from at least a part of the oblique image, which represents the situation of observing the object appearing in the oblique image from the front direction; determining an object area where the object appears in the frontal image based on features extracted from a fixed form part of the document recorded on the object in a sample image in which a given object appears in a known area and features extracted from at least a part of the frontal image; and determining whether the object appearing in the oblique image is the same as the object appearing in the frontal image by comparing the object area in the frontal image with the area in the transformed image corresponding to the object area.

[0017] In addition, the program of the present invention causes a computer to execute the following processes: obtaining a frontal image, which represents the situation of observing the recording surface of a document recorded in an object from the front direction; obtaining an oblique image, which represents the situation of observing the recording surface from the oblique direction; generating a transformed image by transforming the oblique image based on features extracted from at least a part of the frontal image and features extracted from at least a part of the oblique image, which represents the situation of observing the object appearing in the oblique image from the front direction; determining an object area in which the object appears in the frontal image based on features extracted from a fixed form part of a document recorded on the object in a sample image in which a given object appears in a known area and features extracted from at least a part of the frontal image; and determining whether the object appearing in the oblique image is the same as the object appearing in the frontal image by comparing the object area in the frontal image with the area in the transformed image corresponding to the object area. BRIEF DESCRIPTION OF THE DRAWINGS

[0018] Figure 1 This is a diagram showing an example of the overall configuration of an image processing system according to one embodiment of the present invention.

[0019] Figure 2 This is a diagram showing an example of a situation in which a driver's license is photographed.

[0020] Figure 3 This is a diagram showing an example of a front image.

[0021] Figure 4 This is a diagram showing an example of an oblique image.

[0022] Figure 5 This is a diagram showing an example of extracting character data.

[0023] Figure 6 This is a functional block diagram showing an example of functions of a server according to one embodiment of the present invention.

[0024] Figure 7 A diagram showing an example of a converted image.

[0025] Figure 8 A diagram showing an example of a sample image.

[0026] Fig. 9 This is a diagram showing an example of a front image.

[0027] Fig.10 A diagram showing an example of a converted image.

[0028] Fig.11 This is a diagram schematically showing an example of learning of a machine learning model.

[0029] Fig.12 This is a flowchart showing an example of the flow of processing performed in the server according to one embodiment of the present invention.

[0030] Description of symbols

[0031] 1: image processing system; 10: server; 10a: processor; 10b: storage unit; 10c: communication unit; 12: user terminal; 12a: processor; 12b: storage unit; 12c: communication unit; 12d: operation unit; 12e: display unit; 12f: shooting unit; 14: computer network; 20: driver's license; 22: front image; 24: oblique image; 30: image acquisition unit; 32: transformation image generation unit; 34: sample image storage unit; 36: object area determination unit; 38: comparison unit; 40: machine learning model; 42: thickness determination unit; 44: character recognition unit; 50: transformation image; 52: sample image; 54: driver's license. DETAILED DESCRIPTION

[0032] Hereinafter, an embodiment of the present invention will be described in detail with reference to the accompanying drawings.

[0033] Figure 1 FIG. 1 is a diagram showing an example of the overall structure of an image processing system 1 according to an embodiment of the present invention. Figure 1As shown, in the image processing system 1 of this embodiment, for example, a server 10 and a user terminal 12 are included. The server 10 and the user terminal 12 are connected to a computer network 14 such as the Internet. Therefore, the server 10 and the user terminal 12 can communicate with each other via the computer network 14. Figure 1 In the figure, one server 10 and one user terminal 12 are shown respectively, but they may be multiple.

[0034] The server 10 is a computer system such as a server computer, for example, Figure 1 As shown, it includes a processor 10a, a storage unit 10b and a communication unit 10c.

[0035] The processor 10a is, for example, a program-controlled device such as a microprocessor that operates according to a program installed in the server 10. The storage unit 10b is, for example, a storage element such as a ROM, a RAM, a solid-state drive (SSD), etc. The storage unit 10b stores programs executed by the processor 10a. The communication unit 10c is, for example, a communication interface for wired communication or wireless communication, and transmits and receives data to and from the user terminal 12 via the computer network 14.

[0036] The user terminal 12 is a computer for user operation, such as a mobile phone (including a smart phone), a portable information terminal (including a tablet computer), a personal computer, etc. Figure 1 As shown, the user terminal 12 includes, for example, a processor 12a, a storage unit 12b, a communication unit 12c, an operation unit 12d, a display unit 12e, and a camera unit 12f.

[0037] The processor 12a is, for example, a program-controlled device such as a microprocessor that operates according to a program installed in the user terminal 12. The storage unit 12b is, for example, a storage element such as a ROM, a RAM, a solid-state drive (SSD), etc. The storage unit 12b stores programs executed by the processor 12a. The communication unit 12c is, for example, a communication interface for wired communication or wireless communication, and transmits and receives data to and from the server 10 via the computer network 14.

[0038] The operating unit 12d is an input device, such as a touch panel, a pointing device such as a mouse, a keyboard, etc. The operating unit 12d transmits the operation content to the processor 12a. The display unit 12e is, for example, a liquid crystal display unit or an organic EL display unit. The shooting unit 12f includes at least one camera, such as a CMOS image sensor or a CCD image sensor. The shooting unit 12f shoots a static image or a dynamic image to generate image data. In addition, in this embodiment, the case where the shooting unit 12f is included in the user terminal 12 is described, but the shooting unit 12f can also be located outside the user terminal 12.

[0039] In addition, the programs and data described as being stored in the storage units 10b and 12b can also be provided from other computers via the network. In addition, the hardware structure of the server 10 and the user terminal 12 is not limited to the above-mentioned examples, and various hardware can be applied. For example, the server 10 and the user terminal 12 may also include a reading unit for reading a computer-readable information storage medium (for example, an optical drive, a memory card slot), and an input / output unit for inputting / outputting data with an external device (for example, a USB port). For example, the programs and data stored in the information storage medium can also be provided to the server 10 and the user terminal 12 via the reading unit and the input / output unit.

[0040] In this embodiment, in order to open a bank account, sign an insurance contract, etc. via the Internet, the user captures an image of an object such as a person identification document using the imaging unit 12 f and uploads the captured image to the server 10 .

[0041] The identity verification document can be any document that can verify the user, such as a driver's license, insurance card, resident card, or passport. In the following description, it is assumed that the identity verification document of this embodiment is a driver's license. In addition, driver's licenses have various formats according to each country and region, but for the convenience of description, a driver's license in a virtual format is cited as an example.

[0042] Figure 2 2 is a diagram showing a situation where a driver's license 20 is photographed. Figure 2 As shown, for example, the user uses the camera 12f of the user terminal 12 to photograph the driver's license 20 placed on the table. In this embodiment, the user photographs the driver's license 20 from the substantially front (directly above) side. Figure 3 The front image 22 shown is taken into the imaging unit 12f. In addition, the user can capture the driver's license 20 from an oblique direction. Figure 4 The oblique image 24 shown is captured by the imaging unit 12f. In the present embodiment, the resolution of the front image 22 and the oblique image 24 is ensured to a level that allows optical character recognition, and the imaging unit 12f is also focused on the driver's license 20.

[0043] Then, the user terminal 12 of the present embodiment uploads the front image 22 and the oblique image 24 captured by the imaging unit 12 f to the server 10 .

[0044] Then, the server 10 performs electronic identity verification (electronic Know Your Customer (eKYC)) using the uploaded front image 22 and tilted image 24 .

[0045] For example, the server 10 performs optical character recognition on the front image 22 to extract characters such as the name, address, and date of birth printed on the object. Figure 5 It is shown from Figure 3 FIG. 2 is a diagram showing an example of extracted character data of characters extracted from the front image 22 shown.

[0046] Furthermore, the server 10 uses the learned machine learning model to determine whether the driver's license 20 captured in the oblique image 24 has a certain thickness, and here, for example, determines whether it is thicker than a predetermined thickness.

[0047] Furthermore, the server 10 compares the front image 22 and the oblique image 24 to determine whether the object appearing in the front image 22 is the same as the object appearing in the oblique image 24 .

[0048] In electronic identity verification, a user may provide an illegal image such as an image obtained by photographing or scanning a copy of the driver's license 20 or a forged or tampered image instead of the original image of the driver's license 20. In such a case, in electronic identity verification, the user may not find that the provided image is an illegal image.

[0049] In the present embodiment, electronic identity verification is performed using the front image 22 and the oblique image 24 obtained by photographing the same object, thereby enabling strict electronic identity verification.

[0050] Hereinafter, the functions of the server 10 and the processing executed in the server 10 according to the present embodiment will be further described.

[0051] Figure 6 1 is a functional block diagram showing an example of functions installed in the server 10 of the present embodiment. Figure 6 All the features shown, in addition, can also be installed in addition to Figure 6 Functions other than those shown.

[0052] like Figure 6 As shown, in the server 10 of this embodiment, the functions include, for example, an image acquisition unit 30, a transformed image generation unit 32, a sample image storage unit 34, an object region determination unit 36, a comparison unit 38, a machine learning model 40, a thickness determination unit 42, and a character recognition unit 44. The image acquisition unit 30 is mainly implemented based on the communication unit 10c. The transformed image generation unit 32, the object region determination unit 36, the comparison unit 38, the thickness determination unit 42, and the character recognition unit 44 are mainly implemented based on the processor 10a. The sample image storage unit 34 is mainly implemented based on the storage unit 10b. The machine learning model 40 is mainly implemented based on the processor 10a and the storage unit 10b.

[0053] The above functions can also be implemented by executing a program containing instructions corresponding to the above functions installed in the server 10 as a computer in the processor 10a. In addition, the program can also be provided to the server 10 via a computer-readable information storage medium such as an optical disk, a magnetic disk, a magnetic tape, a magneto-optical disk, or via the Internet.

[0054] In the present embodiment, the image acquisition unit 30 acquires, for example, a front image 22, which represents a situation where a recording surface on which a document is recorded in an object such as a person identification document is viewed from the front direction. The image acquisition unit 30 acquires, for example, a front image 22 sent from the user terminal 12. In addition, the front image 22 of the present embodiment does not need to represent a situation where the recording surface is viewed strictly from the front, and it only needs to represent a situation where the recording surface is viewed from approximately the front to the extent that optical character recognition can be performed.

[0055] In addition, the document of this embodiment has a predetermined format and a predetermined layout. That is, it is assumed that what information is drawn where in the document is known in advance.

[0056] In addition, the document of the present embodiment includes a fixed-form portion and a non-fixed-form portion. The fixed-form portion refers to a portion whose content is fixed and is a portion whose content is shared with other documents. In other words, the fixed-form portion is a portion whose content does not change regardless of the document, and is a portion whose content does not change regardless of the user. For example, the fixed-form portion is a format portion in a document, and is a portion that depicts specific characters, symbols, graphics, frame lines, illustrations, or images. The fixed-form portion can also be referred to as a portion that contains information inherent to the document.

[0057] in the case of Figure 3 For example, a title such as "DRIVER LICENSE" is an example of a fixed-form portion. In addition, item names such as "NAME", "BIRTH DAY", "ADDRESS", "DATE", "EXPIRES" and "Number" are examples of fixed-form portions. In addition, a country name such as "Japan" is an example of a fixed-form portion. In addition, an organization name such as "Tokyo Metropolitan Public Safety Commission" is an example of a fixed-form portion. In addition, fixed-form portions are not limited to the characters described above. Figure 3 The image showing the Japanese flag in the driver's license 20 is also an example of a fixed format portion. In addition, the frame surrounding the above-mentioned item name and the like is also an example of a fixed format portion.

[0058] The non-fixed form part refers to a part whose content is not fixed and is not shared with other documents. In other words, the non-fixed form part is a part whose content changes for each document and for each user. For example, the non-fixed form part is a part of the document other than the format part and is a part where information such as user identification information, attributes and other personal information is printed. The non-fixed form part can also be called a part containing information inherent to the user.

[0059] exist Figure 3 In the example, a name such as “YAMADA TARO” is an example of a non-fixed form portion. Furthermore, a date of birth such as “June 23, 1980” is an example of a non-fixed form portion. Furthermore, an address such as “1-2-3ABCCity Tokyo” is an example of a non-fixed form portion. Furthermore, an issue date such as “July 25, 2015” is an example of a non-fixed form portion. Furthermore, an expiration date such as “July 25, 2020” is an example of a non-fixed form portion. Furthermore, a driver’s license number such as “1234 5678 9012” is an example of a non-fixed form portion. In addition, the non-fixed form portion is not limited to the characters described above, Figure 3 The facial photograph of the user in the driver's license 20 is also an example of a non-fixed form part. In addition, if the driver's license 20 contains information indicating the ID or physical characteristics of the user, such information is also an example of a non-fixed form part.

[0060] In the present embodiment, the image acquisition unit 30 acquires, for example, an oblique image 24 showing a state where the recording surface is viewed from an oblique direction. The image acquisition unit 30 acquires, for example, the oblique image 24 transmitted from the user terminal 12 .

[0061] In the present embodiment, for example, the transformed image generating unit 32 generates a Figure 7 The transformed image 50 shown in the figure shows a state where the object appearing in the oblique image 24 is viewed from the front direction. The transformed image generation unit 32 generates the oblique image 24 by transforming the oblique image 24 based on the features extracted from at least a part of the front image 22 and the features extracted from at least a part of the oblique image 24, for example. Figure 7 The transformed image 50 is shown.

[0062] Here, the extracted features are, for example, image feature quantities extracted by using algorithms such as SIFT, SURF, or A-KAZE installed in OpenCV, including position coordinates of each feature point among a plurality of feature points and the feature quantity of the feature point. The feature quantity is, for example, a numerical value output from the above-mentioned algorithm, which is a numerical value obtained by quantifying the features of the local shape and color of the object.

[0063] The transformed image generation unit 32 extracts feature point groups from the entire front image 22 and the entire oblique image 24, for example, by using the above-mentioned algorithm. The transformed image generation unit 32 extracts, for example, several tens to several thousand or more feature points from each of the front image 22 and the oblique image 24.

[0064] Then, the transformed image generation unit 32 matches the feature point group by determining the feature point in the oblique image 24 corresponding to each feature point in the front image 22. When matching, it is sufficient to establish an association between feature points with similar feature quantities. In addition, similar feature quantities refer to similar values ​​of feature quantities, which means that the difference in feature quantities is small (for example, the difference is the smallest). Then, through this matching, the feature points in the front image 22 are associated with the feature points in the oblique image 24.

[0065] Then, the transformed image generation unit 32 calculates the transformation matrix based on the matching result of the feature point group. The transformation matrix is ​​calculated in such a way that the positions of the feature points in the oblique image 24 are close to the positions of the feature points of the matching object in the frontal image 22. The method of obtaining the transformation matrix itself can apply various methods, for example, the calculation formula of the transformation matrix in affine transformation, linear transformation or projective transformation can also be used.

[0066] Then, the transformed image generation unit 32 generates a transformed image 50 by transforming the oblique image 24 according to the transformation matrix. Figure 7 As shown, the document appearing in the transformed image 50 becomes the same as that in Figure 3 The document appears in a substantially similar state in the front side image 22 shown.

[0067] Furthermore, the feature point group may be extracted from the entire image or only from a partial area.

[0068] Here, the case of using a feature point group is taken as an example, but the transformed image generating unit 32 may transform the oblique image 24 based on information that may be a feature of the image, and may use information other than the feature point group.

[0069] In addition, the transformation method of the oblique image 24 is not limited to affine transformation, linear transformation or projection transformation. In the transformation of the oblique image 24, rotation, enlargement / reduction or movement may also be used. In addition, a combination of part or all of affine transformation, linear transformation, projection transformation, rotation, enlargement / reduction and movement may also be used.

[0070] In this embodiment, the sample image storage unit 34 stores, for example, Figure 8 The sample image 52 shown. The sample image 52 of the present embodiment refers to an image in which there is no or basically no deformation or curvature of the document recorded on the recording surface. In other words, the sample image 52 is an image of the document taken in from the front direction or the approximately front direction. The front direction is a direction in which the angle formed with the recording surface of the document is perpendicular, and can also be called directly opposite. The approximately front direction is a direction in which the angle can be regarded as perpendicular, for example, a direction in which the angle is greater than 80°. The format of the sample image 52 is the same as the format of the front image 22. Therefore, the fixed form part of the sample image 52 is the same as the fixed form part of the front image 22, and the non-fixed form part of the sample image 52 is different from the non-fixed form part of the taken-in image. In addition, the sample image 52 may not include the non-fixed form part. That is, the sample image 52 may be only the format part.

[0071] like Figure 8 As shown, the sample image 52 is maintained in the shape of a driver's license 54 with rounded corners, and is in a state of no or almost no deformation. In addition, the direction of the driver's license 54 is not offset, and is in a state of no or almost no bending. Therefore, the characters of the sample image 52 are not deformed or bent, and can be said to be in a state suitable for optical character recognition. For example, the sample image 52 is prepared in advance by the administrator of the image processing system 1. For example, the administrator takes in an object with a document recorded on it through an image reading device such as a scanner or a shooting device and generates a sample image 52, which is registered in the sample image storage unit 34.

[0072] In addition, the background of the sample image 52 is preferably a single color such as black or white.

[0073] In addition, the sample image 52 of the present embodiment is an image in which a given object (here, for example, a driver's license 54) appears in a known area. Hereinafter, this area is referred to as a sample object area R1. That is, the position, shape, and size of the sample object area R1 in the sample image 52 are known in advance. In addition, the sample object area R1 of the present embodiment is a rectangular area surrounding the driver's license 54 that appears in the sample image 52. Then, the sample object area data representing the sample object area R1 is pre-stored in the sample image storage unit 34. The object area data is, for example, data representing the coordinate values ​​of the vertices of the sample object area R1 (here, for example, the four vertices P1, P2, P3, and P4).

[0074] In the present embodiment, the object region identifying unit 36 ​​identifies, for example, a region where the object appears in the front image 22. Fig. 9 The object region determination unit 36 ​​determines the object region R2 based on the features extracted from the fixed format portion of the document recorded in the object in the sample image 52 and the features extracted from at least a portion of the front image 22, for example. Fig. 9 The front object region R2 is determined as shown.

[0075] Here, as described above, the extracted features are, for example, image feature quantities extracted by using algorithms such as SIFT, SURF, or A-KAZE installed in OpenCV. The extracted features include the position coordinates of each feature point and the feature quantity of the feature point with respect to a plurality of feature points. The feature quantity is, for example, a numerical value output from the above-mentioned algorithm, which is a numerical value obtained by quantifying the features of the local shape and color of an object.

[0076] The object region identification unit 36 ​​matches the feature point group by, for example, identifying the feature point in the front image 22 corresponding to each feature point of the fixed form part extracted from the sample image 52. Then, through this matching, the feature point in the fixed form part of the sample image 52 is associated with the feature point in the front image 22.

[0077] Then, the object region determination unit 36 ​​determines the front object region R2 in the front image 22 corresponding to the sample object region R1 based on the matching result and the object region data stored in the sample image storage unit 34. For example, the object region determination unit 36 ​​determines the vertices (in Fig. 9In the example, the coordinate values ​​of the four vertices P5, P6, P7 and P8). In addition, the quadrilateral front object area R2 of the present embodiment may not be a rectangle. For example, when the front image 22 does not represent the situation where the recording surface is strictly observed from the front, the front object area R2 may not be a rectangle.

[0078] In the present embodiment, the object region identification unit 36 ​​identifies, for example, a region ( Fig.10 In the present embodiment, for example, by arranging the front image 22 and the transformed image 50 in the same coordinate system, the transformed object region R3 in the transformed image 50 corresponding to the front object region R2 in the front image 22 can be determined. Here, for example, the vertices of the transformed object region R3 (in Fig.10 In the example, the coordinate values ​​of the four vertices P9, P10, P11 and P12. In addition, similar to the front object region R2, the quadrilateral transformation object region R3 of this embodiment may not be a rectangle.

[0079] In the present embodiment, the comparison unit 38 determines whether the object appearing in the oblique image 24 is the same as the object appearing in the front image 22 by comparing the front object region R2 in the front image 22 and the conversion object region R3 in the conversion image 50 , for example.

[0080] In the present embodiment, for example, the shape and size of the front object region R2 are identical to the shape and size of the conversion object region R3, and for each pixel in the front object region R2, the pixel in the conversion object region R3 corresponding to the pixel can be determined. Then, for example, for each pixel in the front object region R2, the difference between the pixel value of the pixel and the pixel value of the pixel in the conversion object region R3 corresponding to the pixel can be calculated. Then, the total of the differences in the pixel values ​​calculated for each pixel in the front object region R2 can be calculated.

[0081] Then, when the calculated total is smaller than a predetermined value, the comparison unit 38 determines that the object appearing in the oblique image 24 is the same as the object appearing in the frontal image 22. Otherwise, it is determined that the object appearing in the oblique image 24 is different from the object appearing in the frontal image 22.

[0082] In addition, in the present embodiment, the comparison unit 38 may perform comparison (face authentication) between the image of the user's face appearing in the front image 22 and the image of the user's face appearing in the transformed image 50 in addition to the texture matching as described above. Then, when the total of the calculated pixel value differences is smaller than a predetermined value and the face authentication has succeeded, it may be determined that the object appearing in the oblique image 24 is the same as the object appearing in the front image 22. Then, when the total of the calculated pixel value differences is not smaller than a predetermined value or the face authentication has failed, it may be determined that the object appearing in the oblique image 24 is different from the object appearing in the front image 22.

[0083] Here, when determining that the object appearing in the oblique image 24 is different from the object appearing in the frontal image 22 , the comparison unit 38 may notify the user of a request to re-upload the frontal image 22 and the oblique image 24 .

[0084] In this embodiment, the machine learning model 40 is, for example, a machine learning model such as a CNN (Convolutional Neural Network). In this embodiment, for example, it is assumed that the machine learning model 40 is learned in advance by the manager of the image processing system 1. Fig.11 As shown, for example, in this embodiment, a plurality of training data including learning input images and teacher data are prepared in advance. The learning input image is, for example, an image obtained by photographing an original object such as a driver's license from an oblique direction or an image obtained by photographing a thinner paper from an oblique direction, or an image obtained by photographing various objects from an oblique direction.

[0085] Then, for example, a learning input image obtained by photographing the original of the object is associated with teacher data indicating a positive example (for example, teacher data with a value of 1). On the other hand, a learning input image obtained by photographing an object that is not the original of the object is associated with teacher data indicating a negative example (for example, teacher data with a value of 0). In this way, a plurality of training data including learning input images and teacher data can be generated, and the teacher data is associated with the learning input image.

[0086] Then, the learning of the machine learning model 40 is performed using the output data which is the output when the learning input image included in the training data is input to the machine learning model 40. Here, for example, the difference between the output data which is the output when the learning input image included in the training data is input to the machine learning model 40 and the teacher data included in the learning data can also be determined. Then, the following teacher learning can also be performed: based on the determined difference, the value of the parameter of the machine learning model 40 is updated using a method such as the error back propagation method (back propagation).

[0087] In addition, the machine learning model 40 does not need to be a classification model as described above, and may be a regression model. In this case, the values ​​of the teacher data and the output data may be values ​​representing the thickness of the object itself.

[0088] In the present embodiment, for example, the thickness determination unit 42 determines whether the thickness of the object appearing in the oblique image 24 is thicker than a predetermined thickness based on the oblique image 24. Here, for example, when it is determined that the object appearing in the oblique image 24 is the same as the object appearing in the frontal image 22, the determination by the thickness determination unit 42 may be performed.

[0089] Here, for example, the machine learning model 40 is the above-mentioned classification model. In this case, when the output when the oblique image 24 is input to the learned machine learning model 40 is "1", it is determined that the thickness of the object appearing in the oblique image 24 is thicker than the prescribed thickness. Then, when the output when the oblique image 24 is input to the learned machine learning model 40 is "0", it is determined that the thickness of the object appearing in the oblique image 24 is thinner than the prescribed thickness.

[0090] In addition, for example, the machine learning model 40 is the above-mentioned regression model. In this case, when the output when the oblique image 24 is input to the learned machine learning model 40 is greater than a predetermined value, it is determined that the thickness of the object appearing in the oblique image 24 is thicker than the predetermined thickness. Then, when the output when the oblique image 24 is input to the learned machine learning model 40 is less than the predetermined value, it is determined that the thickness of the object appearing in the oblique image 24 is thinner than the predetermined thickness.

[0091] Here, when determining that the thickness of the object appearing in the oblique image 24 is thinner than a predetermined thickness, the thickness determination unit 42 may notify the user of a request to re-upload the front image 22 and the oblique image 24 .

[0092] In the present embodiment, the character recognition unit 44 recognizes characters included in the document written in the front object region R2 by, for example, optical character recognition or the like. The character recognition unit 44 generates, for example, Figure 5 Extract character data as shown.

[0093] Here, refer to Fig.12 An example of the flow of processing performed in the server 10 of this embodiment is described with reference to the flowchart shown.

[0094] First, the image acquisition unit 30 receives the image from the user terminal 12 Figure 3 The front image 22 and Figure 4 The tilted image 24 (S101) is shown.

[0095] Then, the transformed image generating unit 32 generates a frontal image 22 and an oblique image 24 by transforming the oblique image 24 based on the frontal image 22 and the oblique image 24 received in the process shown in S101. Figure 7 The transformed image 50 is shown (S102).

[0096] Then, the object region identification unit 36 Figure 8 The sample image 52 shown, the object region data, and the front image 22 received in the process shown in S101 are used to specify the front object region R2 in the front image 22 (S103).

[0097] Then, the object region identification unit 36 ​​identifies the transformation object region R3 in the transformation image 50 generated in the process shown in S102 , which corresponds to the front object region R2 identified in the process shown in S103 ( S104 ).

[0098] Then, the comparison unit 38 performs a comparison process to determine whether the object appearing in the oblique image 24 received in the process shown in S101 is the same as the object appearing in the front image 22 received in the process shown in S101 (S105). Here, for example, the determination is performed based on the pixel value of each pixel in the front object region R2 determined in the process shown in S103 and the pixel value of each pixel in the conversion object region R3 determined in the process shown in S104.

[0099] Then, the thickness determination unit 42 performs a thickness determination process of determining whether the thickness of the object appearing in the oblique image 24 is thicker than a predetermined thickness based on the oblique image 24 received in the process shown in S101 ( S106 ).

[0100] Then, the character recognition unit 44 performs character recognition processing to generate Figure 5 In the character recognition process, the extracted character data is recognized by optical character recognition or the like, and the characters included in the document written in the front object region R2 specified in the process shown in S103 are recognized (S107). Then, the process shown in this process example ends.

[0101] exist Fig.12 In the process shown, as shown in S102 and S103, after the transformed image generation unit 32 generates the transformed image 50, the object region determination unit 36 ​​determines the front object region R2. Here, the order of the process shown in S102 and the process shown in S103 may be reversed, and the transformed image generation unit 32 may generate the transformed image 50 after the object region determination unit 36 ​​determines the front object region R2.

[0102] Then, in this case, the transformed image generation unit 32 may generate the transformed image 50 by transforming the oblique image 24 based on the features extracted from the front object region R2 determined by the object region determination unit 36 ​​and the features extracted from at least a part of the oblique image 24. In this way, since the number of feature points used for matching is reduced compared to the case where feature points are extracted from the entire front image 22, it is possible to obtain the transformed image 50 with less effort. Fig.12 The process shown in S102 generates the transformed image 50 with a light processing load.

[0103] In the present embodiment, as described above, in the electronic identity verification, it is confirmed that the object appearing in the oblique image 24 has a certain degree of thickness and that the object appearing in the front image 22 is the same as the object appearing in the oblique image 24. Thus, according to the present embodiment, it is possible to strictly confirm that the user holds the original of the object, and as a result, the electronic identity verification can be performed more strictly.

[0104] In addition, the background, the size of the area where the object appears, the illumination method of the ambient light, the color and brightness, etc. are different from image to image. For example, the background when the front image 22 is taken is sometimes different from the background when the oblique image 24 is taken. Therefore, by simply comparing the front image 22 and the oblique image 24, it is impossible to accurately determine whether the object appearing in the front image 22 is the same as the object appearing in the oblique image 24.

[0105] In this embodiment, since the front object region R2 and the conversion object region R3 determined based on the sample image 52 are compared as described above, it is possible to accurately determine whether the object appearing in the front image 22 is the same as the object appearing in the oblique image 24 .

[0106] Furthermore, the present invention is not limited to the above-mentioned embodiments.

[0107] For example, the role sharing between the server 10 and the user terminal 12 is not limited to the above-described content. Figure 6 Some or all of the functions shown may be implemented in the user terminal 12 .

[0108] In addition, the front image 22 does not need to be captured by the imaging unit 12f, but may be read by a scanner.

[0109] In addition, the specific character strings and numerical values ​​described above and the specific character strings and numerical values ​​in the drawings are merely examples, and the present invention is not limited to these character strings and numerical values.

Claims

1. A control system, characterized in that: The control system contains: a front-side image acquisition unit that acquires a front-side image showing a state where a recording surface on which a document is recorded in the object is viewed from the front direction; an oblique image acquisition unit that acquires an oblique image showing a state in which the recording surface is viewed from an oblique direction; a transformed image generating unit that transforms the oblique image based on features extracted from at least a portion of the frontal image and features extracted from at least a portion of the oblique image, thereby generating a transformed image representing a state in which the object appearing in the oblique image is viewed from a frontal direction; an area determination unit for determining an object area in which a given object appears in the front image based on features extracted from a fixed-form portion of a document recorded on the object in a sample image in which the object appears in a known area and features extracted from at least a portion of the front image; as well as A comparison unit determines whether the object appearing in the oblique image is the same as the object appearing in the frontal image by comparing the object region in the frontal image with a region in the transformed image corresponding to the object region.

2. The control system according to claim 1, characterized in that The region specifying unit specifies the object region after the transformed image is generated by the transformed image generating unit.

3. The control system according to claim 1, characterized in that: The transformed image generating unit generates the transformed image after the object region is determined by the region determining unit.

4. The control system according to claim 3, characterized in that: The transformed image generating unit generates the transformed image by transforming the oblique image based on features extracted from the object region determined by the region determining unit and features extracted from at least a portion of the oblique image.

5. The control system according to claim 1, characterized in that: The comparison system further includes a thickness determination unit configured to determine, based on the oblique image, whether a thickness of an object appearing in the oblique image is thicker than a predetermined thickness.

6. The control system according to claim 1, characterized in that: The matching system further includes a character recognition unit that recognizes characters included in the document written in the target object area.

7. A control method, characterized in that The control method comprises the following steps: Acquire a front image, the front image showing a state where a recording surface of a document recorded in an object is viewed from the front direction; obtaining an oblique image, the oblique image showing a situation of observing the recording surface from an oblique direction; transforming the oblique image based on features extracted from at least a portion of the frontal image and features extracted from at least a portion of the oblique image, thereby generating a transformed image representing a situation in which the object appearing in the oblique image is observed from a frontal direction; Determine an object region in which a given object appears in the front image based on features extracted from a fixed-format portion of a document describing the object in a sample image in which the object appears in a known region and features extracted from at least a portion of the front image; as well as By comparing the object region in the front image with the region in the transformed image corresponding to the object region, it is determined whether the object appearing in the oblique image is the same as the object appearing in the front image.

8. An information storage medium storing a program, wherein the program causes a computer to execute the following process: Acquire a front image, the front image showing a state where a recording surface of a document recorded in an object is viewed from the front direction; obtaining an oblique image, the oblique image showing a situation of observing the recording surface from an oblique direction; transforming the oblique image based on features extracted from at least a portion of the frontal image and features extracted from at least a portion of the oblique image, thereby generating a transformed image representing a situation in which the object appearing in the oblique image is observed from a frontal direction; Determining an object region in which a given object appears in the frontal image based on features extracted from a fixed-form portion of a document containing the object in a sample image in which the object appears in a known region and features extracted from at least a portion of the frontal image; and By comparing the object region in the front image with the region in the transformed image corresponding to the object region, it is determined whether the object appearing in the oblique image is the same as the object appearing in the front image.

Images