Methods, systems, and apparatus for detecting photo substitution in photo identification documents

Abstract

Computer-implemented methods, systems, and apparatuses for detecting photo substitution in photo identity documents, including computer programs encoded on computer storage media for implementing methods for detecting photo substitution in photo identity documents, are disclosed. One of the methods includes receiving image data of a cropped photo region of an ID, and predicting an ID category based on the image data of the cropped photo region by a multi-category classification model. The ID includes a photo, and the cropped photo region includes at least the photo. The predicted ID category corresponds to one category in a predetermined set of categories in the multi-category classification model. The method further includes comparing the predicted ID category to an indicated category of the ID, and in response to the predicted ID category not matching the indicated category of the ID, determining that the cropped photo region includes a substituted photo.

Description

Technical Field

[0001] This article relates generally, but not exclusively, to methods and systems for detecting photo replacement in photo ID documents. Background Technology

[0002] Electronic Know Your Customer (eKYC) is a digital due diligence process performed electronically by businesses to verify the authenticity of their customers and assess the potential risks of unauthorized intent in business relationships. To complete the eKYC process, individuals may need to submit an image of a government-approved photo ID to prove their identity. In this context, identity fraud could occur if fraudsters use another person's personal information without authorization to commit fraud or other crimes.

[0003] One of the most common forms of identity fraud involving photo IDs is replacing the photo of the ID card holder with another person's. Artificial intelligence (AI) technology has been used to detect this type of fraud. Traditional methods for detecting photo replacement in ID cards involve training a binary classification model that outputs two categories: the real ID and the fake ID with the replaced photo, treating it as a binary classification task. However, the data labeling work involved in training the model is quite time-consuming. Given an ID card in card form, creating a fake ID sample requires at least four steps: obtaining the ID card, printing another person's photo, placing the printed photo on top of the original ID card, taking a picture of the synthesized fake ID image, and saving it for training. To obtain a highly accurate model, at least several thousand samples are needed, which can be very time-consuming and cannot be completed in a short period. Summary of the Invention

[0004] The described embodiments provide methods, apparatus, and systems for detecting photo replacement in photo identification documents (IDs). In some embodiments, the method can train a multi-class classification model based on samples of cropped photo regions of the ID, each class corresponding to a category of the ID (e.g., driver's license from country A, ID card from country B, passport from country C, etc.). Samples can be obtained by cropping photo regions from the image of the ID and saving them as new images. In some embodiments, the cropped photo regions may also include some background around the photo to comprehensively describe the style and characteristics of the photo regions of a particular category of ID. In some embodiments, convolutional neural networks (CNNs) can be used to train the classification model.

[0005] In some implementations, when an image of an ID is received, a photo region can be cropped from the image and used as input to a trained multi-class classification model. In other implementations, the photo region can be obtained by first preprocessing the ID image to generate an aligned image of the ID, and then cropping the photo region from the aligned image based on the ID category and the photo location information corresponding to that ID category. The model then predicts the ID category based on the cropped photo region. If a real ID is used, the predicted category of the ID should match the indicated ID category. Conversely, if a fake ID with a replacement photo is used, the predicted category of the ID should differ from the indicated category of the ID because the cropped photo region (including the replacement photo) looks different from the cropped photo region of the indicated ID category.

[0006] According to one embodiment, a method for detecting photo replacement in a photo ID is provided. The method includes: receiving image data of a cropped photo region of the ID, wherein the ID includes a photo, and the cropped photo region includes at least the photo; predicting an ID category based on the image data of the cropped photo region using a multi-class classification model, wherein the predicted ID category corresponds to a category in a predetermined category set in the multi-class classification model; comparing the predicted ID category with an indication category of the ID; and determining that the cropped photo region includes a replacement photo in response to a mismatch between the predicted ID category and the indication category of the ID.

[0007] In some implementations, users can enter the indicated category of the ID during the eKYC process or select an ID category from a list of ID categories. Possible ID categories may include national identity cards, passports, and driver's licenses from one or more countries. Accordingly, the multi-category classification model may have a predetermined set of categories representing different ID categories. In one implementation, when a cropped photo region of a new ID is received, the model can output a score for each category and determine the category with the highest score as the ID category based on the cropped photo region. In some implementations, a predetermined threshold (e.g., 95 out of 100) can be set for the highest score, such that if the highest score reaches the threshold, a conclusion is reached that no photo replacement is needed.

[0008] According to other embodiments, one or more of these general and specific embodiments may be implemented using an apparatus, system, method, or computer-readable medium, or any combination of an apparatus, system, method, and computer-readable medium, comprising multiple modules. The above and other described embodiments may optionally include some or all of the following embodiments, or may exclude the following embodiments. Attached Figure Description

[0009] The embodiments and implementations are provided by way of example only, and those skilled in the art will better understand and readily comprehend these embodiments and implementations by reading the following written description in conjunction with the accompanying drawings, in which:

[0010] Figure 1 This is a flowchart illustrating an example of a method for detecting photo replacement in a photo ID document according to an embodiment.

[0011] Figure 2 This illustrates an embodiment. Figure 1 A flowchart illustrating an example implementation of the method.

[0012] Figure 3 This is a schematic diagram of an example of an ID card alignment according to an embodiment.

[0013] Figure 4 This is a schematic diagram of a module example of an apparatus for detecting photo replacement in a photo ID according to an embodiment.

[0014] Figure 5 Suitable for execution according to the embodiments Figure 1 and Figure 2 A block diagram of a computer system example of at least some steps of the example method shown.

[0015] Those skilled in the art will understand that the elements in the accompanying drawings are shown for simplicity and clarity and are not necessarily drawn to scale. For example, the dimensions of some elements in the illustrations, block diagrams, or flowcharts may be exaggerated relative to other elements to aid in understanding the present embodiment. Detailed Implementation

[0016] The embodiments are described by way of example only with reference to the accompanying drawings. The same reference numerals and characters in the drawings denote the same elements or equivalents.

[0017] Some parts of the following description are presented explicitly or implicitly based on algorithms and functional or symbolic representations for manipulating data in computer memory. These algorithmic descriptions and functional or symbolic representations are means by which those skilled in the art of data processing most effectively communicate the essence of their work to others skilled in the art. Here, an algorithm is generally considered to be a self-consistent sequence of steps that leads to a desired result. These steps are steps that require physical manipulation of physical quantities such as electrical, magnetic, or optical signals that can be stored, transmitted, combined, compared, and otherwise manipulated.

[0018] Unless otherwise specifically stated and as will be apparent from the following, it should be understood that throughout this specification, discussions using terms such as “receive,” “obtain,” “determine,” “predict,” “calculate,” “train,” “match,” “generate,” “detect,” “classify,” “set,” “limit,” “compare,” “process,” “update,” “enter,” “select,” “authenticate,” “provide,” and “input” refer to the actions and processes of a computer system or similar electronic device in manipulating data represented as physical quantities within the computer system and converting it into other data similarly represented as physical quantities within the computer system or other information storage, transmission, or display devices.

[0019] This specification also discloses means for performing the methods. Such means may be specifically configured for a desired purpose, or may include a computer or other device selectively activated or reconfigured by a computer program stored in a computer. The algorithms and displays presented herein are not inherently related to any particular computer or other device. Various machines can be used with the program in accordance with the teachings herein. Alternatively, the construction of more specialized means for performing the desired method steps may be suitable. The following description will show the structure of a computer suitable for performing the various methods / processes described herein.

[0020] Furthermore, this specification implicitly discloses a computer program, as it will be apparent to those skilled in the art that the various steps of the methods described herein can be implemented via computer code. The computer program is not intended to be limited to any particular programming language or its implementation. It should be understood that various programming languages ​​and their encodings can be used to implement the teachings contained herein. Moreover, the computer program is not intended to be limited to any particular control flow. Many other variations of the computer program can be used with different control flows without departing from the scope of this document.

[0021] Furthermore, one or more steps of a computer program can be executed in parallel rather than sequentially. Such a computer program can be stored on any computer-readable medium. Computer-readable media can include storage devices such as disks or optical discs, memory chips, or other storage devices suitable for interfacing with a computer. Computer-readable media can also include hardwired media, such as those exemplified in Internet systems, or wireless media, such as those exemplified in GSM mobile phone systems. When a computer program is loaded onto and executed on such a computer, it effectively implements the means for carrying out the steps of the preferred method.

[0022] This document can also be implemented as a hardware module. More specifically, in a hardware sense, a module is a functional hardware unit designed to be used in conjunction with other components or modules. For example, a module can be implemented using discrete electronic components, or it can be part of an entire electronic circuit (e.g., an application-specific integrated circuit (ASIC) or a field-programmable gate array (FPGA)). Many other possibilities exist. Those skilled in the art will understand that the system can also be implemented as a combination of hardware and software modules.

[0023] Photo replacement detection during the process of verifying the authenticity of identity documents can be considered a form of fraud detection or fake identity detection, where the legitimacy of the user is verified and potential fraudsters are detected before any fraudulent activity is carried out. Effective identity authentication can enhance the data security of a system by allowing only authenticated users access to their protected resources. Examples seek to provide methods and systems for detecting photo replacement in photo IDs, thereby detecting fake ID images uploaded by fraudsters. Advantageously, financial risks such as money laundering and fraud can be effectively reduced or eliminated.

[0024] The techniques described herein produce one or more technical effects. A method for detecting photo replacement using a multi-class classification model is provided. In an implementation, the multi-class classification model may include predetermined groups of categories corresponding to various ID categories. Unlike binary classification techniques, which categorize IDs into real IDs and fake IDs (i.e., IDs with replaced photos) and involve centralized data annotation work, training a multi-class classification model does not require data annotation of real / fake IDs. Furthermore, the multi-class classification model does not require fake ID samples for training, which advantageously eliminates the effort of generating a large number of fake ID samples to achieve high accuracy.

[0025] Figure 1 This is a flowchart 100 illustrating an example method for detecting photo replacement in a photo ID, the method comprising the following steps:

[0026] -110: Receive image data of a cropped photo region of ID, wherein ID includes a photo, and the cropped photo region includes at least the photo;

[0027] -120: Predict ID category based on image data of cropped photo regions using a multi-class classification model, where the predicted ID category corresponds to one of the predetermined category groups in the multi-class classification model;

[0028] -130: Predict ID category based on image data of cropped photo regions using a multi-class classification model;

[0029] -135: Determines whether the predicted ID category matches the ID's indicated category; and

[0030] -150: In response to a mismatch between the predicted ID category and the ID's indicated category, determine that the cropped photo area includes the replacement photo.

[0031] In step 110, image data of a cropped photo region of the photo ID is received. The photo ID can be in the form of a card (e.g., a national ID card or driver's license), document (e.g., a passport or birth certificate), etc. The cropped photo region includes at least the photo on the photo ID. In some embodiments, the area of ​​the cropped photo region can be larger than the size of the photo, thereby additionally including some background around the photo. Advantageously, certain characteristics of a specific category of ID can be included in the cropped photo region, which can well describe the style of the photo region and provide useful information in the classification process. These characteristics can include colors and patterns in the photo region, security features, background color of the photo, etc. Depending on the category of ID, some IDs may also have text around the photo, such as the ID owner's printed name, gender, date of birth, country of birth, issuing country, or ID number. Some IDs may have images around the photo, such as the ID owner's signature, the coat of arms of the issuing country, or a smaller version of the photo. Because the characteristics and style of IDs are unique and difficult to forge, including these features in the cropped photo region helps to effectively detect photo substitution on the ID.

[0032] In order to include the background surrounding the photograph, in some implementations, it is assumed that the height of the photograph is [missing information]. H And the width is W The height of the cropped photo area can be determined as follows: H and R H *H Between, the width can be determined as W and R W *W between( R H , R W >1). For example, the height of the cropped photo area can be 1.2* H The width is 1.2* W Understandable, as long as the photo is included. R H and R W The predetermined ratios can be the same or different. The implementation method is not limited.

[0033] In step 120, based on the image data of the cropped photo region, an ID category is predicted using a multi-class classification model. This model may have a predetermined set of categories, each corresponding to a specific category of ID (e.g., Category 1 - Country B Registered ID Card, Category 15 - Country D Permanent ID Card, Category 30 - Country A Driver's License, Category 55 - Country E ID Card, Category 71 - Country F Passport, etc.). Based on the classification results, the predicted ID category will be one of the predetermined categories. In some implementations, predicting the ID category based on the image data of the cropped photo region includes: inputting the image data into the multi-class classification model and outputting a score for each category in the predetermined category set. The corresponding score can reflect the probability that the cropped photo region belongs to the corresponding ID category. The category with the highest score can then be determined as the predicted ID category. It is understood that this is one of many possible implementations for predicting ID categories and is not limited thereto.

[0034] In step 130, the predicted ID category is compared with the ID's indicator category. In some implementations, the user can enter or select the ID's indicator category during the authentication process (e.g., eKYC). Alternatively, the authentication system can instruct the user to upload an ID of a specific category, which will be the ID's indicator category. In step 135, the predicted ID category output by the model is compared with the ID's indicator category. If the predicted ID category does not match the ID's indicator category, the method proceeds to step 150 and determines that the cropped photo region includes a replacement photo. If the predicted ID category matches the ID's indicator category, the method can proceed to step 160 and determine that the photo in the cropped photo region is the actual photo on the photo ID.

[0035] Optionally, after determining that the predicted ID category matches the ID's indicated category, the method may include additional steps 140 and 145 before inferring whether the photo in the cropped photo region has been replaced. In some implementations, the additional steps involve pre-determining a threshold relative to the output score and comparing the highest score to the predetermined threshold. If the highest score is below the predetermined threshold, the method proceeds to step 150 and determines that the cropped photo region includes the replaced photo. If the highest score is equal to or higher than the predetermined threshold, the method may proceed to step 160 and determine that the photo in the cropped photo region is the real photo on the photo ID. These additional steps 140 and 145 are advantageous when a replaced photo is used and the multi-class classification model outputs low scores for all categories (e.g., ≤40 out of 100), meaning the probability that the cropped photo region belongs to any category ID is low. In this case, the ID category will still be predicted in step 120 by selecting the highest score (e.g., 40), and the predicted ID category is likely to match the ID's indicated category. With an additional step, in response to a predicted ID category matching the ID's indicated category, if the highest score (e.g., 40) is below a preset threshold (e.g., 95 out of 100), the method will still proceed to step 150 and determine that the cropped photo region includes the replacement photo. It is understood that setting and adjusting the threshold can reduce false positives in photo replacement detection.

[0036] This paper also provides methods, apparatus, and systems for training multi-class classification models using samples of ID-based cropped photo regions. Assuming that... N Samples of cropped photo regions from each category of photo ID can be used to train the image. N A classification model with multiple categories, each corresponding to a photo ID. In some embodiments, a convolutional neural network (CNN), one of the deep neural networks widely used for analyzing visual images, can be used to train the classification model. It is understood that multi-class classification models can be trained and developed efficiently without requiring any data labeling as needed in binary classification tasks. Furthermore, more image data of cropped photo regions can be collected when implementing the classification model to detect photo replacements for received IDs. Optionally, samples of cropped photo regions used to train the classification model can be updated by including new cropped photo regions containing the received IDs, which can enhance future training models and improve accuracy.

[0037] Figure 2 To show Figure 1 Example flowchart 200 for the implementation of the method. The left half of the flowchart points to training the multi-class classification model. At the beginning of the process, it provides... NAn image set similar to an identity card. The example shown is image 210 of a national permanent ID card (NID card). A cropped photo area 215 can be obtained by cropping from image 210. It can be understood that the cropped photo area 215 includes the photo as well as some background around the photo. Similarly, it can be obtained from... N Image set acquisition for photo ID N Samples of cropped photo regions. Then, a classification model is trained to classify them. N The photos are categorized by region.

[0038] The trained model can be used to detect photo replacements for received IDs. A cropped photo region 225 is obtained from the image 220 of the received ID. In this case, the received ID is a fake D-country ID, which includes a replacement photo from an E-country unified multipurpose ID card. By inputting the cropped photo region 225 into a classification model, the model predicts the ID category as E-country unified multipurpose ID, which differs from the indicated category of the D-country ID. Therefore, it is determined that the received ID is a fake ID with a replacement photo.

[0039] In this implementation, the received ID image 220 can be uploaded via a webpage or mobile application. The cropped photo area 225 can be obtained by cropping a photo area 225 from the ID image 220 based on the ID's indication category and the corresponding photo location information. For example, if the received ID is indicated as a country ID card (D), and the photo area on the country ID card can be within the range of (0.10*h) → 0.75*h, 0.05*w → (0.35*w), where (h, w) are the (height, width) of the ID card, then the photo location information can be used to crop the photo area from the image of the country ID card (D).

[0040] In some implementations, obtaining the cropped photo region may also include preprocessing the ID image to generate an aligned ID image. Figure 3 This is a schematic diagram 300 illustrating an ID card alignment example. As shown, an image 302 including an ID card 310 is received, and in this image, the ID card 310 is tilted. In the next step, the four corner points of the ID card 310 can be detected using machine learning techniques, such as a CNN regression model. By performing alignment, an aligned ID card image 310A can be generated. Then, the cropped photo region can be obtained by cropping the region based on photo location information of a specific ID category using the method described above. The technique of cropping a photo region from a photo ID can be used for the received ID and the image set of IDs used to train the model.

[0041] It is understood that training the model and using the model for photo replacement detection can be two separate processes, performed by the same party or different parties. Furthermore, photo replacement detection methods can be implemented individually or in combination with other authentication and verification methods. The implementation is not limited.

[0042] Figure 4 This is a schematic diagram of an example device 400, including a module for detecting photo replacement in a photo ID. Device 400 includes at least a receiving module 410, a classification module 420, a comparison module 430, and a determination module 440. (See reference...) Figure 1 and Figure 2 The receiving module 410 can be configured to receive image data of a cropped photo region of an ID and receive information about the indication category of an ID entered / selected by the user. The classification module 420 can be configured to predict the ID category based on the image data of the cropped photo region received by the receiving module 410. The comparison module 430 can be configured to compare the predicted ID category with the indication category of the ID and compare the score corresponding to the predicted ID category with a predetermined threshold. The determination module 440 can be configured to determine whether the cropped photo region includes a replacement photo based on whether the predicted ID category matches the indication category of the ID and whether the score is higher than the predetermined threshold. The apparatus 400 may also include a training module configured to train a multi-class classification model. The apparatus 400 may also include an ID image processing module configured to preprocess the received ID image, generate an aligned ID image, and a cropped photo region. The apparatus 400 may also include an output module configured to output the photo replacement detection result obtained by the determination module 440. One or more of these modules, or any combination thereof, may be part of an apparatus for detecting photo replacement in a photo ID.

[0043] The systems, apparatus, modules, or units shown in the preceding embodiments can be implemented using computer chips or entities, or by using products with specific functions. Typical embodiments of the devices are computers (and these computers can be personal computers), laptop computers, cellular phones, camera phones, smartphones, personal digital assistants, media players, navigation devices, email receiving and sending devices, game consoles, tablet computers, wearable devices, or any combination of these devices. Modules described as independent components may or may not be physically independent, and components shown as modules may or may not be physical modules, may be located in one location, or may be distributed across multiple network modules. Some or all modules may be selected according to actual needs to achieve the objectives of the solutions herein. Those skilled in the art will understand and implement the embodiments of this application without inventive effort.

[0044] Figure 5 It is applicable to execution Figure 1 and Figure 2 A block diagram of an exemplary computer system 500 illustrating at least some steps of the exemplary method shown. The following description of the computer system / computing device 500 is provided by way of example only and is not intended to be limiting.

[0045] like Figure 5 As shown, the example computing device 500 includes a processor 502 for executing software routines. Although a single processor is shown for clarity, the computing device 500 may also include a multiprocessor system. The processor 502 is connected to a communication infrastructure 506 for communicating with other components of the computing device 500. The communication infrastructure 506 may include, for example, a communication bus, crossbar, or network.

[0046] The computing device 500 also includes a main memory 504 such as random access memory (RAM) and an auxiliary memory 510. The auxiliary memory 510 may include, for example, a storage drive 512, which may be a hard disk drive, a solid-state drive, or a hybrid drive, and / or a removable storage drive 514, which may include a magnetic tape drive, an optical disc drive, a solid-state storage drive (e.g., a USB flash drive, a flash memory device, a solid-state drive, or a memory card), etc. The removable storage drive 514 reads from and / or writes to the removable storage medium 518 in a well-known manner. The removable storage medium 518 may include magnetic tape, optical disc, non-volatile memory storage media, etc., which are read from and written to by the removable storage drive 514. As understood by those skilled in the art, the removable storage medium 518 includes a computer-readable storage medium storing computer-executable program code instructions and / or data.

[0047] In an alternative implementation, the auxiliary storage 510 may additionally or alternatively include other similar means for allowing computer programs or other instructions to be loaded into the computing device 500. Such means may include, for example, removable storage unit 522 and interface 520. Examples of removable storage unit 522 and interface 520 include program cassettes and cassette interfaces (e.g., found in video game console devices), removable memory chips (e.g., EPROM or PROM) and associated receptacles, removable solid-state storage drives (e.g., USB flash drives, flash memory devices, solid-state drives, or memory cards), and other removable storage units 522 and interfaces 520 that allow software and data to be transferred from the removable storage unit 522 to the computer system 500.

[0048] The computing device 500 also includes at least one communication interface 524. The communication interface 524 allows the transfer of software and data between the computing device 500 and external devices via a communication path 526. In various embodiments herein, the communication interface 524 allows the transfer of data between the computing device 500 and a data communication network (e.g., a public or private data communication network). The communication interface 524 can be used to exchange data between different computing devices 500 that form part of an interconnected computer network. Examples of the communication interface 524 may include a modem, a network interface (e.g., an Ethernet card), a communication port (e.g., serial, parallel, printer, GPIB, IEEE 1394, RJ45, USB), an antenna with associated circuitry, etc. The communication interface 524 may be wired or wireless. The software and data transferred via the communication interface 524 are in the form of signals, which may be electronic, electromagnetic, optical, or other signals that can be received by the communication interface 524. These signals are provided to the communication interface via the communication path 526.

[0049] like Figure 5 As shown, the computing device 500 also includes a display interface 528 and an audio interface 532. The display interface 528 performs operations for presenting images to an associated display 530, and the audio interface 532 performs operations for playing audio content via an associated speaker 534.

[0050] As used herein, the term "computer program product" may refer in part to removable storage medium 518, removable storage unit 522, hard disk installed in storage drive 512, or carrier-borne software via communication path 526 (wireless link or cable) to communication interface 524. Computer-readable storage medium means any non-transitory, non-volatile tangible storage medium that provides recorded instructions and / or data to computing device 500 for execution and / or processing. Examples of such storage media include magnetic tape, CD-ROM, DVD, and Blu-ray. TM Disks, hard disk drives, ROMs or integrated circuits, solid-state storage drives (e.g., USB flash drives, flash devices, solid-state drives or memory cards), hybrid drives, magneto-optical disks, or computer-readable cards (e.g., PCMCIA cards), whether such devices are inside or outside the computing device 500. Examples of temporary or non-tangible computer-readable transmission media that may also be involved in providing software, applications, instructions and / or data to the computing device 500 include radio or infrared transmission channels and network connections to another computer or networked device, as well as the Internet or intranet including email transmissions and information recorded on websites.

[0051] Computer programs (also referred to as computer program code) are stored in main memory 504 and / or secondary memory 510. The computer programs can also be received via communication interface 524. When executed, these computer programs enable computing device 500 to perform one or more features of the embodiments discussed herein. In various embodiments, when executed, the computer programs enable processor 607 to perform the features of the embodiments described above. Therefore, such a computer program represents a controller of computer system 500.

[0052] The software can be stored in a computer program product and loaded into a computing device 500 using a removable storage drive 514, a storage drive 512, or an interface 520. The computer program product can be a non-transitory computer-readable medium. Alternatively, the computer program product can be downloaded to the computer system 500 via communication path 526. When executed by the processor 502, the software causes the computing device 500 to perform necessary operations to perform, such as... Figure 1 and Figure 2 The method shown.

[0053] It should be understood that Figure 5 The embodiments described herein are merely examples to illustrate the operation and structure of system 500. Therefore, in some embodiments, one or more features of computing device 500 may be omitted. Furthermore, in some embodiments, one or more features of computing device 500 may be combined together. Additionally, in some embodiments, one or more features of computing device 500 may be divided into one or more component parts.

[0054] It should be understood that Figure 5 The components shown are designed to provide means for performing various functions and operations of the system described in the above embodiments.

[0055] Those skilled in the art will understand that various changes and / or modifications can be made to this specification as shown in the specific embodiments without departing from the scope of the broadly described specification. Therefore, these embodiments are to be considered illustrative rather than restrictive in all respects.

Claims

1. A computer-implemented method for detecting photo substitution in photo identity documents, comprising: receiving image data of a cropped photo region of an identity document, wherein the identity document includes a photo and the cropped photo region includes at least the photo; predicting, by a multi-class classification model, an identity document class based on the image data of the cropped photo region, wherein the predicted identity document class corresponds to one of a predetermined class set in the multi-class classification model; comparing the predicted identity document class to an indicated class of the identity document; and in response to the predicted identity document class not matching the indicated class of the identity document, determining that the cropped photo region includes a substituted photo relative to a cropped photo region of an identity document corresponding to the indicated class.

2. The method of claim 1, wherein, the cropped photo region further includes a background surrounding the photo.

3. The method of claim 2, wherein: The photo has a height H and a width W , The cropped photo region has a height between H and R H *H and a width between W and R W *W R H and R W is a predetermined ratio greater than 1.​ 4. The method according to any of the preceding claims 1 - 3, wherein, receiving image data of a cropped photo region of an identity document, comprises: receiving an image of the identity document; preprocessing the image of the identity document to generate an aligned image of the identity document; and cropping a photo region from the aligned image of the identity document based on the indicated class of the identity document and photo location information corresponding to the indicated class of the identity document.

5. The method according to any of the preceding claims 1 - 3, wherein, predicting the identity document class based on the image data of the cropped photo region, comprises: inputting the image data into the multi-class classification model; outputting a score for each class in the predetermined class set; and determining the identity document class, wherein the identity document class corresponds to the class in the predetermined class set having the highest score.

6. The method of claim 5, further comprising: setting a threshold for the highest score; and in response to the predicted identity document class matching the indicated class of the identity document and the highest score being below the threshold, determining that the cropped photo region includes a substituted photo. the indicated class of the identity document is entered or selected by a user.

7. The method according to any of the preceding claims 1-3, wherein, the predetermined class set includes national identity documents, passports, or driver’s licenses from one or more countries.

8. The method according to any of the preceding claims 1 - 3, wherein, the multi-class classification model is trained by:

9. The method according to any of the preceding claims 1-3, wherein, obtaining a cropped photo region from the sample image; and Provided are N sample images of photo identification documents of the category; obtaining the cropped photo region from the sample image, comprises: preprocessing at least one of the sample images to generate an aligned image of the photo identity document; and training a class classification model based on the cropped photo region N a class classification model, wherein the N a class corresponds to the N a class photo identification document.

10. The method of claim 9, wherein, cropping a photo region from the aligned image of the photo identity document based on the class of the photo identity document and photo location information corresponding to the class of the photo identity document.

11. The method of claim 9, further comprising: updating the cropped photo region from the sample image used to train the classification model by including the received image data of the cropped photo region of the identity document. training the N-class classification model based on the cropped photo region is based on a convolutional neural network algorithm.

13. A computer-implemented system for detecting photo substitution in photo identity documents, comprising:

12. The method of claim 9, wherein, one or more processors; and ​ ​ ​ one or more computer-readable memories coupled to the one or more processors and having instructions stored thereon that are executed by the one or more processors to perform the method of any one of claims 1 to 12.

14. An apparatus for detecting photo substitution in a photo identification document, comprising a plurality of modules for performing the method of any one of claims 1 to 12.

Images