Techniques for matching different input data

Abstract

Disclosed are systems and methods for training a generative adversarial network (GAN) to transform one type of image (e.g., a selfie) to a second type of image (e.g., an ID document image). After training, the GAN can be used to generate an augmented training set, which includes image pairs (e.g., an image of a first type paired with an image of the second type, an image of the second type generated from an image of the first type paired with an image of the second type). The augmented training data set can be used to train a matching model to identify when subsequent input images (e.g., a selfie and an ID image, an ID image generated from a selfie and an actual ID image) match.

Description

[0001] Cross-reference to related applications

[0002] This international application claims priority to U.S. Patent Application No. 62 / 717,630, filed on August 10, 2018, the disclosure of which is incorporated herein by reference in its entirety for all purposes. Background Art

[0003] Many activities in daily life (including transactions, service access and transportation) require individuals to verify their identity by presenting identification (ID) documents (e.g., passports, driver's licenses, etc.). Typically, a person needs to verify that the identification document matches the person in front of him. An automated system for matching ID documents with the person present in real time will speed up the verification process and eliminate the burden on the operator. However, there are several unique challenges in providing such systems. For example, ID document images are usually of low quality due to compression. Watermarks and / or glare in the ID image may also make it difficult to analyze and process the image.

[0004] Embodiments of the invention address these and other problems individually and collectively. Summary of the invention

[0005] Embodiments of the present invention relate to methods, systems, devices, and computer-readable media that can be used to accurately match an image of an ID document with an image of an ID document holder. For example, a user can capture a self-portrait image using an image capture device (e.g., a camera of their smartphone, a camera provided at a self-service machine, etc.). The user can also provide an image of their ID document that is captured and / or scanned (e.g., by taking a photo of their ID with a camera of their smartphone, by using a scanner and / or camera provided at a query kiosk, etc.). Using the technology provided herein, a data processing computer can be used to match an ID document image with a self-portrait image to determine with high accuracy whether the images depict the same person.

[0006] One embodiment of the present invention relates to a method, which includes receiving input data including a first input image and a second input image by a data processing computer. The method may also include providing the first input image and the second input image as input to a machine learning model by the data processing computer. In some embodiments, the machine learning model may be trained by the following operations: i) obtaining an initial training set including a first group of images of a first type and a second group of images of a second type by the data processing computer; ii) training a neural network to generate an output image of a second type from an input image of the first type; iii) generating an augmented training set by the data processing computer at least in part based on the first group of images of the first type and the neural network; and iv) training the machine learning model by the data processing computer to identify whether two input images match, the machine learning model being trained using the augmented training set. The method may also include performing at least one operation by the data processing computer in response to receiving an output of the machine learning model indicating that the first input image matches the second input image.

[0007] Another embodiment of the present invention relates to a data processing computer, which may include one or more processors and one or more memories storing computer-executable instructions, wherein the computer-executable instructions are executed by the one or more processors so that the data processing computer performs the method described above.

[0008] In some embodiments, the neural network may include a cycle-consistent adversarial network, and training the neural network may include training a first neural network to generate a first type of output image from a second type of input image, and training a second neural network to generate a second type of output image from a first type of input image. In some embodiments, the neural network is a cycle-consistent generative adversarial network. As described herein, a cycle-consistent generative adversarial network may also include corresponding first and second discriminator networks. The first discriminator network may be configured to identify whether a generated output image of the first type is generated or real, and the second discriminator network may be configured to identify whether a generated output image of the second type is generated or real.

[0009] In some embodiments, the method may further include validating the first neural network by providing a first set of input images of a first type to the first neural network to obtain a set of generated images of a second type; providing the set of generated images of the second type to generate a second set of generated images of the first type; and comparing the first set of input images of the first type with the second set of generated images of the first type.

[0010] In some embodiments, the first set of images and the second set of images may not be paired in the initial training set. The augmented training set may include pairs of images. For example, a pair of images of the augmented training set may include a first image in the first set of images and a second image generated by the neural network from the first image. In some embodiments, the first image may be of a first type and the second image may be of a second type.

[0011] In some embodiments, training a machine learning model to identify whether two input images match may include: training the machine learning model using image pairs of an augmented training set and a supervised learning algorithm.

[0012] In some embodiments, the augmented training set may include pairs of images, each pair including two images of the second type. At least one pair of images may include an image generated by the neural network from one of the first set of images.

[0013] In some embodiments, the method may further include transforming a first input image received as input data from a first type to a second type using a neural network. The first input image may be transformed before providing the first input image and the second input image as input to the machine learning model.

[0014] In some embodiments, the first set of images includes portrait images of the user captured by the user (e.g., a "selfie" captured by a camera of the user's device, a portrait image captured by another device such as a kiosk or camera provided by another entity, etc.), and the second set of images includes images captured from an identification card.

[0015] These and other embodiments of the invention are described in more detail below with reference to the accompanying drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0016] Figure 1 A block diagram is shown of an exemplary system and method for matching different input data in accordance with some embodiments.

[0017] Figure 2 A block diagram is shown illustrating another exemplary system and method for matching different input data in accordance with some embodiments.

[0018] Figure 3 A block diagram of an exemplary data processing computer is shown in accordance with some embodiments.

[0019] Figure 4 A block diagram showing an exemplary generative adversarial network.

[0020] Figure 5 A block diagram illustrating an exemplary recurrent generative adversarial network for generating image-to-image translations in accordance with some embodiments.

[0021] Figure 6A block diagram illustrating an exemplary technique for validating two generative adversarial networks in accordance with some embodiments.

[0022] Figure 7 A block diagram is shown illustrating an exemplary process for training a matching model according to some embodiments. DETAILED DESCRIPTION

[0023] As described above, an individual may need to provide an identification (ID) document, such as a passport, driver's license, state-issued ID card, etc., to verify their identity. For example, when boarding an airplane, an individual may need to present their ID with their ticket at a security checkpoint. Typically, someone (e.g., a security guard) needs to verify that the person standing in front of them is the same person depicted on the ID document. This can result in a cumbersome process for verifying identification, which can cause frustration and possible negative impact on the person being verified.

[0024] Automation of this process is not straightforward. While users can now easily provide an image of themselves and / or their ID document using, for example, the camera on their smartphone, it is not easy to determine whether the image of the person matches the image depicted on the ID document. Images of ID documents may be of poor quality or may include watermarks and / or glare. Additionally, due to privacy concerns, it is difficult to obtain a dataset that includes known portrait image / ID document image pairs.

[0025] The processes described herein can be used to provide an efficient process to accurately determine in real time whether a person image matches an ID image with high accuracy. Utilizing these techniques can speed up the verification process and eliminate the burden on operators.

[0026] Before discussing detailed embodiments of the present invention, some description of certain terminology may be helpful.

[0027] A "computing device" may be any suitable electronic device operated by a user. A user device may be, for example, a smartphone, a smartwatch, a laptop, a desktop, or a game console. In some cases, a computing device may be owned by a user or provided by another entity.

[0028] A "neural network" is a machine learning network modeled after the human brain. Such artificial neural networks provide algorithms that allow computers to learn by incorporating new data. A neural network may consist of many perceptrons, each performing simple signal processing and connected to each other in a large mesh network. Neural networks cannot be directly programmed for a task. Instead, they learn information using supervised and / or unsupervised learning.

[0029] "Supervised learning" is a machine learning algorithm that uses a labeled data set to learn a mapping function between input variables and output variables. The goal is to roughly estimate the mapping function so that the output variable can be predicted from new input data. Some example supervised learning algorithms include linear regression, random forests, and support vector machines.

[0030] "Unsupervised learning" is a machine learning algorithm that models the underlying structure or distribution of a dataset in order to learn more about the data. In unsupervised learning, the dataset only has input data, and the output data is not known in advance. Some example unsupervised learning algorithms include k-means clustering and the Apriori algorithm.

[0031] A "convolutional neural network" is a neural network that takes an image as input and assigns importance (e.g., learnable weights / biases) to various aspects / objects in the image. Convolutional neural networks can be used for image processing, image classification, and facial recognition systems.

[0032] "Generative Adversarial Networks" (GANs) are used for generative modeling using deep learning methods such as convolutional neural networks. Generative modeling is an unsupervised learning task in machine learning that involves automatically discovering and learning rules or patterns in input data in a way that allows the model to be used to generate or output new examples that are likely to have been derived from the original data set. GANs consist of two sub-models: a generator model that is trained to generate new data examples, and a discriminator model that is trained to classify examples as real or fake. The two models are trained and competed together in a zero-sum game until the discriminator model is fooled more than a certain threshold percentage of the time, meaning that the generator model generates examples that appear to be credible.

[0033] A "cycle-consistent generative adversarial network" (also known as a "cycleGAN") is a generative adversarial network that uses two generative models and two discriminator models. The cycleGAN can be used for image-to-image translation to learn a function that transforms an input image from one domain to another. As a non-limiting example, a cycleGAN can be used to learn how to change an image of a zebra to depict an image of a horse. The cycleGAN learns this function through a training dataset that includes unpaired data. In other words, the training dataset may include a set of images of a first domain (e.g., zebras) and a set of images of a second domain (e.g., horses), but the images of the first domain (e.g., a specific zebra) are not paired or otherwise associated with the images of the second domain (e.g., a specific horse). For additional information about cycleGAN, see Zhu, Park, Isla, and Efros, “Unpaired Image-to-Image Translation using Cycle-Consistent Adversarial Networks,” published on November 15, 2018 at https: / / arxiv.org / pdf / 1703.10593.pdf, which is incorporated by reference.

[0034] A "server computer" is typically a powerful computer or cluster of computers. For example, a server computer may be a mainframe, a cluster of minicomputers, or a group of servers that function as a unit. In one example, a server computer may be a database server coupled to a network server.

[0035] "Processor" may refer to any suitable one or more data computing devices. A processor may include one or more microprocessors that work together to perform the desired functions. A processor may include a CPU that includes at least one high-speed data processor sufficient to execute program components for executing user and / or system generated requests. The CPU may be a microprocessor such as AMD's Athlon, Duron, and / or Opteron; IBM and / or Motorola's PowerPC; IBM and Sony's Cell processor; Intel's Celeron, Itanium, Pentium, Xeon, and / or XScale; and / or similar processors.

[0036] A "memory" may be any suitable device or devices that can store electronic data. Suitable memory may include a non-transitory computer-readable medium that stores instructions that can be executed by a processor to implement the desired method. Examples of memory may include one or more memory chips, disk drives, and the like. Such memory may operate using any suitable electrical, optical, and / or magnetic operating modes.

[0037] Figure 1 A block diagram illustrating an exemplary system 100 and method for matching different input data according to some embodiments. The system 100 may be used to facilitate Figure 1 Data communications between the various computers depicted in . System 100 includes a computing device 102, a data processing computer 104, a matching engine 106, a training data set data storage area 108, and an augmented training data set data storage area 110. In some embodiments, the matching engine 106 can be a component of the data processing computer 104. The training data set data storage area 108 and the augmented training data set data storage area 110 can be the same data storage area or different data storage areas. In some embodiments, the computing device 102 and the data processing computer 104 can be the same thing. Each of these systems and computers can be in operative communication with each other. For example, these systems and computers can communicate via one or more data networks such as, but not limited to, the Internet, a wireless communication network, a cellular communication network, etc. Typically, Figure 1 The components in the embodiment may communicate using any suitable communication protocol through any suitable communication medium. To simplify the description, Figure 1 A certain number of components are shown in the drawings. However, it should be understood that embodiments of the present invention may include more than one of each component. In addition, some embodiments of the present invention may include more than one of each component. Figure 1 All parts less or more than the parts shown.

[0038] The data processing computer 104 can be in any suitable form. For example, the data processing computer 104 can be a server computer configured to provide the functionality discussed herein. In some embodiments, the data processing computer 104 can be a computing device, such as a laptop, desktop, kiosk, smart phone, tablet computer, etc. In some embodiments, the data processing computer 104 can be configured to obtain input data, such as input data 112 and 114 discussed in more detail below. For example, the data processing computer 104 can be configured with one or more image capture devices, such as a camera, scanner, etc.

[0039] Computing device 102 may likewise be in any suitable form. For example, computing device 102 may be a smartphone, a personal digital assistant (PDA), a tablet computer, a laptop computer, a desktop computer, a digital camera, etc. In some embodiments, computing device 102 may be configured with or configured to access one or more image capture devices, such as cameras, scanners, etc.

[0040] As a non-limiting example, the data processing computer 104 may be a server computer operated on behalf of an entity (e.g., a security organization at an airport or transportation terminal, a merchant, a government agency, etc.). For illustration, the data processing computer may be a server computer operated on behalf of a security organization that is responsible for verifying the identity of passengers at security checkpoints in an airport. Figure 1 The method discussed provides a method for verifying the identity of a passenger.

[0041] The method may begin at step 1, where a training data set may be obtained and stored in the training data set data storage area 108. In some embodiments, the training data set may include a collection of images of a first type, such as images of a person including at least a portion of the person's face (hereinafter referred to as "portrait images"). These images may be captured by the image subject themselves (sometimes referred to as "selfies"), or the images may be captured by a person / device different from the image subject. The training data set may also include a collection of images of a second type, such as ID documents or portions of ID documents that include images of people. As used herein, ID documents are intended to refer to passports, driver's licenses, state-issued identification cards, debit and / or credit cards, or any suitable document that includes an image of the document holder. The training data set contained in the training data set data storage area 108 may include almost no image pairs. That is, images of the first type may not be paired or otherwise associated with images of the second type. The training data set may be provided by any suitable source, such as a user of the data processing computer 104. In some embodiments, the training data set may be obtained using a web crawler or other data collection algorithm that can access various websites on the Internet to identify human images and / or ID document images. In the training data set, the number of portrait images need not equal the number of ID document images, but the sets may have the same number of images.

[0042] In step 2, the data processing computer 104 may retrieve a training data set and initiate a process of training a generative adversarial network, such as a cycleGAN. Using the training data set obtained from the training data set data storage area 108, the data processing computer 104 may train the cycleGAN to identify two transformation functions. One transformation function may specify an operation to be performed to transform an image of a first type (e.g., a portrait image) into an image of a second type (e.g., an ID document image). The second transformation function may specify an operation to be performed to transform an image of a second type (e.g., an ID document image) into an image of the first type (e.g., a portrait image). Figure 4 and 5 Discuss the training process in more detail.

[0043] In step 3, the data processing computer 104 may perform a verification process to verify the accuracy of the two transformation functions determined in step 2. The verification process may include using a first transformation function to transform a first image of a first domain (e.g., a portrait image) into a second image of a second domain (e.g., an ID document image). The transformed image may then be provided as an input to a second transformation function to transform the transformed image back to the first domain (e.g., back to a portrait image). The resulting image may be compared with the original image of the first domain to determine whether the image matches within a certain threshold degree. A similar process may be performed to verify the second transformation function by transforming the image of the second domain to the first domain and then back using a second transformation function. The resulting image may be compared with the original image to determine whether the image matches within a certain threshold degree. If both of the resulting images match the original images of their corresponding domains, the data processing computer 104 may deem the first and second transformation functions to have been verified. If one or both of the resulting images fail to match the original image, the data processing computer 104 may continue to train cycleGAN to improve the accuracy of the transformation function. It may be combined with Figure 6 This process is described further.

[0044] After the first and second transformation functions have been verified (e.g., accurate to a predetermined threshold amount), the data processing computer 104 may be configured to generate an augmented training data set at step 4. The augmented training data set may include image pairs associated with each other. For example, each image of the first domain (e.g., a portrait image) may be transformed into an image of the second domain (e.g., an ID document image) using a first transformation function. Each image of the first domain may be associated / paired with a resulting image of the second domain obtained by applying the first transformation function to the image of the first domain. Similarly, each image of the second domain (e.g., an ID document) may be transformed into an image of the first domain (e.g., a portrait image) using a second transformation function. Each image of the second domain may be associated / paired with a resulting image of the first domain obtained by applying the second transformation function to the image of the second domain. Each pair discussed above may be marked as a match. In some embodiments, the data processing computer 104 may provide additional image pairs, including images of the first domain and images of the second domain that are intentionally mismatched. These pairs may be marked as mismatches. Such augmented training data sets of matching and mismatching image pairs may be stored in an augmented training data set data storage area 110, which may be the same or different data storage area as the training data set data storage area 108.

[0045] At step 5, the matching engine 106 (e.g., a component of the data processing computer 104 or another computing device) may obtain the augmented training data set from the augmented training data set data store 110 or directly from the data processing computer 104. In some embodiments, the matching engine 106 may be configured to utilize any suitable machine learning algorithm to train a matching model to identify whether two input images match each other. In some embodiments, the matching engine 106 may utilize supervised learning techniques and the augmented training data set discussed above to identify when the input images match each other.

[0046] Subsequently, in step 6, user 103 may utilize computing device 102 (or multiple computing devices) to collect input data, such as input data A 112 and input data B 114. In some embodiments, input data A 112 may be an image of user 103 (e.g., an image including a portion of the user's face). For example, user 103 may utilize a camera of computing device 102 to capture an image including at least a portion of his or her face (e.g., a "selfie," also referred to as a "self-portrait image"). In some embodiments, computing device 102 may be owned and operated by user 103, while in other examples, computing device 102 may be provided by a different entity. Input data A 112 may be obtained by user 103 himself or another person. Input data B 114 may be an image of an ID document (e.g., an image of a driver's license). An image of an ID document may include an image of a person. In some embodiments, user 103 may utilize a camera and / or scanner of computing device 102 (or another computing device) to capture input data B 114. If different computing devices are utilized, it should be understood that each computing device may be connected to the computing device 102 over any suitable communication network (e.g., the Internet, A wireless communication network, a cellular communication network, etc.) is communicatively connected to the matching engine 106.

[0047] At step 7, input data A 112 and input data B 114 may be transmitted to the matching engine 106. In some embodiments, the matching engine 106 may transmit via an application programming interface when the matching engine 106 is running on the computing device 102. In other embodiments, when the matching engine 106 is running on a device other than the computing device 102, the input data A 112 and input data B 114 may be transmitted via any suitable communication protocol. In some embodiments, the matching engine 106 may run on a data processing computer 104 (e.g., a server computer).

[0048] At step 8, the matching engine 106 may provide input data A 112 and input data B 114 as inputs to the matching model trained at step 5. The matching model may be configured to provide as an output a determination that two instances of input data (e.g., input data A 112 and input data B 114) match or do not match. As depicted, the "accept" output indicates that input data A 112 matches input data B 114 (e.g., it is determined that the images depict the same person), while the "reject" output indicates that input data A 112 does not match input data B 114 (e.g., it is determined that the images depict different people or at least not the same person). Although not depicted here, the matching engine 106 may be configured to provide the output back to the computing device 102, which in turn may be configured to perform or more operations. As a non-limiting example, the computing device 102 may be configured to provide a notification of the image match on the computing device 102. In some embodiments, computing device 102 may be configured to provide a code, image, barcode, etc., which, when read by another computing device (e.g., a security gate), may indicate that the person has authenticated themselves by providing a real-time picture of themselves that matches the image provided on their ID document. Any suitable operations may be performed based on the output provided by matching engine 106. In some embodiments, if it is determined that the images do not match each other, user 103 may be denied access to an area (e.g., a check-in area at an airport), service, transaction (e.g., purchase of government controlled substances), etc.

[0049] Figure 2 A block diagram of another exemplary system 200 and method for matching different input data according to some embodiments is shown. The system 200 may be similar to Figure 1 The system 200 can be used to facilitate Figure 2 The system 200 includes a computing device 202, a data processing computer 204, a matching engine 206, a training data set data storage area 208, and an augmented training data set data storage area 210. Each of the components 202-210 may be Figure 1 In some embodiments, matching engine 206 may be a component of data processing computer 204. Figure 2 In the example depicted in , the transformation engine 207 can be a component of the data processing computer 204. The training data set data storage area 208 and the augmented training data set data storage area 210 can be the same data storage area or different data storage areas. In some embodiments, the computing device 202 and the data processing computer 204 can be the same thing. Each of these systems and computers can be in operative communication with each other. For example, these systems and computers can communicate through one or more data networks such as, but not limited to, the Internet, a wireless communication network, a cellular communication network, etc. In general, Figure 2 The components in the embodiment may communicate using any suitable communication protocol through any suitable communication medium. To simplify the description, Figure 2 A certain number of components are shown in the drawings. However, it should be understood that embodiments of the present invention may include more than one of each component. In addition, some embodiments of the present invention may include more than one of each component. Figure 2 All parts less or more than the parts shown.

[0050] The computing device 202 may be in any suitable form. For example, the computing device 202 may be a smartphone, a personal digital assistant (PDA), a tablet computer, a laptop computer, a desktop computer, a digital camera, etc. In some embodiments, the computing device 202 may be configured with or configured to access one or more image capture devices, such as a camera, a scanner, etc.

[0051] The data processing computer 204 may be Figure 1 and 2 In some embodiments, the data processing computer 204 may be a server computer operated on behalf of an entity (e.g., a security organization at an airport or transportation terminal, a merchant, a government agency, etc.). Figure 2 The method discussed provides a method for verifying whether two different input images match.

[0052] The method may begin at step 1, where a training data set may be obtained and stored in the training data set data storage area 208. In some embodiments, the training data set may include a collection of images of a first type, such as images of a person including at least a portion of the person's face (hereinafter referred to as "portrait images"). These images may be captured by the image subject themselves (sometimes referred to as "selfies"), or the images may be captured by a person / device different from the image subject. The training data set may also include a collection of images of a second type, such as ID documents or portions of ID documents that include images of people. As used herein, ID documents are intended to refer to passports, driver's licenses, state-issued identification cards, debit and / or credit cards, or any suitable document that includes an image of the document holder. The training data set contained in the training data set data storage area 208 may include almost no image pairs. That is, images of the first type may not be paired or otherwise associated with images of the second type. The training data set may be provided by any suitable source, such as a user of the data processing computer 204. In some embodiments, the training data set may be obtained using a web crawler or other data collection algorithm that can access various websites on the Internet to identify human images and / or ID document images. In the training data set, the number of portrait images need not equal the number of ID document images, but the sets may have the same number of images.

[0053] In step 2, the data processing computer 204 may retrieve a training data set and initiate a process of training a generative adversarial network, such as cycleGAN. Using the training data set obtained from the training data set data storage area 208, the data processing computer 204 (e.g., the transformation engine 207) may train the cycleGAN to identify two transformation functions. One transformation function may specify an operation to be performed to transform an image of a first type (e.g., a portrait image) into an image of a second type (e.g., an ID document image). The second transformation function may specify an operation to be performed to transform an image of a second type (e.g., an ID document image) into an image of a first type (e.g., a portrait image). The training process may be similar to the above combined Figure 4 and 5 The process described.

[0054] In step 3, the data processing computer 104 may perform a verification process to verify the accuracy of the two transformation functions determined in step 2. The verification process may be similar to the above combined Figure 6 The verification process discussed.

[0055] After the first and second transformation functions have been verified (e.g., are accurate to a predetermined threshold amount), the data processing computer 204 may be configured to generate an augmented training data set at step 5. The augmented training data set may include pairs of images associated with each other. For example, each image of the first domain (e.g., a portrait image) may be transformed into an image of the second domain (e.g., an ID document image) using the first transformation function. Each transformed image may be paired with another instance of the transformed image and marked as a match. Each transformed image may also be paired with one of the original images of the second domain and marked as a mismatch. Thus, the augmented training data set may include pairs of images of the second domain (e.g., ID document images), some of which include one or more images generated by transforming images of the first domain into the second domain.

[0056] At step 5, the matching engine 206 (e.g., a component of the data processing computer 204 or another computing device) may obtain the augmented training data set from the augmented training data set data store 210 or directly from the data processing computer 204. In some embodiments, the matching engine 106 may be configured to train a matching model to identify whether two input images match each other using any suitable machine learning algorithm. In some embodiments, the matching engine 206 may use supervised learning techniques and the augmented training data set discussed above to identify when the input images match each other. The training process for training this matching model may be similar to Figure 7 Process 700.

[0057] Subsequently, in step 6, user 203 may utilize computing device 202 (or multiple computing devices) to collect input data, such as input data A 212 and input data B 214. In some embodiments, input data A 212 may be an image of user 203 (e.g., an image including a portion of the user's face). For example, user 203 may utilize a camera of computing device 202 to capture an image including at least a portion of his or her face (e.g., a "selfie," also referred to as a "self-portrait image"). In some embodiments, computing device 202 may be owned and operated by user 203, while in other examples, computing device 202 may be provided by a different entity. Input data A 212 may be obtained by user 203 himself or another person. Input data B 214 may be an image of an ID document (e.g., a driver's license image). An image of an ID document may include an image of a person. In some embodiments, user 203 may utilize a camera and / or scanner of computing device 202 (or another computing device) to capture input data B 214. If different computing devices are utilized, it should be understood that each computing device may be connected to the computing device 202 over any suitable communication network (e.g., the Internet, A wireless communication network, a cellular communication network, etc.) is communicatively connected to the matching engine 106.

[0058] At step 7, input data A 212 and input data B 214 may be transmitted to the transformation engine 207. The transformation engine 207 may be configured to apply a first function to transform input data A 212 into generated input data B 216. In other words, input data A 212, which is an image of a first type / domain (e.g., a portrait image such as a selfie), may be transformed into input data B 216 of a second type / domain (e.g., an ID document image generated from a portrait image). In some embodiments, both generated input data B 216 and input data B 214 may be of the second type / domain.

[0059] At step 8, the generated input data B 216 and input data B 214 may be transmitted to the matching engine 206. In some embodiments, the matching engine 206 may be transmitted via an application programming interface when the matching engine 206 is running on the same device as the transformation engine 207. In other embodiments, when the matching engine 206 is running on a device different from the computing device 202, the generated input data B 216 and input data B 214 may be transmitted via any suitable communication protocol. In some embodiments, the transformation engine 207 and / or the matching engine 206 may be running at the computing device 202, and / or some portion of the transformation engine 207 and / or the matching engine 206 may be running at a server computer such as the data processing computer 204.

[0060] At step 9, the matching engine 206 may provide the generated input data B 216 and the input data B 214 as input data to the matching model trained at step 5. The matching model may be configured to provide as an output a determination that two instances of input data (e.g., the generated input data B 216 and the input data B 214) match or do not match. As depicted, the "accept" output indicates that the generated input data B 216 matches the input data B 214 (e.g., it is determined that the images depict the same person), while the "reject" output indicates that the generated input data B 216 does not match the input data B 214 (e.g., it is determined that the images depict different people or at least not the same person). Although not depicted here, the matching engine 206 may be configured to provide the output back to the computing device 202, which in turn may be configured to perform one or more operations. As a non-limiting example, the computing device 202 may be configured to provide a notification of the image match on the computing device 202. In some embodiments, computing device 202 may be configured to provide a code, image, barcode, etc., which, when read by another computing device (e.g., a security gate), may indicate that the person has authenticated themselves by providing a real-time picture of themselves that matches the image provided on their ID document. Any suitable operations may be performed based on the output provided by matching engine 206. In some embodiments, if it is determined that the images do not match each other, user 203 may be denied access to an area (e.g., a check-in area at an airport), service, transaction (e.g., purchase of government controlled substances), etc.

[0061] exist Figure 2 The embodiment according to the present invention is shown in Figure 1 1. The data processing computer 104 may include a data storage area 104A, a processor 104B, a network interface 104C, and a computer readable medium 104D.

[0062] The computer readable medium 104D may include several components, such as a processing module 104E, a transformation engine 104F, and a matching engine 104G. More or fewer components are contemplated. It should also be appreciated that Figure 2 The components depicted in the can be combined to perform the functions described herein. The computer-readable medium 104D can also include code that can be executed by the processor 104B to implement the methods discussed herein.

[0063] In some embodiments, data storage area 104A may be an example of training data set data storage area 104H (e.g., Figure 1 and 2 ), and / or augmenting the training dataset data storage area 104I (e.g., Figure 1 and 2In some embodiments, the training dataset data store 104H and / or the augmented training dataset data store 104I may be external to but available for use by the data processing computer 104 and / or any suitable component thereof.

[0064] The processing module 104E may include code that, when executed, causes the processor 104B to receive an initial training data set. Figure 1 and 2 As described in , the initial training data set may include a set of images of a first type / domain and a set of images of a second type / domain. In some embodiments, the processing module 104E may be configured to store the training data set in the training data set data storage area 104H. In some embodiments, the processing module 104E may also be configured to receive input data (e.g., two images). In some embodiments, the input data may be received using the network interface 104C. In other embodiments, the processing module 104E may provide any suitable interface (e.g., an image capture interface, an application program interface, etc.) through which the input data may be received. The processing module 104E may be configured to cause the processor 104B to provide the input data to the transformation engine 104F.

[0065] In some embodiments, the transformation engine 104F may be configured to train one or more generative adversarial networks. For example, the transformation engine 104F may be configured to cause the processor 104B to train a generative model using a training data set to generate a second type of image from a first type of image. The transformation engine 104F may also be configured to cause the processor 104B to perform operations to train a discriminator model to classify generated images as "real" or "fake / generated." In a similar manner, the transformation engine 104F may be configured to cause the processor 104B to train a second generative model using a training data set to generate a first type of image from a second type of image. The transformation engine 104F may also be configured to cause the processor 104B to perform operations to train a second discriminator model to classify generated images generated by the second generative model as "real" or "fake / generated." The process of training these GANs may be described below in conjunction with Figures 4 to 6 Discussed in more detail. After training, the transformation engine 104F can store the transformation functions of the first and second generative models. One transformation function can specify the operation performed on an image of a first type / domain to transform the image into an image of a second type / domain. The second transformation function can specify the operation performed on an image of a second type / domain to transform the image into an image of a first type / domain.

[0066] In some embodiments, the transformation engine 104F may be configured to cause the processor 104B to perform operations to generate an augmented data set. For example, the transformation engine 104F may cause the processor 104B to generate a corresponding image of the second type / domain using each image of the first type / domain of the training data set and the first transformation function. The transformation engine 104F may be configured to cause the processor 104B to mark the corresponding image pair as "matching". Similarly, the second transformation function may be used to transform the image of the second type / domain of the training data set to generate a corresponding image of the first type / domain. These pairs may also be marked as matching. The transformation engine 104F may be configured to cause the processor 104B to generate a mismatch by pairing the image of the first type / domain of the training data set with the image of the second type obtained from the initial training data set (e.g., the image of the second type not generated using the first transformation function). In some embodiments, the transformation engine 104F may be configured to cause the processor 104B to store the resulting image (referred to herein as "augmented training data set") in the augmented training data set data storage area 104I.

[0067] In some embodiments, the transformation engine 104F may be configured to cause the processor 104B to transform an input image in a pair of input images from a first type / domain to a second type / domain. The transformation engine 104F may be configured to cause the processor 104B to provide the transformed image of the second type and another image of the second type from the input image to the matching engine 104G.

[0068] In some embodiments, the matching engine 104G may be configured to cause the processor 104B to obtain the augmented training data set from the augmented training data set data storage area 104I and / or directly from the transformation engine 104F. In some embodiments, the matching engine 104G may be configured with code that, when executed, causes the processor 104B to train and / or maintain a matching model (e.g., Figure 7 The matching engine 104G may be configured to cause the processor 104B to perform a combination of Figure 6 The training process 700 discussed herein identifies whether two input images match in order to train a matching model. In some embodiments, the matching engine 104G may be configured to cause the processor 104B to transmit an output (e.g., an indication of a match or mismatch) to any suitable computing system. In some embodiments, the matching engine 104G may cause the processor 104B to transmit the output via the network interface 104C. The network interface 104C may be any suitable interface corresponding to any suitable communication network, such as the Internet, a cellular network, a wireless network, and the like.

[0069] Figure 4A block diagram of an exemplary generative adversarial network 400 is shown. Generative adversarial network 400 can be used to capture the characteristics of an image of a second domain so as to train a model (e.g., identify a transformation function) to transform an image from a first domain to a second domain in the absence of any previously paired / labeled training examples. Generative adversarial network 400 includes generative network 402 and discriminator network 404. Generative network 402 and discriminator network 404 can each be an example of a neural network. Generative network 402 can be trained to generate new images of a domain from input data 406. Trainable discriminator network 404 identifies whether the generated image is real or fake (e.g., generated by generative network 402).

[0070] At step 1, input data 406 may be used as input to generative network 402. Input data 406 may correspond to a fixed length vector of random noise. In some embodiments, input data 406 may correspond to an image in a first domain. Generative network 402 may generate an image (e.g., generated input data 408) using this random noise at step 2. Generated input data 408 may be generated as an example of an image in a second domain.

[0071] In step 3, the discriminator network 404 may obtain a ground truth dataset 410. The ground truth dataset 410 may include a set of images of the second domain. The discriminator network 404 may be trained with the ground truth dataset 410 to classify an input image as "real" (e.g., in the same domain as the ground truth data) or "false" (e.g., not in the same domain as the ground truth data). The discriminator network 404 may be trained using any suitable supervised or unsupervised machine learning technique and the ground truth dataset 410.

[0072] At step 4, the discriminator network may classify the generated image 408 as "fake" (e.g., not belonging to the first domain) or "true" (e.g., belonging to the first domain). The determination of true (e.g., 1) or false (e.g., 0) may be provided in conjunction with the generated image 408 in two separate feedback loops. For example, at step 5, the output and generated image 408 may be provided to the discriminator network 404 as additional training data to improve the accuracy of the discriminator network in identifying true images from false images. The same output and generated image 408 may be provided back to the generative network 402 at step 6. This data may be used to improve the ability of the generative network to generate better samples that are more similar to other images in the first domain.

[0073] Steps 1 to 6 may be performed any suitable number of times to improve each network in the generative adversarial network over time. The generative adversarial network can be viewed as a zero-sum problem. When the generative network 402 is able to deceive the discriminator network 404, the generative network and / or the model parameters of the generative network 402 are rewarded unchanged, but the discriminator network 404 is penalized and its model parameters are updated. Therefore, Figure 4 The networks compete with each other during the training phase.

[0074] Figure 5 A block diagram of an exemplary cycle generative adversarial network (cycleGAN) 500 for generating image-to-image translation according to some embodiments is shown. The cycleGAN 500 may be a Figure 2 The transformation engine 207 and / or Figure 3 500 may include two different generative adversarial networks (GANs). For example, a first generative adversarial network (GAN) may include a generative network 502 and a discriminator network 504. A second GAN may include a generative network 506 and a discriminator network 508. Each of the first and second GANs may be Figure 4 An example of GAN 400.

[0075] In some embodiments, Figure 4 The generative network 502 may be configured / trained as described above to generate images for domain Y (e.g., generated ID images 510). Domain Y may correspond to a type of image (e.g., ID document images, such as real ID images 512 that are not generated by the generative network 502 but are captured by a camera or scanner). The discriminator network 504 may be configured / trained to classify images as true (e.g., belonging to domain Y) or false (e.g., not belonging to domain Y). Figure 4 The generative network 506 is configured / trained as described to generate images of domain X (e.g., generated portrait images 514). Domain X may correspond to a second type of image (e.g., portrait images, such as real portrait images 516 that are not generated by the generative network 506 but are captured with a camera). At any suitable time, the GAN may be validated.

[0076] Figure 6 FIG. 1 shows a method for verifying two generative adversarial networks (e.g., Figure 5 5. A block diagram of an exemplary technique 600 for cycleGAN (GAN) using a generative network 502 is shown. Function G may represent a transformation function provided by generative network 502, while function F may represent a transformation function provided by generative network 506. At 602, cycleGAN 500 is shown in a minimalist form.

[0077] In some embodiments, during the training phase of cycleGAN 500, each real portrait image 516 (e.g., where image X is an example) may be transformed from domain X to domain Y using the transformation function G of the generative network 502. This transformation produces an image The transformed image can then be transformed from domain Y back to domain X using the transformation function F of the generative network 506. This transformation will produce image x. These transformations are depicted at 604. Image x and image Similarly, each real ID image 512 (e.g., image y) can be transformed from domain Y to domain X using the transformation function F of the generative network 506. This transformation produces image x. The transformed image can then be transformed from domain X back to domain Y using the transformation function G of the generative network 502. This transformation produces image y. These transformations are depicted at 606. Image y can be compared to image It should be understood that the and depicted at 606 It is not intended to depict the same image. Similarly, the image at 604 and the image at 606 is not intended to depict the same image. In some embodiments, a cycle consistency loss is applied to ensure that the transformed image retains information about the original image. The resulting image (at 604 and 606 ) match the original image (x and y, respectively), the more accurate the transformation functions of the generative networks 502 and 506 may be. When the resulting image matches the original image within a certain threshold, both GANs may be considered accurate enough to be deployed / used.

[0078] In some embodiments, the applied loss function may use distance as a cycle loss. However, in the problem of ID / portrait conversion, it may not be realistic to expect to recover a high-quality portrait image from a compressed ID document image. Therefore, in some embodiments, instead of cycle consistency, a perceptual loss function may be used. It is known that some adversarial loss functions may suffer from mode collapse. Therefore, in some embodiments, Wasserstein loss may be applied in combination with gradient penalty, which may improve the performance of image generation tasks.

[0079] Figure 6 The techniques described in may be performed any suitable number of times as desired to train cycleGAN 500 to be more accurate.

[0080] Figure 7 An example process 700 for training a matching model 702 is shown in accordance with at least one embodiment. The process 700 may be performed by a matching engine 704, which is Figure 1 Matching engine 106, Figure 2 The matching engine 206 and / or Figure 3 An example of a matching engine 104G.

[0081] In some embodiments, process 700 may begin at 706, where matching engine 704 (or a component thereof) may obtain a training data set 708. Training data set 708 may include any suitable data with which matching model 702 may be trained to identify whether two input images match. For example, training data set 708 may include an augmented training data set, such as a combination of Figure 1 and 2 In some embodiments, the training data set 708 may include image pairs including images of a first type / domain (e.g., portrait images) and images of a second type / domain (e.g., ID document images). In some embodiments, the images of the second type / domain in some pairs may be generated from corresponding images of the first type / domain (e.g., corresponding portrait images). In other embodiments, the training data set 708 may include image pairs including two images of the same type / domain (e.g., second type / domain, such as ID document images). In some embodiments, at least one image in these pairs may be an image of the second type generated from an image of the first type using the transformation function described above. In either scenario, some pairs may be marked as matches, while other pairs may be marked as mismatches / non-matches.

[0082] Any suitable portion of the training data set 708 may be submitted at 710 and used to train the matching model 702 at 712. In some embodiments, the training may utilize any suitable supervised machine learning technique. Supervised machine learning techniques are intended to refer to any suitable machine learning algorithm that maps inputs to outputs based on example input-output pairs. A supervised learning algorithm (e.g., a decision tree, a Bayesian algorithm, reinforcement learning of an artificial neural network, a distance function such as a nearest neighbor function, a regression algorithm, etc.) may analyze the training data and generate an inference function (also referred to as a "model") that may be used to identify an output for a subsequent input (e.g., output 714). Thus, by executing a supervised learning algorithm on the training data set 708, the matching model 702 may be trained to identify whether two input images match (or do not match). For example, the output 714 may include an "accept" or "reject" value corresponding to a "match" or "mismatch" decision, respectively.

[0083] After training, or at any suitable time, the matching model 702 may be evaluated to assess the quality (e.g., accuracy) of the model. For example, a quality assessment process 716 may be performed. In some embodiments, the quality assessment process 716 may include providing each pair in the training data set 708 to the model to identify whether the output 714 correctly marks the pair as a match (e.g., indicated by an "accept" output) or a mismatch (e.g., indicated by a "reject" output). The output 714 may be compared to the labels provided in the training data set 708 to identify how many of the outputs of the model are accurate. For example, if 90 of 100 outputs (e.g., match / mismatch decisions) accurately reflect the labels provided in the training data set 708, then the matching model 702 may be determined to be 90% accurate. In some embodiments, when the matching model 702 is used for subsequent input image pairs that were not previously labeled, the subsequent image pairs and corresponding output labels may be added to the training data set 708 and used to retrain and / or adjust the matching model 702 (e.g., by completely retraining the matching model 702 or by performing an incremental update of the matching model 702). In some embodiments, subsequent image pairs and corresponding output labels may not be added to the training data set 708 until a user (eg, an administrator, etc.) identifies the label as accurate for a particular image pair.

[0084] Process 700 may be performed any suitable number of times, at any suitable intervals and / or according to any suitable schedule, such that the accuracy of matching model 702 improves over time.

[0085] Technical improvements

[0086] By utilizing the techniques described herein, matching models can be trained to identify matches between portrait images and ID document images, and / or matches between ID document images generated from portrait images and actual ID document images. Although training datasets for training these models may be unavailable or difficult to obtain, cycleGAN can be used to learn transformation functions for transforming images from one domain (e.g., portrait image domain) to another domain (e.g., ID document image domain), and vice versa. Therefore, portrait images, which are much more readily available than ID document images, can be used to generate training datasets to train the matching models discussed herein.

[0087] Any of the computing devices described herein may be examples of computer systems that can be used to implement any of the entities or components described above. The subsystems of such computer systems may be interconnected via a system bus. Additional subsystems include printers, keyboards, storage devices, and monitors that are coupled to a display adapter. Peripheral devices and input / output (I / O) devices are coupled to an I / O controller and may be connected to a computer system by any of a number of means known in the art, such as a serial port. For example, an I / O port or external interface may be used to connect a computer device to a wide area network (such as the Internet), a mouse input device, or a scanner. Interconnection via a system bus may allow a central processing unit to communicate with each subsystem and control the execution of instructions from a system memory or storage device, as well as information exchange between subsystems. System memory and / or storage device may embody a computer-readable medium.

[0088] As described above, the services of the present invention may involve implementing one or more functions, processes, operations, or method steps. In some embodiments, the functions, processes, operations, or method steps may be implemented as a result of executing an instruction set or software code by a suitably programmed computing device, microprocessor, data processor, etc. The instruction set or software code may be stored in a memory or other form of data storage element accessed by a computing device, microprocessor, etc. In other embodiments, the functions, processes, operations, or method steps may be implemented by firmware or a dedicated processor, integrated circuit, etc.

[0089] Any software component or function described in this application can be implemented as a software code executed by a processor using, for example, conventional or object-oriented techniques and using any suitable computer language (e.g., Java, C++, or Perl). The software code can be stored as a series of instructions or commands on a computer-readable medium such as a random access memory (RAM), a read-only memory (ROM), a magnetic medium such as a hard drive or a floppy disk, or an optical medium such as a CD-ROM. Any such computer-readable medium may reside on or within a single computing device, and may exist on or within different computing devices within a system or network.

[0090] The above description is illustrative rather than restrictive. After reading this disclosure, many variations of the present invention will become apparent to those skilled in the art. Therefore, the scope of the present invention should not be determined with reference to the above description, but should be determined with reference to the pending claims and their full scope or equivalent.

[0091] One or more features of any embodiment may be combined with one or more features of any other embodiment without departing from the scope of the present invention.

[0092] Unless clearly indicated to the contrary, recitation of "a", "an", or "the" is intended to mean "one or more".

[0093] All patents, patent applications, publications, and descriptions mentioned above are incorporated herein by reference in their entirety for all purposes. No admission is made that they are prior art.

Claims

1. A computer-implemented method comprising: receiving input data by a data processing computer, the input data comprising a first input image and a second input image; The data processing computer provides the first input image and the second input image as input to a machine learning model to determine whether the first input image matches the second input image, the machine learning model being formed in the following manner: (i) obtaining, by the data processing computer, an initial training set comprising a first set of images of a first type and a second set of images of a second type; (ii) training a neural network to generate second output images of the second type from a first set of images of the first type and to generate first output images of the first type from a second set of images of the second type, the neural network being different from the machine learning model; (iii) generating, by the data processing computer, an augmented training data set to include: (1) a first pair, which includes a pairing of the second output image with a matching image in the first set of images; (2) a second pair, which includes a pairing of the first output image with a matching image in the second set of images, the first pair and the second pair are marked as matching examples; and (3) additional examples obtained from the initial training set, the additional examples including at least one mismatched pair, the mismatched pair including a second image of a second type and a first image of a first type, the at least one mismatched pair being marked as a mismatched example; and (iv) training the machine learning model to identify whether two input images match, the machine learning model being trained using a supervised learning algorithm and the augmented training data set comprising the first pair, the second pair, and the at least one mismatched pair; and At least one operation is performed by the data processing computer in response to receiving an output of the machine learning model indicating that the first input image matches the second input image.

2. The computer-implemented method of claim 1 , wherein the neural network is a cycle-consistent generative adversarial network, and wherein training the neural network comprises: training a first neural network to generate second output images of the second type from a first set of input images of the first type; as well as A second neural network is trained to generate first output images of the first type from a second set of input images of the second type.

3. The computer-implemented method of claim 1 , further comprising: Before generating the augmented training data set, the neural network is validated by: providing a first output image of the first type to the neural network to obtain a set of generated images of the second type; providing the second output image of the second type to the neural network to generate a second set of generated images of the first type; as well as The first set of images of the first type is compared to the second set of generated images of the first type and the second set of images of the second type is compared to the second set of generated images of the second type.

4. The computer-implemented method of claim 1, wherein the initial training set comprises the first set of images and the second set of images that are unpaired.

5. The computer-implemented method of claim 1 , further comprising: The first input image received as input data of the first type is transformed into the second type using the neural network, and the first input image is transformed before providing the first input image and the second input image as the input to the machine learning model.

6. The computer-implemented method of claim 1, wherein the first set of images comprises self-portrait images captured by a user, and wherein the second set of images comprises images captured from an identification card.

7. The computer-implemented method of claim 1, wherein the neural network is a cycle-consistent generative adversarial network.

8. A data processing computer comprising: one or more processors; as well as one or more memories storing computer-executable instructions, wherein execution of the computer-executable instructions by the one or more processors causes the data processing computer to: receiving input data including a first input image and a second input image; The first input image and the second input image are provided as input to a machine learning model to determine whether the first input image matches the second input image, the machine learning model being formed by: (i) obtaining an initial training set comprising a first set of images of a first type and a second set of images of a second type; (ii) training a neural network to generate second output images of the second type from a first set of images of the first type and to generate first output images of the first type from a second set of images of the second type, the neural network being different from the machine learning model; (iii) generating an augmented training data set to include: (1) a first pair, which includes a pairing of the second output image and a matching image in the first set of images; (2) a second pair, which includes a pairing of the first output image and a matching image in the second set of images, the first pair and the second pair are marked as matching examples; and (3) additional examples obtained from the initial training set, the additional examples including at least one mismatched pair, the mismatched pair including a second image of a second type and a first image of a first type, the at least one mismatched pair being marked as a mismatched example; and (iv) training the machine learning model to identify whether two input images match, the machine learning model being trained using a supervised learning algorithm and an augmented training data set including the first pair, the second pair, and the at least one mismatched pair; and At least one operation is performed in response to receiving an output of the machine learning model indicating that the first input image matches the second input image.

9. The data processing computer of claim 8, wherein execution of the computer executable instructions by the one or more processors further causes the data processing computer to collect the first set of images using a web crawler.

10. The data processing computer of claim 8, wherein training the neural network comprises applying an adversarial loss function.

11. The data processing computer according to claim 8, wherein the neural network comprises at least two generative networks and at least two corresponding discriminator networks.

12. The data processing computer according to claim 8, wherein the input data is received from an interface provided by the data processing computer.

13. The data processing computer of claim 8, wherein the input data is received from a computing device different from the data processing computer.

14. The data processing computer of claim 8, wherein the first type corresponds to portrait images, and wherein the first set of images are portrait images.

15. The data processing computer according to claim 8, wherein the second type corresponds to ID document images, and wherein the second group of images are ID document images.

16. The data processing computer according to claim 8, wherein each image in the first set of images and each image in the second set of images includes at least some portion of a subject's face.

17. A data processing computer according to claim 8, wherein performing the at least one operation in response to receiving an output of the machine learning model indicating that the first input image matches the second input image includes at least one of: approving a transaction, or enabling access to a resource or location.

Images