The invention concerns a process for biometric validation designed to detect video images rendered by computing programs or artificial intelligence visual models

Abstract

A computer-implemented method is described and includes receiving, from a computing device the identifying data of a user moving an object. The identifying data including series of video images of the user, the object, and their movement. The method further includes, extracting, by the computing device, one or more biometric identifiers & identifying properties about the user and the object. The method further includes, verifying, by the computing device, using the video images of the movement against identifying data associated with the user and the object, and / or against a database with enrolment data associated with the user and / or the object. The method further includes, providing, by the computing device, a confirmatory indicator associated with the user are not digital video images generated by the computer program. In response to receiving the confirmatory indicator, the method further includes, processing enrollment and other requests associated with an identity authentication system.

Description

TITLE OF YOUR INVENTIONThe invention concerns a process for biometric validation designed to detect video images rendered by computing programs, deepfake packages, or artificial intelligence visual models.TECHNICAL FIELD

[0001] The following disclosure / THIS INVENTION relates generally to the provision of a method for the positive identification of an individual, particularly, but not exclusively, as a means for identity authentication to enrolling or verifying a user as genuine person and not digital video images generated by computer program.

[0002] The invention finds particular, but not exclusive, use as a means for validation of identity through mobile phone or personal devices over Internet for identity authentication purpose. However, the invention is not to be regarded as limited to such applications.BACKGROUND OF THE INVENTION

[0003] Today's digital landscape require robust and secure methods for remote user enrollment, especially in scenarios where the person is not physically present during the registration process. This need arises from the necessity to verify the identity of users in a manner that prevents fraud and ensures trust.

[0004] Some systems authenticate users at enrollment or authentication using biometric information like self-portrait. This includes requiring user to provide real-time selfportrait photos or videos from different angles or asking the user to perform different facial actions within a short video like opening mouth, eye blinking, for liveness detection.

[0005] Some systems require users to provide photos of their identity card or identity document with facial image issued by an authority for validation.

[0006] Some systems verify enrolled users using biometric information, for example, a user may provide facial images or video with facial image when attempts to update the user account or devices.

[0007] Positive identification of an individual is important for preventing scam or frauds. One prior solution proposed for these particular problems is to add methodologies for liveness detection relying on certain actions / physical attribute of the individual. The liveness detection was introduced and intended to prevent users from using photo or images of another person.

[0008] However, the advancement on computing algorithms on image processing, and artificial intelligence on imaging and audio manipulation have evolved over the years, and posed threat to the current identity authentication methodologies.

[0009] Artificial intelligence technologies have evolved and are capable of producing realistic videos with an identity different from the user, by using videos and / or photos of another person fortraining. The trained neural network model (aka, visual model with fake identity) is capable of generating video images to perform actions like opening mouth with the fake identity facial landmarks and speaking voices that can match with required lip movements.

[0010] The advancement on Artificial intelligence (hereafter referred as “Al”) generated virtual image / virtual model of another person becomes harder to detect by human and / or machine, using the video of self-portrait "alone" for liveness detection through video may be insufficient to provide positive identification of individuals.

[0011] This current biometric validation become vulnerable to such attack and lead to mistaken identity authentication.

[0012] The challenge is that the trained Al model becomes increasingly sophisticated with higher resolution image, smoother resulting image, and more natural facial expression, through using high quality data made available for training, adoption of improved algorithms, more powerful graphics processing unit.

[0013] Due to the ease of access of personal videos and / or images from social media and online mediums, this result in increased difficulty to validate the identity of person when encountered in virtual medium is the genuine person or an Al neural network model generated by computing devices. Such encounters can be when encountering user through video conferencing. Alternatively, for a computer process during the validation of user through camera or mobile device.

[0014] The biometric validation using self-portrait is still considered essential or unavoidable in many situations, and artificial intelligence technologies (aka deepfakes) can imitate personal traits and was perceived hard to detect with eyeball checking. There is a need for a new approach to verify the user’s provided biometric information is genuine during the encounters.SUMMARY OF THE INVENTION

[0015] The system and process shall compose of one or more of the following elements. First is capturing the biometric properties of the user. Second is identifying an object and capturing the properties of the object. Third is capturing video images of the movement specified by the validation party.

[0016] This approach requires the user to perform a few steps during the biometric registration and validation process, using an “object” and perform an action specified by the validation party, and video images would be captured for the validation process.

[0017] Unlike a person’s biometric facial information that could be easily accessible from social media or online mediums and obtained for model training. An identified “object”, specified by the validation party, would become a variable that could not be confirmed and prepared in advance for training purpose. Moreover, there are objects considered specific to- and-only-to the user and cannot be easily accessed by other people.

[0018] During the identity validation of the person-in-video, the validating party would identify and request the person-in-video to perform an action with the specified object.

[0019] The validating party shall then verify the resulting videos and images, to identify for abnormality. As the “object” could not be used to train model in advance, the resulting video and / or image shall exhibit defects on different aspects for detection.

[0020] Such abnormality includes but not limited to smeared edges of the face and object, missing parts of the face, or missing parts of the object, or distinct property difference exhibit between the face and object.

[0021] Given the advancement of artificial intelligence on creating pre-trained facial model is foreseeable, the invention provides a practical way to conduct biometric validation, and such validation can be conducted manually or by a computer program.

[0022] The advantageous effects of this invention is that the biometric information accompanied with object can now be validated to enhance the credibility. This checking offers additional reassurance regarding the credibility of digital identity of the person-in- video during real-time engagements.

[0023] By including an object (external element) in the video images of the person- in-video. With the presence of foreign object, artificial intelligence algorithms will have to handle not only the movement of facial landmarks, but also the dynamic image generation of target face with the specified object. First advantage is this prevents a video that could be prepared in advance, and exploit the real-time graphics manipulation power for the overlapping between the foreign object & facial landmarks in the video images.

[0024] Second advantage is disrupting the algorithm of standard dynamic face alignment process. Upon the foreign object covers part of the face and / or facial landmarks, which disrupts the dynamic process of face alignment, which would result in image smearing, missing part of object, missing part of face issues.

[0025] Third advantage is by adding a “foreign” object to enhance the difficulty to pre-train artificial intelligence visual model. Optimization of neural network requires high quality data and extensive pre-training of the Al algorithm. However, the object does not have the pre -determined properties (shape, size, texture, information) available for training purpose.

[0026] In summary, the invention describes systems and processes to enable a computing device, such as a smartphone or related mobile device, to collect biometric based user information together with an assigned object into one or more programs.

[0027] This specification describes systems and processes that includes use of a mobile computing device by a user to submit identity data and object data during an enrollment phase of an identity enrollment program and to submit related identity data by the same user during a corresponding enrollment phase of the identity enrollment program.

[0028] One aspect of the subject matter described in this specification can be embodied in a computer implemented method.

[0029] The method includes, receiving, by a computing device identifying data about a user, the identifying data including digital video captured by the computing device, including the “biometric identifier and a foreign object”.

[0030] Another aspect of the subject matter described in this specification can be embodied in a method that includes, verifying the biometric attribute of a user with a foreign object.

[0031] Implementations of techniques described in this specification include methods, systems, and computer program products. The details of one or more disclosed implementations are provided in the drawings and in the detailed description.

[0032] A preferred embodiment of the present invention will now be described with reference to the accompanying drawings.BRIEF DESCRIPTION OF THE DRAWINGS

[0033] FIG 1 illustrates an example system for acquiring information for user during enrollment process of a program, and validation of user identity upon validation

[0034] FIG 2 illustrates the example system and involved elements for acquiring information for user during enrollment process of a program, and validation of user identity upon validation

[0035] FIG 3 illustrates some potential video images captured as validation data when the person in the video moves the object to overlap with the person’s face for capturing videos images

[0036] FIG 4 illustrates some potential video images captured as biometric data, object data, and validation data.

[0037] FIG 5 illustrates an example process for acquiring information for validation data

[0038] FIG 6, FIG 7, FIG 8 illustrates an example manual supported process for collecting data for the remote person in the video.DETAILED DESCRIPTION OF EMBODIMENTS

[0039] For systems that need to identify the user belonged to a genuine person, systems in general would adopt biometric recognition and / or verifying characteristics of individual, such as facial landmarks, through capturing videos or images provided by user during enrollment in the system.

[0040] Facial landmarks are key points on the face that are used to localize and represent salient regions of the face. These key points play a discriminative role or can serve as anchor points on a face. Examples of facial landmarks include eyes, eyebrows, nose, mouth, jawline, and others.

[0041] During an enrollment procedure, the biometric systems typically requires the user undergo registration to provide one or more pieces of identifying information, through video images captured by users’ mobile device into the system. And a means to prevent user providing another person’s photo, some biometric system would require the person toconduct designated action for liveness features like eye blinking, opening mouth, or speaking a sentence to attest the image provided belonged to a genuine person, and the facial landmarks.

[0042] With the recent advancement in Artificial intelligence made impersonating other persons with the subject person’s photo and video clips posed threats on the reliability of registration process.

[0043] Current methods of enrolling users in biometric systems are dependent on the quality and accuracy of the video images provided through users devices. The current enrollment process captures biometric data & personal data from identity documents as-is from mobile device. The user biometric enrollment was not designed to detect and prevent impersonation by artificial intelligence technologies. And the video images of identity documents may not be reliable unless the data has been validated by the identity card issuing authority.

[0044] In order to improve the efficiency and accuracy of the captured information, an embodiment of the present invention provides a method for detecting the abnormal video images of the user, to improve the reliability of identity verification. Such method and techniques may be implemented in a new enrollment registration process, or combine with any existing enrollment or validation process or routine to reduce the potential threat of impersonation.

[0045] This specification describes systems and processes that provide users with a secure biometric identity verification process. Implementation of these systems and processes described allow enrollment program applicants (e.g. users) to use personal mobile devices to collect biometric based enrollment data to prevent system being compromised by the use of artificial intelligence technologies (e.g. deepfake).

[0046] The described subject matter encompasses one or more methods that enable customers to validate personal biographic data, digital self-portrait video images, with legal identity documents, or other information for identity proofing services. The method can be embodied and executed, at least in part, in application such as executable program code forcollecting and transmitting data from a device such as smartphone, tablet, or service workstation.

[0047] Below description covers various situations when user’s biometric identity have to be validated. In some situation where user has already completed biometric registration on the system, system would validate using users registered biometric information for identifying if the user is the genuine person.

[0048] FIG. 1 illustrates an example system 100, acquiring user information during an enrollment process of an user identity registration program. System 100 generally includes a user personal device 101, a user device data acquisition module 111, a user device data identification module 113, a biometric data acquisition module 121, a biometric data extraction module 122, a biometric data identification module 123, an object data acquisition module 131, an object data extraction module 132, an object data identification module 133, a validation data acquisition module 141, a validation data extraction module 142, a validation data identification module 143, a data correlation module 151, an abnormal data identification module 160. The system further includes a data and properties database module 152, and a document and object database module 153. Examples of user devices 101 include smartphones, mobile devices, laptops, tablets, desktop computers, and other computing devices.

[0049] In general, system 100 can be implemented, in part, by execution of program code in the form of an executable program, otherwise known as an “app”, that can be launched or executed from user device 101. Upon execution of the application program code, the application can then establish a data connection with the one or more modules and storage memory of system 100.

[0050] User device data acquisition module 111 can be a module to acquire the information of device that the user is using. For example, module 111 can be an application to collect the device manufacturer, operating system release version, camera module, camera resolution, IP address, of the user device.

[0051] User device data identification module 113 can be a module to acquire the properties of user device and properties of video images captured by user device. Forexample, module 113 can lookup the device manufacturer and model number to identify the camera modules, supported video and / or photo resolution. Module 113 can be an application to identify the properties of video to be captured by the user device like resolution, color profde, encoding format.

[0052] Biometric data acquisition module 121 can be a module to acquire certain elementary or identifying features of a user. For example, module 121 can be a digital video image acquisition or camera application configured to capture digital video image of the user from various angles.

[0053] Biometric data extraction module 122 can be a module to acquire certain identifying biometric features of user. For example, module 122 can acquire the information of user’s face, outline, contour line, texture, and facial landmarks, such as eyes, eyebrows, nose, mouth, ear, jawline, moles, lentigo, tattoo, hair, and others.

[0054] Biometric data identification module 123 can be a module to identify the properties of facial landmarks extracted in 122, and identify the also the correlation between facial landmarks. For example, module 123 can be an application to identify and proximity & differences across facial landmarks.

[0055] The data correlation module 151 can be a module to correlate the information collected from 121, 122, 123 and associate with the user, and record the information in the Data and properties database 152, and Document and object database 153.

[0056] Object data acquisition module 131 can be a module to acquire certain elementary or identifying features of an object specified. For example, module 131 can be a digital video image acquisition or camera application configured to capture various angles of an identity document, or a definite object designated or assigned by the module.

[0057] In some implementation, the “object” can be a standard document and can be an object with a physical state or it can be a piece of information displayed on a physical device.

[0058] In some implementation, the “object” can be assigned by the module during the process, and can be an object with a physical state or it can be a piece of information displayed on a physical device.

[0059] Object data extraction module 132 can be a module adapted to acquire certain identifying features of an object. For example, module 132 can acquire the object information such as the nature, dimension, contour line of the object, color, texture, and / or information displayed on the object if it is a scanned document such as text and images printed on the object.

[0060] Object data identification module 133 can be a module to identify the properties of object extracted in 132, and identify the also the genuine properties of the object. For example, module 133 can lookup database if the object was issued by the company itself, it could identify whether the information extracted from module 132 is consistent with the lookup information.

[0061] In some implementation, the object can be a regulated document that could be validated by the issuing company of regulated bodies for its authenticity. For example, if the object is staff card, module 133 can lookup company database for the image of staff card from historical record , and validate the authenticity of the object.

[0062] In some implementation, the object can be specified by the computer program in static or dynamic manner. The objects specified could be general available (e.g. post-it, name card, dollar bill), or restricted access in public market (e.g. staff card), or uniquely issued by regulated authority (e.g. identity card, passport, driver license).

[0063] The data correlation module 151 will can be a module to correlate the information collected from 131, 132, 133 and associate with the user, and record the information in the Data and properties database 152, and Document and object database 153.

[0064] Collecting validation data and subsequently analysis involve collecting biometric information and object data. Validation data acquisition module 141 can be a module to acquire video images of both user (the subject of biometric data collected inmodule 121) and the object (the subject of object data acquired in module 131) together, with an action track assigned randomly in real-time.

[0065] For example, as shown in FIG 2, module 141 can be a digital video image acquisition or camera application configured to capture digital video images of the user moving the object across is an action path (e.g. horizontally, vertically, diagonally) across user’s face in front of the camera.

[0066] Validation data extraction module 142 can be a module to extract certain identifying biometric features of user, certain identifying features of an object, and the areas where user and object coincide in the video image.

[0067] Validation data identification module 143 can be a module to identify the properties of the user, the object, to identify the action track, to identify properties of the user and object upon when they coincide.

[0068] Adopting actions that instructed by the computer program with static and / or dynamic element. The action movements could be moving object between camera & the person with requested action trail (e.g. horizontal / vertical / circular), and / or involve dynamic elements shown on a device held by the person (e.g. SMS / Instant message / QR code / Graphics).

[0069] The data correlation module 151 can be a module to correlate the information collected from 141, 142, 143 and associate with the user, and record the information in the Data and properties database 152, and Document and object database 153.

[0070] For example, as shown in FIG 3, the biometric data of user collected from video images captured in 121, 122, 123, the data of object collected from video images captured in 131, 132, 133, the validation data collected from video images captured in 141, 142, 143 would be associated to the user in module 151.

[0071] The images of person and objects could be captured in module 121 and module 131, or these could be the images captured previously during the registration process, and retrieved from database.

[0072] The abnormal data identification module 160 can be a module to identify the abnormal data based upon the data collected and correlated in 151, extracted and processed in 151, and stored in 152, and 153.

[0073] In images A , B , C , D , E , captured in FIG 2 would be validated and compared against the image of the user, and image of object for abnormal symptom.

[0074] In some situation where the user image involved impersonation, the images could exhibit inconsistency when one or more of the facial landmarks are being covered by the object, resulting in an unnatural facial image of impersonated target, or smeared skin or overflow of facial image and / or object image.

[0075] In some implementation, the user image data and properties can be validated with the user device properties, in some situation where user image involved impersonation, the resulting image captured from 141 to 143 exhibit properties that are inconsistent with the user device properties, such as quality of facial landmarks against device captured video images quality.

[0076] In some implementation, the user image data and properties can be validated with the user image data captured previously during enrollment process and stored in the data and properties database 152 and document and object database 153. In some situation where user image involved impersonation, the resulting image captured from 141 to 143 could exhibit inconsistency.

[0077] In some implementation, the user data and properties can be validated against other users data and properties stored in the data and properties database 152 and document and object database 153. In some situation where user image involved impersonation, the result may contains duplicate registration of person with biometric data that resemble each other.

[0078] In some implementation, the object data and properties can be validated with the object data captured previously during enrollment process and stored in the data and properties database 152 and document and object database 153.

[0079] The embodiment can be implemented in full or in part. As shown in FIG 4, where the validation would capture biometric data capture, object data, and validation data for abnormal symptom identification.

[0080] In FIG 5, where a person has not conducted biometric registration enrollment with system, system does not have the person’s registered biometric information.

[0081] In FIG 6, when there is need arise for validating the user in the video for potential use of impersonated video, the person followed user request to identify certain object.

[0082] In FIG 7, the remote person in the video followed instruction to move the object along an action track agreed with user.

[0083] The system or the user shall then conduct validation to identify if the user is a genuine person.

[0084] As shown in FIG 8, where the validation conducted only with validation data, and rely on the validation data alone for abnormal symptom identification. The process can be an application, or supported by manual process.

[0085] Only few example implementations are disclosed. Modifications, variations, and other potential enhancements can be made based on the disclosed.

Claims

CLAIMS1. A computer - implemented method comprising:Receiving, by a computing device• identifying data about a person, the identifying data including digital video images of the person; and• identifying data about an object, the identifying data including digital video images of the object; the object may be specified by the user or computing program; and• identifying data about a person performing an action with the object, the identifying data including digital video images of the person and object overlapped together; the action may be specified by the user or computing program;Extracting, by the computing device• identifying data about a person, some of the biometric identifiers about the user;• identifying data about the object, some of the properties of the object;• identifying data about the person and object together, some of the biometric identifiers about the user, and some of the properties of the object;Identifying, by the computing device• identifying data about the biometric data of the user, and the correlation between those data;• identifying data about the properties of the object, and the correlation between those properties;• identifying data about the data with both person and object together, and the correlation between those data;Retrieving, by the computing device• identifying data about a person from previous registration, on the biometric identifiers about the user;• identifying data about the object from previous registration, on the properties about the object; some of the common properties about the object;Verifying, by the computing device• abnormality identified associated in the image with a person & object together;• abnormality identified about the person, or some of the properties of the person; like unmatched biometric properties where overlapped with the object; and• abnormality identified about the object, or some of the properties of the object; like unmatched object properties where overlapped with the person; and• abnormality identified about the properties of the person or the objectProviding, by the computing device• an indicator associated with the process result; for enrollment associated with the person or identity validation request for the person.

2. The computing -implemented method of claim 1, further comprising:Verifying, by the computing device, the biometric data of a person against other person biometric data in the database.

3. A system comprising:One or more processing device used by the person as subject of validation for data acquisition;Acquiring, by the computing device• identifying data about a person, the identifying data including digital video images of the person; and• identifying data about an object, the identifying data including digital video images of the object; the object may be specified by the user or computing program; and• identifying data about a person performing an action with the object, the identifying data including digital video images of the person and object overlapped together; the action may be specified by the user or computing program;Receiving, by the computing device• an indicator returned from validation party with the result of validation;One or more processing device used by validation party for person genuine identity validation comprising:Receiving, by a computing device• identifying data about a person, the identifying data including digital video images of the person; and• identifying data about an object, the identifying data including digital video images of the object; the object may be specified by the user or computing program; and• identifying data about a person performing an action with the object, the identifying data including digital video images of the person and object overlapped together; the action may be specified by the user or computing program;Extracting, by the computing device identifying data about a person, some of the biometric identifiers about the user;• identifying data about the object, some of the properties of the object;• identifying data about the person and object together, some of the biometric identifiers about the user, and some of the properties of the object;Identifying, by the computing device• identifying data about the biometric data of the user, and the correlation between those data;• identifying data about the properties of the object, and the correlation between those properties;• identifying data about the data with both person and object together, and the correlation between those data;Retrieving, by the computing device• identifying data about a person from previous registration, on the biometric identifiers about the user;• identifying data about the object from previous registration, on the properties about the object; some of the common properties about the object;Verifying, by the computing device• abnormality identified associated in the image with a person & object together;• abnormality identified about the person, or some of the properties of the person; like unmatched biometric properties where overlapped with the object; and• abnormality identified about the object, or some of the properties of the object; like unmatched object properties where overlapped with the person; and abnormality identified about the properties of the person or the objectProviding, by the computing device an indicator associated with the process result;4. An electronic system comprising:One or more processing device used by the person as subject of validation for data acquisition;Acquiring, by the computing device• identifying data about a person, the identifying data including digital video images of the person; and• identifying data about an object, the identifying data including digital video images of the object; the object may be specified by the user or computing program; and• identifying data about a person performing an action with the object, the identifying data including digital video images of the person and object overlapped together; the action may be specified by the user or computing program;Receiving, by the computing device• an indicator returned from validation party with the result of validation;One or more processing device used by validation party for person genuine identity validation comprising:Receiving, by a computing device• identifying data about a person, the identifying data including digital video images of the person; and• identifying data about an object, the identifying data including digital video images of the object; the object may be specified by the user or computing program; and• identifying data about a person performing an action with the object, the identifying data including digital video images of the person and object overlapped together; the action may be specified by the user or computing program;Extracting, by the computing device• identifying data about a person, some of the biometric identifiers about the user;• identifying data about the object, some of the properties of the object;• identifying data about the person and object together, some of the biometric identifiers about the user, and some of the properties of the object;Identifying, by the computing device• identifying data about the biometric data of the user, and the correlation between those data;• identifying data about the properties of the object, and the correlation between those properties;• identifying data about the data with both person and object together, and the correlation between those data;Retrieving, by the computing device• identifying data about a person from previous registration, on the biometric identifiers about the user;• identifying data about the object from previous registration, on the properties about the object; some of the common properties about the object;Verifying, by the computing device abnormality identified associated in the image with a person & object together;• abnormality identified about the person, or some of the properties of the person; like unmatched biometric properties where overlapped with the object; and• abnormality identified about the object, or some of the properties of the object; like unmatched object properties where overlapped with the person; and• abnormality identified about the properties of the person or the objectProviding, by the computing device• an indicator associated with the process result;5. An electronic system comprising:Receiving, by a computing device or online systems• identifying data about a person, the identifying data including a digital self-image of the person; and• identifying data about an object, the identifying data including a digital image of the object; the object may be specified by the user or computing program; and• identifying data about a person performing action with an object; the action may be specified by the user or computing program; and• identifying data about a person & object together, the identifying data including a digital self-images of the person and object overlapped together;Extracting, by the computing device or online systems• identifying data about a person, some of the biometric identifiers about the user;• identifying data about the object, some of the properties of the object;Retrieving, by the computing device• identifying data about a person from previous registration, some of the biometric identifiers about the user;• identifying data about the object from previous registration, some of the properties about the object; or identifying data about the object from record, some of the common properties about the object;Verifying, by the computing device or online systems• abnormality identified associated in the image with a person & object together;• abnormality identified about the person, or some of the properties of the person; like smearing edges of the person when overlapped with the object; and• abnormality identified about the object, or some of the properties of the object; like smearing edges of the object when overlapped with the person; and• abnormality identified about the differences on properties of the person and the objectReceiving, by the computing device, or online systems• a confirmation indicator associated with the user;6. A method comprising:Acquiring from the video images, the biometric data of the person, the object data, and the identifying data with the person and object overlapped together while performing a specific action;Verifying, the person in the video, is the genuine user with the biometric data acquired7. The computer implemented method of claim 1, further comprising:The computing device capturing the video and images of the person and the object transmits the data to a remote system to perform validationon the receiving biometric data associated with the person, to determine whether the digital self-image of the person is genuine8. The computer implemented method of claim 1, further comprising:The computing device perform validation receiving properties data associated with both the person & the object, to determine whether the digital self-image is genuine.

Images